From patchwork Thu Jul 13 21:12:16 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 9839565 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4AE18602BD for ; Thu, 13 Jul 2017 18:10:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B62622872A for ; Thu, 13 Jul 2017 21:12:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AAFFC28774; Thu, 13 Jul 2017 21:12:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 42BF22872A for ; Thu, 13 Jul 2017 21:12:30 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8438D80C26; Thu, 13 Jul 2017 21:12:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 8438D80C26 Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 8438D80C26 Authentication-Results: mx1.redhat.com; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NnKnO2kH" Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C075960BEC; Thu, 13 Jul 2017 21:12:29 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 2F7874A492; Thu, 13 Jul 2017 21:12:29 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v6DLCRE8032199 for ; Thu, 13 Jul 2017 17:12:27 -0400 Received: by smtp.corp.redhat.com (Postfix) id A940617B76; Thu, 13 Jul 2017 21:12:27 +0000 (UTC) Delivered-To: dm-devel@redhat.com Received: from mx1.redhat.com (ext-mx02.extmail.prod.ext.phx2.redhat.com [10.5.110.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A30BF17AE9 for ; Thu, 13 Jul 2017 21:12:26 +0000 (UTC) Received: from mail-qk0-f194.google.com (mail-qk0-f194.google.com [209.85.220.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id BCFD780C12 for ; Thu, 13 Jul 2017 21:12:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com BCFD780C12 Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx02.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=snitzer@gmail.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com BCFD780C12 Received: by mail-qk0-f194.google.com with SMTP id v17so7455260qka.3; Thu, 13 Jul 2017 14:12:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=kkdOfXwSxdgGFeSA2Xs2lF1AKjT8Kf1r0ulAQOfJXnk=; b=NnKnO2kHnZx7k05G9IiSDum+xaSZpfA6dUvw8Q2v4Az2DyuXjJxgA94AL7KEjE9/n4 HE8lj4nZlO9rjMNVuhYZRsAa3sQNJy01Q6oC51WPA8aL+818/CkbqBae701Fv9spoCVN vB3Tyqn+or0OkwSgt4QtvhWUwwtdmmNpQ0RccLtHzDJsKI7yQX/pJHTGFHwqL+Hh6n1h HpAeMHQtXDrEAgcRz1OHI3316KHwRaMp8uEhqKwFMqZVHze4Dyxb9/jZVbTa7AeoMbzj I29p3P8cWX7PsmW1Bh7j6RjAETY0mOG2ABInJhJcbaefBpKxSLAjdbjQB2fiTtF7PSME hDRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=kkdOfXwSxdgGFeSA2Xs2lF1AKjT8Kf1r0ulAQOfJXnk=; b=s4YcOJ2Q7RN4EcSIfLaSF5oMF7SO9s+RRWmqXls7Zv2W16PMkwHNIRckAGaopP/3vE +UDKu5jUnzP89PEEI2QeGlYvwukT36C/HaY10extf9/Tveg3kJpXXUdXAAoW+7V6rT0D Frw4OCd3W0HQg3ZEwqDdKLo7S3gzobkaasSRhCXfFIPb+ztCdJeGc2qos3eFyVVpUfyT 2dPSYwBDclJIua6gD/3S5lV1Z2wLGcAAkZ+Yz246prdoY7GLgB0bNws4+iRvYMXiQ+Do zZEG2B1NWPn67N9gHwEvWpCWglno3RApsKoUSzsVrLmLTRRBE4z/mhVuED4Mzu2slwYs DMvg== X-Gm-Message-State: AIVw111gy5LrdWUTa4ad2aTWa5Cg/bLETX+jMnigC/8y1xP01aVYI7EV /XGQ15FRD/Le9og7pZY= X-Received: by 10.55.20.230 with SMTP id 99mr7604694qku.236.1499980343870; Thu, 13 Jul 2017 14:12:23 -0700 (PDT) Received: from localhost ([66.187.232.66]) by smtp.gmail.com with ESMTPSA id y58sm5371464qtc.36.2017.07.13.14.12.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 13 Jul 2017 14:12:22 -0700 (PDT) From: Mike Snitzer To: hch@lst.de Date: Thu, 13 Jul 2017 17:12:16 -0400 Message-Id: <20170713211217.52361-2-snitzer@redhat.com> In-Reply-To: <20170713211217.52361-1-snitzer@redhat.com> References: <20170713211217.52361-1-snitzer@redhat.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Thu, 13 Jul 2017 21:12:25 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Thu, 13 Jul 2017 21:12:25 +0000 (UTC) for IP:'209.85.220.194' DOMAIN:'mail-qk0-f194.google.com' HELO:'mail-qk0-f194.google.com' FROM:'snitzer@gmail.com' RCPT:'' X-RedHat-Spam-Score: 1.47 * (BAYES_50, DKIM_SIGNED, DKIM_VALID, FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, RCVD_IN_SORBS_SPAM, SPF_PASS) 209.85.220.194 mail-qk0-f194.google.com 209.85.220.194 mail-qk0-f194.google.com X-RedHat-Possible-Forgery: Mike Snitzer X-Scanned-By: MIMEDefang 2.78 on 10.5.110.26 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: dm-devel@redhat.com Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, linux-scsi@vger.kernel.org Subject: [dm-devel] [for-4.14 RFC PATCH 1/2] dm rq: avoid deadlock if dm-mq is stacked on old .request_fn device(s) X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Thu, 13 Jul 2017 21:12:31 +0000 (UTC) X-Virus-Scanned: ClamAV using ClamSMTP Conditionally use __blk_put_request() or blk_put_request() instead of just blk_put_request() in multipath_release_clone(). Otherwise a deadlock will occur because scsi_end_request() will take the clone request's queue_lock, around its call to blk_finish_request(), and then the later blk_put_request() also tries to take it: [12749.916332] queued_spin_lock_slowpath+0xb/0xf [12749.916335] _raw_spin_lock_irqsave+0x37/0x40 [12749.916339] blk_put_request+0x39/0x60 [12749.916342] multipath_release_clone+0xe/0x10 [dm_multipath] [12749.916350] dm_softirq_done+0x156/0x240 [dm_mod] [12749.916353] __blk_mq_complete_request+0x90/0x140 [12749.916355] blk_mq_complete_request+0x16/0x20 [12749.916360] dm_complete_request+0x23/0x40 [dm_mod] [12749.916365] end_clone_request+0x1d/0x20 [dm_mod] [12749.916367] blk_finish_request+0x83/0x120 [12749.916370] scsi_end_request+0x12d/0x1d0 [12749.916371] scsi_io_completion+0x13c/0x630 [12749.916374] ? set_next_entity+0x7c/0x780 [12749.916376] scsi_finish_command+0xd9/0x120 [12749.916378] scsi_softirq_done+0x12a/0x150 [12749.916380] blk_done_softirq+0x9e/0xd0 [12749.916382] __do_softirq+0xc9/0x269 [12749.916384] run_ksoftirqd+0x29/0x50 [12749.916385] smpboot_thread_fn+0x110/0x160 [12749.916387] kthread+0x109/0x140 [12749.916389] ? sort_range+0x30/0x30 [12749.916390] ? kthread_park+0x60/0x60 [12749.916391] ret_from_fork+0x25/0x30 This "fix" is gross in that the long-term fitness of stacking blk-mq DM multipath (dm-mq) ontop of old .request_fn devices is questionable. The above stack trace shows just how ugly it is to have old .request_fn SCSI code cascade into blk-mq code during DM multipath request completion. Signed-off-by: Mike Snitzer --- drivers/md/dm-mpath.c | 16 ++++++++++++++-- drivers/md/dm-rq.c | 8 +++++--- drivers/md/dm-target.c | 4 ++-- include/linux/device-mapper.h | 3 ++- 4 files changed, 23 insertions(+), 8 deletions(-) diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c index 0e8ab5b..34cf7b6 100644 --- a/drivers/md/dm-mpath.c +++ b/drivers/md/dm-mpath.c @@ -520,9 +520,21 @@ static int multipath_clone_and_map(struct dm_target *ti, struct request *rq, return DM_MAPIO_REMAPPED; } -static void multipath_release_clone(struct request *clone) +static void multipath_release_clone(struct dm_target *ti, struct request *clone) { - blk_put_request(clone); + struct multipath *m = ti->private; + struct request_queue *q = clone->q; + + if (!q->mq_ops && m->queue_mode == DM_TYPE_MQ_REQUEST_BASED) { + /* + * dm-mq on .request_fn already holds clone->q->queue_lock + * via blk_finish_request()... + * - true for .request_fn SCSI, but is it _always_ true? + */ + lockdep_assert_held(q->queue_lock); + __blk_put_request(q, clone); + } else + blk_put_request(clone); } /* diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c index c6ebc5b..95bb44c 100644 --- a/drivers/md/dm-rq.c +++ b/drivers/md/dm-rq.c @@ -220,11 +220,12 @@ static void dm_end_request(struct request *clone, blk_status_t error) { int rw = rq_data_dir(clone); struct dm_rq_target_io *tio = clone->end_io_data; + struct dm_target *ti = tio->ti; struct mapped_device *md = tio->md; struct request *rq = tio->orig; blk_rq_unprep_clone(clone); - tio->ti->type->release_clone_rq(clone); + ti->type->release_clone_rq(ti, clone); rq_end_stats(md, rq); if (!rq->q->mq_ops) @@ -267,6 +268,7 @@ static void dm_mq_delay_requeue_request(struct request *rq, unsigned long msecs) static void dm_requeue_original_request(struct dm_rq_target_io *tio, bool delay_requeue) { + struct dm_target *ti = tio->ti; struct mapped_device *md = tio->md; struct request *rq = tio->orig; int rw = rq_data_dir(rq); @@ -274,7 +276,7 @@ static void dm_requeue_original_request(struct dm_rq_target_io *tio, bool delay_ rq_end_stats(md, rq); if (tio->clone) { blk_rq_unprep_clone(tio->clone); - tio->ti->type->release_clone_rq(tio->clone); + ti->type->release_clone_rq(ti, tio->clone); } if (!rq->q->mq_ops) @@ -488,7 +490,7 @@ static int map_request(struct dm_rq_target_io *tio) case DM_MAPIO_REMAPPED: if (setup_clone(clone, rq, tio, GFP_ATOMIC)) { /* -ENOMEM */ - ti->type->release_clone_rq(clone); + ti->type->release_clone_rq(ti, clone); return DM_MAPIO_REQUEUE; } diff --git a/drivers/md/dm-target.c b/drivers/md/dm-target.c index c0d7e60..adbd17b 100644 --- a/drivers/md/dm-target.c +++ b/drivers/md/dm-target.c @@ -138,12 +138,12 @@ static int io_err_clone_and_map_rq(struct dm_target *ti, struct request *rq, return DM_MAPIO_KILL; } -static void io_err_release_clone_rq(struct request *clone) +static void io_err_release_clone_rq(struct dm_target *ti, struct request *clone) { } static long io_err_dax_direct_access(struct dm_target *ti, pgoff_t pgoff, - long nr_pages, void **kaddr, pfn_t *pfn) + long nr_pages, void **kaddr, pfn_t *pfn) { return -EIO; } diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index 0c1b50ad..f2ca0ab 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -61,7 +61,8 @@ typedef int (*dm_clone_and_map_request_fn) (struct dm_target *ti, struct request *rq, union map_info *map_context, struct request **clone); -typedef void (*dm_release_clone_request_fn) (struct request *clone); +typedef void (*dm_release_clone_request_fn) (struct dm_target *ti, + struct request *clone); /* * Returns: