From patchwork Wed Jul 20 14:27:28 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Snitzer X-Patchwork-Id: 9239689 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 49FFE6077C for ; Wed, 20 Jul 2016 14:27:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3AF7B268AE for ; Wed, 20 Jul 2016 14:27:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2EF1E27BF7; Wed, 20 Jul 2016 14:27:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 88D4C268AE for ; Wed, 20 Jul 2016 14:27:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754173AbcGTO1a (ORCPT ); Wed, 20 Jul 2016 10:27:30 -0400 Received: from mx1.redhat.com ([209.132.183.28]:46787 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753212AbcGTO1a (ORCPT ); Wed, 20 Jul 2016 10:27:30 -0400 Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D53AD4DB13; Wed, 20 Jul 2016 14:27:29 +0000 (UTC) Received: from localhost (vpn-55-241.rdu2.redhat.com [10.10.55.241]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u6KERSob022676 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 20 Jul 2016 10:27:29 -0400 Date: Wed, 20 Jul 2016 10:27:28 -0400 From: Mike Snitzer To: Bart Van Assche Cc: device-mapper development , linux-scsi@vger.kernel.org Subject: Re: dm-mq and end_clone_request() Message-ID: <20160720142727.GA57399@redhat.com> References: <4ed669ed-beae-76a8-b806-a284565b327a@sandisk.com> <20160720140815.GA19045@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20160720140815.GA19045@redhat.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Wed, 20 Jul 2016 14:27:29 +0000 (UTC) Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Wed, Jul 20 2016 at 10:08am -0400, Mike Snitzer wrote: > On Tue, Jul 19 2016 at 6:57pm -0400, > Bart Van Assche wrote: > > > Hello Mike, > > > > If I run a fio data integrity test against kernel v4.7-rc7 then I > > see often that fio reports I/O errors if a path is removed despite > > queue_if_no_path having been set in /etc/multipath.conf. Further > > analysis showed that this happens because during SCSI device removal > > a SCSI device enters state SDEV_CANCEL before the block layer queue > > is marked as "dying". In that state I/O requests submitted to that > > SCSI device are failed with -EIO. The behavior for > > end_clone_request() in drivers/md/dm.c for such requests is as ... > > - With multiqueue support enabled, pass the "error" argument to > > dm_complete_request(). > > The error arg is passed to dm_complete_request() regardless of queue > type but it is only immediately used by the blk-mq API (via > blk_mq_complete_request). > > > Shouldn't end_clone_request() requeue failed requests in both cases > > instead of passing the I/O error to the submitter only if multiqueue > > is enabled? > > Pretty sure you'll find it is _not_ blk-mq that is passing the error > up. (But if I'm proven wrong that will be welcomed news). > > The error passed to dm_complete_request() is always used to set > tio->error which is later used by dm_done(). DM core handles errors > later via softirq in dm_done() -- where the error is passed into the > target_type's rq_end_io hook. > > So in DM multipath you'll see do_end_io() we do finally act on the error > we got from the lower layer. And if the error is -EIO, noretry_error() > will return true and -EIO will be returned up the IO stack. For some reason I thought -EIO was considered not retryable. That's obviously wrong (e.g. noretry_error() doesn't seize on -EIO). > In the end we're relying on SCSI to properly categorize the underlying > faults as retryable vs not -- via SCSI's differentiated IO errors. > > Unfortunately I'm not seeing anything that DM multipath can do > differently here. -EIO is _always_ propagated up. > > It is strange that all the dm-mq testing that has been done didn't ever > catch this. The mptest testsuite is a baseline for validating DM > multipath (and request-based DM core) changes. But I've also had Red > Hat's QE hammer dm-mq with heavy IO (in terms of the "dt" utility) on a > larger NetApp testbed in the face of regular controller faults. > > Must be this scenario of SDEV_CANCEL is a race that is relatively > unique/rare to your testbed? > > This raises the question: should SCSI be returning something other than > -EIO for this case? E.g. an error that is retryable? So it must be that blk-mq is somehow returning -EIO earlier based on rq->errors that is established during blk_mq_complete_request(). Please try this patch (not happy with it since it assumes all request-based DM targets will handle IO errors -- which is fine for now since DM multipath is the only one). Could be you've already tried this? Does it fix your problem? --- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c index 7a96618..347ff25 100644 --- a/drivers/md/dm-rq.c +++ b/drivers/md/dm-rq.c @@ -414,7 +414,7 @@ static void dm_complete_request(struct request *rq, int error) if (!rq->q->mq_ops) blk_complete_request(rq); else - blk_mq_complete_request(rq, error); + blk_mq_complete_request(rq, 0); } /*