From patchwork Wed Nov 25 09:04:34 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hannes Reinecke X-Patchwork-Id: 7696571 Return-Path: X-Original-To: patchwork-linux-block@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 2333DBF90C for ; Wed, 25 Nov 2015 09:05:19 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 4ED57208B1 for ; Wed, 25 Nov 2015 09:05:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7207320867 for ; Wed, 25 Nov 2015 09:05:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932141AbbKYJEq (ORCPT ); Wed, 25 Nov 2015 04:04:46 -0500 Received: from mx2.suse.de ([195.135.220.15]:44349 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932133AbbKYJEj (ORCPT ); Wed, 25 Nov 2015 04:04:39 -0500 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id BCFDBAC8C; Wed, 25 Nov 2015 09:02:56 +0000 (UTC) Message-ID: <565579A2.4000005@suse.de> Date: Wed, 25 Nov 2015 10:04:34 +0100 From: Hannes Reinecke User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: emilne@redhat.com CC: Christoph Hellwig , Michael Ellerman , Mark Salter , "James E. J. Bottomley" , brking , linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-block@vger.kernel.org, Mike Snitzer , Jun'ichi Nomura , Jens Axboe Subject: Re: kernel BUG at drivers/scsi/scsi_lib.c:1096! References: <1447838334.1564.2.camel@ellerman.id.au> <1447855399.3974.24.camel@redhat.com> <1447894964.15206.0.camel@ellerman.id.au> <20151119082325.GA11419@infradead.org> <564DEC41.5010600@suse.de> <1448030316.4067.18.camel@localhost.localdomain> <564F3453.9040603@suse.de> <1448033323.4067.24.camel@localhost.localdomain> In-Reply-To: <1448033323.4067.24.camel@localhost.localdomain> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Spam-Status: No, score=-7.5 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 11/20/2015 04:28 PM, Ewan Milne wrote: > On Fri, 2015-11-20 at 15:55 +0100, Hannes Reinecke wrote: >> Can't we have a joint effort here? >> I've been spending a _LOT_ of time trying to debug things here, but >> none of the ideas I've come up with have been able to fix anything. > > Yes. I'm not the one primarily looking at it, and we don't have a > reproducer in-house. We just have the one dump right now. > >> >> I'm almost tempted to increase the count from scsi_alloc_sgtable() >> by one and be done with ... >> > > That might not fix it if it is a problem with the merge code, though. > And indeed, it doesn't. Seems I finally found the culprit. What happens is this: We have two paths, with these seg_boundary_masks: path-1: seg_boundary_mask = 65535, path-2: seg_boundary_mask = 4294967295, consequently the DM request queue has this: md-1: seg_boundary_mask = 65535, What happens now is that a request is being formatted, and sent to path 2. During submission req->nr_phys_segments is formatted with the limits of path 2, arriving at a count of 3. Now the request gets retried on path 1, but as the NOMERGE request flag is set req->nr_phys_segments is never updated. But blk_rq_map_sg() ignores all counters, and just uses the bi_vec directly, resulting in a count of 4 -> boom. So the culprit here is the NOMERGE flag, which is evaluated via ->dm_dispatch_request() ->blk_insert_cloned_request() ->blk_rq_check_limits() If the above assessment is correct, the following patch should fix it: rq->cmd_flags)) { Mike? Jens? Can you comment on it? Cheers, Hannes diff --git a/block/blk-core.c b/block/blk-core.c index 801ced7..12cccd6 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1928,7 +1928,7 @@ EXPORT_SYMBOL(submit_bio); */ int blk_rq_check_limits(struct request_queue *q, struct request *rq) { - if (!rq_mergeable(rq)) + if (rq->cmd_type != REQ_TYPE_FS) return 0; if (blk_rq_sectors(rq) > blk_queue_get_max_sectors(q,