From patchwork Tue Jul 8 10:28:13 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Junichi Nomura X-Patchwork-Id: 4503791 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Original-To: patchwork-dm-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 247759F392 for ; Tue, 8 Jul 2014 10:47:11 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 3E26120259 for ; Tue, 8 Jul 2014 10:47:10 +0000 (UTC) Received: from mx5-phx2.redhat.com (mx5-phx2.redhat.com [209.132.183.37]) by mail.kernel.org (Postfix) with ESMTP id 3AFD820270 for ; Tue, 8 Jul 2014 10:47:09 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by mx5-phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s68Agmhm005884; Tue, 8 Jul 2014 06:42:49 -0400 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id s68AglWn004425 for ; Tue, 8 Jul 2014 06:42:47 -0400 Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.21]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s68Agl08026153; Tue, 8 Jul 2014 06:42:47 -0400 Received: from tyo201.gate.nec.co.jp (TYO201.gate.nec.co.jp [210.143.35.51]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s68AgfsD014953 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 8 Jul 2014 06:42:43 -0400 Received: from mailgate3.nec.co.jp ([10.7.69.160]) by tyo201.gate.nec.co.jp (8.13.8/8.13.4) with ESMTP id s68AT807007609; Tue, 8 Jul 2014 19:29:08 +0900 (JST) Received: from mailsv.nec.co.jp (imss63.nec.co.jp [10.7.69.158]) by mailgate3.nec.co.jp (8.11.7/3.7W-MAILGATE-NEC) with ESMTP id s68AT8r06665; Tue, 8 Jul 2014 19:29:08 +0900 (JST) Received: from mail01b.kamome.nec.co.jp (mail01b.kamome.nec.co.jp [10.25.43.2]) by mailsv.nec.co.jp (8.13.8/8.13.4) with ESMTP id s68ASp4U021749; Tue, 8 Jul 2014 19:29:08 +0900 (JST) Received: from bpxc99gp.gisp.nec.co.jp ([10.38.151.138] [10.38.151.138]) by mail03.kamome.nec.co.jp with ESMTP id BT-MMP-404307; Tue, 8 Jul 2014 19:28:14 +0900 Received: from BPXM12GP.gisp.nec.co.jp ([169.254.2.164]) by BPXC10GP.gisp.nec.co.jp ([10.38.151.138]) with mapi id 14.02.0328.011; Tue, 8 Jul 2014 19:28:14 +0900 From: Junichi Nomura To: Mike Snitzer , Paul Mackerras Thread-Topic: Regression in 3.15 on POWER8 with multipath SCSI Thread-Index: AQHPmpdSO1G1Y6CCYUGnzA1eamu97Q== Date: Tue, 8 Jul 2014 10:28:13 +0000 Message-ID: <11AF7C027C4C02408624617A4986078401323542@BPXM12GP.gisp.nec.co.jp> References: <20140630103058.GA17747@iris.ozlabs.ibm.com> <20140701193907.GA15306@redhat.com> In-Reply-To: <20140701193907.GA15306@redhat.com> Accept-Language: ja-JP, en-US Content-Language: ja-JP X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.34.125.85] Content-ID: <459D64762E434A4A97AED6B1302C1D28@gisp.nec.co.jp> MIME-Version: 1.0 X-RedHat-Spam-Score: -4.602 (BAYES_00, DCC_REPUT_00_12, RCVD_IN_DNSWL_MED, SPF_HELO_PASS, SPF_PASS) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Scanned-By: MIMEDefang 2.68 on 10.5.110.21 X-MIME-Autoconverted: from quoted-printable to 8bit by lists01.pubmisc.prod.ext.phx2.redhat.com id s68AglWn004425 X-loop: dm-devel@redhat.com Cc: Vladimir Davydov , "bvanassche@acm.org" , "linux-kernel@vger.kernel.org" , "linuxppc-dev@ozlabs.org" , "dm-devel@redhat.com" , Andrew Morton , Linus Torvalds Subject: Re: [dm-devel] Regression in 3.15 on POWER8 with multipath SCSI X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk Reply-To: device-mapper development List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_NONE, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 07/02/14 04:39, Mike Snitzer wrote: > On Mon, Jun 30 2014 at 6:30am -0400, > Paul Mackerras wrote: > >> I have a machine on which 3.15 usually fails to boot, and 3.14 boots >> every time. The machine is a POWER8 2-socket server with 20 cores >> (thus 160 CPUs), 128GB of RAM, and 7 SCSI disks connected via a >> hardware-RAID-capable adapter which appears as two IPR controllers >> which are both connected to each disk. I am booting from a disk that >> has Fedora 20 installed on it. >> >> After over two weeks of bisections, I can finally point to the commits >> that cause the problems. The culprits are: >> >> 3e9f1be1 dm mpath: remove process_queued_ios() >> e8099177 dm mpath: push back requests instead of queueing >> bcccff93 kobject: don't block for each kobject_uevent >> >> The interesting thing is that neither e8099177 nor bcccff93 cause >> failures on their own, but with both commits in there are failures >> where the system will fail to find /home on some occasions. >> >> With 3e9f1be1 included, the system appears to be prone to a deadlock >> condition which typically causes the boot process to hang with this >> message showing: >> >> A start job is running for Monitoring of LVM2 mirror...rogress polling >> >> (with a [*** ] thing before it where the asterisks move back and >> forth). >> >> If I revert 63d832c3 ("dm mpath: really fix lockdep warning") , >> 4cdd2ad7 ("dm mpath: fix lock order inconsistency in >> multipath_ioctl"), 3e9f1be1 and bcccff93, in that order, I get a >> kernel that will boot every time. The first two are later commits >> that fix some problems with 3e9f1be1 (though not the problems I am >> seeing). >> >> Can anyone see any reason why e8099177 and bcccff93 would interfere >> with each other? > > No, not seeing any obvious relation. > > But even though you listed e8099177 as a culprit you didn't list it as a > commit you reverted. Did you leave e8099177 simply because attempting > to revert it fails (if you don't first revert other dm-mpath.c commits)? > > (btw, Bart Van Assche also has issues with commit e8099177 due to hangs > during cable pull testing of mpath devices -- Bart: curious to know if > your cable pull tests pass if you just revert bcccff93). It seems Bart's issue has gone with the attached patch: http://www.redhat.com/archives/dm-devel/2014-July/msg00035.html Could you try if it makes any difference on your issue? The problem is dm-mpath's state machine stall due to e8099177 but ioctl to the device can kick the state machine running again. That might be related to why bcccff93 affects the reproducibility. Also, 3e9f1be1 integrates some codes into the one which is affected by this problem. So it makes sense why the problem becomes easier to occur with that. - Jun'ichi Nomura, NEC Corporation pg_ready() checks the current state of the multipath and may return false even if a new IO is needed to change the state. OTOH, if multipath_busy() returns busy, a new IO will not be sent to multipath target and the state change won't happen. That results in lock up. The intent of multipath_busy() is to avoid unnecessary cycles of dequeue + request_fn + requeue if it is known that multipath device will requeue. Such situation would be: - path group is being activated - there is no path and the multipath is setup to requeue if no path This patch should fix the problem introduced as a part of this commit: commit e809917735ebf1b9a56c24e877ce0d320baee2ec dm mpath: push back requests instead of queueing --- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c index ebfa411..d58343e 100644 --- a/drivers/md/dm-mpath.c +++ b/drivers/md/dm-mpath.c @@ -1620,8 +1620,9 @@ static int multipath_busy(struct dm_target *ti) spin_lock_irqsave(&m->lock, flags); - /* pg_init in progress, requeue until done */ - if (!pg_ready(m)) { + /* pg_init in progress or no paths available */ + if (m->pg_init_in_progress || + (!m->nr_valid_paths && m->queue_if_no_path)) { busy = 1; goto out; }