From patchwork Tue Aug 8 23:10:45 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 9889759 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8EB4860363 for ; Wed, 9 Aug 2017 07:51:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7C16C268AE for ; Wed, 9 Aug 2017 07:51:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6F1D828A2C; Wed, 9 Aug 2017 07:51:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id D7B4E268AE for ; Wed, 9 Aug 2017 07:51:47 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id DA188769E8; Wed, 9 Aug 2017 07:51:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com DA188769E8 Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=wdc.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com DA188769E8 Authentication-Results: mx1.redhat.com; dkim=fail reason="signature verification failed" (1024-bit key) header.d=sharedspace.onmicrosoft.com header.i=@sharedspace.onmicrosoft.com header.b="E2rTXfpQ" Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1DCF95DD69; Wed, 9 Aug 2017 07:51:44 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 7038B1803B22; Wed, 9 Aug 2017 07:51:41 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v78NCPvw029615 for ; Tue, 8 Aug 2017 19:12:26 -0400 Received: by smtp.corp.redhat.com (Postfix) id EE1A177D6D; Tue, 8 Aug 2017 23:12:25 +0000 (UTC) Delivered-To: dm-devel@redhat.com Received: from mx1.redhat.com (ext-mx05.extmail.prod.ext.phx2.redhat.com [10.5.110.29]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E37F077D6B for ; Tue, 8 Aug 2017 23:12:23 +0000 (UTC) Received: from esa5.hgst.iphmx.com (esa5.hgst.iphmx.com [216.71.153.144]) (using TLSv1.2 with cipher RC4-SHA (112/128 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 78AD337EEA for ; Tue, 8 Aug 2017 23:10:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 78AD337EEA Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=wdc.com Authentication-Results: ext-mx05.extmail.prod.ext.phx2.redhat.com; spf=pass smtp.mailfrom=Bart.VanAssche@wdc.com DKIM-Filter: OpenDKIM Filter v2.11.0 mx1.redhat.com 78AD337EEA X-IronPort-AV: E=Sophos;i="5.41,345,1498492800"; d="scan'208";a="40189694" Received: from mail-cys01nam02lp0053.outbound.protection.outlook.com (HELO NAM02-CY1-obe.outbound.protection.outlook.com) ([207.46.163.53]) by ob1.hgst.iphmx.com with ESMTP; 09 Aug 2017 07:10:47 +0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sharedspace.onmicrosoft.com; s=selector1-wdc-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=fWW/4ae2pXC2dYzxJ11f7HnWtFmgD+jdKdHFj/Ow5hA=; b=E2rTXfpQi0Rb5maxAT3u5xjDkk1YIfSQoZDs7CFZcS4BLJPBNsH+5dxfVPa7Q5CRcAIGCtsXjaw4SqVAQ8dDrTuBE8qIibnvaLd+azpJW0eDuHLC6UDpHZYI91T6xhY6c7RzK7QwAwhSoRUKu1mT2F2nV+OkvPKciOmpNm4mrQU= Received: from CY1PR0401MB1536.namprd04.prod.outlook.com (10.163.19.154) by CY1PR0401MB1533.namprd04.prod.outlook.com (10.163.19.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.1.1320.16; Tue, 8 Aug 2017 23:10:45 +0000 Received: from CY1PR0401MB1536.namprd04.prod.outlook.com ([10.163.19.154]) by CY1PR0401MB1536.namprd04.prod.outlook.com ([10.163.19.154]) with mapi id 15.01.1320.018; Tue, 8 Aug 2017 23:10:46 +0000 From: Bart Van Assche To: "dm-devel@redhat.com" , "linux-scsi@vger.kernel.org" , "linux-block@vger.kernel.org" , "ming.lei@redhat.com" Thread-Topic: [v4.13-rc BUG] system lockup when running big buffered write(4M) to IB SRP via mpath Thread-Index: AQHTEFEYgOBxouQM3EyyuMfVVLhEg6J7FnuA Date: Tue, 8 Aug 2017 23:10:45 +0000 Message-ID: <1502233843.2686.4.camel@wdc.com> References: <20170808141715.GB22763@ming.t460p> In-Reply-To: <20170808141715.GB22763@ming.t460p> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Bart.VanAssche@wdc.com; x-originating-ip: [63.163.107.100] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; CY1PR0401MB1533; 20:hV2CzIjnyKxxsS8oDplJYDicn/2nXme9TB1I07odZ7hC2WLptQj2RFg2+lsmYbDJUstX0EGTNo3sfYEdLnPvb95oeA/hV771Phd7FxKHcjkWTIk9YdLJpzPGZjwCOCS0jhPddcFksgRrEJHnIMQKpo5Yy/jRUcaWuSfAFUOav90= x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 5fcda801-d78b-4af4-3d15-08d4deb2b362 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(48565401081)(300000503095)(300135400095)(2017052603031)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:CY1PR0401MB1533; x-ms-traffictypediagnostic: CY1PR0401MB1533: wdcipoutbound: EOP-TRUE x-exchange-antispam-report-test: UriScan:; x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001)(93006095)(93001095)(100000703101)(100105400095)(6055026)(6041248)(20161123558100)(20161123555025)(20161123562025)(20161123564025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:CY1PR0401MB1533; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:CY1PR0401MB1533; x-forefront-prvs: 03932714EB x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(39410400002)(39400400002)(39850400002)(39840400002)(39450400003)(39860400002)(24454002)(377424004)(189002)(199003)(52314003)(25786009)(3280700002)(53936002)(99286003)(478600001)(33646002)(54356999)(3660700001)(8676002)(76176999)(2501003)(36756003)(189998001)(50986999)(102836003)(6116002)(8936002)(575784001)(86362001)(3846002)(6246003)(66066001)(105586002)(38730400002)(305945005)(97736004)(2201001)(81166006)(4326008)(72206003)(7736002)(2950100002)(101416001)(229853002)(2906002)(81156014)(6512007)(14454004)(5660300001)(6486002)(77096006)(103116003)(2900100001)(6506006)(6436002)(106356001)(68736007); DIR:OUT; SFP:1102; SCL:1; SRVR:CY1PR0401MB1533; H:CY1PR0401MB1536.namprd04.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: wdc.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-ID: <1AF7A6A916E81F479B612A2B4CDAADFF@namprd04.prod.outlook.com> MIME-Version: 1.0 X-OriginatorOrg: wdc.com X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Aug 2017 23:10:45.9624 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0401MB1533 X-Greylist: Delayed for 101:46:14 by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Tue, 08 Aug 2017 23:12:14 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Tue, 08 Aug 2017 23:12:14 +0000 (UTC) for IP:'216.71.153.144' DOMAIN:'esa5.hgst.iphmx.com' HELO:'esa5.hgst.iphmx.com' FROM:'Bart.VanAssche@wdc.com' RCPT:'' X-RedHat-Spam-Score: -1.5 (DCC_REPUT_00_12, DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_LOW) 216.71.153.144 esa5.hgst.iphmx.com 216.71.153.144 esa5.hgst.iphmx.com X-Scanned-By: MIMEDefang 2.78 on 10.5.110.29 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-MIME-Autoconverted: from base64 to 8bit by lists01.pubmisc.prod.ext.phx2.redhat.com id v78NCPvw029615 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Wed, 09 Aug 2017 03:51:40 -0400 Cc: "loberman@redhat.com" Subject: Re: [dm-devel] [v4.13-rc BUG] system lockup when running big buffered write(4M) to IB SRP via mpath X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Wed, 09 Aug 2017 07:51:46 +0000 (UTC) X-Virus-Scanned: ClamAV using ClamSMTP On Tue, 2017-08-08 at 22:17 +0800, Ming Lei wrote: > Laurence and I see a system lockup issue when running concurrent > big buffered write(4M bytes) to IB SRP on v4.13-rc3. > [ ... ] > #cat hammer_write.sh > #!/bin/bash > while true; do > dd if=/dev/zero of=/dev/mapper/$1 bs=4096k count=800 > done Hello Laurence, Is your goal perhaps to simulate a DDN workload? In that case I think you need oflag=direct to the dd argument list such that the page cache writeback code does not alter the size of the write requests. Anyway, this test should not trigger a lockup. Can you check whether the patch below makes the soft lockup complaints disappear (without changing the hammer_write.sh test script)? Thanks, Bart. ---------------------------------------------------------------------------- [PATCH] block: Make blk_mq_delay_kick_requeue_list() rerun the queue at a quiet time Drivers like dm-mpath requeue requests if no paths are available and if configured to do so. If the queue depth is sufficiently high and the queue rerunning delay sufficiently short then .requeue_work can be queued so often that other work items queued on the same work queue do not get executed. Avoid that this happens by only rerunning the queue after no blk_mq_delay_kick_requeue_list() calls have occurred during @msecs milliseconds. Since the device mapper core is the only user of blk_mq_delay_kick_requeue_list(), modify the implementation of this function instead of creating a new function. Signed-off-by: Bart Van Assche --- block/blk-mq.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel diff --git a/block/blk-mq.c b/block/blk-mq.c index 041f7b7fa0d6..8bfea36e92f9 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -679,8 +679,8 @@ EXPORT_SYMBOL(blk_mq_kick_requeue_list); void blk_mq_delay_kick_requeue_list(struct request_queue *q, unsigned long msecs) { - kblockd_schedule_delayed_work(&q->requeue_work, - msecs_to_jiffies(msecs)); + kblockd_mod_delayed_work_on(WORK_CPU_UNBOUND, &q->requeue_work, + msecs_to_jiffies(msecs)); } EXPORT_SYMBOL(blk_mq_delay_kick_requeue_list);