From patchwork Mon Mar 9 01:04:51 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Junichi Nomura X-Patchwork-Id: 5963311 X-Patchwork-Delegate: christophe.varoqui@free.fr Return-Path: X-Original-To: patchwork-dm-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id A6F919F2A9 for ; Mon, 9 Mar 2015 01:14:19 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id A0BD22024F for ; Mon, 9 Mar 2015 01:14:18 +0000 (UTC) Received: from mx4-phx2.redhat.com (mx4-phx2.redhat.com [209.132.183.25]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AC459201F5 for ; Mon, 9 Mar 2015 01:14:16 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by mx4-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id t2918Rlr025920; Sun, 8 Mar 2015 21:08:28 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id t2918PiQ004559 for ; Sun, 8 Mar 2015 21:08:25 -0400 Received: from mx1.redhat.com (ext-mx12.extmail.prod.ext.phx2.redhat.com [10.5.110.17]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t2918PWd027272; Sun, 8 Mar 2015 21:08:25 -0400 Received: from tyo202.gate.nec.co.jp (TYO202.gate.nec.co.jp [210.143.35.52]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id t2918Mmo011772 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 8 Mar 2015 21:08:24 -0400 Received: from mailgate3.nec.co.jp ([10.7.69.197]) by tyo202.gate.nec.co.jp (8.13.8/8.13.4) with ESMTP id t2918F8S001574; Mon, 9 Mar 2015 10:08:15 +0900 (JST) Received: from mailsv3.nec.co.jp (imss62.nec.co.jp [10.7.69.157]) by mailgate3.nec.co.jp (8.11.7/3.7W-MAILGATE-NEC) with ESMTP id t2918Ec10759; Mon, 9 Mar 2015 10:08:14 +0900 (JST) Received: from mail03.kamome.nec.co.jp (mail03.kamome.nec.co.jp [10.25.43.7]) by mailsv3.nec.co.jp (8.13.8/8.13.4) with ESMTP id t2918DYA020800; Mon, 9 Mar 2015 10:08:14 +0900 (JST) Received: from bpxc99gp.gisp.nec.co.jp ([10.38.151.137] [10.38.151.137]) by mail02.kamome.nec.co.jp with ESMTP id BT-MMP-2643128; Mon, 9 Mar 2015 10:04:52 +0900 Received: from BPXM12GP.gisp.nec.co.jp ([169.254.2.79]) by BPXC09GP.gisp.nec.co.jp ([10.38.151.137]) with mapi id 14.03.0174.002; Mon, 9 Mar 2015 10:04:51 +0900 From: Junichi Nomura To: device-mapper development , Mike Snitzer Thread-Topic: [dm-devel] [PATCH 6/8] dm: don't start current request if it would've merged with the previous Thread-Index: AQHQWgULSxJXWIHVBEugz2DqrJGIJQ== Date: Mon, 9 Mar 2015 01:04:51 +0000 Message-ID: <54FCF1B2.8030007@ce.jp.nec.com> References: <1425430031-78140-1-git-send-email-snitzer@redhat.com> <1425430031-78140-7-git-send-email-snitzer@redhat.com> In-Reply-To: <1425430031-78140-7-git-send-email-snitzer@redhat.com> Accept-Language: ja-JP, en-US Content-Language: ja-JP X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.34.125.85] Content-ID: MIME-Version: 1.0 X-RedHat-Spam-Score: -4.602 (BAYES_00, DCC_REPUT_00_12, RCVD_IN_DNSWL_MED, SPF_HELO_PASS, SPF_PASS) 210.143.35.52 TYO202.gate.nec.co.jp 210.143.35.52 TYO202.gate.nec.co.jp X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Scanned-By: MIMEDefang 2.68 on 10.5.110.17 X-MIME-Autoconverted: from quoted-printable to 8bit by lists01.pubmisc.prod.ext.phx2.redhat.com id t2918PiQ004559 X-loop: dm-devel@redhat.com Cc: "axboe@kernel.dk" , "jmoyer@redhat.com" , Shiva Krishna Merla Subject: Re: [dm-devel] [PATCH 6/8] dm: don't start current request if it would've merged with the previous X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk Reply-To: device-mapper development List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 03/04/15 09:47, Mike Snitzer wrote: > Request-based DM's dm_request_fn() is so fast to pull requests off the > queue that steps need to be taken to promote merging by avoiding request > processing if it makes sense. > > If the current request would've merged with previous request let the > current request stay on the queue longer. Hi Mike, Looking at this thread, I think there are 2 different problems mixed. Firstly, "/dev/skd" is STEC S1120 block driver, which doesn't have lld_busy function. So back pressure doesn't propagate to request-based dm device and dm feeds as many request as possible to the lower driver. ("pulling too fast" situation) If you still have access to the device, can you try the patch like the attached one? Secondly, for this comment from Merla ShivaKrishna: > Yes, Indeed this the exact issue we saw at NetApp. While running sequential > 4K write I/O with large thread count, 2 paths yield better performance than > 4 paths and performance drastically drops with 4 paths. The device queue_depth > as 32 and with blktrace we could see better I/O merging happening and average > request size was > 8K through iostat. With 4 paths none of the I/O gets merged and > always average request size is 4K. Scheduler used was noop as we are using SSD > based storage. We could get I/O merging to happen even with 4 paths but with lower > device queue_depth of 16. Even then the performance was lacking compared to 2 paths. Have you tried increasing nr_requests of the dm device? E.g. setting nr_requests to 256. 4 paths with each queue depth 32 means that it can have 128 I/Os in flight. With the default value of nr_requests 128, the request queue is almost always empty and I/O merge could not happen. Increasing nr_requests of the dm device allows some more requests queued, thus the chance of merging may increase. Reducing the lower device queue depth could be another solution. But if the depth is too low, you might not be able to keep the optimal speed. ---- Jun'ichi Nomura, NEC Corporation [PATCH] skd: Add lld_busy function for request-based stacking driver --- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel diff --git a/drivers/block/skd_main.c b/drivers/block/skd_main.c index 1e46eb2..0e8f466 100644 --- a/drivers/block/skd_main.c +++ b/drivers/block/skd_main.c @@ -565,6 +565,16 @@ skd_prep_discard_cdb(struct skd_scsi_request *scsi_req, blk_add_request_payload(req, page, len); } +static int skd_lld_busy(struct request_queue *q) +{ + struct skd_device *skdev = q->queuedata; + + if (skdev->in_flight >= skdev->cur_max_queue_depth) + return 1; + + return 0; +} + static void skd_request_fn_not_online(struct request_queue *q); static void skd_request_fn(struct request_queue *q) @@ -4419,6 +4429,8 @@ static int skd_cons_disk(struct skd_device *skdev) /* set sysfs ptimal_io_size to 8K */ blk_queue_io_opt(q, 8192); + /* register feed back function for stacking driver */ + blk_queue_lld_busy(q, skd_lld_busy); + /* DISCARD Flag initialization. */ q->limits.discard_granularity = 8192; q->limits.discard_alignment = 0;