From patchwork Tue Nov 6 21:34:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 10671533 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 41A1815A6 for ; Tue, 6 Nov 2018 21:56:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 326E028FE3 for ; Tue, 6 Nov 2018 21:56:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2664429143; Tue, 6 Nov 2018 21:56:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,SUBJ_OBFU_PUNCT_FEW autolearn=ham version=3.3.1 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C2D5D28FE3 for ; Tue, 6 Nov 2018 21:56:09 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8A333308424D; Tue, 6 Nov 2018 21:56:08 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 56CDA5D757; Tue, 6 Nov 2018 21:56:08 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id DFBBD4BB79; Tue, 6 Nov 2018 21:56:07 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id wA6Ld2ZZ004236 for ; Tue, 6 Nov 2018 16:39:02 -0500 Received: by smtp.corp.redhat.com (Postfix) id 029C45C880; Tue, 6 Nov 2018 21:39:02 +0000 (UTC) Delivered-To: dm-devel@redhat.com Received: from leontynka.twibright.com (ovpn-204-55.brq.redhat.com [10.40.204.55]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C9D351710F; Tue, 6 Nov 2018 21:39:00 +0000 (UTC) Received: from debian-a64.vm ([192.168.208.2]) by leontynka.twibright.com with smtp (Exim 4.89) (envelope-from ) id 1gK93q-0005NV-8y; Tue, 06 Nov 2018 22:38:59 +0100 Received: by debian-a64.vm (sSMTP sendmail emulation); Tue, 06 Nov 2018 22:38:53 +0100 Message-Id: <20181106213853.335317680@debian-a64.vm> User-Agent: quilt/0.65 Date: Tue, 06 Nov 2018 22:34:59 +0100 From: Mikulas Patocka To: Mike Snitzer , "Alasdair G. Kergon" MIME-Version: 1.0 Content-Disposition: inline; filename=dm-avoid-indirect-call.patch X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: dm-devel@redhat.com Cc: dm-devel@redhat.com, Mikulas Patocka Subject: [dm-devel] [patch 1/5] dm: avoid indirect call in __dm_make_request X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.40]); Tue, 06 Nov 2018 21:56:09 +0000 (UTC) X-Virus-Scanned: ClamAV using ClamSMTP Indirect calls are inefficient because of retpolines that are used for spectre workaround. This patch replaces an indirect call with a condition (that can be predicted by the branch predictor). Signed-off-by: Mikulas Patocka --- drivers/md/dm.c | 27 ++++++--------------------- 1 file changed, 6 insertions(+), 21 deletions(-) -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel Index: linux-2.6/drivers/md/dm.c =================================================================== --- linux-2.6.orig/drivers/md/dm.c 2018-10-16 23:19:49.900000000 +0200 +++ linux-2.6/drivers/md/dm.c 2018-10-16 23:19:51.000000000 +0200 @@ -1687,8 +1687,7 @@ out: typedef blk_qc_t (process_bio_fn)(struct mapped_device *, struct dm_table *, struct bio *); -static blk_qc_t __dm_make_request(struct request_queue *q, struct bio *bio, - process_bio_fn process_bio) +static blk_qc_t dm_make_request(struct request_queue *q, struct bio *bio) { struct mapped_device *md = q->queuedata; blk_qc_t ret = BLK_QC_T_NONE; @@ -1708,26 +1707,15 @@ static blk_qc_t __dm_make_request(struct return ret; } - ret = process_bio(md, map, bio); + if (dm_get_md_type(md) == DM_TYPE_NVME_BIO_BASED) + ret = __process_bio(md, map, bio); + else + ret = __split_and_process_bio(md, map, bio); dm_put_live_table(md, srcu_idx); return ret; } -/* - * The request function that remaps the bio to one target and - * splits off any remainder. - */ -static blk_qc_t dm_make_request(struct request_queue *q, struct bio *bio) -{ - return __dm_make_request(q, bio, __split_and_process_bio); -} - -static blk_qc_t dm_make_request_nvme(struct request_queue *q, struct bio *bio) -{ - return __dm_make_request(q, bio, __process_bio); -} - static int dm_any_congested(void *congested_data, int bdi_bits) { int r = bdi_bits; @@ -2235,12 +2223,9 @@ int dm_setup_md_queue(struct mapped_devi break; case DM_TYPE_BIO_BASED: case DM_TYPE_DAX_BIO_BASED: - dm_init_normal_md_queue(md); - blk_queue_make_request(md->queue, dm_make_request); - break; case DM_TYPE_NVME_BIO_BASED: dm_init_normal_md_queue(md); - blk_queue_make_request(md->queue, dm_make_request_nvme); + blk_queue_make_request(md->queue, dm_make_request); break; case DM_TYPE_NONE: WARN_ON_ONCE(true); From patchwork Tue Nov 6 21:35:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 10671525 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6F243109C for ; Tue, 6 Nov 2018 21:55:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5ED0428FE3 for ; Tue, 6 Nov 2018 21:55:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5083B29143; Tue, 6 Nov 2018 21:55:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.4 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,SUBJ_OBFU_PUNCT_FEW,SUBJ_OBFU_PUNCT_MANY autolearn=ham version=3.3.1 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E80D028FE3 for ; Tue, 6 Nov 2018 21:55:06 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B372C8830A; Tue, 6 Nov 2018 21:55:04 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 718C860C46; Tue, 6 Nov 2018 21:55:04 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id C8CFA180B5B7; Tue, 6 Nov 2018 21:55:03 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id wA6Ld6YI004253 for ; Tue, 6 Nov 2018 16:39:06 -0500 Received: by smtp.corp.redhat.com (Postfix) id 330B216D26; Tue, 6 Nov 2018 21:39:06 +0000 (UTC) Delivered-To: dm-devel@redhat.com Received: from leontynka.twibright.com (ovpn-204-55.brq.redhat.com [10.40.204.55]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0FCDE45B8; Tue, 6 Nov 2018 21:39:02 +0000 (UTC) Received: from debian-a64.vm ([192.168.208.2]) by leontynka.twibright.com with smtp (Exim 4.89) (envelope-from ) id 1gK93r-0005NY-Gw; Tue, 06 Nov 2018 22:39:00 +0100 Received: by debian-a64.vm (sSMTP sendmail emulation); Tue, 06 Nov 2018 22:38:54 +0100 Message-Id: <20181106213854.621440000@debian-a64.vm> User-Agent: quilt/0.65 Date: Tue, 06 Nov 2018 22:35:00 +0100 From: Mikulas Patocka To: Mike Snitzer , "Alasdair G. Kergon" MIME-Version: 1.0 Content-Disposition: inline; filename=dm-dont-rewrite-in_flight.patch X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-loop: dm-devel@redhat.com Cc: dm-devel@redhat.com, Mikulas Patocka Subject: [dm-devel] [patch 2/5] dm: dont rewrite dm_disk(md)->part0.in_flight X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Tue, 06 Nov 2018 21:55:06 +0000 (UTC) X-Virus-Scanned: ClamAV using ClamSMTP generic_start_io_acct and generic_end_io_acct already update the variable in_flight using atomic operations, so we don't have to overwrite them again. Signed-off-by: Mikulas Patocka --- drivers/md/dm.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel Index: linux-2.6/drivers/md/dm.c =================================================================== --- linux-2.6.orig/drivers/md/dm.c 2018-10-10 22:24:34.090000000 +0200 +++ linux-2.6/drivers/md/dm.c 2018-10-10 23:06:24.420000000 +0200 @@ -612,8 +612,7 @@ static void start_io_acct(struct dm_io * generic_start_io_acct(md->queue, bio_op(bio), bio_sectors(bio), &dm_disk(md)->part0); - atomic_set(&dm_disk(md)->part0.in_flight[rw], - atomic_inc_return(&md->pending[rw])); + atomic_inc(&md->pending[rw]); if (unlikely(dm_stats_used(&md->stats))) dm_stats_account_io(&md->stats, bio_data_dir(bio), @@ -642,7 +641,6 @@ static void end_io_acct(struct dm_io *io * a flush. */ pending = atomic_dec_return(&md->pending[rw]); - atomic_set(&dm_disk(md)->part0.in_flight[rw], pending); pending += atomic_read(&md->pending[rw^0x1]); /* nudge anyone waiting on suspend queue */ From patchwork Tue Nov 6 21:35:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 10671537 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5DCB215A6 for ; Tue, 6 Nov 2018 21:56:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4C2B428FE3 for ; Tue, 6 Nov 2018 21:56:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4059A29168; Tue, 6 Nov 2018 21:56:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B1AF428FE3 for ; Tue, 6 Nov 2018 21:56:19 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8895C3082AF5; Tue, 6 Nov 2018 21:56:18 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 407ED101F94C; Tue, 6 Nov 2018 21:56:18 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 027DD180B5B7; Tue, 6 Nov 2018 21:56:18 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id wA6Ld7lT004259 for ; Tue, 6 Nov 2018 16:39:07 -0500 Received: by smtp.corp.redhat.com (Postfix) id 2C30A6129B; Tue, 6 Nov 2018 21:39:07 +0000 (UTC) Delivered-To: dm-devel@redhat.com Received: from leontynka.twibright.com (ovpn-204-55.brq.redhat.com [10.40.204.55]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 49532607D5; Tue, 6 Nov 2018 21:39:03 +0000 (UTC) Received: from debian-a64.vm ([192.168.208.2]) by leontynka.twibright.com with smtp (Exim 4.89) (envelope-from ) id 1gK93s-0005Nb-Ov; Tue, 06 Nov 2018 22:39:01 +0100 Received: by debian-a64.vm (sSMTP sendmail emulation); Tue, 06 Nov 2018 22:38:56 +0100 Message-Id: <20181106213855.868640480@debian-a64.vm> User-Agent: quilt/0.65 Date: Tue, 06 Nov 2018 22:35:01 +0100 From: Mikulas Patocka To: Mike Snitzer , "Alasdair G. Kergon" MIME-Version: 1.0 Content-Disposition: inline; filename=dm-remove-pending.patch X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: dm-devel@redhat.com Cc: dm-devel@redhat.com, Mikulas Patocka Subject: [dm-devel] [patch 3/5] dm: remove the pending variable X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Tue, 06 Nov 2018 21:56:19 +0000 (UTC) X-Virus-Scanned: ClamAV using ClamSMTP Remove the "pending" variable and use dm_disk(md)->part0.in_flight. These variables hold identical values. Signed-off-by: Mikulas Patocka --- drivers/md/dm-core.h | 1 - drivers/md/dm-rq.c | 4 ++-- drivers/md/dm.c | 34 ++++++++++++++++------------------ 3 files changed, 18 insertions(+), 21 deletions(-) -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel Index: linux-2.6/drivers/md/dm-core.h =================================================================== --- linux-2.6.orig/drivers/md/dm-core.h 2018-10-25 00:47:12.000000000 +0200 +++ linux-2.6/drivers/md/dm-core.h 2018-10-25 00:47:12.000000000 +0200 @@ -65,7 +65,6 @@ struct mapped_device { */ struct work_struct work; wait_queue_head_t wait; - atomic_t pending[2]; spinlock_t deferred_lock; struct bio_list deferred; Index: linux-2.6/drivers/md/dm.c =================================================================== --- linux-2.6.orig/drivers/md/dm.c 2018-10-25 00:47:12.000000000 +0200 +++ linux-2.6/drivers/md/dm.c 2018-10-25 00:47:28.000000000 +0200 @@ -597,23 +597,20 @@ static void free_tio(struct dm_target_io int md_in_flight(struct mapped_device *md) { - return atomic_read(&md->pending[READ]) + - atomic_read(&md->pending[WRITE]); + return atomic_read(&dm_disk(md)->part0.in_flight[READ]) + + atomic_read(&dm_disk(md)->part0.in_flight[WRITE]); } static void start_io_acct(struct dm_io *io) { struct mapped_device *md = io->md; struct bio *bio = io->orig_bio; - int rw = bio_data_dir(bio); io->start_time = jiffies; generic_start_io_acct(md->queue, bio_op(bio), bio_sectors(bio), &dm_disk(md)->part0); - atomic_inc(&md->pending[rw]); - if (unlikely(dm_stats_used(&md->stats))) dm_stats_account_io(&md->stats, bio_data_dir(bio), bio->bi_iter.bi_sector, bio_sectors(bio), @@ -625,27 +622,30 @@ static void end_io_acct(struct dm_io *io struct mapped_device *md = io->md; struct bio *bio = io->orig_bio; unsigned long duration = jiffies - io->start_time; - int pending; - int rw = bio_data_dir(bio); + /* + * make sure that atomic_dec in generic_end_io_acct is not reordered + * with previous writes + */ + smp_mb__before_atomic(); generic_end_io_acct(md->queue, bio_op(bio), &dm_disk(md)->part0, io->start_time); + /* + * generic_end_io_acct does atomic_dec, this barrier makes sure that + * atomic_dec is not reordered with waitqueue_active + */ + smp_mb__after_atomic(); if (unlikely(dm_stats_used(&md->stats))) dm_stats_account_io(&md->stats, bio_data_dir(bio), bio->bi_iter.bi_sector, bio_sectors(bio), true, duration, &io->stats_aux); - /* - * After this is decremented the bio must not be touched if it is - * a flush. - */ - pending = atomic_dec_return(&md->pending[rw]); - pending += atomic_read(&md->pending[rw^0x1]); - /* nudge anyone waiting on suspend queue */ - if (!pending) - wake_up(&md->wait); + if (unlikely(waitqueue_active(&md->wait))) { + if (!md_in_flight(md)) + wake_up(&md->wait); + } } /* @@ -1899,8 +1899,6 @@ static struct mapped_device *alloc_dev(i if (!md->disk) goto bad; - atomic_set(&md->pending[0], 0); - atomic_set(&md->pending[1], 0); init_waitqueue_head(&md->wait); INIT_WORK(&md->work, dm_wq_work); init_waitqueue_head(&md->eventq); Index: linux-2.6/drivers/md/dm-rq.c =================================================================== --- linux-2.6.orig/drivers/md/dm-rq.c 2018-10-25 00:47:12.000000000 +0200 +++ linux-2.6/drivers/md/dm-rq.c 2018-10-25 00:47:12.000000000 +0200 @@ -182,7 +182,7 @@ static void rq_completed(struct mapped_d struct request_queue *q = md->queue; unsigned long flags; - atomic_dec(&md->pending[rw]); + atomic_dec(&dm_disk(md)->part0.in_flight[rw]); /* nudge anyone waiting on suspend queue */ if (!md_in_flight(md)) @@ -536,7 +536,7 @@ static void dm_start_request(struct mapp blk_start_request(orig); else blk_mq_start_request(orig); - atomic_inc(&md->pending[rq_data_dir(orig)]); + atomic_inc(&dm_disk(md)->part0.in_flight[rq_data_dir(orig)]); if (md->seq_rq_merge_deadline_usecs) { md->last_rq_pos = rq_end_sector(orig); From patchwork Tue Nov 6 21:35:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 10671527 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 868AF109C for ; Tue, 6 Nov 2018 21:55:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7662E28FE3 for ; Tue, 6 Nov 2018 21:55:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6A58E29143; Tue, 6 Nov 2018 21:55:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E1F5828FE3 for ; Tue, 6 Nov 2018 21:55:17 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 131402F3D; Tue, 6 Nov 2018 21:55:17 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id CFA11AF8D9; Tue, 6 Nov 2018 21:55:16 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 8CDB3180B5B7; Tue, 6 Nov 2018 21:55:16 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id wA6Ld7xs004262 for ; Tue, 6 Nov 2018 16:39:07 -0500 Received: by smtp.corp.redhat.com (Postfix) id 3CBFB607DE; Tue, 6 Nov 2018 21:39:07 +0000 (UTC) Delivered-To: dm-devel@redhat.com Received: from leontynka.twibright.com (ovpn-204-55.brq.redhat.com [10.40.204.55]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A4E6660C4D; Tue, 6 Nov 2018 21:39:04 +0000 (UTC) Received: from debian-a64.vm ([192.168.208.2]) by leontynka.twibright.com with smtp (Exim 4.89) (envelope-from ) id 1gK93u-0005Ne-1P; Tue, 06 Nov 2018 22:39:03 +0100 Received: by debian-a64.vm (sSMTP sendmail emulation); Tue, 06 Nov 2018 22:38:57 +0100 Message-Id: <20181106213857.112447880@debian-a64.vm> User-Agent: quilt/0.65 Date: Tue, 06 Nov 2018 22:35:02 +0100 From: Mikulas Patocka To: Mike Snitzer , "Alasdair G. Kergon" MIME-Version: 1.0 Content-Disposition: inline; filename=dm-percpu.patch X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: dm-devel@redhat.com Cc: dm-devel@redhat.com, Mikulas Patocka Subject: [dm-devel] [patch 4/5] dm: use percpu counters X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Tue, 06 Nov 2018 21:55:17 +0000 (UTC) X-Virus-Scanned: ClamAV using ClamSMTP Use percpu inflight counters to avoid cache line bouncing and improve performance. Signed-off-by: Mikulas Patocka --- drivers/md/dm-core.h | 5 ++++ drivers/md/dm-rq.c | 10 +++++++-- drivers/md/dm.c | 52 +++++++++++++++++++++++++++++++++++---------------- 3 files changed, 49 insertions(+), 18 deletions(-) -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel Index: linux-2.6/drivers/md/dm-core.h =================================================================== --- linux-2.6.orig/drivers/md/dm-core.h 2018-11-06 00:45:06.320000000 +0100 +++ linux-2.6/drivers/md/dm-core.h 2018-11-06 00:45:35.640000000 +0100 @@ -24,6 +24,10 @@ struct dm_kobject_holder { struct completion completion; }; +struct dm_percpu { + unsigned inflight[2]; +}; + /* * DM core internal structure that used directly by dm.c and dm-rq.c * DM targets must _not_ deference a mapped_device to directly access its members! @@ -63,6 +67,7 @@ struct mapped_device { /* * A list of ios that arrived while we were suspended. */ + struct dm_percpu __percpu *counters; struct work_struct work; wait_queue_head_t wait; spinlock_t deferred_lock; Index: linux-2.6/drivers/md/dm.c =================================================================== --- linux-2.6.orig/drivers/md/dm.c 2018-11-06 00:45:06.320000000 +0100 +++ linux-2.6/drivers/md/dm.c 2018-11-06 22:31:33.980000000 +0100 @@ -597,19 +597,33 @@ static void free_tio(struct dm_target_io int md_in_flight(struct mapped_device *md) { - return atomic_read(&dm_disk(md)->part0.in_flight[READ]) + - atomic_read(&dm_disk(md)->part0.in_flight[WRITE]); + int cpu; + unsigned sum = 0; + for_each_possible_cpu(cpu) { + struct dm_percpu *p = per_cpu_ptr(md->counters, cpu); + sum += p->inflight[READ] + p->inflight[WRITE]; + } + return (int)sum; } static void start_io_acct(struct dm_io *io) { struct mapped_device *md = io->md; struct bio *bio = io->orig_bio; + struct hd_struct *part; + int sgrp, cpu; io->start_time = jiffies; - generic_start_io_acct(md->queue, bio_op(bio), bio_sectors(bio), - &dm_disk(md)->part0); + part = &dm_disk(md)->part0; + sgrp = op_stat_group(bio_op(bio)); + cpu = part_stat_lock(); + part_round_stats(md->queue, cpu, part); + part_stat_inc(cpu, part, ios[sgrp]); + part_stat_add(cpu, part, sectors[sgrp], bio_sectors(bio)); + part_stat_unlock(); + + this_cpu_inc(md->counters->inflight[bio_data_dir(bio)]); if (unlikely(dm_stats_used(&md->stats))) dm_stats_account_io(&md->stats, bio_data_dir(bio), @@ -622,25 +636,25 @@ static void end_io_acct(struct dm_io *io struct mapped_device *md = io->md; struct bio *bio = io->orig_bio; unsigned long duration = jiffies - io->start_time; + struct hd_struct *part; + int sgrp, cpu; - /* - * make sure that atomic_dec in generic_end_io_acct is not reordered - * with previous writes - */ - smp_mb__before_atomic(); - generic_end_io_acct(md->queue, bio_op(bio), &dm_disk(md)->part0, - io->start_time); - /* - * generic_end_io_acct does atomic_dec, this barrier makes sure that - * atomic_dec is not reordered with waitqueue_active - */ - smp_mb__after_atomic(); + part = &dm_disk(md)->part0; + sgrp = op_stat_group(bio_op(bio)); + cpu = part_stat_lock(); + part_stat_add(cpu, part, nsecs[sgrp], jiffies_to_nsecs(duration)); + part_round_stats(md->queue, cpu, part); + part_stat_unlock(); + + smp_wmb(); + this_cpu_dec(md->counters->inflight[bio_data_dir(bio)]); if (unlikely(dm_stats_used(&md->stats))) dm_stats_account_io(&md->stats, bio_data_dir(bio), bio->bi_iter.bi_sector, bio_sectors(bio), true, duration, &io->stats_aux); + smp_mb(); /* nudge anyone waiting on suspend queue */ if (unlikely(waitqueue_active(&md->wait))) { if (!md_in_flight(md)) @@ -1828,6 +1842,8 @@ static void cleanup_mapped_device(struct if (md->queue) blk_cleanup_queue(md->queue); + free_percpu(md->counters); + cleanup_srcu_struct(&md->io_barrier); if (md->bdev) { @@ -1899,6 +1915,10 @@ static struct mapped_device *alloc_dev(i if (!md->disk) goto bad; + md->counters = alloc_percpu(struct dm_percpu); + if (!md->counters) + goto bad; + init_waitqueue_head(&md->wait); INIT_WORK(&md->work, dm_wq_work); init_waitqueue_head(&md->eventq); Index: linux-2.6/drivers/md/dm-rq.c =================================================================== --- linux-2.6.orig/drivers/md/dm-rq.c 2018-10-30 15:09:45.770000000 +0100 +++ linux-2.6/drivers/md/dm-rq.c 2018-11-06 00:53:13.870000000 +0100 @@ -172,6 +172,12 @@ static void rq_end_stats(struct mapped_d } } +static unsigned rq_md_in_flight(struct mapped_device *md) +{ + return atomic_read(&dm_disk(md)->part0.in_flight[READ]) + + atomic_read(&dm_disk(md)->part0.in_flight[WRITE]); +} + /* * Don't touch any member of the md after calling this function because * the md may be freed in dm_put() at the end of this function. @@ -185,7 +191,7 @@ static void rq_completed(struct mapped_d atomic_dec(&dm_disk(md)->part0.in_flight[rw]); /* nudge anyone waiting on suspend queue */ - if (!md_in_flight(md)) + if (!rq_md_in_flight(md)) wake_up(&md->wait); /* @@ -674,7 +680,7 @@ static void dm_old_request_fn(struct req pos = blk_rq_pos(rq); if ((dm_old_request_peeked_before_merge_deadline(md) && - md_in_flight(md) && rq->bio && !bio_multiple_segments(rq->bio) && + rq_md_in_flight(md) && rq->bio && !bio_multiple_segments(rq->bio) && md->last_rq_pos == pos && md->last_rq_rw == rq_data_dir(rq)) || (ti->type->busy && ti->type->busy(ti))) { blk_delay_queue(q, 10); From patchwork Tue Nov 6 21:35:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 10671509 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9513614E2 for ; Tue, 6 Nov 2018 21:52:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 83E6A2AD58 for ; Tue, 6 Nov 2018 21:52:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 783552ADCD; Tue, 6 Nov 2018 21:52:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id DBF1B2AD58 for ; Tue, 6 Nov 2018 21:52:14 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C06378830A; Tue, 6 Nov 2018 21:52:13 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1DDF060C5C; Tue, 6 Nov 2018 21:52:13 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 6F40D180B5B7; Tue, 6 Nov 2018 21:52:12 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id wA6LdAIB004285 for ; Tue, 6 Nov 2018 16:39:10 -0500 Received: by smtp.corp.redhat.com (Postfix) id 79FE65D76C; Tue, 6 Nov 2018 21:39:10 +0000 (UTC) Delivered-To: dm-devel@redhat.com Received: from leontynka.twibright.com (ovpn-204-55.brq.redhat.com [10.40.204.55]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 75D495D760; Tue, 6 Nov 2018 21:39:05 +0000 (UTC) Received: from debian-a64.vm ([192.168.208.2]) by leontynka.twibright.com with smtp (Exim 4.89) (envelope-from ) id 1gK93v-0005Nh-AW; Tue, 06 Nov 2018 22:39:04 +0100 Received: by debian-a64.vm (sSMTP sendmail emulation); Tue, 06 Nov 2018 22:38:58 +0100 Message-Id: <20181106213858.391264280@debian-a64.vm> User-Agent: quilt/0.65 Date: Tue, 06 Nov 2018 22:35:03 +0100 From: Mikulas Patocka To: Mike Snitzer , "Alasdair G. Kergon" MIME-Version: 1.0 Content-Disposition: inline; filename=dm-inflight.patch X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-loop: dm-devel@redhat.com Cc: dm-devel@redhat.com, Mikulas Patocka Subject: [dm-devel] [patch 5/5] block: use a driver-specific handler for the "inflight" value X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Tue, 06 Nov 2018 21:52:14 +0000 (UTC) X-Virus-Scanned: ClamAV using ClamSMTP Device mapper was converted to percpu inflight counters. In order to display the correct values in the "inflight" sysfs file, we need a custom callback that sums the percpu counters. Signed-off-by: Mikulas Patocka --- block/blk-settings.c | 6 ++++++ block/genhd.c | 5 +++++ drivers/md/dm.c | 18 ++++++++++++++++++ include/linux/blkdev.h | 4 ++++ 4 files changed, 33 insertions(+) -- dm-devel mailing list dm-devel@redhat.com https://www.redhat.com/mailman/listinfo/dm-devel Index: linux-2.6/block/genhd.c =================================================================== --- linux-2.6.orig/block/genhd.c 2018-11-06 22:31:46.350000000 +0100 +++ linux-2.6/block/genhd.c 2018-11-06 22:31:46.350000000 +0100 @@ -85,6 +85,11 @@ void part_in_flight(struct request_queue void part_in_flight_rw(struct request_queue *q, struct hd_struct *part, unsigned int inflight[2]) { + if (q->get_inflight_fn) { + q->get_inflight_fn(q, inflight); + return; + } + if (q->mq_ops) { blk_mq_in_flight_rw(q, part, inflight); return; Index: linux-2.6/include/linux/blkdev.h =================================================================== --- linux-2.6.orig/include/linux/blkdev.h 2018-11-06 22:31:46.350000000 +0100 +++ linux-2.6/include/linux/blkdev.h 2018-11-06 22:31:46.350000000 +0100 @@ -324,6 +324,7 @@ typedef int (lld_busy_fn) (struct reques typedef int (bsg_job_fn) (struct bsg_job *); typedef int (init_rq_fn)(struct request_queue *, struct request *, gfp_t); typedef void (exit_rq_fn)(struct request_queue *, struct request *); +typedef void (get_inflight_fn)(struct request_queue *, unsigned int [2]); enum blk_eh_timer_return { BLK_EH_DONE, /* drivers has completed the command */ @@ -466,6 +467,8 @@ struct request_queue { exit_rq_fn *exit_rq_fn; /* Called from inside blk_get_request() */ void (*initialize_rq_fn)(struct request *rq); + /* Called to get the "inflight" values */ + get_inflight_fn *get_inflight_fn; const struct blk_mq_ops *mq_ops; @@ -1232,6 +1235,7 @@ extern int blk_queue_dma_drain(struct re dma_drain_needed_fn *dma_drain_needed, void *buf, unsigned int size); extern void blk_queue_lld_busy(struct request_queue *q, lld_busy_fn *fn); +extern void blk_queue_get_inflight(struct request_queue *q, get_inflight_fn *fn); extern void blk_queue_segment_boundary(struct request_queue *, unsigned long); extern void blk_queue_virt_boundary(struct request_queue *, unsigned long); extern void blk_queue_prep_rq(struct request_queue *, prep_rq_fn *pfn); Index: linux-2.6/block/blk-settings.c =================================================================== --- linux-2.6.orig/block/blk-settings.c 2018-11-06 22:31:46.350000000 +0100 +++ linux-2.6/block/blk-settings.c 2018-11-06 22:31:46.350000000 +0100 @@ -79,6 +79,12 @@ void blk_queue_lld_busy(struct request_q } EXPORT_SYMBOL_GPL(blk_queue_lld_busy); +void blk_queue_get_inflight(struct request_queue *q, get_inflight_fn *fn) +{ + q->get_inflight_fn = fn; +} +EXPORT_SYMBOL_GPL(blk_queue_get_inflight); + /** * blk_set_default_limits - reset limits to default values * @lim: the queue_limits structure to reset Index: linux-2.6/drivers/md/dm.c =================================================================== --- linux-2.6.orig/drivers/md/dm.c 2018-11-06 22:31:46.350000000 +0100 +++ linux-2.6/drivers/md/dm.c 2018-11-06 22:31:46.350000000 +0100 @@ -662,6 +662,23 @@ static void end_io_acct(struct dm_io *io } } +static void dm_get_inflight(struct request_queue *q, unsigned int inflight[2]) +{ + struct mapped_device *md = q->queuedata; + int cpu; + + inflight[READ] = inflight[WRITE] = 0; + for_each_possible_cpu(cpu) { + struct dm_percpu *p = per_cpu_ptr(md->counters, cpu); + inflight[READ] += p->inflight[READ]; + inflight[WRITE] += p->inflight[WRITE]; + } + if ((int)inflight[READ] < 0) + inflight[READ] = 0; + if ((int)inflight[WRITE] < 0) + inflight[WRITE] = 0; +} + /* * Add the bio to the list of deferred io. */ @@ -2242,6 +2259,7 @@ int dm_setup_md_queue(struct mapped_devi case DM_TYPE_NVME_BIO_BASED: dm_init_normal_md_queue(md); blk_queue_make_request(md->queue, dm_make_request); + blk_queue_get_inflight(md->queue, dm_get_inflight); break; case DM_TYPE_NONE: WARN_ON_ONCE(true);