From patchwork Tue Apr 12 08:56:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12810237 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E90B6C433EF for ; Tue, 12 Apr 2022 08:56:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649753803; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=NmBb6M7rFf0wxbZpcL8Xm5cl5FqUfBVQPnxJETvDMvw=; b=WvtTPDh0jOjjdupQUCiKsdeMr1LgvvaYQV+DU3G/jMJgCkZDqoawUvu5V8sVWeXgdUDql1 cM5s14k3fm5x6srjK7zLlV/oKoAI3FjGI8+MaxhjFIWI+USaD9+SgsKCCv2bTOZAW5/NpX 00U60KEO5cJ+GGNfd0TvY7ELQrD9h+s= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-365-gNfyZK3mN6KMfvRrJ8F1EA-1; Tue, 12 Apr 2022 04:56:40 -0400 X-MC-Unique: gNfyZK3mN6KMfvRrJ8F1EA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 12194811E83; Tue, 12 Apr 2022 08:56:39 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 010C21400AFB; Tue, 12 Apr 2022 08:56:39 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 9B5171949762; Tue, 12 Apr 2022 08:56:38 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 8394D1947BBF for ; Tue, 12 Apr 2022 08:56:37 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 68B7540CFD1E; Tue, 12 Apr 2022 08:56:37 +0000 (UTC) Received: from localhost (ovpn-8-19.pek2.redhat.com [10.72.8.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9E23C40CFD1D; Tue, 12 Apr 2022 08:56:36 +0000 (UTC) From: Ming Lei To: Jens Axboe , Mike Snitzer Date: Tue, 12 Apr 2022 16:56:09 +0800 Message-Id: <20220412085616.1409626-2-ming.lei@redhat.com> In-Reply-To: <20220412085616.1409626-1-ming.lei@redhat.com> References: <20220412085616.1409626-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.1 Subject: [dm-devel] [PATCH 1/8] block: replace disk based account with bdev's X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Damien Le Moal , Ming Lei Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com 'block device' is generic type for interface, and gendisk becomes more one block layer internal type, so replace disk based account interface with bdec's. Also add 'start_time' parameter to bdev_start_io_acct() so that we can cover device mapper's io accounting by the two bdev based interface. Signed-off-by: Ming Lei --- block/blk-core.c | 15 ++++++++------- drivers/block/zram/zram_drv.c | 5 +++-- include/linux/blkdev.h | 7 ++++--- 3 files changed, 15 insertions(+), 12 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 937bb6b86331..a3ae13b129ff 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1063,12 +1063,13 @@ unsigned long bio_start_io_acct(struct bio *bio) } EXPORT_SYMBOL_GPL(bio_start_io_acct); -unsigned long disk_start_io_acct(struct gendisk *disk, unsigned int sectors, - unsigned int op) +unsigned long bdev_start_io_acct(struct block_device *bdev, + unsigned int sectors, unsigned int op, + unsigned long start_time) { - return __part_start_io_acct(disk->part0, sectors, op, jiffies); + return __part_start_io_acct(bdev, sectors, op, start_time); } -EXPORT_SYMBOL(disk_start_io_acct); +EXPORT_SYMBOL(bdev_start_io_acct); static void __part_end_io_acct(struct block_device *part, unsigned int op, unsigned long start_time) @@ -1091,12 +1092,12 @@ void bio_end_io_acct_remapped(struct bio *bio, unsigned long start_time, } EXPORT_SYMBOL_GPL(bio_end_io_acct_remapped); -void disk_end_io_acct(struct gendisk *disk, unsigned int op, +void bdev_end_io_acct(struct block_device *bdev, unsigned int op, unsigned long start_time) { - __part_end_io_acct(disk->part0, op, start_time); + __part_end_io_acct(bdev, op, start_time); } -EXPORT_SYMBOL(disk_end_io_acct); +EXPORT_SYMBOL(bdev_end_io_acct); /** * blk_lld_busy - Check if underlying low-level drivers of a device are busy diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index e9474b02012d..adb5209a556a 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1675,9 +1675,10 @@ static int zram_rw_page(struct block_device *bdev, sector_t sector, bv.bv_len = PAGE_SIZE; bv.bv_offset = 0; - start_time = disk_start_io_acct(bdev->bd_disk, SECTORS_PER_PAGE, op); + start_time = bdev_start_io_acct(bdev->bd_disk->part0, + SECTORS_PER_PAGE, op, jiffies); ret = zram_bvec_rw(zram, &bv, index, offset, op, NULL); - disk_end_io_acct(bdev->bd_disk, op, start_time); + bdev_end_io_acct(bdev->bd_disk->part0, op, start_time); out: /* * If I/O fails, just return error(ie, non-zero) without diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 60d016138997..f680ba6f0ab2 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1491,9 +1491,10 @@ static inline void blk_wake_io_task(struct task_struct *waiter) wake_up_process(waiter); } -unsigned long disk_start_io_acct(struct gendisk *disk, unsigned int sectors, - unsigned int op); -void disk_end_io_acct(struct gendisk *disk, unsigned int op, +unsigned long bdev_start_io_acct(struct block_device *bdev, + unsigned int sectors, unsigned int op, + unsigned long start_time); +void bdev_end_io_acct(struct block_device *bdev, unsigned int op, unsigned long start_time); void bio_start_io_acct_time(struct bio *bio, unsigned long start_time); From patchwork Tue Apr 12 08:56:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12810238 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7651C4167E for ; Tue, 12 Apr 2022 08:57:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649753828; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=KtxJr51rdJ/8zkVd3HiHAHlCtKhYX5OOlOe62y4vEa4=; b=iGjFivZIqYvSQcaCTaLov5BJ9LoohcHTHwwvYpX3fzEykzKeZseQCNIr95Sa5tGRvArcHo k/4leqOdU85jSRa8onDxk3IEgGMnP4oDoP+siR7SiDUvOyWz1ard5JbcftTl1/Xb39Za15 GvM66JkvbMdzD0Fpegw5HxGoDmci6do= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-643-kG87rOsuPHGENYAOyyOdag-1; Tue, 12 Apr 2022 04:57:06 -0400 X-MC-Unique: kG87rOsuPHGENYAOyyOdag-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 74EB73803907; Tue, 12 Apr 2022 08:57:04 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2F64E1400AFB; Tue, 12 Apr 2022 08:57:04 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id DC3731940365; Tue, 12 Apr 2022 08:57:03 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 1C5691947BBF for ; Tue, 12 Apr 2022 08:57:03 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id E0A922166B1E; Tue, 12 Apr 2022 08:57:02 +0000 (UTC) Received: from localhost (ovpn-8-19.pek2.redhat.com [10.72.8.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id 540482166B1B; Tue, 12 Apr 2022 08:56:39 +0000 (UTC) From: Ming Lei To: Jens Axboe , Mike Snitzer Date: Tue, 12 Apr 2022 16:56:10 +0800 Message-Id: <20220412085616.1409626-3-ming.lei@redhat.com> In-Reply-To: <20220412085616.1409626-1-ming.lei@redhat.com> References: <20220412085616.1409626-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 Subject: [dm-devel] [PATCH 2/8] dm: don't pass bio to __dm_start_io_acct and dm_end_io_acct X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Damien Le Moal , Ming Lei Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com dm->orig_bio is always passed to __dm_start_io_acct and dm_end_io_acct, so it isn't necessary to take one bio parameter for the two helpers. Signed-off-by: Ming Lei --- drivers/md/dm.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 3c5fad7c4ee6..62f7af815ef8 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -526,16 +526,13 @@ static void dm_io_acct(bool end, struct mapped_device *md, struct bio *bio, bio->bi_iter.bi_size = bi_size; } -static void __dm_start_io_acct(struct dm_io *io, struct bio *bio) +static void __dm_start_io_acct(struct dm_io *io) { - dm_io_acct(false, io->md, bio, io->start_time, &io->stats_aux); + dm_io_acct(false, io->md, io->orig_bio, io->start_time, &io->stats_aux); } static void dm_start_io_acct(struct dm_io *io, struct bio *clone) { - /* Must account IO to DM device in terms of orig_bio */ - struct bio *bio = io->orig_bio; - /* * Ensure IO accounting is only ever started once. * Expect no possibility for race unless DM_TIO_IS_DUPLICATE_BIO. @@ -555,12 +552,12 @@ static void dm_start_io_acct(struct dm_io *io, struct bio *clone) spin_unlock_irqrestore(&io->lock, flags); } - __dm_start_io_acct(io, bio); + __dm_start_io_acct(io); } -static void dm_end_io_acct(struct dm_io *io, struct bio *bio) +static void dm_end_io_acct(struct dm_io *io) { - dm_io_acct(true, io->md, bio, io->start_time, &io->stats_aux); + dm_io_acct(true, io->md, io->orig_bio, io->start_time, &io->stats_aux); } static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio) @@ -875,14 +872,14 @@ static void dm_io_complete(struct dm_io *io) io_error = io->status; if (dm_io_flagged(io, DM_IO_ACCOUNTED)) - dm_end_io_acct(io, bio); + dm_end_io_acct(io); else if (!io_error) { /* * Must handle target that DM_MAPIO_SUBMITTED only to * then bio_endio() rather than dm_submit_bio_remap() */ - __dm_start_io_acct(io, bio); - dm_end_io_acct(io, bio); + __dm_start_io_acct(io); + dm_end_io_acct(io); } free_io(io); smp_wmb(); From patchwork Tue Apr 12 08:56:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12810240 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B8C95C433EF for ; Tue, 12 Apr 2022 08:57:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649753838; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=vMhjms0E2gSV96dapVnhhioWpXGd9PiB2zNNauuoono=; b=KASLsmKMoCdLZn+fBH+3JZ1SagNZ4pM4EeayC++D3JZR2Q2Kxegf/n1TzaLhxt0f7p9Zqr uMGgC+i60viYkBUpciznGM35WIwdVTfR6U/b5y813Xry5u7cj5az6JHWzQ1V/gi43Kaicg QqmXqSYX1QdqvWjYspTI+nthtbY3Ygc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-633-BPxgMM_APi6-UTpeXGRguw-1; Tue, 12 Apr 2022 04:57:10 -0400 X-MC-Unique: BPxgMM_APi6-UTpeXGRguw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8F0C71014A67; Tue, 12 Apr 2022 08:57:08 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7C7DF2166BDC; Tue, 12 Apr 2022 08:57:08 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 1FCEE1949762; Tue, 12 Apr 2022 08:57:08 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 0E8151947BBF for ; Tue, 12 Apr 2022 08:57:07 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id F2A111400AFB; Tue, 12 Apr 2022 08:57:06 +0000 (UTC) Received: from localhost (ovpn-8-19.pek2.redhat.com [10.72.8.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0D73E145BA42; Tue, 12 Apr 2022 08:57:05 +0000 (UTC) From: Ming Lei To: Jens Axboe , Mike Snitzer Date: Tue, 12 Apr 2022 16:56:11 +0800 Message-Id: <20220412085616.1409626-4-ming.lei@redhat.com> In-Reply-To: <20220412085616.1409626-1-ming.lei@redhat.com> References: <20220412085616.1409626-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 Subject: [dm-devel] [PATCH 3/8] dm: pass 'dm_io' instance to dm_io_acct directly X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Damien Le Moal , Ming Lei Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com All the other 4 parameters are retrieved from the 'dm_io' instance, so not necessary to pass all four to dm_io_acct(). Signed-off-by: Ming Lei --- drivers/md/dm.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 62f7af815ef8..ed85cd1165a4 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -498,9 +498,12 @@ static bool bio_is_flush_with_data(struct bio *bio) return ((bio->bi_opf & REQ_PREFLUSH) && bio->bi_iter.bi_size); } -static void dm_io_acct(bool end, struct mapped_device *md, struct bio *bio, - unsigned long start_time, struct dm_stats_aux *stats_aux) +static void dm_io_acct(struct dm_io *io, bool end) { + struct dm_stats_aux *stats_aux = &io->stats_aux; + unsigned long start_time = io->start_time; + struct mapped_device *md = io->md; + struct bio *bio = io->orig_bio; bool is_flush_with_data; unsigned int bi_size; @@ -528,7 +531,7 @@ static void dm_io_acct(bool end, struct mapped_device *md, struct bio *bio, static void __dm_start_io_acct(struct dm_io *io) { - dm_io_acct(false, io->md, io->orig_bio, io->start_time, &io->stats_aux); + dm_io_acct(io, false); } static void dm_start_io_acct(struct dm_io *io, struct bio *clone) @@ -557,7 +560,7 @@ static void dm_start_io_acct(struct dm_io *io, struct bio *clone) static void dm_end_io_acct(struct dm_io *io) { - dm_io_acct(true, io->md, io->orig_bio, io->start_time, &io->stats_aux); + dm_io_acct(io, true); } static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio) From patchwork Tue Apr 12 08:56:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12810239 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8355EC433FE for ; Tue, 12 Apr 2022 08:57:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649753837; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=HWUjNqR6a0KAVssoySThku2mZK95OEt5SIaY1cRN/28=; b=Q88MlarQgumSBGhLWAM6MUra0eCzkbf3QCR+Q+IW0yB3UoWQ6nRr5fxIfJCJeQtgetd8Nd svsWicnO0naSUIjbGZOjTY50/zwwx6PIwoVarQtB15R8k3JrGFStlQr+K0bcEPEkAnxNPY NNVe6KeOaa9uLhoAILYckgUNrJ/rw+k= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-115-R91ZGTpBOZW5orOtMbb-DQ-1; Tue, 12 Apr 2022 04:57:14 -0400 X-MC-Unique: R91ZGTpBOZW5orOtMbb-DQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A7B49800B28; Tue, 12 Apr 2022 08:57:12 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9534441639E; Tue, 12 Apr 2022 08:57:12 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 61AC81940365; Tue, 12 Apr 2022 08:57:12 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id A033C1947BBF for ; Tue, 12 Apr 2022 08:57:10 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 7F96BC44B1E; Tue, 12 Apr 2022 08:57:10 +0000 (UTC) Received: from localhost (ovpn-8-19.pek2.redhat.com [10.72.8.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id 83520C44B1D; Tue, 12 Apr 2022 08:57:09 +0000 (UTC) From: Ming Lei To: Jens Axboe , Mike Snitzer Date: Tue, 12 Apr 2022 16:56:12 +0800 Message-Id: <20220412085616.1409626-5-ming.lei@redhat.com> In-Reply-To: <20220412085616.1409626-1-ming.lei@redhat.com> References: <20220412085616.1409626-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.8 Subject: [dm-devel] [PATCH 4/8] dm: switch to bdev based io accounting interface X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Damien Le Moal , Ming Lei Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com DM won't account sectors in flush IO, also we can retrieve sectors from 'dm_io' for avoiding to allocate & update new original bio, which will be done in the following patch. So switch to bdev based io accounting interface. Signed-off-by: Ming Lei --- drivers/md/dm.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index ed85cd1165a4..31eacc0e93ed 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -504,29 +504,24 @@ static void dm_io_acct(struct dm_io *io, bool end) unsigned long start_time = io->start_time; struct mapped_device *md = io->md; struct bio *bio = io->orig_bio; - bool is_flush_with_data; - unsigned int bi_size; + unsigned int sectors; /* If REQ_PREFLUSH set save any payload but do not account it */ - is_flush_with_data = bio_is_flush_with_data(bio); - if (is_flush_with_data) { - bi_size = bio->bi_iter.bi_size; - bio->bi_iter.bi_size = 0; - } + if (bio_is_flush_with_data(bio)) + sectors = 0; + else + sectors = bio_sectors(bio); if (!end) - bio_start_io_acct_time(bio, start_time); + bdev_start_io_acct(bio->bi_bdev, sectors, bio_op(bio), + start_time); else - bio_end_io_acct(bio, start_time); + bdev_end_io_acct(bio->bi_bdev, bio_op(bio), start_time); if (unlikely(dm_stats_used(&md->stats))) dm_stats_account_io(&md->stats, bio_data_dir(bio), bio->bi_iter.bi_sector, bio_sectors(bio), end, start_time, stats_aux); - - /* Restore bio's payload so it does get accounted upon requeue */ - if (is_flush_with_data) - bio->bi_iter.bi_size = bi_size; } static void __dm_start_io_acct(struct dm_io *io) From patchwork Tue Apr 12 08:56:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12810241 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9E8EBC433EF for ; Tue, 12 Apr 2022 08:57:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649753842; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=I/wbdTybM5EoAkgoFwuMIDJZitQqZGJdZIyDrPUbn/A=; b=fffAqvaMQ+DlMSzKOr0TW5yeYuJy/xymI6YVeCjPMtLzVNoXlwBwgFoQ+njppQbVLulzfo 32Z3X94V/XwOqsWGlvCEZyZ8G7QIutJYrpf23xp+Jo3kB6ao/GFMzDYBVEEVxwgkAUIz3Y cKtPS4YTdaa9Hlmb0ldoW8qYf2YCv9A= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-399-sC_Cl6hVPdqQuDuqGRt0VA-1; Tue, 12 Apr 2022 04:57:19 -0400 X-MC-Unique: sC_Cl6hVPdqQuDuqGRt0VA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D39303C14842; Tue, 12 Apr 2022 08:57:17 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id C27AD54ACAF; Tue, 12 Apr 2022 08:57:17 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 9AFDF1949762; Tue, 12 Apr 2022 08:57:17 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 1AB8D1947BBF for ; Tue, 12 Apr 2022 08:57:16 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id DF5D240C1241; Tue, 12 Apr 2022 08:57:15 +0000 (UTC) Received: from localhost (ovpn-8-19.pek2.redhat.com [10.72.8.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id E878040470F2; Tue, 12 Apr 2022 08:57:13 +0000 (UTC) From: Ming Lei To: Jens Axboe , Mike Snitzer Date: Tue, 12 Apr 2022 16:56:13 +0800 Message-Id: <20220412085616.1409626-6-ming.lei@redhat.com> In-Reply-To: <20220412085616.1409626-1-ming.lei@redhat.com> References: <20220412085616.1409626-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Subject: [dm-devel] [PATCH 5/8] dm: always setup ->orig_bio in alloc_io X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Damien Le Moal , Ming Lei Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com The current DM codes setup ->orig_bio after __map_bio() returns, and not only cause kernel panic for dm zone, but also a bit ugly and tricky, especially the waiting until ->orig_bio is set in dm_submit_bio_remap(). The reason is that one new bio is cloned from original FS bio to represent the mapped part, which just serves io accounting. Now we have switched to bdev based io accounting interface, and we can retrieve sectors/bio_op from both the real original bio and the added fields of .sector_offset & .sectors easily, so the new cloned bio isn't necessary any more. Not only fixes dm-zone's kernel panic, but also cleans up dm io accounting & split a bit. Signed-off-by: Ming Lei --- drivers/md/dm-core.h | 8 ++++++- drivers/md/dm.c | 51 ++++++++++++++++++++++---------------------- 2 files changed, 32 insertions(+), 27 deletions(-) diff --git a/drivers/md/dm-core.h b/drivers/md/dm-core.h index 4277853c7535..aefb080c230d 100644 --- a/drivers/md/dm-core.h +++ b/drivers/md/dm-core.h @@ -247,7 +247,12 @@ struct dm_io { blk_short_t flags; atomic_t io_count; struct mapped_device *md; + + /* The three fields represent mapped part of original bio */ struct bio *orig_bio; + unsigned int sector_offset; /* offset to end of orig_bio */ + unsigned int sectors; + blk_status_t status; spinlock_t lock; unsigned long start_time; @@ -264,7 +269,8 @@ struct dm_io { */ enum { DM_IO_START_ACCT, - DM_IO_ACCOUNTED + DM_IO_ACCOUNTED, + DM_IO_SPLITTED }; static inline bool dm_io_flagged(struct dm_io *io, unsigned int bit) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 31eacc0e93ed..df1d013fb793 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -509,8 +509,10 @@ static void dm_io_acct(struct dm_io *io, bool end) /* If REQ_PREFLUSH set save any payload but do not account it */ if (bio_is_flush_with_data(bio)) sectors = 0; - else + else if (likely(!(dm_io_flagged(io, DM_IO_SPLITTED)))) sectors = bio_sectors(bio); + else + sectors = io->sectors; if (!end) bdev_start_io_acct(bio->bi_bdev, sectors, bio_op(bio), @@ -518,10 +520,21 @@ static void dm_io_acct(struct dm_io *io, bool end) else bdev_end_io_acct(bio->bi_bdev, bio_op(bio), start_time); - if (unlikely(dm_stats_used(&md->stats))) + if (unlikely(dm_stats_used(&md->stats))) { + sector_t sector; + + if (likely(!dm_io_flagged(io, DM_IO_SPLITTED))) { + sector = bio->bi_iter.bi_sector; + sectors = bio_sectors(bio); + } else { + sector = bio_end_sector(bio) - io->sector_offset; + sectors = io->sectors; + } + dm_stats_account_io(&md->stats, bio_data_dir(bio), - bio->bi_iter.bi_sector, bio_sectors(bio), + sector, sectors, end, start_time, stats_aux); + } } static void __dm_start_io_acct(struct dm_io *io) @@ -576,7 +589,7 @@ static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio) io->status = 0; atomic_set(&io->io_count, 1); this_cpu_inc(*md->pending_io); - io->orig_bio = NULL; + io->orig_bio = bio; io->md = md; io->map_task = current; spin_lock_init(&io->lock); @@ -1222,13 +1235,6 @@ void dm_submit_bio_remap(struct bio *clone, struct bio *tgt_clone) /* Still in target's map function */ dm_io_set_flag(io, DM_IO_START_ACCT); } else { - /* - * Called by another thread, managed by DM target, - * wait for dm_split_and_process_bio() to store - * io->orig_bio - */ - while (unlikely(!smp_load_acquire(&io->orig_bio))) - msleep(1); dm_start_io_acct(io, clone); } @@ -1557,7 +1563,6 @@ static void dm_split_and_process_bio(struct mapped_device *md, struct dm_table *map, struct bio *bio) { struct clone_info ci; - struct bio *orig_bio = NULL; int error = 0; init_clone_info(&ci, md, map, bio); @@ -1573,22 +1578,16 @@ static void dm_split_and_process_bio(struct mapped_device *md, if (error || !ci.sector_count) goto out; - /* - * Remainder must be passed to submit_bio_noacct() so it gets handled - * *after* bios already submitted have been completely processed. - * We take a clone of the original to store in ci.io->orig_bio to be - * used by dm_end_io_acct() and for dm_io_complete() to use for - * completion handling. - */ - orig_bio = bio_split(bio, bio_sectors(bio) - ci.sector_count, - GFP_NOIO, &md->queue->bio_split); - bio_chain(orig_bio, bio); - trace_block_split(orig_bio, bio->bi_iter.bi_sector); + /* setup the mapped part for accounting */ + dm_io_set_flag(ci.io, DM_IO_SPLITTED); + ci.io->sectors = bio_sectors(bio) - ci.sector_count; + ci.io->sector_offset = bio_end_sector(bio) - bio->bi_iter.bi_sector; + + bio_trim(bio, ci.io->sectors, ci.sector_count); + trace_block_split(bio, bio->bi_iter.bi_sector); + bio_inc_remaining(bio); submit_bio_noacct(bio); out: - if (!orig_bio) - orig_bio = bio; - smp_store_release(&ci.io->orig_bio, orig_bio); if (dm_io_flagged(ci.io, DM_IO_START_ACCT)) dm_start_io_acct(ci.io, NULL); From patchwork Tue Apr 12 08:56:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12810242 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D0F4FC433F5 for ; Tue, 12 Apr 2022 08:57:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649753845; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=3icUmd176o97ELEaf+bWqv6t3mzigSy6jJ2H7Fj4CK8=; b=ZIFtl+qe0O+DPvclxU7UVqQfPEVtViKyjs0iLGDNh9fv2TYAVka5Fb5mMS6hgwYSlL5smA TnmuV6UtzzZEhv74PYLO2KOParT61KdogrferiZtqzQ3g4eyTeDibrAVFRr5sBg2oNpyYf XZ94t3m7zxIhminTu30IanQkEgQQNWo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-393-17V1EuQuOPqsjyJkkibjgw-1; Tue, 12 Apr 2022 04:57:22 -0400 X-MC-Unique: 17V1EuQuOPqsjyJkkibjgw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 557971014A63; Tue, 12 Apr 2022 08:57:21 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 401BA401DFD; Tue, 12 Apr 2022 08:57:21 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 05B4C1949762; Tue, 12 Apr 2022 08:57:21 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id CA6D11947BBF for ; Tue, 12 Apr 2022 08:57:19 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id A9DAF401DFD; Tue, 12 Apr 2022 08:57:19 +0000 (UTC) Received: from localhost (ovpn-8-19.pek2.redhat.com [10.72.8.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id E5922401DF3; Tue, 12 Apr 2022 08:57:17 +0000 (UTC) From: Ming Lei To: Jens Axboe , Mike Snitzer Date: Tue, 12 Apr 2022 16:56:14 +0800 Message-Id: <20220412085616.1409626-7-ming.lei@redhat.com> In-Reply-To: <20220412085616.1409626-1-ming.lei@redhat.com> References: <20220412085616.1409626-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Subject: [dm-devel] [PATCH 6/8] dm: don't grab target io reference in dm_zone_map_bio X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Damien Le Moal , Ming Lei Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com dm_zone_map_bio() is only called from __map_bio in which the io's reference is grabbed already, and the reference won't be released until the bio is submitted, so no necessary to do it dm_zone_map_bio any more. Reviewed-by: Damien Le Moal Tested-by: Damien Le Moal Signed-off-by: Ming Lei --- drivers/md/dm-core.h | 7 ------- drivers/md/dm-zone.c | 10 ---------- drivers/md/dm.c | 7 ++++++- 3 files changed, 6 insertions(+), 18 deletions(-) diff --git a/drivers/md/dm-core.h b/drivers/md/dm-core.h index aefb080c230d..811c0ccbc63d 100644 --- a/drivers/md/dm-core.h +++ b/drivers/md/dm-core.h @@ -283,13 +283,6 @@ static inline void dm_io_set_flag(struct dm_io *io, unsigned int bit) io->flags |= (1U << bit); } -static inline void dm_io_inc_pending(struct dm_io *io) -{ - atomic_inc(&io->io_count); -} - -void dm_io_dec_pending(struct dm_io *io, blk_status_t error); - static inline struct completion *dm_get_completion_from_kobject(struct kobject *kobj) { return &container_of(kobj, struct dm_kobject_holder, kobj)->completion; diff --git a/drivers/md/dm-zone.c b/drivers/md/dm-zone.c index c1ca9be4b79e..85d3c158719f 100644 --- a/drivers/md/dm-zone.c +++ b/drivers/md/dm-zone.c @@ -545,13 +545,6 @@ int dm_zone_map_bio(struct dm_target_io *tio) return DM_MAPIO_KILL; } - /* - * The target map function may issue and complete the IO quickly. - * Take an extra reference on the IO to make sure it does disappear - * until we run dm_zone_map_bio_end(). - */ - dm_io_inc_pending(io); - /* Let the target do its work */ r = ti->type->map(ti, clone); switch (r) { @@ -580,9 +573,6 @@ int dm_zone_map_bio(struct dm_target_io *tio) break; } - /* Drop the extra reference on the IO */ - dm_io_dec_pending(io, sts); - if (sts != BLK_STS_OK) return DM_MAPIO_KILL; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index df1d013fb793..3c3ba6b4e19b 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -937,11 +937,16 @@ static inline bool dm_tio_is_normal(struct dm_target_io *tio) !dm_tio_flagged(tio, DM_TIO_IS_DUPLICATE_BIO)); } +static void dm_io_inc_pending(struct dm_io *io) +{ + atomic_inc(&io->io_count); +} + /* * Decrements the number of outstanding ios that a bio has been * cloned into, completing the original io if necc. */ -void dm_io_dec_pending(struct dm_io *io, blk_status_t error) +static void dm_io_dec_pending(struct dm_io *io, blk_status_t error) { /* Push-back supersedes any I/O errors */ if (unlikely(error)) { From patchwork Tue Apr 12 08:56:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12810243 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6EDA3C433EF for ; Tue, 12 Apr 2022 08:57:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649753854; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=a9QCQNQxTPBSYKeVFBX4MwO10L91IpNOg/wp1FFx/NU=; b=D7RBcBGH11Z8tFo8Pd0rB/qRAyB81DdLMmX7kFqCZ+Vq83e4/XBqDjWPhyKWJNGbC9c7LW PLXAFEDxYMEgro0XhTfHIKdrgfJaDGI9eNBeF2G4M1iiAZ4iVasC0f1UJoLZWmw9WT3kNs y7GQoxbak0W38QoYMQAteNRuBhIh3Ng= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-594-z7yteZtyO6aYYlIrsky8PQ-1; Tue, 12 Apr 2022 04:57:31 -0400 X-MC-Unique: z7yteZtyO6aYYlIrsky8PQ-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8A5583C14849; Tue, 12 Apr 2022 08:57:29 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 79E94416376; Tue, 12 Apr 2022 08:57:29 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 409C81940365; Tue, 12 Apr 2022 08:57:29 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 8B60C1949762 for ; Tue, 12 Apr 2022 08:57:28 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 7E585145BA52; Tue, 12 Apr 2022 08:57:28 +0000 (UTC) Received: from localhost (ovpn-8-19.pek2.redhat.com [10.72.8.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9EE96145BA42; Tue, 12 Apr 2022 08:57:27 +0000 (UTC) From: Ming Lei To: Jens Axboe , Mike Snitzer Date: Tue, 12 Apr 2022 16:56:15 +0800 Message-Id: <20220412085616.1409626-8-ming.lei@redhat.com> In-Reply-To: <20220412085616.1409626-1-ming.lei@redhat.com> References: <20220412085616.1409626-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 Subject: [dm-devel] [PATCH 7/8] dm: improve target io referencing X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Damien Le Moal , Ming Lei Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.10 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Currently target io's reference counter is grabbed before calling __map_bio(), this way isn't efficient since we can move this grabbing into alloc_io(). Meantime it becomes typical async io reference counter model: one is for submission side, the other is for completion side, and the io won't be completed until both two sides are done. Signed-off-by: Ming Lei --- drivers/md/dm.c | 51 ++++++++++++++++++++++++++++++++++++------------- 1 file changed, 38 insertions(+), 13 deletions(-) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 3c3ba6b4e19b..2987f7cf7b47 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -587,7 +587,9 @@ static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio) io = container_of(tio, struct dm_io, tio); io->magic = DM_IO_MAGIC; io->status = 0; - atomic_set(&io->io_count, 1); + + /* one is for submission, the other is for completion */ + atomic_set(&io->io_count, 2); this_cpu_inc(*md->pending_io); io->orig_bio = bio; io->md = md; @@ -937,11 +939,6 @@ static inline bool dm_tio_is_normal(struct dm_target_io *tio) !dm_tio_flagged(tio, DM_TIO_IS_DUPLICATE_BIO)); } -static void dm_io_inc_pending(struct dm_io *io) -{ - atomic_inc(&io->io_count); -} - /* * Decrements the number of outstanding ios that a bio has been * cloned into, completing the original io if necc. @@ -1276,7 +1273,6 @@ static void __map_bio(struct bio *clone) /* * Map the clone. */ - dm_io_inc_pending(io); tio->old_sector = clone->bi_iter.bi_sector; if (unlikely(swap_bios_limit(ti, clone))) { @@ -1358,11 +1354,12 @@ static void alloc_multiple_bios(struct bio_list *blist, struct clone_info *ci, } } -static void __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti, +static int __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti, unsigned num_bios, unsigned *len) { struct bio_list blist = BIO_EMPTY_LIST; struct bio *clone; + int ret = 0; switch (num_bios) { case 0: @@ -1371,15 +1368,19 @@ static void __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti, clone = alloc_tio(ci, ti, 0, len, GFP_NOIO); dm_tio_set_flag(clone_to_tio(clone), DM_TIO_IS_DUPLICATE_BIO); __map_bio(clone); + ret = 1; break; default: alloc_multiple_bios(&blist, ci, ti, num_bios, len); while ((clone = bio_list_pop(&blist))) { dm_tio_set_flag(clone_to_tio(clone), DM_TIO_IS_DUPLICATE_BIO); __map_bio(clone); + ret += 1; } break; } + + return ret; } static void __send_empty_flush(struct clone_info *ci) @@ -1399,8 +1400,19 @@ static void __send_empty_flush(struct clone_info *ci) ci->bio = &flush_bio; ci->sector_count = 0; - while ((ti = dm_table_get_target(ci->map, target_nr++))) - __send_duplicate_bios(ci, ti, ti->num_flush_bios, NULL); + while ((ti = dm_table_get_target(ci->map, target_nr++))) { + int bios; + + atomic_add(ti->num_flush_bios, &ci->io->io_count); + bios = __send_duplicate_bios(ci, ti, ti->num_flush_bios, NULL); + atomic_sub(ti->num_flush_bios - bios, &ci->io->io_count); + } + + /* + * alloc_io() takes one extra reference for submission, so the + * reference won't reach 0 after the following subtraction + */ + atomic_sub(1, &ci->io->io_count); bio_uninit(ci->bio); } @@ -1409,6 +1421,7 @@ static void __send_changing_extent_only(struct clone_info *ci, struct dm_target unsigned num_bios) { unsigned len; + int bios; len = min_t(sector_t, ci->sector_count, max_io_len_target_boundary(ti, dm_target_offset(ti, ci->sector))); @@ -1420,7 +1433,13 @@ static void __send_changing_extent_only(struct clone_info *ci, struct dm_target ci->sector += len; ci->sector_count -= len; - __send_duplicate_bios(ci, ti, num_bios, &len); + atomic_add(num_bios, &ci->io->io_count); + bios = __send_duplicate_bios(ci, ti, num_bios, &len); + /* + * alloc_io() takes one extra reference for submission, so the + * reference won't reach 0 after the following subtraction + */ + atomic_sub(num_bios - bios + 1, &ci->io->io_count); } static bool is_abnormal_io(struct bio *bio) @@ -1603,9 +1622,15 @@ static void dm_split_and_process_bio(struct mapped_device *md, * Add every dm_io instance into the hlist_head which is stored in * bio->bi_private, so that dm_poll_bio can poll them all. */ - if (error || !ci.submit_as_polled) + if (error || !ci.submit_as_polled) { + /* + * In case of submission failure, the extra reference for + * submitting io isn't consumed yet + */ + if (error) + atomic_dec(&ci.io->io_count); dm_io_dec_pending(ci.io, errno_to_blk_status(error)); - else + } else dm_queue_poll_io(bio, ci.io); } From patchwork Tue Apr 12 08:56:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12810244 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6D7A5C433EF for ; Tue, 12 Apr 2022 08:57:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649753856; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=hLtZIZI5eMXlkvCkHO6DuGZbHKPVT4ZiDlWBAUYfipc=; b=fYX54MiMO/rMEbAORZloc4aK+LtoDi6egP55Yk6G/MGwwUIZV06E1ITi0+Nn3ORYdD7S5j x7RCiWhvnIM2bN9SOh9398dDBAX5hNM0oetB8q2t/X/aneYsjPJD6Dsp23tLcABuimxIVw XgGEkaiMbHQcU2kJacc3zfry3irdsDk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-473-RhOL4a0qPkWPlnY4JOoKrg-1; Tue, 12 Apr 2022 04:57:35 -0400 X-MC-Unique: RhOL4a0qPkWPlnY4JOoKrg-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8D97E833966; Tue, 12 Apr 2022 08:57:33 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7C42AC44CC2; Tue, 12 Apr 2022 08:57:33 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 24AC81940365; Tue, 12 Apr 2022 08:57:33 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 121481949762 for ; Tue, 12 Apr 2022 08:57:32 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id E7D75C44CC2; Tue, 12 Apr 2022 08:57:31 +0000 (UTC) Received: from localhost (ovpn-8-19.pek2.redhat.com [10.72.8.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id 206CBC44B1D; Tue, 12 Apr 2022 08:57:30 +0000 (UTC) From: Ming Lei To: Jens Axboe , Mike Snitzer Date: Tue, 12 Apr 2022 16:56:16 +0800 Message-Id: <20220412085616.1409626-9-ming.lei@redhat.com> In-Reply-To: <20220412085616.1409626-1-ming.lei@redhat.com> References: <20220412085616.1409626-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.8 Subject: [dm-devel] [PATCH 8/8] dm: put all polled io into one single list X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, dm-devel@redhat.com, Damien Le Moal , Ming Lei Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.8 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com If bio_split() isn't involved, it is a bit overkill to link dm_io into hlist, given there is only single dm_io in the list, so convert to single list for holding all dm_io instances associated with this bio. Signed-off-by: Ming Lei --- drivers/md/dm-core.h | 2 +- drivers/md/dm.c | 46 +++++++++++++++++++++++--------------------- 2 files changed, 25 insertions(+), 23 deletions(-) diff --git a/drivers/md/dm-core.h b/drivers/md/dm-core.h index 811c0ccbc63d..7f51957913e8 100644 --- a/drivers/md/dm-core.h +++ b/drivers/md/dm-core.h @@ -257,7 +257,7 @@ struct dm_io { spinlock_t lock; unsigned long start_time; void *data; - struct hlist_node node; + struct dm_io *next; struct task_struct *map_task; struct dm_stats_aux stats_aux; /* last member of dm_target_io is 'struct bio' */ diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 2987f7cf7b47..db23efd6bbf6 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1492,7 +1492,7 @@ static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti, } /* - * Reuse ->bi_private as hlist head for storing all dm_io instances + * Reuse ->bi_private as dm_io list head for storing all dm_io instances * associated with this bio, and this bio's bi_private needs to be * stored in dm_io->data before the reuse. * @@ -1500,14 +1500,14 @@ static bool __process_abnormal_io(struct clone_info *ci, struct dm_target *ti, * touch it after splitting. Meantime it won't be changed by anyone after * bio is submitted. So this reuse is safe. */ -static inline struct hlist_head *dm_get_bio_hlist_head(struct bio *bio) +static inline struct dm_io **dm_poll_list_head(struct bio *bio) { - return (struct hlist_head *)&bio->bi_private; + return (struct dm_io **)&bio->bi_private; } static void dm_queue_poll_io(struct bio *bio, struct dm_io *io) { - struct hlist_head *head = dm_get_bio_hlist_head(bio); + struct dm_io **head = dm_poll_list_head(bio); if (!(bio->bi_opf & REQ_DM_POLL_LIST)) { bio->bi_opf |= REQ_DM_POLL_LIST; @@ -1517,19 +1517,20 @@ static void dm_queue_poll_io(struct bio *bio, struct dm_io *io) */ io->data = bio->bi_private; - INIT_HLIST_HEAD(head); - /* tell block layer to poll for completion */ bio->bi_cookie = ~BLK_QC_T_NONE; + + io->next = NULL; } else { /* * bio recursed due to split, reuse original poll list, * and save bio->bi_private too. */ - io->data = hlist_entry(head->first, struct dm_io, node)->data; + io->data = (*head)->data; + io->next = *head; } - hlist_add_head(&io->node, head); + *head = io; } /* @@ -1682,18 +1683,16 @@ static bool dm_poll_dm_io(struct dm_io *io, struct io_comp_batch *iob, static int dm_poll_bio(struct bio *bio, struct io_comp_batch *iob, unsigned int flags) { - struct hlist_head *head = dm_get_bio_hlist_head(bio); - struct hlist_head tmp = HLIST_HEAD_INIT; - struct hlist_node *next; - struct dm_io *io; + struct dm_io **head = dm_poll_list_head(bio); + struct dm_io *list = *head; + struct dm_io *tmp = NULL; + struct dm_io *curr, *next; /* Only poll normal bio which was marked as REQ_DM_POLL_LIST */ if (!(bio->bi_opf & REQ_DM_POLL_LIST)) return 0; - WARN_ON_ONCE(hlist_empty(head)); - - hlist_move_list(head, &tmp); + WARN_ON_ONCE(!list); /* * Restore .bi_private before possibly completing dm_io. @@ -1704,24 +1703,27 @@ static int dm_poll_bio(struct bio *bio, struct io_comp_batch *iob, * clearing REQ_DM_POLL_LIST here. */ bio->bi_opf &= ~REQ_DM_POLL_LIST; - bio->bi_private = hlist_entry(tmp.first, struct dm_io, node)->data; + bio->bi_private = list->data; - hlist_for_each_entry_safe(io, next, &tmp, node) { - if (dm_poll_dm_io(io, iob, flags)) { - hlist_del_init(&io->node); + for (curr = list, next = curr->next; curr; curr = next, next = + curr ? curr->next : NULL) { + if (dm_poll_dm_io(curr, iob, flags)) { /* * clone_endio() has already occurred, so passing * error as 0 here doesn't override io->status */ - dm_io_dec_pending(io, 0); + dm_io_dec_pending(curr, 0); + } else { + curr->next = tmp; + tmp = curr; } } /* Not done? */ - if (!hlist_empty(&tmp)) { + if (tmp) { bio->bi_opf |= REQ_DM_POLL_LIST; /* Reset bio->bi_private to dm_io list head */ - hlist_move_list(&tmp, head); + *head = tmp; return 0; } return 1;