From patchwork Mon Mar 27 08:40:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13192075 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA443C77B6C for ; Wed, 29 Mar 2023 08:16:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230393AbjC2IQS (ORCPT ); Wed, 29 Mar 2023 04:16:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230346AbjC2IQM (ORCPT ); Wed, 29 Mar 2023 04:16:12 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4E4144AC for ; Wed, 29 Mar 2023 01:16:03 -0700 (PDT) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230329081602epoutp014c193abdb79d30edabe2aa14987f230f~Q1dLBklor1190311903epoutp01c for ; Wed, 29 Mar 2023 08:16:02 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230329081602epoutp014c193abdb79d30edabe2aa14987f230f~Q1dLBklor1190311903epoutp01c DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1680077762; bh=8LMsuFU+adOGECMdeeltyN6nn0NGSJ2FHXwHcZ2lfa0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=esYXPDxSUXLO6DiWbogdjDMgpv3j8uhnPYQ6H3RmmlVcSDBVXRYEM59za3V2P4RGo mqjmMCRtzzLmMudX2gDe563aWzh/GLNKhQaipgOZsiuhQuu9ccXkQZE06IKC7mB491 weFnvP2SQr858Hs/icmGHQjbqlcaOc7Eu6LrTdsY= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230329081601epcas5p32acd68a630e2d9664166d41744fec1d9~Q1dKTAu2J2887628876epcas5p3d; Wed, 29 Mar 2023 08:16:01 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.180]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4PmfWg4sGYz4x9Pt; Wed, 29 Mar 2023 08:15:59 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id FA.B5.55678.FB3F3246; Wed, 29 Mar 2023 17:15:59 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230327084216epcas5p3945507ecd94688c40c29195127ddc54d~QOhgrkaAa0488004880epcas5p37; Mon, 27 Mar 2023 08:42:16 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230327084216epsmtrp10833f46bb72e7e16fb46bdb08d3d4324~QOhgqQCmH3087630876epsmtrp1m; Mon, 27 Mar 2023 08:42:16 +0000 (GMT) X-AuditID: b6c32a4a-909fc7000000d97e-04-6423f3bf0833 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id FF.B5.31821.8E651246; Mon, 27 Mar 2023 17:42:16 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230327084212epsmtip141d949ff8ed23c9813a82bf4362350a0~QOhdMdMe-3056830568epsmtip1I; Mon, 27 Mar 2023 08:42:12 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v8 1/9] block: Introduce queue limits for copy-offload support Date: Mon, 27 Mar 2023 14:10:49 +0530 Message-Id: <20230327084103.21601-2-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230327084103.21601-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Tf1CTZRzved+xH9r0BQkeKGm+mgs7YLOxHjjQvDh7L7kLz0sLKnxj7wEy tt1+SGlXiwGKCSO8KAeCBMcOLMSByE/tINFBRApSCIQgSwwHDKLTW1gbg/K/z/f7fD7fH5/n vlzcJ48TyE1VaBm1gpaT7DWsxs7g4JArC5tloubcTeh8dxeOMguWcHRu1MhG053zABXNPcKR s7cPR+0zxV5o6PtmDLV9U4ih6nNXMdRa7sDQ1X/sbFTYMQiQ7ZYJQ+23X0Jt7VYW6m8pYaOy KhsHdZwyYKhp8jOAGp1lOKqdnmWh67efRX1L17xehVT/wB7KNNbLpppNoxyq77cLLKq/V0dZ anLZVH3lp1TrkJ5N5RlmXITsMS9q9vItNpXfUAOo+p6j1IIliLJM2rG49fFpUSkMLWPUAkaR pJSlKpKjyT37El9LDJeKxCHiCPQKKVDQ6Uw0GRMbF7I7Ve7ygRQcpuU6VyqO1mjIsB1RaqVO ywhSlBptNMmoZHKVRBWqodM1OkVyqILRRopFou3hLuLBtBSz3V91fe+H1vJOoAd9MScAjwsJ CZwaeABOgDVcH6IVwC+79bgnmAdwYKST7Wb5EAsAXir1WlX01PYAT74FwOyZNI/AgEGjYZLl fmATQvjDvezlsr5EDg4dtlyWO8CJOxgsy6xclm8g4mCd+cGygkW8AOvbzrryXC6fiICnP5e6 ISTCoHHM283gEZGwpKl7eQg+4Q2tpz29cOJ5aLhYvDw1JMw8uDhYx/ZoY2D58IueoTfAP641 cDw4EN435qzgZPiw34Z5sAoaui4DD94Js7uNuLsMTgTD8y1hnvRGl0G1mKftOpjnnFyR8mFT 6Som4bHqkhUMYftP+hVMwYKlSo7HtzwAHd+GFACB6YltTE9sY/q/81mA14AARqVJT2Y04art Cibjvx9OUqZbwPJdbHujCYzfmQvtABgXdADIxUlfvnOQlPnwZfRHRxi1MlGtkzOaDhDuMvsL PPCZJKXrsBTaRLEkQiSRSqWSiJelYtKfL4y2JvkQybSWSWMYFaNe1WFcXqAeU3rfqxJvNmce 4vn9cjzmTETAonnxxpEfxWf8Dn9SDG1Pmx4vkhMl7/n6nUzY6Xj3/u+7hVyrzqsajObOqmOi hmMvpI7kvl48va72oZ/5pIHcoqqIf/8vi8Do/GociTOc7UVdf2vn9MVbDowIt/IzChu4nLW7 coKfyq9IyrZb/QNij+8f493of7O0YpS4GfZzVroph6aH/hy/tDZoeNxmD8262/hIOFVDs/e+ VR0s/njuVJWoon5fHadIILQnbMz/db8p4THv7akg3tfh73wXmiIxzX7QeSAr4Kj80I7exK1K 08jk/BWLQ5l67OCuidLDkfGbJjLXXxQO18w8Z3VUSu/ezCRZmhRavA1Xa+h/AePZOzWgBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02RbUhTURyHOfder9fV6DpNj5qKq1wYrTKVU/RikXErkt6oMESnXlTadG2+ VWRLi8oyzdBqCZaY5SItlzptis20nMgiW29kYzWzsM1lVCxTc5PAb78/z/OcL4fCeU8Jfyo9 I4uVZYjEfJJDNHfxg5Z92ReSsmLU5oMa9D04KiidwNHdwRISjXSNAVRhd+BovN+Ao3bbdTf0 trMVQ9rqMgzV3e3G0KOb3zHUPWUlUZnuFUBDRiWG2t8tRdr2XgINtFWSqKp2yB3pLhdiSGM5 CVDzeBWO6kdGCfTsXQAyTDx1i4bMwMvtjNLUTzKtykF3xvDhAcEM9GczjapzJKOuOcE8eqsg meJC27Rw2uTGjHYYSebiQxVg1H3HmB+NQUyjxYrtnBfHWZvCitNzWNny9YmctNtWX+mzXXm9 N7uAAhg2FwEPCtIRsK++DxQBDsWjNQB+H/9IzAAI9V9rwcz2gnWTw+4z0kkM3ipocUkkLYBP hk+7am+6FIcvTArSeeC0DYMqc62b0/KiY2HJlVOugqAXQ7X2xnRBUVx6Nbx2Pso5Ib0clpg8 nYYHvQZWavSukjdtlJxpw5ybS3vC3msW1ys4HQwLm67jpYBWzkLKWegGwFTAj5XKJakS+Upp eAabK5SLJPLsjFRhcqakEbj+OixMA7Qqu1AHMAroAKRwvjdXvT0khcdNER05ysoyE2TZYlau AwEUwfflPi/qTeDRqaIs9hDLSlnZf4pRHv4KLCL+njE4PvaOIHLH79jPmnp33/3mqZ4xH5X3 mZ/icONknU0efiBalrkbBputvy1is7EiZ2e710YJHbHquVdEq9LaY3jyynLstf9cfENzUujZ T3ul+YyiM3Cyu2mJXpVT/r52X0vMnLRtw0nRwDP/l8RRuagpsXoULy5fVFm6JHSdRdhRY9+4 4HN1R9HhvK0NuOEqJzQ9iBeTe5CKedxQFaxytJn4iWPqeJMjZ8f9zmRBZD6hX+hToD/1o3lZ 8ZY9zFB4QpQ5juyeyowUmg+A6G/c446kCcFgYJzW9MG+t7zMcim5xv5JcPBCpN+mq2M8g+XN KvHxP/PzW/4GFATyCXmaaGUYLpOL/gEgU2CjWgMAAA== X-CMS-MailID: 20230327084216epcas5p3945507ecd94688c40c29195127ddc54d X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230327084216epcas5p3945507ecd94688c40c29195127ddc54d References: <20230327084103.21601-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Nitesh Shetty Add device limits as sysfs entries, - copy_offload (RW) - copy_max_bytes (RW) - copy_max_bytes_hw (RO) Above limits help to split the copy payload in block layer. copy_offload: used for setting copy offload(1) or emulation(0). copy_max_bytes: maximum total length of copy in single payload. copy_max_bytes_hw: Reflects the device supported maximum limit. Reviewed-by: Hannes Reinecke Signed-off-by: Nitesh Shetty Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- Documentation/ABI/stable/sysfs-block | 36 ++++++++++++++++ block/blk-settings.c | 24 +++++++++++ block/blk-sysfs.c | 64 ++++++++++++++++++++++++++++ include/linux/blkdev.h | 12 ++++++ include/uapi/linux/fs.h | 3 ++ 5 files changed, 139 insertions(+) diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block index c57e5b7cb532..f5c56ad91ad6 100644 --- a/Documentation/ABI/stable/sysfs-block +++ b/Documentation/ABI/stable/sysfs-block @@ -155,6 +155,42 @@ Description: last zone of the device which may be smaller. +What: /sys/block//queue/copy_offload +Date: November 2022 +Contact: linux-block@vger.kernel.org +Description: + [RW] When read, this file shows whether offloading copy to + device is enabled (1) or disabled (0). Writing '0' to this + file will disable offloading copies for this device. + Writing any '1' value will enable this feature. If device + does not support offloading, then writing 1, will result in + error. + + +What: /sys/block//queue/copy_max_bytes +Date: November 2022 +Contact: linux-block@vger.kernel.org +Description: + [RW] While 'copy_max_bytes_hw' is the hardware limit for the + device, 'copy_max_bytes' setting is the software limit. + Setting this value lower will make Linux issue smaller size + copies from block layer. + + +What: /sys/block//queue/copy_max_bytes_hw +Date: November 2022 +Contact: linux-block@vger.kernel.org +Description: + [RO] Devices that support offloading copy functionality may have + internal limits on the number of bytes that can be offloaded + in a single operation. The `copy_max_bytes_hw` + parameter is set by the device driver to the maximum number of + bytes that can be copied in a single operation. Copy + requests issued to the device must not exceed this limit. + A value of 0 means that the device does not + support copy offload. + + What: /sys/block//queue/crypto/ Date: February 2022 Contact: linux-block@vger.kernel.org diff --git a/block/blk-settings.c b/block/blk-settings.c index 896b4654ab00..350f3584f691 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -59,6 +59,8 @@ void blk_set_default_limits(struct queue_limits *lim) lim->zoned = BLK_ZONED_NONE; lim->zone_write_granularity = 0; lim->dma_alignment = 511; + lim->max_copy_sectors_hw = 0; + lim->max_copy_sectors = 0; } /** @@ -82,6 +84,8 @@ void blk_set_stacking_limits(struct queue_limits *lim) lim->max_dev_sectors = UINT_MAX; lim->max_write_zeroes_sectors = UINT_MAX; lim->max_zone_append_sectors = UINT_MAX; + lim->max_copy_sectors_hw = ULONG_MAX; + lim->max_copy_sectors = ULONG_MAX; } EXPORT_SYMBOL(blk_set_stacking_limits); @@ -183,6 +187,22 @@ void blk_queue_max_discard_sectors(struct request_queue *q, } EXPORT_SYMBOL(blk_queue_max_discard_sectors); +/** + * blk_queue_max_copy_sectors_hw - set max sectors for a single copy payload + * @q: the request queue for the device + * @max_copy_sectors: maximum number of sectors to copy + **/ +void blk_queue_max_copy_sectors_hw(struct request_queue *q, + unsigned int max_copy_sectors) +{ + if (max_copy_sectors >= MAX_COPY_TOTAL_LENGTH) + max_copy_sectors = MAX_COPY_TOTAL_LENGTH; + + q->limits.max_copy_sectors_hw = max_copy_sectors; + q->limits.max_copy_sectors = max_copy_sectors; +} +EXPORT_SYMBOL_GPL(blk_queue_max_copy_sectors_hw); + /** * blk_queue_max_secure_erase_sectors - set max sectors for a secure erase * @q: the request queue for the device @@ -578,6 +598,10 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, t->max_segment_size = min_not_zero(t->max_segment_size, b->max_segment_size); + t->max_copy_sectors = min(t->max_copy_sectors, b->max_copy_sectors); + t->max_copy_sectors_hw = min(t->max_copy_sectors_hw, + b->max_copy_sectors_hw); + t->misaligned |= b->misaligned; alignment = queue_limit_alignment_offset(b, start); diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index 1a743b4f2958..dccb162cf318 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -213,6 +213,63 @@ static ssize_t queue_discard_zeroes_data_show(struct request_queue *q, char *pag return queue_var_show(0, page); } +static ssize_t queue_copy_offload_show(struct request_queue *q, char *page) +{ + return queue_var_show(blk_queue_copy(q), page); +} + +static ssize_t queue_copy_offload_store(struct request_queue *q, + const char *page, size_t count) +{ + s64 copy_offload; + ssize_t ret = queue_var_store64(©_offload, page); + + if (ret < 0) + return ret; + + if (copy_offload && !q->limits.max_copy_sectors_hw) + return -EINVAL; + + if (copy_offload) + blk_queue_flag_set(QUEUE_FLAG_COPY, q); + else + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + + return count; +} + +static ssize_t queue_copy_max_hw_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors_hw << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_store(struct request_queue *q, + const char *page, size_t count) +{ + s64 max_copy; + ssize_t ret = queue_var_store64(&max_copy, page); + + if (ret < 0) + return ret; + + if (max_copy & (queue_logical_block_size(q) - 1)) + return -EINVAL; + + max_copy >>= SECTOR_SHIFT; + if (max_copy > q->limits.max_copy_sectors_hw) + max_copy = q->limits.max_copy_sectors_hw; + + q->limits.max_copy_sectors = max_copy; + return count; +} + static ssize_t queue_write_same_max_show(struct request_queue *q, char *page) { return queue_var_show(0, page); @@ -591,6 +648,10 @@ QUEUE_RO_ENTRY(queue_nr_zones, "nr_zones"); QUEUE_RO_ENTRY(queue_max_open_zones, "max_open_zones"); QUEUE_RO_ENTRY(queue_max_active_zones, "max_active_zones"); +QUEUE_RW_ENTRY(queue_copy_offload, "copy_offload"); +QUEUE_RO_ENTRY(queue_copy_max_hw, "copy_max_bytes_hw"); +QUEUE_RW_ENTRY(queue_copy_max, "copy_max_bytes"); + QUEUE_RW_ENTRY(queue_nomerges, "nomerges"); QUEUE_RW_ENTRY(queue_rq_affinity, "rq_affinity"); QUEUE_RW_ENTRY(queue_poll, "io_poll"); @@ -638,6 +699,9 @@ static struct attribute *queue_attrs[] = { &queue_discard_max_entry.attr, &queue_discard_max_hw_entry.attr, &queue_discard_zeroes_data_entry.attr, + &queue_copy_offload_entry.attr, + &queue_copy_max_hw_entry.attr, + &queue_copy_max_entry.attr, &queue_write_same_max_entry.attr, &queue_write_zeroes_max_entry.attr, &queue_zone_append_max_entry.attr, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index e3242e67a8e3..200338f2ec2e 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -298,6 +298,9 @@ struct queue_limits { unsigned int discard_alignment; unsigned int zone_write_granularity; + unsigned long max_copy_sectors_hw; + unsigned long max_copy_sectors; + unsigned short max_segments; unsigned short max_integrity_segments; unsigned short max_discard_segments; @@ -564,6 +567,7 @@ struct request_queue { #define QUEUE_FLAG_NOWAIT 29 /* device supports NOWAIT */ #define QUEUE_FLAG_SQ_SCHED 30 /* single queue style io dispatch */ #define QUEUE_FLAG_SKIP_TAGSET_QUIESCE 31 /* quiesce_tagset skip the queue*/ +#define QUEUE_FLAG_COPY 32 /* supports copy offload */ #define QUEUE_FLAG_MQ_DEFAULT ((1UL << QUEUE_FLAG_IO_STAT) | \ (1UL << QUEUE_FLAG_SAME_COMP) | \ @@ -584,6 +588,7 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q); test_bit(QUEUE_FLAG_STABLE_WRITES, &(q)->queue_flags) #define blk_queue_io_stat(q) test_bit(QUEUE_FLAG_IO_STAT, &(q)->queue_flags) #define blk_queue_add_random(q) test_bit(QUEUE_FLAG_ADD_RANDOM, &(q)->queue_flags) +#define blk_queue_copy(q) test_bit(QUEUE_FLAG_COPY, &(q)->queue_flags) #define blk_queue_zone_resetall(q) \ test_bit(QUEUE_FLAG_ZONE_RESETALL, &(q)->queue_flags) #define blk_queue_dax(q) test_bit(QUEUE_FLAG_DAX, &(q)->queue_flags) @@ -902,6 +907,8 @@ extern void blk_queue_chunk_sectors(struct request_queue *, unsigned int); extern void blk_queue_max_segments(struct request_queue *, unsigned short); extern void blk_queue_max_discard_segments(struct request_queue *, unsigned short); +extern void blk_queue_max_copy_sectors_hw(struct request_queue *q, + unsigned int max_copy_sectors); void blk_queue_max_secure_erase_sectors(struct request_queue *q, unsigned int max_sectors); extern void blk_queue_max_segment_size(struct request_queue *, unsigned int); @@ -1221,6 +1228,11 @@ static inline unsigned int bdev_discard_granularity(struct block_device *bdev) return bdev_get_queue(bdev)->limits.discard_granularity; } +static inline unsigned int bdev_max_copy_sectors(struct block_device *bdev) +{ + return bdev_get_queue(bdev)->limits.max_copy_sectors; +} + static inline unsigned int bdev_max_secure_erase_sectors(struct block_device *bdev) { diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index b7b56871029c..b3ad173f619c 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -64,6 +64,9 @@ struct fstrim_range { __u64 minlen; }; +/* maximum total copy length */ +#define MAX_COPY_TOTAL_LENGTH (1 << 27) + /* extent-same (dedupe) ioctls; these MUST match the btrfs ioctl definitions */ #define FILE_DEDUPE_RANGE_SAME 0 #define FILE_DEDUPE_RANGE_DIFFERS 1 From patchwork Mon Mar 27 08:40:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13192076 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F11BBC761A6 for ; Wed, 29 Mar 2023 08:16:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230377AbjC2IQi (ORCPT ); Wed, 29 Mar 2023 04:16:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230374AbjC2IQQ (ORCPT ); Wed, 29 Mar 2023 04:16:16 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 202EF4697 for ; Wed, 29 Mar 2023 01:16:08 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230329081606epoutp01f311bc3b0c3f3d418ce4fec26e3a7bbd~Q1dPBlruL1316713167epoutp01h for ; Wed, 29 Mar 2023 08:16:06 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230329081606epoutp01f311bc3b0c3f3d418ce4fec26e3a7bbd~Q1dPBlruL1316713167epoutp01h DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1680077766; bh=S/AESpOblqaglCp0pwrLwPKzeuihkrjZZs90eduqZX4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jfZPa1G1poSkhrXhJeYAoz8yX+DERZO6k6+F+IwBAmeMiHqNFum74zg3ptojasqeR JyC+5LKG5FHIbqkuwZrPMkLsSTC2Rj8UGh/czY0SRnJnMGpQjG/yPGEfQZ6ps1t/Lq w4UiVs3x6hMXDVRF3u7CXG2nz3EKKpEAOdEyFOJM= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230329081605epcas5p42a743f8306957971e642e6df25dfe8b3~Q1dOYEUoY0856608566epcas5p4Q; Wed, 29 Mar 2023 08:16:05 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.182]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4PmfWm0jcSz4x9Px; Wed, 29 Mar 2023 08:16:04 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 59.FF.06765.3C3F3246; Wed, 29 Mar 2023 17:16:03 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230327084226epcas5p28e667b25cbb5e4b0e884aa2ca89cbfff~QOhpuUFL-2983129831epcas5p2K; Mon, 27 Mar 2023 08:42:26 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230327084226epsmtrp158fd5a02f82d941bca7d3593b5824a0b~QOhptJug73087530875epsmtrp1s; Mon, 27 Mar 2023 08:42:26 +0000 (GMT) X-AuditID: b6c32a4b-20fff70000011a6d-ac-6423f3c342ae Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id F2.C5.31821.1F651246; Mon, 27 Mar 2023 17:42:26 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230327084222epsmtip1ce77c8d1469b570810959246667d4d05~QOhmVZrxS3003330033epsmtip1i; Mon, 27 Mar 2023 08:42:22 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v8 2/9] block: Add copy offload support infrastructure Date: Mon, 27 Mar 2023 14:10:50 +0530 Message-Id: <20230327084103.21601-3-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230327084103.21601-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xbZRjGc87pDRLYgQ39YDrIWRgDhLYC9SvCZBHmMTDDnGbLjIGmPbaE 3ugp4+YiA2FsGZfhAFdw6CQYQEHuhdIpFMpgYdynYAaI1GVDYAOVmQ61tKD77/e+7/PkvXz5 OJj7FbYXJ0mppTRKkZxgOTM6TP6Hg/o3Dkp4t8/ApmEzBnNKtjDYcK+YBZdN6wgsf/QXBq0j oxg0rlYy4cz3XSjsuVGKwrqGARQavniMwoF/VliwtO8uAi3TOhQaZwNhj3GIASe7q1iwutbC hn2f5KJQv3QegR3Wagw2Lq8x4K3Z/XB0a5AZBcjJqVhSNz/CIrt099jk6Fwzg5wcSSVb6i+y yNaaj0jDTDaLLMxdtQny5pnk2s1pFlnUVo+QrbezyI2WA2TL0goav+dMcoSMEkkojQ+lFKsk SUppJBF7MuH1hDABjx/EF8JXCB+lSEFFEtFx8UHHkuS2MxA+Z0XyVFsqXkTTBPdIhEaVqqV8 ZCpaG0lQaolcHaoOpkUKOlUpDVZS2nA+j/dymE2YmCz76kEhW/30VPp41XksG5l74xLixAF4 KFgrfIpdQpw57rgBAR+bLzAcwToCruv1bEewgYBrtZcZu5bWsss7qm4ElJuLEEeQi4KZ32aY 2yoW7gf67+fZC/vwfAw8tly0WzB8AQXVOTW2CoezFz8GvvlFvG1g4L7AnH/f3sIFF4LlijV0 WwJwLiied9tOO+HhoEo/zHRI3MDQtSW7HMO9QW57pX0JgDc4gemCMaZj1GhQalpgOXgveDjY xnawF3hQnL/DUvBk0oI6WA1yzTcRB78G8oaLse0ZMNwfNHVzHekXQdlwI+ro6woKrUs7Vheg v77LBLhQV7XDABjvZO8wCTo2J3ZOWoiA/rZydgnio3tmH90z++j+b/05gtUjnpSaVkgpOkwd oqTS/ntmsUrRgtj/RkCsHllceBTch6AcpA8BHIzY52K9S0jcXSSijExKo0rQpMopug8Js937 CublIVbZPpdSm8APFfJCBQJBqDBEwCeed/GLHBK741KRlkqmKDWl2fWhHCevbNRDvegX49zf HHjO4HsqoHd/V9OqtYfHi62Ibk9/4dd2Drcz4kZgadwhusRboj/hlvh2eEFaTNxPvv5Wlun9 lEVDY3rQQNQtjx8Ff9RYRq7WSsL+bg6qFzYQAa+enn2zKaU3z1CR3NsZKOr94a2N9+a4DzN/ f+6Q+YPyxKkQ04T8u4yKgZyGw8fvEE88g1NrhByFTP/z5rtXzeNDB8pkJzOsRGbOpm7cdDrt z7rBT79W+Gd56qbcjWSnZdZT6r0ZeaLuaAPlajn6zpLrHiO3+HjM1tkV76j1suwjLu1jZd8m ekrDCTpzbCLfcM4pZf6lqBVXvfOHleLCLwtUn2UV5bQLpOkEg5aJ+AGYhhb9Cxl5cTmkBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02Re0hTcRTH+917d3ddqNdp9StDc/RimmWF/YqS6GE/EnuagVC69OZzOrfU HlTaUkkyh1PDbWBZWE4z3Ep89tjK1DAtNdBwo1SsZGpGZdjLKYH/nXM+n+/hwGFIYQu1hIlN PMXJEyUJIlpA1ZhFHmsmQr2i1k12A3S/rZlEl1S/SVTRn0ejEfMEQEXjP0k01d5BoqZRLQ/1 PqkjUGNpPoHKK54TqOHmFwI9/2ujUb7pLUBDPRoCNfV5o8amVgp11etoVFI2xEcmtZJAtYMZ ANVMlZCoamSMQi197qjj9wvedoi7uoOwxtpO4zpNPx93WKop3NWegg36KzQ23r6IG3rTaZyr HJ0WMq08PPaoh8bXHugBNr48h78aPLBh0EYccA4TbI3iEmJTOfnagAhBzJ1PuXzZr6OnX+sy yHRg2ZMDHBjIboTGwqtUDhAwQrYWwE7bV94sgLDtUxmYrV1h+Z9h/qyUQcC/VZl8O6DZVfDZ cCawAzdWRcI31nTa3pDsKAH178umVzGMKxsI7w1E2gMUuwI2Zw1T9tqR3QxHro8RdgWya2Ge 1cU+dmC3QF1t28wRwmklL7uemNVdYGvx4EyUZD2h8qGWVAFWMwdp5qAbgNCDxZxMIY2WKvxk 6xO5NF+FRKpISYz2jUySGsDMs8XiWtCoH/c1AYIBJgAZUuTmaAzyihI6RknOnOXkSeHylARO YQLuDCVa5NiZ0xouZKMlp7h4jpNx8v+UYByWpBMPAy90ZjvtfyzsO276Hhbsn3xQVR4XMWo+ vI/6jvY67SweEmTuCDhUWlCpi9hA31h6M7T6yM4V7vu9fd4FH1M7LVCvY8lfsfvC1bIsUmvs 5/uH3F0+JB4YXvRssXcqLzQ+lczvuVq118/fdsEy79EU5l6GdfMzCup9GqTXV8tMJUWRLU/V CwbqPNShytw4lyJV8Daf5ObPBZULjXd/vPNzPuEZ4/zNXN1mQIWTpebx3UktV7TFSSG+aQdt dKsHscvy6jxOszbeUsT3fDgyv+byppOioA2Tkw3ZgpU7zu1eE/PCZ5ln4JY/Jw4nfzRfTM4K fPKTr1O6WlJdr8kqn+J+sYhSxEj8xKRcIfkH4ykxpFsDAAA= X-CMS-MailID: 20230327084226epcas5p28e667b25cbb5e4b0e884aa2ca89cbfff X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230327084226epcas5p28e667b25cbb5e4b0e884aa2ca89cbfff References: <20230327084103.21601-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Nitesh Shetty Introduce blkdev_issue_copy which takes similar arguments as copy_file_range and performs copy offload between two bdevs. Introduce REQ_COPY copy offload operation flag. Create a read-write bio pair with a token as payload and submitted to the device in order. Read request populates token with source specific information which is then passed with write request. This design is courtesy Mikulas Patocka's token based copy Larger copy will be divided, based on max_copy_sectors limit. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- block/blk-lib.c | 236 ++++++++++++++++++++++++++++++++++++++ block/blk.h | 2 + include/linux/blk_types.h | 25 ++++ include/linux/blkdev.h | 3 + 4 files changed, 266 insertions(+) diff --git a/block/blk-lib.c b/block/blk-lib.c index e59c3069e835..cbc6882d1e7a 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -115,6 +115,242 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, } EXPORT_SYMBOL(blkdev_issue_discard); +/* + * For synchronous copy offload/emulation, wait and process all in-flight BIOs. + * This must only be called once all bios have been issued so that the refcount + * can only decrease. This just waits for all bios to make it through + * bio_copy_*_write_end_io. IO errors are propagated through cio->io_error. + */ +static int cio_await_completion(struct cio *cio) +{ + int ret = 0; + + if (cio->endio) + return 0; + + if (atomic_read(&cio->refcount)) { + __set_current_state(TASK_UNINTERRUPTIBLE); + blk_io_schedule(); + } + + ret = cio->comp_len; + kfree(cio); + + return ret; +} + +static void blk_copy_offload_write_end_io(struct bio *bio) +{ + struct copy_ctx *ctx = bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (bio->bi_status) { + clen = (bio->bi_iter.bi_sector << SECTOR_SHIFT) - cio->pos_out; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + } + __free_page(bio->bi_io_vec[0].bv_page); + bio_put(bio); + + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } +} + +static void blk_copy_offload_read_end_io(struct bio *read_bio) +{ + struct copy_ctx *ctx = read_bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (read_bio->bi_status) { + clen = (read_bio->bi_iter.bi_sector << SECTOR_SHIFT) + - cio->pos_in; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + __free_page(read_bio->bi_io_vec[0].bv_page); + bio_put(ctx->write_bio); + bio_put(read_bio); + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } + return; + } + + schedule_work(&ctx->dispatch_work); + bio_put(read_bio); +} + +static void blk_copy_dispatch_work_fn(struct work_struct *work) +{ + struct copy_ctx *ctx = container_of(work, struct copy_ctx, + dispatch_work); + + submit_bio(ctx->write_bio); +} + +/* + * __blk_copy_offload - Use device's native copy offload feature. + * we perform copy operation by sending 2 bio. + * 1. First we send a read bio with REQ_COPY flag along with a token and source + * and length. Once read bio reaches driver layer, device driver adds all the + * source info to token and does a fake completion. + * 2. Once read operation completes, we issue write with REQ_COPY flag with same + * token. In driver layer, token info is used to form a copy offload command. + * + * returns the length of bytes copied or negative error value + */ +static int __blk_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask) +{ + struct cio *cio; + struct copy_ctx *ctx; + struct bio *read_bio, *write_bio; + struct page *token; + sector_t copy_len; + sector_t rem, max_copy_len; + + cio = kzalloc(sizeof(struct cio), GFP_KERNEL); + if (!cio) + return -ENOMEM; + atomic_set(&cio->refcount, 0); + cio->waiter = current; + cio->endio = end_io; + cio->private = private; + + max_copy_len = min(bdev_max_copy_sectors(bdev_in), + bdev_max_copy_sectors(bdev_out)) << SECTOR_SHIFT; + + cio->pos_in = pos_in; + cio->pos_out = pos_out; + cio->comp_len = len; + for (rem = len; rem > 0; rem -= copy_len) { + copy_len = min(rem, max_copy_len); + + token = alloc_page(gfp_mask); + if (unlikely(!token)) + goto err_token; + + ctx = kzalloc(sizeof(struct copy_ctx), gfp_mask); + if (!ctx) + goto err_ctx; + read_bio = bio_alloc(bdev_in, 1, REQ_OP_READ | REQ_COPY + | REQ_SYNC | REQ_NOMERGE, gfp_mask); + if (!read_bio) + goto err_read_bio; + write_bio = bio_alloc(bdev_out, 1, REQ_OP_WRITE + | REQ_COPY | REQ_SYNC | REQ_NOMERGE, gfp_mask); + if (!write_bio) + goto err_write_bio; + + ctx->cio = cio; + ctx->write_bio = write_bio; + INIT_WORK(&ctx->dispatch_work, blk_copy_dispatch_work_fn); + + __bio_add_page(read_bio, token, PAGE_SIZE, 0); + read_bio->bi_iter.bi_size = copy_len; + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_end_io = blk_copy_offload_read_end_io; + read_bio->bi_private = ctx; + + __bio_add_page(write_bio, token, PAGE_SIZE, 0); + write_bio->bi_iter.bi_size = copy_len; + write_bio->bi_end_io = blk_copy_offload_write_end_io; + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_private = ctx; + + atomic_inc(&cio->refcount); + submit_bio(read_bio); + pos_in += copy_len; + pos_out += copy_len; + } + + /* Wait for completion of all IO's*/ + return cio_await_completion(cio); + +err_write_bio: + bio_put(read_bio); +err_read_bio: + kfree(ctx); +err_ctx: + __free_page(token); +err_token: + cio->comp_len = min_t(sector_t, cio->comp_len, (len - rem)); + return cio_await_completion(cio); +} + +static inline int blk_copy_sanity_check(struct block_device *bdev_in, + loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, + size_t len) +{ + unsigned int align = max(bdev_logical_block_size(bdev_out), + bdev_logical_block_size(bdev_in)) - 1; + + if (bdev_read_only(bdev_out)) + return -EPERM; + + if ((pos_in & align) || (pos_out & align) || (len & align) || !len || + len >= MAX_COPY_TOTAL_LENGTH) + return -EINVAL; + + return 0; +} + +static inline bool blk_check_copy_offload(struct request_queue *q_in, + struct request_queue *q_out) +{ + return blk_queue_copy(q_in) && blk_queue_copy(q_out); +} + +/* + * @bdev_in: source block device + * @pos_in: source offset + * @bdev_out: destination block device + * @pos_out: destination offset + * @len: length in bytes to be copied + * @end_io: end_io function to be called on completion of copy operation, + * for synchronous operation this should be NULL + * @private: end_io function will be called with this private data, should be + * NULL, if operation is synchronous in nature + * @gfp_mask: memory allocation flags (for bio_alloc) + * + * Returns the length of bytes copied or a negative error value + * + * Description: + * Copy source offset from source block device to destination block + * device. length of a source range cannot be zero. Max total length of + * copy is limited to MAX_COPY_TOTAL_LENGTH + */ +int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask) +{ + struct request_queue *q_in = bdev_get_queue(bdev_in); + struct request_queue *q_out = bdev_get_queue(bdev_out); + int ret = -EINVAL; + + ret = blk_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len); + if (ret) + return ret; + + if (blk_check_copy_offload(q_in, q_out)) + ret = __blk_copy_offload(bdev_in, pos_in, bdev_out, pos_out, + len, end_io, private, gfp_mask); + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_issue_copy); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/block/blk.h b/block/blk.h index d65d96994a94..684b8fa121db 100644 --- a/block/blk.h +++ b/block/blk.h @@ -311,6 +311,8 @@ static inline bool bio_may_exceed_limits(struct bio *bio, break; } + if (unlikely(op_is_copy(bio->bi_opf))) + return false; /* * All drivers must accept single-segments bios that are <= PAGE_SIZE. * This is a quick and dirty check that relies on the fact that diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index a0e339ff3d09..7f586c4b9954 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -423,6 +423,7 @@ enum req_flag_bits { */ /* for REQ_OP_WRITE_ZEROES: */ __REQ_NOUNMAP, /* do not free blocks when zeroing */ + __REQ_COPY, /* copy request */ __REQ_NR_BITS, /* stops here */ }; @@ -452,6 +453,7 @@ enum req_flag_bits { #define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV) #define REQ_SWAP (__force blk_opf_t)(1ULL << __REQ_SWAP) +#define REQ_COPY ((__force blk_opf_t)(1ULL << __REQ_COPY)) #define REQ_FAILFAST_MASK \ (REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER) @@ -478,6 +480,11 @@ static inline bool op_is_write(blk_opf_t op) return !!(op & (__force blk_opf_t)1); } +static inline bool op_is_copy(blk_opf_t op) +{ + return (op & REQ_COPY); +} + /* * Check if the bio or request is one that needs special treatment in the * flush state machine. @@ -537,4 +544,22 @@ struct blk_rq_stat { u64 batch; }; +typedef void (cio_iodone_t)(void *private, int comp_len); + +struct cio { + struct task_struct *waiter; /* waiting task (NULL if none) */ + atomic_t refcount; + loff_t pos_in; + loff_t pos_out; + size_t comp_len; + cio_iodone_t *endio; /* applicable for async operation */ + void *private; /* applicable for async operation */ +}; + +struct copy_ctx { + struct cio *cio; + struct work_struct dispatch_work; + struct bio *write_bio; +}; + #endif /* __LINUX_BLK_TYPES_H */ diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 200338f2ec2e..1bb43697d43d 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1054,6 +1054,9 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop); int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp); +int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Mon Mar 27 08:40:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13192077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FDC9C761A6 for ; Wed, 29 Mar 2023 08:16:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230480AbjC2IQ4 (ORCPT ); Wed, 29 Mar 2023 04:16:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230397AbjC2IQV (ORCPT ); Wed, 29 Mar 2023 04:16:21 -0400 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3318949C3 for ; Wed, 29 Mar 2023 01:16:13 -0700 (PDT) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20230329081611epoutp02123172a8f2e10164862618ff78bd9515~Q1dTyW2lR2902829028epoutp02B for ; Wed, 29 Mar 2023 08:16:11 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20230329081611epoutp02123172a8f2e10164862618ff78bd9515~Q1dTyW2lR2902829028epoutp02B DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1680077771; bh=9UUtJzqB/vtEDJqb2nlZEgTkIsQjAV6HpudQvYQnF5E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AdVEIYIoaP0MF1p36nYSgEpwWDydmEXZIbL6NG2T1QX16eeJaZFcMwSXw2wtPnWha /ZlvXXXX0W+gNkbYd6SEnbkHZ1WRtYSTNEEr2II1x4xjxJ8v1tWDewjqhBV1mFGk8D n/fWRXcnDDRXDytBkrz6HAXR9rUSp/0bD/5t8rN4= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230329081610epcas5p2d7561bc6f101ea54fb5213140dac3bf1~Q1dS9dGIM1664716647epcas5p27; Wed, 29 Mar 2023 08:16:10 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.179]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4PmfWr1zKQz4x9QD; Wed, 29 Mar 2023 08:16:08 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 0D.FF.06765.8C3F3246; Wed, 29 Mar 2023 17:16:08 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20230327084235epcas5p495559f907ce39184da72a412c5691e43~QOhyUneER1922519225epcas5p4T; Mon, 27 Mar 2023 08:42:35 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230327084235epsmtrp272391a7d39c94643d70c5131fbca4a5f~QOhyTjvI92690526905epsmtrp2m; Mon, 27 Mar 2023 08:42:35 +0000 (GMT) X-AuditID: b6c32a4b-20fff70000011a6d-b8-6423f3c85b85 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id D2.3C.18071.BF651246; Mon, 27 Mar 2023 17:42:35 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230327084231epsmtip15b2b6358f9832c39a98a2988b37b33a1~QOhu6xmSe2834728347epsmtip1d; Mon, 27 Mar 2023 08:42:31 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Vincent Fu , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v8 3/9] block: add emulation for copy Date: Mon, 27 Mar 2023 14:10:51 +0530 Message-Id: <20230327084103.21601-4-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230327084103.21601-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02TfzCbdxzH7/s8+UWne0TLVzodj/UchqRFv6rMrd32bHqbq223H7cjk4cg kiyJru1sfqTR0qWsuq6CorV2uGPDShHTaKk41U650VNmUaVFMdaeU0ua2Prf6/O59/v7+fG9 DwfnnmbzOIlSFa2QCiUky55xqcPbx+/6oqeIf/HJBlRr7MRRVv4qjqpH8ljoQccCQKcfPcHR Sm8fjvSzRUw01H4ZQ63nTmKosvoahlrK5zF0bW2GhU4aBgGaGNBhSD/si1r13QzU31zMQqUX JtjIUKDGUJMpE6BLK6U4qnkwx0DXh7egv44fA6hvtYsZ4Ur1346kdKO9LOqyboRN9d39hUH1 96ZSdVU5LKq+Ip1qGcpgUVr1rFmgGWVSc20DLOpEQxWg6nu+ohbrtlJ1phks6sVPkneLaaGI VrjT0jiZKFGaEEZGRsfsiQkK5gv8BCFoJ+kuFabQYeTefVF+byZKzMsg3Q8IJanmVJRQqSQD wncrZKkq2l0sU6rCSFouksgD5f5KYYoyVZrgL6VVuwR8/vYgszA2WdxjzAVyY8TBM0+b2Rkg OygX2HEgEQiXCtuYucCewyVaALzdtoRZgwUAcwqmcIuKSywD+MM573WH8ewUyyrSA1jXeA9Y RWoMTt8iLcwivODVSQ2wiDYR2Ticn8hhWAKc0ODQ1N7EtqiciCDYdbHimZtBbIOTT++wLOxA hMCynmIzc8zlAmDeqKMlbUfsgsVNRqZV4gi7C00MC+PEy1D9axFueR8StXZwyDAHrK3uhVnD apaVneB0VwPbyjy4OKu35RPg4/4JzMpyqO5ss3lfgxpjHm7pASe8YW1zgDXtBr831mDWuhuh dsVkszrAprPrTMKjlcU2hlB/I8PGFCz5O9u2Xi2ArRfa2fnAXffcPLrn5tH9X7oM4FXAlZYr UxJoZZB8h5T+8r9fjpOl1IFnB+IT2QTGxx75GwDGAQYAOTi5yWFlkBRxHUTCQ4dphSxGkSqh lQYQZN73dzhvc5zMfGFSVYwgMIQfGBwcHBiyI1hAujh4hXXHcYkEoYpOpmk5rVj3YRw7Xga2 dacnXvhbB3TO+kkpjvL49p721c/HRiuSPoh/P3pPo29J/Hhow9vB9++uZW97Pab2fjzv8adj goD5L/r2b6hO2Vz0oXhLw589sQ2N8741zq6D6Uf2Lad5X3Hanz6g154oV3XeEXh+7fHu75yp xTXksj060/m9NM8ISThW8xG/bNyn/OEbeUTXqVnuzZxbk7zzbzWHMk9VuWEH08pfCcv+MZln WDi/8UhLbt1nqw9NWp+jq+IMzDHt4xv/eF05vJg1blo6M5M2cTXTLcn5QP0MeySrsiM84nhS AbjZ+4dL/gv8Ug9lR8lLY2WHJKsRoSbZtH+zxin+5zaRBtpLv2kNjF1OnHunlGQoxUKBD65Q Cv8Fi8niWqkEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02Ra0hTYRzGec85O+e4GJ1ml1fNzNGorLxExpuEmlAdsg8SaRiUjXZwyznX pl0/uDS7LEoxum1imlZqYbmZmXMRq2Vu2cSVaWV2UZZZZuYkk1Y6Cfr28Px+PP8PfxoXPiMC abkym1MrJQoRyScaHooWrJhICZVG5t1m0C37YxzlFf3G0Y2eQhINPhwB6PzwOI4m2pw4sgwZ eKj7wT0MNV8pxlD1DRuGzOXfMWT785VExdZOgPpf6DFkebUMNVtaCeRqKiHR5Wv9FLKezcdQ Y98RgBomLuOodvAbgZ68CkIfT50AyPm7hRcfwLqeJ7L63jaSvafvoVjn2zqCdbXlsMaakyRr qsxlzd1akj2dPzQpFPTy2G/3X5DsmfoawJoch9kfxgWsse8rljRzO3+tlFPI93HqiNhdfJnD rgMqe/yBi94mSguOReuAHw2ZVdBeOkDqAJ8WMmYA+yo6yWkAoX3gGpjO/rDa66ampSMYfDx+ lJgCJLMYPnIXgCkwmynCYUev1jeFM+dw6DF/8U35M9Gw5Xqlb4pgxNDtfe3rBcwaWOYomcz0 5IkIWNg7a6r2Y2JgSaOdN5WFk0rh8SZsWp8FWy/1+Q7jTAjMv2PAiwCj/w/p/0NlAKsBAZxK k5meqYlSRSm5/eEaSaYmR5kevjsr0wh8Lw9b2gju1gyHWwFGAyuANC6aLTAlhkqFAqnk4CFO nZWmzlFwGisIognRPEG7rjVNyKRLsrkMjlNx6n8Uo/0CtZjB5alfV7JSVlA0ESftCI+1icFo V+rGKwLK9T0kqONzZLBj4TvxDnvQMmdenMcQVeUIDj5dNSe1as/80qRN6tzRhL1S5dXm7qR2 e0aic+6SRdHZh1/Hem7qyqVK1bERWUKqw52cG0KVbY2rNPI9pp+dxZb38vVv6iLCLjRF140N hw6cRzFpB9DH+RcGqSX99Suy2s/OWPQrI/DSqpeecW31h+CM+JgAhzhyy85t8jrpTzmVwmtY 11U21PLlnX9Xbc+GpybB/R7zIZuCeO81yKiKNeq5+YbBkaEKd3kyerow4MdiW2pB7SfF6IwE LzF2ZvlmS8xqmFA1JjaoRLs3iAiNTBIVhqs1kr9diRZoYQMAAA== X-CMS-MailID: 20230327084235epcas5p495559f907ce39184da72a412c5691e43 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230327084235epcas5p495559f907ce39184da72a412c5691e43 References: <20230327084103.21601-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Nitesh Shetty For the devices which does not support copy, copy emulation is added. It is required for in-kernel users like fabrics, where file descriptor is not available and hence they can't use copy_file_range. Copy-emulation is implemented by reading from source into memory and writing to the corresponding destination asynchronously. Also emulation is used, if copy offload fails or partially completes. Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu Signed-off-by: Anuj Gupta --- block/blk-lib.c | 171 ++++++++++++++++++++++++++++++++++++++++- block/blk-map.c | 4 +- include/linux/blkdev.h | 3 + 3 files changed, 175 insertions(+), 3 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index cbc6882d1e7a..a21819e59b29 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -289,6 +289,169 @@ static int __blk_copy_offload(struct block_device *bdev_in, loff_t pos_in, return cio_await_completion(cio); } +static void *blk_alloc_buf(sector_t req_size, sector_t *alloc_size, + gfp_t gfp_mask) +{ + int min_size = PAGE_SIZE; + void *buf; + + while (req_size >= min_size) { + buf = kvmalloc(req_size, gfp_mask); + if (buf) { + *alloc_size = req_size; + return buf; + } + /* retry half the requested size */ + req_size >>= 1; + } + + return NULL; +} + +static void blk_copy_emulate_write_end_io(struct bio *bio) +{ + struct copy_ctx *ctx = bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (bio->bi_status) { + clen = (bio->bi_iter.bi_sector << SECTOR_SHIFT) - cio->pos_out; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + } + kvfree(page_address(bio->bi_io_vec[0].bv_page)); + bio_map_kern_endio(bio); + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } +} + +static void blk_copy_emulate_read_end_io(struct bio *read_bio) +{ + struct copy_ctx *ctx = read_bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (read_bio->bi_status) { + clen = (read_bio->bi_iter.bi_sector << SECTOR_SHIFT) - + cio->pos_in; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + __free_page(read_bio->bi_io_vec[0].bv_page); + bio_map_kern_endio(read_bio); + kfree(ctx); + + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } + } + schedule_work(&ctx->dispatch_work); + kfree(read_bio); +} + +/* + * If native copy offload feature is absent, this function tries to emulate, + * by copying data from source to a temporary buffer and from buffer to + * destination device. + * returns the length of bytes copied or negative error value + */ +static int __blk_copy_emulate(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask) +{ + struct request_queue *in = bdev_get_queue(bdev_in); + struct request_queue *out = bdev_get_queue(bdev_out); + struct bio *read_bio, *write_bio; + void *buf = NULL; + struct copy_ctx *ctx; + struct cio *cio; + sector_t buf_len, req_len, rem = 0; + sector_t max_src_hw_len = min_t(unsigned int, + queue_max_hw_sectors(in), + queue_max_segments(in) << (PAGE_SHIFT - SECTOR_SHIFT)) + << SECTOR_SHIFT; + sector_t max_dst_hw_len = min_t(unsigned int, + queue_max_hw_sectors(out), + queue_max_segments(out) << (PAGE_SHIFT - SECTOR_SHIFT)) + << SECTOR_SHIFT; + sector_t max_hw_len = min_t(unsigned int, + max_src_hw_len, max_dst_hw_len); + + cio = kzalloc(sizeof(struct cio), GFP_KERNEL); + if (!cio) + return -ENOMEM; + atomic_set(&cio->refcount, 0); + cio->pos_in = pos_in; + cio->pos_out = pos_out; + cio->waiter = current; + cio->endio = end_io; + cio->private = private; + + for (rem = len; rem > 0; rem -= buf_len) { + req_len = min_t(int, max_hw_len, rem); + + buf = blk_alloc_buf(req_len, &buf_len, gfp_mask); + if (!buf) + goto err_alloc_buf; + + ctx = kzalloc(sizeof(struct copy_ctx), gfp_mask); + if (!ctx) + goto err_ctx; + + read_bio = bio_map_kern(in, buf, buf_len, gfp_mask); + if (IS_ERR(read_bio)) + goto err_read_bio; + + write_bio = bio_map_kern(out, buf, buf_len, gfp_mask); + if (IS_ERR(write_bio)) + goto err_write_bio; + + ctx->cio = cio; + ctx->write_bio = write_bio; + INIT_WORK(&ctx->dispatch_work, blk_copy_dispatch_work_fn); + + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_iter.bi_size = buf_len; + read_bio->bi_opf = REQ_OP_READ | REQ_SYNC; + bio_set_dev(read_bio, bdev_in); + read_bio->bi_end_io = blk_copy_emulate_read_end_io; + read_bio->bi_private = ctx; + + write_bio->bi_iter.bi_size = buf_len; + write_bio->bi_opf = REQ_OP_WRITE | REQ_SYNC; + bio_set_dev(write_bio, bdev_out); + write_bio->bi_end_io = blk_copy_emulate_write_end_io; + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_private = ctx; + + atomic_inc(&cio->refcount); + submit_bio(read_bio); + + pos_in += buf_len; + pos_out += buf_len; + } + + /* Wait for completion of all IO's*/ + return cio_await_completion(cio); + +err_write_bio: + bio_put(read_bio); +err_read_bio: + kfree(ctx); +err_ctx: + kvfree(buf); +err_alloc_buf: + cio->comp_len -= min_t(sector_t, cio->comp_len, len - rem); + return cio_await_completion(cio); +} + static inline int blk_copy_sanity_check(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len) @@ -338,15 +501,21 @@ int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, struct request_queue *q_in = bdev_get_queue(bdev_in); struct request_queue *q_out = bdev_get_queue(bdev_out); int ret = -EINVAL; + bool offload = false; ret = blk_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len); if (ret) return ret; - if (blk_check_copy_offload(q_in, q_out)) + offload = blk_check_copy_offload(q_in, q_out); + if (offload) ret = __blk_copy_offload(bdev_in, pos_in, bdev_out, pos_out, len, end_io, private, gfp_mask); + if ((ret != len) || !offload) + ret = __blk_copy_emulate(bdev_in, pos_in + ret, bdev_out, + pos_out + ret, len - ret, end_io, private, gfp_mask); + return ret; } EXPORT_SYMBOL_GPL(blkdev_issue_copy); diff --git a/block/blk-map.c b/block/blk-map.c index 7b12f4bb4d4c..be2414904319 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -362,7 +362,7 @@ static void bio_invalidate_vmalloc_pages(struct bio *bio) #endif } -static void bio_map_kern_endio(struct bio *bio) +void bio_map_kern_endio(struct bio *bio) { bio_invalidate_vmalloc_pages(bio); bio_uninit(bio); @@ -379,7 +379,7 @@ static void bio_map_kern_endio(struct bio *bio) * Map the kernel address into a bio suitable for io to a block * device. Returns an error pointer in case of error. */ -static struct bio *bio_map_kern(struct request_queue *q, void *data, +struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, gfp_t gfp_mask) { unsigned long kaddr = (unsigned long)data; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 1bb43697d43d..a54153610800 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1057,6 +1057,9 @@ int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); +struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, + gfp_t gfp_mask); +void bio_map_kern_endio(struct bio *bio); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Mon Mar 27 08:40:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13192078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44407C77B6C for ; Wed, 29 Mar 2023 08:17:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230412AbjC2IQ6 (ORCPT ); Wed, 29 Mar 2023 04:16:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230400AbjC2IQV (ORCPT ); Wed, 29 Mar 2023 04:16:21 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A90D7272E for ; Wed, 29 Mar 2023 01:16:17 -0700 (PDT) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230329081616epoutp031090b9ed9bc7e7daefb7168d64133601~Q1dX7_uAU0376903769epoutp03Q for ; Wed, 29 Mar 2023 08:16:16 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230329081616epoutp031090b9ed9bc7e7daefb7168d64133601~Q1dX7_uAU0376903769epoutp03Q DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1680077776; bh=WvdhcAqig0ZvA8R/13pJCcDVxh9uWqwE8wBUKwSBVOY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kGq/yGzeaRJYweIrpaIlz/UzpYKqGinMCxyLbVDC8ZL1lk4sFHtnxTmh6b+c7kRte QgCG4kzsfm/G0frDRThj1Vt8xie0ftL9u1SKbpy9ESp/u8O+7n4fySGNA+6Siki8BB mXJQoYryzrJ6lg+T6JExY2cBVynAlOvwATrXEuTE= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230329081615epcas5p14300f1ae4f62c3d77912cb5bed32fae8~Q1dXNOwWs0780307803epcas5p1G; Wed, 29 Mar 2023 08:16:15 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.179]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4PmfWx0ScCz4x9QN; Wed, 29 Mar 2023 08:16:13 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 86.C5.55678.CC3F3246; Wed, 29 Mar 2023 17:16:12 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230327084244epcas5p1b0ede867e558ff6faf258de3656a8aa4~QOh6-F6-N3074330743epcas5p1C; Mon, 27 Mar 2023 08:42:44 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230327084244epsmtrp1cff790198889bdcd3d04f686e0b3d729~QOh6_Nzdt3087530875epsmtrp14; Mon, 27 Mar 2023 08:42:44 +0000 (GMT) X-AuditID: b6c32a4a-909fc7000000d97e-35-6423f3cced03 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 66.3C.18071.40751246; Mon, 27 Mar 2023 17:42:44 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230327084241epsmtip156d3b8206492a759af558e21583618b1~QOh3v8lla2834728347epsmtip1f; Mon, 27 Mar 2023 08:42:41 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v8 4/9] fs, block: copy_file_range for def_blk_ops for direct block device. Date: Mon, 27 Mar 2023 14:10:52 +0530 Message-Id: <20230327084103.21601-5-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230327084103.21601-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTVxzHPfeWPjCwy8N4hnt0l0wGCLRYysHZsWS63MmMbHMuM3HY0Tta 6Wt9yJhbeJRiwGARg4uFiAN0QCdsII6HJaYKyGskA0R0BebKKnSCSKZbwLCWls3/Pr9vft/f 6+Sw8eBTrDC2TKmjNUqxnGT6M65cj4yKGVwKl/CMdQLU1N+Do/zSpziy2E1M5Lr+CKAzD//B 0fLQMI6s8xV+aOJaO4auVpdhqN7SjaHObxcx1L36gInKbLcAmhkzY8h6JxpdtfYx0EhHJRNV XZxhIdtpA4baHHkAXVmuwlGja4GBbt7Zgoaf9vq9CamR0RTKPDXEpNrNdhY1PPkjgxoZ0lPN DUVMqqU2h+qcyGVSJYZ5d4Jxyo9a6BpjUicvNwCqZeAYtdT8EtXseIClPncwc6eUFktoDZdW pqskMmWGiEz5IO2ttAQhjx/DT0KJJFcpVtAicte7qTFvy+TuO5Dco2K53i2lirVaMu6NnRqV XkdzpSqtTkTSaolcLVDHasUKrV6ZEaukdTv4PF58gjvxcKa0ytaOqb9/7Yum3hOsXND9SjHg sCEhgPk9fSwPBxOdAA5OHioG/m5+BGCe/TvgDR4DeKFlEqw7rt07x/Q6rADaR+K8SQYMroyN rpViEhHwhtO45g4lCnG4OFPE8AQ4MY3BqvzatVIhxCG4Wlq2VopBvArHLC7cwwFEEhxoPOnW 2e52cdA0FeSROcQOWNnW7+dNCYJ9Zx0MD+PEy9DQWoF76kPCwoFne4dY3lF3wbumVoaXQ+Bc 72WfHgZnTYU+zoB/j8xgXlZDQ0+Xb81kaOw34Z4ZcCISNnXEeeUXYXl/I+btGwhLlh0+awBs O7fOJDxeX+ljCK0/5/qYgsMd55nea5UAONn9hFEKuOZn9jE/s4/5/9bnAd4AnqfVWkUGrU1Q xyvprP9eOV2laAZrfyNqTxv4bfphrA1gbGADkI2ToQHLt0hJcIBEnP0lrVGlafRyWmsDCe57 n8LDNqWr3J9LqUvjC5J4AqFQKEjaLuSTmwMiRH3pwUSGWEdn0rSa1qz7MDYnLBermF36y+HY GHgwmW3ZH8c7UjOu2RY/uKGupODokwmOtkjxOD36vkjWAz4OL2HVJkoD5aOWlf0FDZ/U3VyI cn4999mF0+8nFhxvH/x194Ya26fZ9vHSshrZT/z60OKLmVyH/cbv43LRloJ901/VhHy+8RtJ MV+z3f+FvMnw1Zyt8XOjP9TObr7n4p9Q35b+cmTRGPS6c+LPxuigVfY7m+46hV1ThzX5ya0T iXvG9nVGuLauRO7tnBlQhFuc7Xppimzso8TsY5fu16rY5dIDRMWZCjor2k+1GujqSdE7RbY/ PjzAcQYZa7K6ZKrFwmbu3pXq+b7w0Orx93K2WW5fElhNhqPlJEMrFfOjcI1W/C99hxNMpAQA AA== X-Brightmail-Tracker: H4sIAAAAAAAAA02Rf0yMcRzH+36fp6en7ObpMN+E9PiVX6VlfLXE/HzU0ObnxDjd48RdtefK jxintKjR6bbShfJjpy5OdUp1hd05uXI7K2fTKNnRSElmRoWuZvPfZ5/X6/3eZ/vQhPgpOYmO T0jmhQSJnKV8yGoLO3UBuT1QuvC7aha+2/SEwGnqIQKXvcmhcLelH+C8vp8EHrA7CNzQW+iJ Xz2qhbj+ei7EpWVWiE3XvkJs/dND4VzzS4DfO7UQN7TNw/UNNhK31l2mcJHuvRc2a9IhrnGd Brh6oIjAhu4vJH7a5o8dQ42eKxDX+iKa03bYKa5W+8aLc7RXkFyrPYWr1J+jOOPNU5zplYri zqf3DgsZHZ7clwdOirtwTw84Y/Nx7lvlVK7S1QNjxu70iZDy8vjDvBASudfnQJG5FibdDjp6 tzHbSwWsgVnAm0bMIvTo3VUqC/jQYsYE0OvMNM9RgFDTRx0Ynceh0t9dXqPSaYjUli7CDShm NnrclQHcYDyjJlBLh2qkimB6IdJ36oaraHocE4vaM0+4AyQzEznLukfCImYpajZcoNwKYkJQ Toeve+3NhKPLNU0jR4iHlZzMOjiq+yJbgYt0zwQTgNKrCgk1YLT/Ie1/qBhAPfDjk5QKmUIZ mhSawB8JVkoUypQEWXBcoqISjDx77pwacF/fF2wGkAZmgGiCHS8yRgdKxSKp5FgqLyTuEVLk vNIM/GmSnSh6nmXbI2ZkkmT+EM8n8cI/CmnvSSrol6z5tcy16rPT0E6HWEseyuYVpa4si5ps EKt7VufrGoK62f7VjpO+bdbp36Jdt9gJvQWZ36+0bJ1yv+aMbvHuAtiTPXiJyl9aVVcIBhrT dm1ZV5u/L97biVMXxsTJ7BXFYevzDKYxtoBtnVVb+sf4b1I83Hvk5do4AM9sKrHFc0PGr3PK /3wo1+RudQge0+xnoSiiZeyPMP7G/KgnawbDsy/KGU3A7Nj8XZ2ktHz584ySMGFjf3WgeXuX Rh5xNTJ8iR970NJXkecR+2nCjpmR8npbyvmoOwUwwmNwYkxxYkizzln6NigKr5Ueb2RnKIXN dtOGWx8tp5pSD5qMz/azpPKAJHQuISglfwGRSWizWwMAAA== X-CMS-MailID: 20230327084244epcas5p1b0ede867e558ff6faf258de3656a8aa4 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230327084244epcas5p1b0ede867e558ff6faf258de3656a8aa4 References: <20230327084103.21601-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Nitesh Shetty For direct block device opened with O_DIRECT, use copy_file_range to issue device copy offload, and fallback to generic_copy_file_range incase device copy offload capability is absent. Modify checks to allow bdevs to use copy_file_range. Suggested-by: Ming Lei Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty --- block/blk-lib.c | 22 ++++++++++++++++++++++ block/fops.c | 20 ++++++++++++++++++++ fs/read_write.c | 11 +++++++++-- include/linux/blkdev.h | 3 +++ 4 files changed, 54 insertions(+), 2 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index a21819e59b29..c288573c7e77 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -475,6 +475,28 @@ static inline bool blk_check_copy_offload(struct request_queue *q_in, return blk_queue_copy(q_in) && blk_queue_copy(q_out); } +int blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask) +{ + struct request_queue *in_q = bdev_get_queue(bdev_in); + struct request_queue *out_q = bdev_get_queue(bdev_out); + int ret = -EINVAL; + bool offload = false; + + ret = blk_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len); + if (ret) + return ret; + + offload = blk_check_copy_offload(in_q, out_q); + if (offload) + ret = __blk_copy_offload(bdev_in, pos_in, bdev_out, pos_out, + len, end_io, private, gfp_mask); + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_copy_offload); + /* * @bdev_in: source block device * @pos_in: source offset diff --git a/block/fops.c b/block/fops.c index d2e6be4e3d1c..3b7c05831d5c 100644 --- a/block/fops.c +++ b/block/fops.c @@ -611,6 +611,25 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to) return ret; } +static ssize_t blkdev_copy_file_range(struct file *file_in, loff_t pos_in, + struct file *file_out, loff_t pos_out, + size_t len, unsigned int flags) +{ + struct block_device *in_bdev = I_BDEV(bdev_file_inode(file_in)); + struct block_device *out_bdev = I_BDEV(bdev_file_inode(file_out)); + int comp_len = 0; + + if ((file_in->f_iocb_flags & IOCB_DIRECT) && + (file_out->f_iocb_flags & IOCB_DIRECT)) + comp_len = blkdev_copy_offload(in_bdev, pos_in, out_bdev, + pos_out, len, NULL, NULL, GFP_KERNEL); + if (comp_len != len) + comp_len = generic_copy_file_range(file_in, pos_in + comp_len, + file_out, pos_out + comp_len, len - comp_len, flags); + + return comp_len; +} + #define BLKDEV_FALLOC_FL_SUPPORTED \ (FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE | \ FALLOC_FL_ZERO_RANGE | FALLOC_FL_NO_HIDE_STALE) @@ -694,6 +713,7 @@ const struct file_operations def_blk_fops = { .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, .fallocate = blkdev_fallocate, + .copy_file_range = blkdev_copy_file_range, }; static __init int blkdev_init(void) diff --git a/fs/read_write.c b/fs/read_write.c index 7a2ff6157eda..62e925e9b2f0 100644 --- a/fs/read_write.c +++ b/fs/read_write.c @@ -20,6 +20,7 @@ #include #include #include +#include #include "internal.h" #include @@ -1448,7 +1449,11 @@ static int generic_copy_file_checks(struct file *file_in, loff_t pos_in, return -EOVERFLOW; /* Shorten the copy to EOF */ - size_in = i_size_read(inode_in); + if (S_ISBLK(inode_in->i_mode)) + size_in = bdev_nr_bytes(I_BDEV(file_in->f_mapping->host)); + else + size_in = i_size_read(inode_in); + if (pos_in >= size_in) count = 0; else @@ -1709,7 +1714,9 @@ int generic_file_rw_checks(struct file *file_in, struct file *file_out) /* Don't copy dirs, pipes, sockets... */ if (S_ISDIR(inode_in->i_mode) || S_ISDIR(inode_out->i_mode)) return -EISDIR; - if (!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) + + if ((!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) && + (!S_ISBLK(inode_in->i_mode) || !S_ISBLK(inode_out->i_mode))) return -EINVAL; if (!(file_in->f_mode & FMODE_READ) || diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index a54153610800..468d5f3378e2 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1057,6 +1057,9 @@ int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); +int blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask); struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, gfp_t gfp_mask); void bio_map_kern_endio(struct bio *bio); From patchwork Mon Mar 27 08:40:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13192079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57CF2C6FD18 for ; Wed, 29 Mar 2023 08:17:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230497AbjC2IRP (ORCPT ); Wed, 29 Mar 2023 04:17:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47354 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230427AbjC2IQ0 (ORCPT ); Wed, 29 Mar 2023 04:16:26 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2849C4C19 for ; Wed, 29 Mar 2023 01:16:21 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230329081619epoutp031e61fe80a5c042494da09ab33a36909f~Q1dbI-jgj0422704227epoutp03A for ; Wed, 29 Mar 2023 08:16:19 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230329081619epoutp031e61fe80a5c042494da09ab33a36909f~Q1dbI-jgj0422704227epoutp03A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1680077779; bh=wfIShqGRwy8zbWXdhjKiAuBZYfQu+83aAnldtDJI0aM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ps/Y+y09r386qqNImV95D3afT7nFYAOcjAKvNB3TQJKpYAu+TkgOQ+YTPqcALC1hL vxaRTzflt8iL4TFE27CZo9524GYH7CrEm9uBUKVSstUxCnOWQMEcsdOFfqzguWkYwA Sci3dE0xfFYDKfT520INlkyr/DwUH5Lx3nvntECE= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230329081618epcas5p49a981c0219f0702b8b4e1ca7bca29f56~Q1daWVRVl0904209042epcas5p4p; Wed, 29 Mar 2023 08:16:18 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.181]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4PmfX10952z4x9Px; Wed, 29 Mar 2023 08:16:17 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 7E.8D.10528.0D3F3246; Wed, 29 Mar 2023 17:16:16 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20230327084254epcas5p4c5f324c1501062f743895273c302c0a4~QOiDroNmb1029410294epcas5p4r; Mon, 27 Mar 2023 08:42:54 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230327084254epsmtrp17f8b55ecd893d4cfccb9d8d41b09877e~QOiDqVzDB3087530875epsmtrp1B; Mon, 27 Mar 2023 08:42:54 +0000 (GMT) X-AuditID: b6c32a49-e75fa70000012920-b8-6423f3d0ad11 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id DB.C5.31821.D0751246; Mon, 27 Mar 2023 17:42:53 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230327084250epsmtip18b195389784bd7a60d495205c2edc92c~QOiASmib92502425024epsmtip1X; Mon, 27 Mar 2023 08:42:50 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , =?utf-8?q?Javier_Gonz=C3=A1lez?= , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v8 5/9] nvme: add copy offload support Date: Mon, 27 Mar 2023 14:10:53 +0530 Message-Id: <20230327084103.21601-6-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230327084103.21601-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xbdRTHvff2uQheCrIf6FhzF4NggFZK9yvC2ISZu+AMcSZbiAlc2zte pW362AQzRQoMcMhjbssKG4/NGdgD5ZXy6MRWNmBD5B2IwKY0imxlPN1Wcba06P77nHO+53ce vxwOxvuK7c9JU2hptYKSE6xtjDZL0OshP6/skgkulklgY/8tDOaWbWDw6nQpCy5YlhF49tET DNoHBjFoslUy4WR3Owq76ipQWH+1B4WdtUso/O2XFTbsefaQBSvM4wi0jhlQaJp6A3aZ+hhw pKOKBauvWNnQfFqPQuPc5whss1dj8MbCIgP2Tr0CBzduM/f6kSOj8aRhdoBFthum2eTgzHcM cmRARzY1FLHI5sufkZ2TOSyyRG9zCPJnmeTizTEW+WVLA0I23/mEXGkKIJvmHqIJLyVmRKXS lIxW82mFVClLU6REE/GHkmKTIsQCYYhQAncTfAWVSUcTce8mhLyTJncsg+Afo+Q6hyuB0miI sD1RaqVOS/NTlRptNEGrZHKVSBWqoTI1OkVKqILWRgoFgjcjHMLkjNTT6/NMlfH4x2unclk5 yN3kYoTLAbgI3OlrxoqRbRwe3omAwVor02UsI6C7pBJxGSsIGFg+z9pK+Xt53h3oQMD9hWK2 M8DD9SiYGDnhZBYeCH78PX9T5IMXYGDJWsRwGhheg4GBsS6GU+WNi8FGwbcOFYfDwF8DT/LS neiBS4B56IATAR4GSme9nGIuHgmqjP1MJ3vgXqDv/NzmIxi+E+hbKzFXbxYuuFfu7eI4cOXZ ZYaLvcGft1vYLvYH86UFbk4Bj0esqItVQH/rJuLiGJDfX4o5W8DwINDYEeZy7wBn+m+grrKe oMQ+5071AMaLW0yAk/VVbgbA9FOOm0nw1GJxb7rEsenvS5llCN/w3DiG58Yx/F+6BsEaED9a pclMoTURKqGCPv7fH0uVmU3I5nkEHzAi0/cehZoRlIOYEcDBCB8P+zgh43nIqKxsWq1MUuvk tMaMRDiWXY75vyxVOu5LoU0SiiQCkVgsFknCxUJiu0dgdJ+Uh6dQWjqDplW0eisP5XD9c9D0 F5hjj9Ns62y1yRSia0QT27i+E9Kj4+f+OSHwsYNe+1DtmA3/wdztdfCjXbap/NVjsRHDw6Oi 5YLmKEbkRqvsTHvui4c9Jx5MzuzXURYR0RlUEX8XZD/YZ/6j+sj1znS/v4Zysvb181e/KQwI 5137tTDTFHs/LuvC3jXfJtmOs6sG83xytl6qbfFVhzLUtnOBPd5JdVVZMU/z5HHSI6GJB8en 1i5QhaMdPjtt6xODeG9dkedbfel7Zg6PBn+qrPeeMIcravJO1nywlIFc6104eumU6NBigHX4 a/q65P0vLgV9OFNd7v92pTX9vTzhVM/ucV9jRjP1KoMbWJ03H9O6nWBoUilhMKbWUP8C6tbM CacEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrEIsWRmVeSWpSXmKPExsWy7bCSnC5vuGKKwcEPZhbrTx1jtmia8JfZ YvXdfjaL14c/MVpM+/CT2eL32fPMFnvfzWa1uHlgJ5PFnkWTmCxWrj7KZLF74Ucmi8d3PrNb HP3/ls1i0qFrjBZPr85isth7S9tiz96TLBaXd81hs5i/7Cm7xaHJzUwWO540Mlps+z2f2WLd 6/csFiduSVuc/3uc1UHS4/IVb49Z98+yeeycdZfd4/y9jSwel8+Wemxa1cnmsXlJvcfumw1s Hr3N74AKWu+zerzfd5XNo2/LKkaPzaerPT5vkvPY9OQtUwB/FJdNSmpOZllqkb5dAlfG5G8v WQt2lFd87Wlia2A8k9DFyMkhIWAi8efTS8YuRi4OIYEdjBLXry5lgkhISJx6uYwRwhaWWPnv OTtEUSOTxIqtW9lAEmwC6hJHnreCdYsITGCWuHS/gQ3EYRZYxyyxtnkpK0iVsICZxN+2DUBV HBwsAqoSP1uyQExeAUuJQxc9QUwJAX2J/vuCIMWcAlYSc3acAmsUAqrob98Fdg+vgKDEyZlP WEDKmYHWrp8nBBJmFpCXaN46m3kCo+AsJFWzEKpmIalawMi8ilEytaA4Nz232LDAKC+1XK84 Mbe4NC9dLzk/dxMjOBloae1g3LPqg94hRiYOxkOMEhzMSiK8m70VU4R4UxIrq1KL8uOLSnNS iw8xSnOwKInzXug6GS8kkJ5YkpqdmlqQWgSTZeLglGpgStRP3zHp1c3HM4P0j00OLUj5GNld Iz5jx/353QEbZx897qfx0U6mbI7zfNXP8/y1GHlfbUpNePSy9NaM++cyrXnWvZRmfjnJk9Ni z02TG/OqpfVkdnLOr1h8Zv1NJ0Xjjubd57KXFrlG5IttcTBh7IjOLpPNjjgn8Ik/pGSXkMol brNzZ3Oyu9p0CxUMd+adDdLbtVO/Z/3ro6+TLPsMteZLyuo7TXUvUXW+z3w6dmlV2ort00V0 Hu/m4ZS7PjGWY92xgE2/5e9KG6TyfOD9/Whyi9PuLb0yelKMYukSc8yr96w7tO1Bl7TaU7Hn Ew+HSstdOtlgvTjJMfzK/MjASAVma6FffkXTeP08V7oqsRRnJBpqMRcVJwIAV+kfhXUDAAA= X-CMS-MailID: 20230327084254epcas5p4c5f324c1501062f743895273c302c0a4 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230327084254epcas5p4c5f324c1501062f743895273c302c0a4 References: <20230327084103.21601-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Nitesh Shetty For device supporting native copy, nvme driver receives read and write request with BLK_COPY op flags. For read request the nvme driver populates the payload with source information. For write request the driver converts it to nvme copy command using the source information in the payload and submits to the device. current design only supports single source range. This design is courtesy Mikulas Patocka's token based copy trace event support for nvme_copy_cmd. Set the device copy limits to queue limits. Signed-off-by: Kanchan Joshi Signed-off-by: Nitesh Shetty Signed-off-by: Javier González Signed-off-by: Anuj Gupta --- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 106 +++++++++++++++++++++++++++++++++- drivers/nvme/host/fc.c | 5 ++ drivers/nvme/host/nvme.h | 7 +++ drivers/nvme/host/pci.c | 27 ++++++++- drivers/nvme/host/rdma.c | 7 +++ drivers/nvme/host/tcp.c | 16 +++++ drivers/nvme/host/trace.c | 19 ++++++ include/linux/nvme.h | 43 +++++++++++++- 9 files changed, 223 insertions(+), 8 deletions(-) diff --git a/drivers/nvme/host/constants.c b/drivers/nvme/host/constants.c index bc523ca02254..01be882b726f 100644 --- a/drivers/nvme/host/constants.c +++ b/drivers/nvme/host/constants.c @@ -19,6 +19,7 @@ static const char * const nvme_ops[] = { [nvme_cmd_resv_report] = "Reservation Report", [nvme_cmd_resv_acquire] = "Reservation Acquire", [nvme_cmd_resv_release] = "Reservation Release", + [nvme_cmd_copy] = "Copy Offload", [nvme_cmd_zone_mgmt_send] = "Zone Management Send", [nvme_cmd_zone_mgmt_recv] = "Zone Management Receive", [nvme_cmd_zone_append] = "Zone Management Append", diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 4eb62a7dac44..e0b5381d572b 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -754,6 +754,80 @@ static inline void nvme_setup_flush(struct nvme_ns *ns, cmnd->common.nsid = cpu_to_le32(ns->head->ns_id); } +static inline blk_status_t nvme_setup_copy_read(struct nvme_ns *ns, + struct request *req) +{ + struct bio *bio = req->bio; + struct nvme_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + + memcpy(token->subsys, "nvme", 4); + token->ns = ns; + token->src_sector = bio->bi_iter.bi_sector; + token->sectors = bio->bi_iter.bi_size >> 9; + + return BLK_STS_OK; +} + +static inline blk_status_t nvme_setup_copy_write(struct nvme_ns *ns, + struct request *req, struct nvme_command *cmnd) +{ + struct nvme_copy_range *range = NULL; + struct bio *bio = req->bio; + struct nvme_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + sector_t src_sector, dst_sector, n_sectors; + u64 src_lba, dst_lba, n_lba; + unsigned short nr_range = 1; + u16 control = 0; + + if (unlikely(memcmp(token->subsys, "nvme", 4))) + return BLK_STS_NOTSUPP; + if (unlikely(token->ns != ns)) + return BLK_STS_NOTSUPP; + + src_sector = token->src_sector; + dst_sector = bio->bi_iter.bi_sector; + n_sectors = token->sectors; + if (WARN_ON(n_sectors != bio->bi_iter.bi_size >> 9)) + return BLK_STS_NOTSUPP; + + src_lba = nvme_sect_to_lba(ns, src_sector); + dst_lba = nvme_sect_to_lba(ns, dst_sector); + n_lba = nvme_sect_to_lba(ns, n_sectors); + + if (WARN_ON(!n_lba)) + return BLK_STS_NOTSUPP; + + if (req->cmd_flags & REQ_FUA) + control |= NVME_RW_FUA; + + if (req->cmd_flags & REQ_FAILFAST_DEV) + control |= NVME_RW_LR; + + memset(cmnd, 0, sizeof(*cmnd)); + cmnd->copy.opcode = nvme_cmd_copy; + cmnd->copy.nsid = cpu_to_le32(ns->head->ns_id); + cmnd->copy.sdlba = cpu_to_le64(dst_lba); + + range = kmalloc_array(nr_range, sizeof(*range), + GFP_ATOMIC | __GFP_NOWARN); + if (!range) + return BLK_STS_RESOURCE; + + range[0].slba = cpu_to_le64(src_lba); + range[0].nlb = cpu_to_le16(n_lba - 1); + + cmnd->copy.nr_range = 0; + + req->special_vec.bv_page = virt_to_page(range); + req->special_vec.bv_offset = offset_in_page(range); + req->special_vec.bv_len = sizeof(*range) * nr_range; + req->rq_flags |= RQF_SPECIAL_PAYLOAD; + + cmnd->copy.control = cpu_to_le16(control); + + return BLK_STS_OK; +} + static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req, struct nvme_command *cmnd) { @@ -988,10 +1062,16 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req) ret = nvme_setup_discard(ns, req, cmd); break; case REQ_OP_READ: - ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_read); + if (unlikely(req->cmd_flags & REQ_COPY)) + ret = nvme_setup_copy_read(ns, req); + else + ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_read); break; case REQ_OP_WRITE: - ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_write); + if (unlikely(req->cmd_flags & REQ_COPY)) + ret = nvme_setup_copy_write(ns, req, cmd); + else + ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_write); break; case REQ_OP_ZONE_APPEND: ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_zone_append); @@ -1698,6 +1778,26 @@ static void nvme_config_discard(struct gendisk *disk, struct nvme_ns *ns) blk_queue_max_write_zeroes_sectors(queue, UINT_MAX); } +static void nvme_config_copy(struct gendisk *disk, struct nvme_ns *ns, + struct nvme_id_ns *id) +{ + struct nvme_ctrl *ctrl = ns->ctrl; + struct request_queue *q = disk->queue; + + if (!(ctrl->oncs & NVME_CTRL_ONCS_COPY)) { + blk_queue_max_copy_sectors_hw(q, 0); + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + return; + } + + /* setting copy limits */ + if (blk_queue_flag_test_and_set(QUEUE_FLAG_COPY, q)) + return; + + blk_queue_max_copy_sectors_hw(q, + nvme_lba_to_sect(ns, le16_to_cpu(id->mssrl))); +} + static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b) { return uuid_equal(&a->uuid, &b->uuid) && @@ -1897,6 +1997,7 @@ static void nvme_update_disk_info(struct gendisk *disk, set_capacity_and_notify(disk, capacity); nvme_config_discard(disk, ns); + nvme_config_copy(disk, ns, id); blk_queue_max_write_zeroes_sectors(disk->queue, ns->ctrl->max_zeroes_sectors); } @@ -5345,6 +5446,7 @@ static inline void _nvme_check_size(void) BUILD_BUG_ON(sizeof(struct nvme_download_firmware) != 64); BUILD_BUG_ON(sizeof(struct nvme_format_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_dsm_cmd) != 64); + BUILD_BUG_ON(sizeof(struct nvme_copy_command) != 64); BUILD_BUG_ON(sizeof(struct nvme_write_zeroes_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_abort_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_get_log_page_command) != 64); diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 2ed75923507d..db2e22b4ca7f 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -2807,6 +2807,11 @@ nvme_fc_queue_rq(struct blk_mq_hw_ctx *hctx, if (ret) return ret; + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_end_request(rq, BLK_STS_OK); + return BLK_STS_OK; + } /* * nvme core doesn't quite treat the rq opaquely. Commands such * as WRITE ZEROES will return a non-zero rq payload_bytes yet diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index bf46f122e9e1..257f91ee1f2d 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -500,6 +500,13 @@ struct nvme_ns { }; +struct nvme_copy_token { + char subsys[4]; + struct nvme_ns *ns; + u64 src_sector; + u64 sectors; +}; + /* NVMe ns supports metadata actions by the controller (generate/strip) */ static inline bool nvme_ns_has_pi(struct nvme_ns *ns) { diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index b615906263f3..d30816533922 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -496,16 +496,19 @@ static inline void nvme_sq_copy_cmd(struct nvme_queue *nvmeq, nvmeq->sq_tail = 0; } -static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +static inline void nvme_commit_sq_db(struct nvme_queue *nvmeq) { - struct nvme_queue *nvmeq = hctx->driver_data; - spin_lock(&nvmeq->sq_lock); if (nvmeq->sq_tail != nvmeq->last_sq_tail) nvme_write_sq_db(nvmeq, true); spin_unlock(&nvmeq->sq_lock); } +static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +{ + nvme_commit_sq_db(hctx->driver_data); +} + static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req, int nseg) { @@ -849,6 +852,12 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, struct request *req) if (ret) return ret; + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_start_request(req); + return BLK_STS_OK; + } + if (blk_rq_nr_phys_segments(req)) { ret = nvme_map_data(dev, req, &iod->cmd); if (ret) @@ -895,6 +904,18 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, ret = nvme_prep_rq(dev, req); if (unlikely(ret)) return ret; + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_set_request_complete(req); + blk_mq_end_request(req, BLK_STS_OK); + /* Commit the sq if copy read was the last req in the list, + * as copy read deoesn't update sq db + */ + if (bd->last) + nvme_commit_sq_db(nvmeq); + return ret; + } + spin_lock(&nvmeq->sq_lock); nvme_sq_copy_cmd(nvmeq, &iod->cmd); nvme_write_sq_db(nvmeq, bd->last); diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index bbad26b82b56..a8bf2a87f42a 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -2043,6 +2043,13 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, nvme_start_request(rq); + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_end_request(rq, BLK_STS_OK); + ret = BLK_STS_OK; + goto unmap_qe; + } + if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) && queue->pi_support && (c->common.opcode == nvme_cmd_write || diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 42c0598c31f2..3c238e07b05c 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -2364,6 +2364,11 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns, if (ret) return ret; + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + return BLK_STS_OK; + } + req->state = NVME_TCP_SEND_CMD_PDU; req->status = cpu_to_le16(NVME_SC_SUCCESS); req->offset = 0; @@ -2432,6 +2437,17 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx, nvme_start_request(rq); + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_set_request_complete(rq); + blk_mq_end_request(rq, BLK_STS_OK); + /* if copy read is the last req queue tcp reqs */ + if (bd->last && nvme_tcp_queue_more(queue)) + queue_work_on(queue->io_cpu, nvme_tcp_wq, + &queue->io_work); + return ret; + } + nvme_tcp_queue_request(req, true, bd->last); return BLK_STS_OK; diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c index 1c36fcedea20..da4a7494e5a7 100644 --- a/drivers/nvme/host/trace.c +++ b/drivers/nvme/host/trace.c @@ -150,6 +150,23 @@ static const char *nvme_trace_read_write(struct trace_seq *p, u8 *cdw10) return ret; } +static const char *nvme_trace_copy(struct trace_seq *p, u8 *cdw10) +{ + const char *ret = trace_seq_buffer_ptr(p); + u64 slba = get_unaligned_le64(cdw10); + u8 nr_range = get_unaligned_le16(cdw10 + 8); + u16 control = get_unaligned_le16(cdw10 + 10); + u32 dsmgmt = get_unaligned_le32(cdw10 + 12); + u32 reftag = get_unaligned_le32(cdw10 + 16); + + trace_seq_printf(p, + "slba=%llu, nr_range=%u, ctrl=0x%x, dsmgmt=%u, reftag=%u", + slba, nr_range, control, dsmgmt, reftag); + trace_seq_putc(p, 0); + + return ret; +} + static const char *nvme_trace_dsm(struct trace_seq *p, u8 *cdw10) { const char *ret = trace_seq_buffer_ptr(p); @@ -243,6 +260,8 @@ const char *nvme_trace_parse_nvm_cmd(struct trace_seq *p, return nvme_trace_zone_mgmt_send(p, cdw10); case nvme_cmd_zone_mgmt_recv: return nvme_trace_zone_mgmt_recv(p, cdw10); + case nvme_cmd_copy: + return nvme_trace_copy(p, cdw10); default: return nvme_trace_common(p, cdw10); } diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 779507ac750b..6582b26e532c 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -337,7 +337,7 @@ struct nvme_id_ctrl { __u8 nvscc; __u8 nwpc; __le16 acwu; - __u8 rsvd534[2]; + __le16 ocfs; __le32 sgls; __le32 mnan; __u8 rsvd544[224]; @@ -365,6 +365,7 @@ enum { NVME_CTRL_ONCS_WRITE_ZEROES = 1 << 3, NVME_CTRL_ONCS_RESERVATIONS = 1 << 5, NVME_CTRL_ONCS_TIMESTAMP = 1 << 6, + NVME_CTRL_ONCS_COPY = 1 << 8, NVME_CTRL_VWC_PRESENT = 1 << 0, NVME_CTRL_OACS_SEC_SUPP = 1 << 0, NVME_CTRL_OACS_NS_MNGT_SUPP = 1 << 3, @@ -414,7 +415,10 @@ struct nvme_id_ns { __le16 npdg; __le16 npda; __le16 nows; - __u8 rsvd74[18]; + __le16 mssrl; + __le32 mcl; + __u8 msrc; + __u8 rsvd91[11]; __le32 anagrpid; __u8 rsvd96[3]; __u8 nsattr; @@ -796,6 +800,7 @@ enum nvme_opcode { nvme_cmd_resv_report = 0x0e, nvme_cmd_resv_acquire = 0x11, nvme_cmd_resv_release = 0x15, + nvme_cmd_copy = 0x19, nvme_cmd_zone_mgmt_send = 0x79, nvme_cmd_zone_mgmt_recv = 0x7a, nvme_cmd_zone_append = 0x7d, @@ -819,7 +824,8 @@ enum nvme_opcode { nvme_opcode_name(nvme_cmd_resv_release), \ nvme_opcode_name(nvme_cmd_zone_mgmt_send), \ nvme_opcode_name(nvme_cmd_zone_mgmt_recv), \ - nvme_opcode_name(nvme_cmd_zone_append)) + nvme_opcode_name(nvme_cmd_zone_append), \ + nvme_opcode_name(nvme_cmd_copy)) @@ -996,6 +1002,36 @@ struct nvme_dsm_range { __le64 slba; }; +struct nvme_copy_command { + __u8 opcode; + __u8 flags; + __u16 command_id; + __le32 nsid; + __u64 rsvd2; + __le64 metadata; + union nvme_data_ptr dptr; + __le64 sdlba; + __u8 nr_range; + __u8 rsvd12; + __le16 control; + __le16 rsvd13; + __le16 dspec; + __le32 ilbrt; + __le16 lbat; + __le16 lbatm; +}; + +struct nvme_copy_range { + __le64 rsvd0; + __le64 slba; + __le16 nlb; + __le16 rsvd18; + __le32 rsvd20; + __le32 eilbrt; + __le16 elbat; + __le16 elbatm; +}; + struct nvme_write_zeroes_cmd { __u8 opcode; __u8 flags; @@ -1757,6 +1793,7 @@ struct nvme_command { struct nvme_download_firmware dlfw; struct nvme_format_cmd format; struct nvme_dsm_cmd dsm; + struct nvme_copy_command copy; struct nvme_write_zeroes_cmd write_zeroes; struct nvme_zone_mgmt_send_cmd zms; struct nvme_zone_mgmt_recv_cmd zmr; From patchwork Mon Mar 27 08:40:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13192080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D935C74A5B for ; Wed, 29 Mar 2023 08:17:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230361AbjC2IR0 (ORCPT ); Wed, 29 Mar 2023 04:17:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230364AbjC2IQ2 (ORCPT ); Wed, 29 Mar 2023 04:16:28 -0400 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EF504239 for ; Wed, 29 Mar 2023 01:16:25 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20230329081623epoutp04650b9f05363efd676ffb74b265471e6f~Q1dexJqif1998619986epoutp04- for ; Wed, 29 Mar 2023 08:16:23 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20230329081623epoutp04650b9f05363efd676ffb74b265471e6f~Q1dexJqif1998619986epoutp04- DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1680077783; bh=k1bVpVyxbnP2n/Pv5/agKnJF+GMbukvS2GpzzjdrKh4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fHhgvVeuNZKKRO8Gjt7E7ov+bHqI5idsxMuZeYzKyJKDqIoRVtXEhn1vXJbNjKP1E X8Jw6xDPA2ZbbGTW0raB9X4EJCzRH1rKhGBJQAYrXp4gWqfuEOPeNBE3QronRTT1Lm 1PPH/I+9Cfs4LfgWlfA3zT3lsNFYm/acGar0FhBM= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230329081622epcas5p1a7f50fa27f85dcd3a7b76448190e44b1~Q1deGUzbP2757827578epcas5p1r; Wed, 29 Mar 2023 08:16:22 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.175]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4PmfX460PJz4x9QB; Wed, 29 Mar 2023 08:16:20 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 53.D5.55678.4D3F3246; Wed, 29 Mar 2023 17:16:20 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230327084303epcas5p22fdd3af683d3eb1b3f503bcf045f578a~QOiMcX0NT3264932649epcas5p26; Mon, 27 Mar 2023 08:43:03 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230327084303epsmtrp2de2d978cc2ef367704d325a24699efe9~QOiMbD6sH2690426904epsmtrp2C; Mon, 27 Mar 2023 08:43:03 +0000 (GMT) X-AuditID: b6c32a4a-909fc7000000d97e-59-6423f3d4c087 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 30.D5.31821.71751246; Mon, 27 Mar 2023 17:43:03 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230327084259epsmtip13b16a43ea53bf9752786ce31bd4566df~QOiI4hjQ93003630036epsmtip1L; Mon, 27 Mar 2023 08:42:59 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v8 6/9] nvmet: add copy command support for bdev and file ns Date: Mon, 27 Mar 2023 14:10:54 +0530 Message-Id: <20230327084103.21601-7-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230327084103.21601-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xbZRjGPef0BrNyKCN8oNvwwMLFAGXcPgjsEgk5MlCMzMRJxDN6BEJp u150Q5MVGjAjQy5jcxS2DtgwFAbKbeU2ubgBBUZMYUrZBkNQGXIpiDc2teWA7r/f877Pm/fy 5eNhgmKuGy9doqTlEkpMcOxZbf0+3n7j6x4i4a+j3rDReAeDOUVPMVj3oJADF/vXEHhx9U8M bo6OYbB7uZwNJ3vaUdhVVYLC2rrbKOystKDw9j9LHFjSdw+B8xNaFHabX4Fd3UMsaOqo4EBd zTwX9p3XoNAwl43Atk0dBhsWV1hw0PwiHHs6wD4MSNP4UVI7Pcoh27UPuOTYw69YpGlURTbp z3LI5mtnyM5JNYcs0CxbDbnTbHLl1gSH/KxFj5DNwx+T6017yaa5JTTB4XhGZBpNiWi5Oy1J kYrSJalRxNG3kl9NDgkVBvoFhsMwwl1CZdJRRHRcgl9Muth6B8L9Q0qssoYSKIWCCDgYKZeq lLR7mlShjCJomUgsC5b5K6hMhUqS6i+hlRGBQuGBEKvx/Yy0ypourqzu8Kn+KSc18jAkH7Hj ATwYmLNbuPmIPU+AdyLgeoMZZcQaAupbqzFGrFuFrgDbKeno1bOZRAcChmcsHEZoUHBpZAWx uTi4F/jmp1zEltiN52HAMn+WZRMYPoMCXc61LZcTHgfKZnO2mIXvB2r9EtvGfDwclNQ0Wvvx rP0CQOG0oy1sh0eACoNx2+IIhsrmWDbG8H1A01q+NSvAv7ADlxcuspnaaHBd48CM7QQeD9g2 tbEbWF/u5jCcCv4wzaMMy4Dmzi2E4UMg11i4NQKG+4DGjgAmvAdcMDagTNsXQMHm3HYpHxiu 7DABPq2t2GYAuu+qt5kERRsWFnOsAgS0azTcIsRd+8w62mfW0f7f+iqC6RFXWqbITKUVIbID Evqj/145RZrZhGz9Dd9YA/JoZtW/D0F5SB8CeBixm795jxAJ+CLqdBYtlybLVWJa0YeEWM9d jLk5p0itn0uiTA4MDhcGh4aGBocHhQYSLnyvqKEUAZ5KKekMmpbR8p06lGfnpkar9rnEN+gW JqJvHJ+ajBip3lWq9UqnltY9B74Pey09H6vg9bjE1p9M8rZPahG81KsZyZqr5dIt44bT8b15 iQOJcb6l8Zd+dBj3wXU8y5mi12OGEwYb8fsHK94eXXV1LU86Ye/6uX/bd19PfftD1q7ZvLJK 4zu1T24cOfa7h6fzz7GC5//2TXx3dvmJeMH8S9R753qF3vcLJ2qIKg+1cxD9pZFck/GXij0x 0+XI3yxCUPJXT/XsYO6jnuas59jkm9l7WvPhoWkqqH6xNuykUWLKV+nMsn41ctUD3XB8/Eb7 J85kZuWmeP9N/FjPzZh61fmX9YZK9G7pqam95zaOmK+c+KCUYCnSqEBfTK6g/gX3wOwgpAQA AA== X-Brightmail-Tracker: H4sIAAAAAAAAA02Ra2xLYRjH855zenY2K6etxLuRsTKio9tc4nWJjBAvI5lLSEZG6bHRtdva zX3UFmSjttQtamwTKT1ziY2lW9eh21gnM3YRSldkLLPsggjFsHaR+PY8/98vz//Dw5DiBiqU 2aHJ4LQaRYqUDqIqaqVh08dsCFdG/2oQoVuND0mUXTBIolJ3Po16aj8DdHbAS6KfTc0ksvdd EKCX9ysJVH3ZSCBLaT2BbCWfCFT/p5dGRsdzgN63mwhkd0WiaruTQq1VhTQqMr8PQI5TOQSy dh4GqOJnEYlu9vRTqME1FjUPPhLEQtzaFodNniYaV5rcAbi54zaFW5sycRmfS+PyK4ew7aWe xoacviHhiEeA+2vaaXzyDg9w+eP9+EtZGC7r7CXiRyUELVByKTt2cdqohVuCkkvM1QFppbF7 al9J9KBjdh4IZCA7C1Y94AV5IIgRs1YAa7xXyWEAYWO3GQzPEmj53RUwLB0m4Jl7xZQP0OwU WNd1BPjAaLaAhC0ePe1bSLaPgPxbs8BnSdiV8Py7bP8pio2Aer7XnwvZudBovjVUxwxVRMF8 j8gXB7LzYKG10a+Ih5T8Y1XEsC6CzvOd/mKSHQ9z7l4gCwBr+g+Z/kPFgOBBCJemUyepdTFp MzTcbrlOodZlapLk21LVZcD/bJnMCqr5AbkDEAxwAMiQ0tHC8rhwpVioVOzdx2lTN2szUzid A4xlKOkY4dM852Yxm6TI4FQcl8Zp/1GCCQzVE7uWR/6pnBayXyv52mrIwSNPKw6xVGyLdfFr Ecy6u6CTFj6JSj9e4vaqZsYcSDz3kRlR0lVY7vWEVCwKv/TN8C7qx/fQsMj+e/xiiZdMCH42 n58/cs5lG7cysd04MZdN335jnMpyxhpR9My0KdyiHi8iE1VH27atlikrR8T9uFNnsaUuWlrT tiQ3fVl81pToA+DFpFylfVBGGw3pc+cpJ8mDnQ9VLpv7keHNsaqENY6ai3jv09hXn6+FCbfW ygfM/fXtdW+7TSJ3W4+JfnwiuWPjutsrnJMHvVm7C6+IpA8+rN953YaDEyZEX5vzdcWETZqD vzLsjrUtrlXd16fGu7KllC5ZESMjtTrFX/CwBMJbAwAA X-CMS-MailID: 20230327084303epcas5p22fdd3af683d3eb1b3f503bcf045f578a X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230327084303epcas5p22fdd3af683d3eb1b3f503bcf045f578a References: <20230327084103.21601-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Nitesh Shetty Add support for handling target command on target. For bdev-ns we call into blkdev_issue_copy, which the block layer completes by a offloaded copy request to backend bdev or by emulating the request. For file-ns we call vfs_copy_file_range to service our request. Currently target always shows copy capability by setting NVME_CTRL_ONCS_COPY in controller ONCS. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- drivers/nvme/target/admin-cmd.c | 9 +++-- drivers/nvme/target/io-cmd-bdev.c | 58 +++++++++++++++++++++++++++++++ drivers/nvme/target/io-cmd-file.c | 52 +++++++++++++++++++++++++++ drivers/nvme/target/loop.c | 6 ++++ drivers/nvme/target/nvmet.h | 1 + 5 files changed, 124 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c index 80099df37314..978786ec6a9e 100644 --- a/drivers/nvme/target/admin-cmd.c +++ b/drivers/nvme/target/admin-cmd.c @@ -433,8 +433,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) id->nn = cpu_to_le32(NVMET_MAX_NAMESPACES); id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES); id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM | - NVME_CTRL_ONCS_WRITE_ZEROES); - + NVME_CTRL_ONCS_WRITE_ZEROES | NVME_CTRL_ONCS_COPY); /* XXX: don't report vwc if the underlying device is write through */ id->vwc = NVME_CTRL_VWC_PRESENT; @@ -536,6 +535,12 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req) if (req->ns->bdev) nvmet_bdev_set_limits(req->ns->bdev, id); + else { + id->msrc = (u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16(BIO_MAX_VECS << + (PAGE_SHIFT - SECTOR_SHIFT)); + id->mcl = cpu_to_le32(le16_to_cpu(id->mssrl)); + } /* * We just provide a single LBA format that matches what the diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c index c2d6cea0236b..0af273097aa4 100644 --- a/drivers/nvme/target/io-cmd-bdev.c +++ b/drivers/nvme/target/io-cmd-bdev.c @@ -46,6 +46,19 @@ void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id) id->npda = id->npdg; /* NOWS = Namespace Optimal Write Size */ id->nows = to0based(bdev_io_opt(bdev) / bdev_logical_block_size(bdev)); + + /*Copy limits*/ + if (bdev_max_copy_sectors(bdev)) { + id->msrc = id->msrc; + id->mssrl = cpu_to_le16((bdev_max_copy_sectors(bdev) << + SECTOR_SHIFT) / bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32(id->mssrl); + } else { + id->msrc = (u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16((BIO_MAX_VECS << PAGE_SHIFT) / + bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32(id->mssrl); + } } void nvmet_bdev_ns_disable(struct nvmet_ns *ns) @@ -184,6 +197,19 @@ static void nvmet_bio_done(struct bio *bio) nvmet_req_bio_put(req, bio); } +static void nvmet_bdev_copy_end_io(void *private, int comp_len) +{ + struct nvmet_req *req = (struct nvmet_req *)private; + + if (comp_len == req->copy_len) { + req->cqe->result.u32 = cpu_to_le32(1); + nvmet_req_complete(req, errno_to_nvme_status(req, 0)); + } else { + req->cqe->result.u32 = cpu_to_le32(0); + nvmet_req_complete(req, blk_to_nvme_status(req, BLK_STS_IOERR)); + } +} + #ifdef CONFIG_BLK_DEV_INTEGRITY static int nvmet_bdev_alloc_bip(struct nvmet_req *req, struct bio *bio, struct sg_mapping_iter *miter) @@ -450,6 +476,34 @@ static void nvmet_bdev_execute_write_zeroes(struct nvmet_req *req) } } +/* At present we handle only one range entry */ +static void nvmet_bdev_execute_copy(struct nvmet_req *req) +{ + struct nvme_copy_range range; + struct nvme_command *cmnd = req->cmd; + int ret; + + + ret = nvmet_copy_from_sgl(req, 0, &range, sizeof(range)); + if (ret) + goto out; + + ret = blkdev_issue_copy(req->ns->bdev, + le64_to_cpu(cmnd->copy.sdlba) << req->ns->blksize_shift, + req->ns->bdev, + le64_to_cpu(range.slba) << req->ns->blksize_shift, + (le16_to_cpu(range.nlb) + 1) << req->ns->blksize_shift, + nvmet_bdev_copy_end_io, (void *)req, GFP_KERNEL); + if (ret) { + req->cqe->result.u32 = cpu_to_le32(0); + nvmet_req_complete(req, blk_to_nvme_status(req, BLK_STS_IOERR)); + } + + return; +out: + nvmet_req_complete(req, errno_to_nvme_status(req, ret)); +} + u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) { switch (req->cmd->common.opcode) { @@ -468,6 +522,10 @@ u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_bdev_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_bdev_execute_copy; + return 0; + default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c index 2d068439b129..69f198ecec77 100644 --- a/drivers/nvme/target/io-cmd-file.c +++ b/drivers/nvme/target/io-cmd-file.c @@ -322,6 +322,49 @@ static void nvmet_file_dsm_work(struct work_struct *w) } } +static void nvmet_file_copy_work(struct work_struct *w) +{ + struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); + int nr_range; + loff_t pos; + struct nvme_command *cmnd = req->cmd; + int ret = 0, len = 0, src, id; + + nr_range = cmnd->copy.nr_range + 1; + pos = le64_to_cpu(req->cmd->copy.sdlba) << req->ns->blksize_shift; + if (unlikely(pos + req->transfer_len > req->ns->size)) { + nvmet_req_complete(req, errno_to_nvme_status(req, -ENOSPC)); + return; + } + + for (id = 0 ; id < nr_range; id++) { + struct nvme_copy_range range; + + ret = nvmet_copy_from_sgl(req, id * sizeof(range), &range, + sizeof(range)); + if (ret) + goto out; + + len = (le16_to_cpu(range.nlb) + 1) << (req->ns->blksize_shift); + src = (le64_to_cpu(range.slba) << (req->ns->blksize_shift)); + ret = vfs_copy_file_range(req->ns->file, src, req->ns->file, + pos, len, 0); +out: + if (ret != len) { + pos += ret; + req->cqe->result.u32 = cpu_to_le32(id); + nvmet_req_complete(req, ret < 0 ? + errno_to_nvme_status(req, ret) : + errno_to_nvme_status(req, -EIO)); + return; + + } else + pos += len; + } + + nvmet_req_complete(req, 0); + +} static void nvmet_file_execute_dsm(struct nvmet_req *req) { if (!nvmet_check_data_len_lte(req, nvmet_dsm_len(req))) @@ -330,6 +373,12 @@ static void nvmet_file_execute_dsm(struct nvmet_req *req) queue_work(nvmet_wq, &req->f.work); } +static void nvmet_file_execute_copy(struct nvmet_req *req) +{ + INIT_WORK(&req->f.work, nvmet_file_copy_work); + queue_work(nvmet_wq, &req->f.work); +} + static void nvmet_file_write_zeroes_work(struct work_struct *w) { struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); @@ -376,6 +425,9 @@ u16 nvmet_file_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_file_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_file_execute_copy; + return 0; default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c index f2d24b2d992f..d18ed8067a15 100644 --- a/drivers/nvme/target/loop.c +++ b/drivers/nvme/target/loop.c @@ -146,6 +146,12 @@ static blk_status_t nvme_loop_queue_rq(struct blk_mq_hw_ctx *hctx, return ret; nvme_start_request(req); + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_set_request_complete(req); + blk_mq_end_request(req, BLK_STS_OK); + return BLK_STS_OK; + } iod->cmd.common.flags |= NVME_CMD_SGL_METABUF; iod->req.port = queue->ctrl->port; if (!nvmet_req_init(&iod->req, &queue->nvme_cq, diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h index 89bedfcd974c..69ed4c8469e5 100644 --- a/drivers/nvme/target/nvmet.h +++ b/drivers/nvme/target/nvmet.h @@ -393,6 +393,7 @@ struct nvmet_req { struct device *p2p_client; u16 error_loc; u64 error_slba; + size_t copy_len; }; #define NVMET_MAX_MPOOL_BVEC 16 From patchwork Mon Mar 27 08:40:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13192081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEAC8C6FD18 for ; Wed, 29 Mar 2023 08:17:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230383AbjC2IRn (ORCPT ); Wed, 29 Mar 2023 04:17:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230467AbjC2IQi (ORCPT ); Wed, 29 Mar 2023 04:16:38 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E21604C34 for ; Wed, 29 Mar 2023 01:16:28 -0700 (PDT) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230329081627epoutp0141f2d96b12788db4ca99990c4fa291a9~Q1dic8koN1329513295epoutp012 for ; Wed, 29 Mar 2023 08:16:27 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230329081627epoutp0141f2d96b12788db4ca99990c4fa291a9~Q1dic8koN1329513295epoutp012 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1680077787; bh=yoRFN8EbW7IWQW15oore9Bn4KQVBbxBNv9lbpiBp+v0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=r011OhMgEwdVDiZn6siDPoHdLoEMOexkyQi9hW1UYtqpmw9+qTAfkc7efsMmG5IxY Q9/ax7kEVW0abCSRQU9ZorR3bSZG35pZhG+/vT3afdFtuYNSrqr6aQkaIdeckjeTpc 3ucFBmNFgOOdjsVQpZeR8GROzgfX+rcAahrq9atc= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230329081626epcas5p3b03079a343e247ea82e50a52a134188d~Q1dhxTPTL0829808298epcas5p3R; Wed, 29 Mar 2023 08:16:26 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.182]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4PmfX902Rlz4x9Q1; Wed, 29 Mar 2023 08:16:25 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id E6.BD.10528.8D3F3246; Wed, 29 Mar 2023 17:16:24 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230327084312epcas5p377810b172aa6048519591518f8c308d0~QOiVHfUy03103731037epcas5p3W; Mon, 27 Mar 2023 08:43:12 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230327084312epsmtrp286ec9b83f4cc477d036bbb430103e0f9~QOiVGSVuc2690526905epsmtrp2G; Mon, 27 Mar 2023 08:43:12 +0000 (GMT) X-AuditID: b6c32a49-c17ff70000012920-d9-6423f3d896ba Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 54.D5.31821.02751246; Mon, 27 Mar 2023 17:43:12 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230327084309epsmtip1fbc0ef7c19d6306421092c5c2f1356d6~QOiR2sZpk3056530565epsmtip1U; Mon, 27 Mar 2023 08:43:09 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v8 7/9] dm: Add support for copy offload. Date: Mon, 27 Mar 2023 14:10:55 +0530 Message-Id: <20230327084103.21601-8-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230327084103.21601-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTVxzHc+5tL4WlcikMj5BpueocEko7Sj04mCbT5RIWQ8K2LBtL19Fr YZS2a4u6sWWVCgihPBQZFhy6GJJCpA4YArU+KPIyyEKFDWaHbiCbjyrURDaCrLVl87/P7/E9 v8fJj4PzqoOiOLkqPaNVyZQUEcLqcsS+Fv+LZ7NcWHFXgqwjAzgqql7BUaurikD3HYsA1T3+ G0fLo2M4srsb2GjqSg+GLn5/DEOW1msYsp1ZwNC11YcEOtY3CdDchBlD9uk4dNE+zELO3kYC NTXPBaG+40YMdc8eBqhruQlHbfcfsdDQdDQaWxlk74a082Y6bZ4ZJegesyuIHvvtBxbtHC2g 21vKCLrj7De0bcpA0Caj25tQPMOmH12aIOjKzhZAd1wvpD3tG+n22YdYRuiHeSk5jEzOaPmM Klstz1UpUqn0TOlb0iSJUBQvSkY7KL5Kls+kUnveyYh/O1fp3QPFPyBTFnhdGTKdjkp4M0Wr LtAz/By1Tp9KMRq5UiPWCHSyfF2BSiFQMfqdIqHw9SRv4id5OQMuN0tTEX3IPtTPNoDDkeUg mANJMbw1VkuUgxAOj7QB+I+lA/MbiwD+OmwNGB4Ae5d+xtYkFybduI95ZC+AR64f9CcZMdha de55gCC3wf75YuALRJAlOFyYK2P5DJy8jcGmorPeCIcTTiZDuwn6BCxyK/TccbJ9bq7XPTV3 wIeQTIBVM2G+jGByJ2zsHmH7mEuGweGTsywf4+QmaPyxAfe9DklrMCyyN7P9je6BdaXfBjgc 3hvsDPJzFPS47YSfFXDJORcYTAONA5eAn3fB4pEq3NcDTsZCa2+C3/0KPDHShvnrroOm5dmA lAu7v1tjCpZaGgMMof2GIcA0HHA2Av+uTACeNp4PqgZ88wvzmF+Yx/x/6dMAbwEbGI0uX8Ho kjQiFXPwv0/OVue3g+ensT2tG7huPxb0AYwD+gDk4FQEd3mSkvO4ctkXXzJatVRboGR0fSDJ u+4aPOrlbLX3tlR6qUicLBRLJBJxcqJERK3nbksdzuaRCpmeyWMYDaNd02Gc4CgDVuM48rQr 9KULlqAua0zoUoFNceavxVff2/Lxicrxhn3szLi2fYfE0TT47N7+slHNncuXI0ueGAWdd231 5nW3JIlotW5PWqjHNchyKbPGS6317+7FpeNLhbGxJoeg+OiE0HtlfE1NxAd7nUMOs7WXG1kd lTxWOUUR6+ul3Frr7g3PylLSe6xvxK+qPj0Vt3gj6w93egf6vbhiV3neSuWOkukaQc65+Sch o+/3yzdlNj6QmmLmuWHSsrbNV58aPiLiHmRt+em8FLmrlXpeOFVrsSj52rnPp0T2ka0n/0y8 6TjarCq0mDJjNhL144YrPV9Ln+Wdmqa1X1217a9dqBenUSxdjky0HdfqZP8CP7GRxaMEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrOIsWRmVeSWpSXmKPExsWy7bCSnK5CuGKKwZ0GTYv1p44xWzRN+Mts sfpuP5vF68OfGC2mffjJbPH77Hlmi73vZrNa3Dywk8liz6JJTBYrVx9lsti98COTxdH/b9ks Jh26xmjx9OosJou9t7Qt9uw9yWJxedccNov5y56yWxya3MxkseNJI6PFtt/zmS3WvX7PYnHi lrTF+b/HWR0kPC5f8faYdf8sm8fOWXfZPc7f28jicflsqcemVZ1sHpuX1HvsvtnA5tHb/A6o oPU+q8f7fVfZPPq2rGL02Hy62uPzJjmPTU/eMgXwR3HZpKTmZJalFunbJXBlHLv7jqWgR7pi 74kjrA2MjWJdjJwcEgImEtuvvWPuYuTiEBLYwSixtPkfM0RCQuLUy2WMELawxMp/z9khihqZ JB4fnscEkmATUJc48ryVESQhIjCBWeLS/QY2EIdZ4B2TxKqHy1i7GDk4hAUsJfb2SoA0sAio Snx+eBkszAsUvvm0DMSUENCX6L8vCFLBKWAlMWfHKVYQWwioor99F9gqXgFBiZMzn7CA2MwC 8hLNW2czT2AUmIUkNQtJagEj0ypGydSC4tz03GLDAqO81HK94sTc4tK8dL3k/NxNjOC41tLa wbhn1Qe9Q4xMHIyHGCU4mJVEeDd7K6YI8aYkVlalFuXHF5XmpBYfYpTmYFES573QdTJeSCA9 sSQ1OzW1ILUIJsvEwSnVwOTB8CBgvtV1R3mz7ZfWLuRps7rF3fvx8B3R078i5h/dtuBj/ul5 jxQWvq6XiPVYq+AjphmT+YQv7N7rvncH6zV5HuU+m3C86MnJ7XYBJVk3BHw7p3+avkZZbqLH lK079+VXfBEKilXmuzQh5qhafK6Pi0iPR4v6rga74+EB3BW7VlRxvv2st1lDSdozWIjxzb3m 59aZx1rUhN5rNSw3qbvBxMFnsTSmuShtjcLdNa6+y1cdfRG77bbl8YWP9qsUTX5qW/Rk8tdp 1255GjkUen256rqD6QX3ZK6i2fM/FicLs16qLLyQqCciJxf77PUVkbWJm2LyXGW5N+fesour nehQbXJinsq0VvuVZh+XPFViKc5INNRiLipOBADBCLKqWgMAAA== X-CMS-MailID: 20230327084312epcas5p377810b172aa6048519591518f8c308d0 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230327084312epcas5p377810b172aa6048519591518f8c308d0 References: <20230327084103.21601-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Nitesh Shetty Before enabling copy for dm target, check if underlying devices and dm target support copy. Avoid split happening inside dm target. Fail early if the request needs split, currently splitting copy request is not supported. Signed-off-by: Nitesh Shetty --- drivers/md/dm-table.c | 42 +++++++++++++++++++++++++++++++++++ drivers/md/dm.c | 7 ++++++ include/linux/device-mapper.h | 5 +++++ 3 files changed, 54 insertions(+) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 7899f5fb4c13..45e894b9e3be 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1863,6 +1863,39 @@ static bool dm_table_supports_nowait(struct dm_table *t) return true; } +static int device_not_copy_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + struct request_queue *q = bdev_get_queue(dev->bdev); + + return !blk_queue_copy(q); +} + +static bool dm_table_supports_copy(struct dm_table *t) +{ + struct dm_target *ti; + unsigned int i; + + for (i = 0; i < t->num_targets; i++) { + ti = dm_table_get_target(t, i); + + if (!ti->copy_offload_supported) + return false; + + /* + * target provides copy support (as implied by setting + * 'copy_offload_supported') + * and it relies on _all_ data devices having copy support. + */ + if (!ti->type->iterate_devices || + ti->type->iterate_devices(ti, + device_not_copy_capable, NULL)) + return false; + } + + return true; +} + static int device_not_discard_capable(struct dm_target *ti, struct dm_dev *dev, sector_t start, sector_t len, void *data) { @@ -1945,6 +1978,15 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, q->limits.discard_misaligned = 0; } + if (!dm_table_supports_copy(t)) { + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + /* Must also clear copy limits... */ + q->limits.max_copy_sectors = 0; + q->limits.max_copy_sectors_hw = 0; + } else { + blk_queue_flag_set(QUEUE_FLAG_COPY, q); + } + if (!dm_table_supports_secure_erase(t)) q->limits.max_secure_erase_sectors = 0; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 2d0f934ba6e6..08ec51000af8 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1693,6 +1693,13 @@ static blk_status_t __split_and_process_bio(struct clone_info *ci) if (unlikely(ci->is_abnormal_io)) return __process_abnormal_io(ci, ti); + if ((unlikely(op_is_copy(ci->bio->bi_opf)) && + max_io_len(ti, ci->sector) < ci->sector_count)) { + DMERR("Error, IO size(%u) > max target size(%llu)\n", + ci->sector_count, max_io_len(ti, ci->sector)); + return BLK_STS_IOERR; + } + /* * Only support bio polling for normal IO, and the target io is * exactly inside the dm_io instance (verified in dm_poll_dm_io) diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index 7975483816e4..44969a20295e 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -380,6 +380,11 @@ struct dm_target { * bio_set_dev(). NOTE: ideally a target should _not_ need this. */ bool needs_bio_set_dev:1; + + /* + * copy offload is supported + */ + bool copy_offload_supported:1; }; void *dm_per_bio_data(struct bio *bio, size_t data_size); From patchwork Mon Mar 27 08:40:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13192082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6A2BC6FD18 for ; Wed, 29 Mar 2023 08:17:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230514AbjC2IRt (ORCPT ); Wed, 29 Mar 2023 04:17:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230396AbjC2IQz (ORCPT ); Wed, 29 Mar 2023 04:16:55 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E75A04C29 for ; Wed, 29 Mar 2023 01:16:31 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230329081630epoutp0117deaa98432fc1efa1cdb841f31aaa43~Q1dlXEZWk1285012850epoutp01Q for ; Wed, 29 Mar 2023 08:16:30 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230329081630epoutp0117deaa98432fc1efa1cdb841f31aaa43~Q1dlXEZWk1285012850epoutp01Q DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1680077790; bh=Xr5pG5YQRmNwtteFma8PSAfaeXx0TOavDZPZ9ALIkAQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RUmu20NuzDnC+HGx33YnJktvWo/SM3cRedufcqsmKYUEXDQCN7EVDMV9PiTLeKYSp VveA0KvJWCb25V1qaeVnvqG7N/AAinKWQw5/uH6H+bhTE/kCvKSuoHXKjbVXy5fuNr ChlUOIkdQ+UsMcnUiVbmTMkH+stddf7pTBErgEV8= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230329081629epcas5p29eacb83ce69432a0ad642a245c809ee4~Q1dkxESEl1635016350epcas5p2t; Wed, 29 Mar 2023 08:16:29 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.181]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4PmfXD1pMqz4x9Pp; Wed, 29 Mar 2023 08:16:28 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 2F.D5.55678.CD3F3246; Wed, 29 Mar 2023 17:16:28 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230327084322epcas5p12f01e676e47d3c8ba880f3f5d58999b4~QOid0ghRO3074330743epcas5p1R; Mon, 27 Mar 2023 08:43:22 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230327084322epsmtrp14a16237c5338352e007443853faae221~QOidzab1c3087630876epsmtrp1O; Mon, 27 Mar 2023 08:43:22 +0000 (GMT) X-AuditID: b6c32a4a-909fc7000000d97e-79-6423f3dc0d61 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id C4.4C.18071.92751246; Mon, 27 Mar 2023 17:43:22 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230327084318epsmtip1e924ae0d72b9b7a598d9de0e0ac183ae~QOiakAl-c3003630036epsmtip1N; Mon, 27 Mar 2023 08:43:18 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v8 8/9] dm: Enable copy offload for dm-linear target Date: Mon, 27 Mar 2023 14:10:56 +0530 Message-Id: <20230327084103.21601-9-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230327084103.21601-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Ta0xbZRjHfc9pT8uweLiYvesca85QBAVabOFljKkZM2cbM3iJH9SBDT2B BmhLL264MJEONggVZJFpQS6KRugyXMe4toQxGQKBjnBZysZtlk1ABhveCcOWFt233/N/nn+e y5uXi/uVcvhcuULLqBXSDIrYxmq+FhIcdnt1j0xoKfJGjf3XcZRXuo4j02QJgRavPQSofOVv HK0N2nBkvV/BRvauNgxZvi7DUL2pB0MdtQ8w1LOxRKCy7nGA5saMGLJOvIAs1j4WGmmvJFD1 d3Mc1H1Oj6FWxycANa9V4+ji4jIL/TSxE9nWe9mvQHpk9AhtnB4k6DbjJIe2TV1i0SODOtrc UEjQl+s+pjvsuQRt0N93FuRPs+nlzjGC/rSpAdCXB07Sq+ZA2uxYwhKfejd9XxojlTFqAaNI UcrkitQ46shbyQeSJVFCUZgoBkVTAoU0k4mj4hMSw16TZzjvQAk+lGbonFKiVKOhIvbvUyt1 WkaQptRo4yhGJctQiVXhGmmmRqdIDVcw2r0ioTBS4iz8ID0t1z5AqH5gn7i7scrOBV2sIuDF haQYnq+/hReBbVw/sgPATrsFcwcPAbRZhgh38AeA9QY7sWWx9q1y3AkrgNbmBeBK+JF6DE6O 8l1MkMHwx3v5wFUUQBbg8MFcIcsV4OQMBqvz6jYd/mQ87DPUYC5mkc9CU3mLU+dweWQMrFEU Aa6zWQQsmfZ1FXiRe2Flaz/bxTzSF/Z96dhcASd3Q/2Vis0VIGnygu2dXRz3oPFw6c6wZ2h/ uNDb5NH5cL6kwMOp8K+ROczNKqi/3gnc/DLM7y/BXTPgZAhsbI9wy7vg5/0XMXdfH2hYc3is PNhatcUUPFNf6WEIrUO5HqbhvZ5lz6kNzuv+focoBQLjY/sYH9vH+H/rGoA3gB2MSpOZymgk qkgFc/y/R05RZprB5tcIPdwKZmdWwrsBxgXdAHJxKoC3Nk7J/HgyafZHjFqZrNZlMJpuIHFe +zOc/3SK0vm3FNpkkThGKI6KihLHvBQlorbzguP6UvzIVKmWSWcYFaPe8mFcL34uVqrzbju6 Wm6pOsM6uHBMeywlZ/aXcPuTsT4jOq9hzqFvbxb8s8N/PYcsaLka32l+u6dWaHguizkRG3ng ktZcm1N1oeK906ySR/B4Ul6gXo4/keu4m7T4RWjSm2L72KNfT8dNZ/k2JVC20v0FU5KAd1Rn DytuFbeMB8nBhiDrmarW+Whlw7n3K4dnp9jz2ljiSn3xSu/VhubdpqYuZbf1+ehD/SEHT/qs h8qGlm2OIPafr18Iq5q48bOJVJcbkkzBjZIXEwLPZ8Z775Tv2c4/RRTSFZNv/Fa78lXdwO3i /KxRfZBp19m47Fdnyk7Jh28mH2V9k11d3nqjSxezEiD+XhTVmEexNGlSUSiu1kj/BXih46aj BAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02Re0hTYRjG/c45OzsuVscp9JVkNMrKanNd4OtedOGUUCPoKmWrnSzTaZuW WayjdqGRzQykpunsYjnD1sy7ps4spy0jU5hdhqJZmJtlRiWzmhX03+/leZ73eeGlcNFjYjJ1 SBXPqlWKaDEpIEobxEHzQrZPU4ZWpWDoXvNjHKWke3BU+EZPov6GzwBlDn7H0Yi9FUc1riwe ctRVYKj6egaGCgobMVSV9wlDjT8HSJRh7QCot92AoZrOOai6xkagtspsEuXm9/KR9XIqhsp7 kgEqHcnFUVG/m0BNnYGo1fOEtwoybS/DGIPTTjIVhjd8pvXtfYJpsycwFtN5kim+eYqpcnAk k5bq+m044+Qx7oftJHPxgQkwxS0nmCFLEGPpGcDkE3YJlinZ6ENHWbV0xV7BQc7RQsaZeYnv fg7xOFBH6IAvBemFsMY2xNcBASWiqwC8OlgG/ggQNn/I/8v+sGC0j+9lEZ2MwdLOMSbpmfBR 3xngDQfQ6Th84eRI74DTLgyauvJ5Xpc/vRba0oyYlwl6BizM9DbwKSG9GBpVOkD93i+Feqef 1+BLL4HZ5c28P1WLof5c5VhQSPtB29WesZtxeipMLcnC0wFt+E8y/CcZAWYCk9g4TUxkjEYW J1OxxyQaRYwmQRUp2R8bYwFjrw6ZXQ7KTIMSK8AoYAWQwsUBwuKwaUqRUKk4nsSqYyPUCdGs xgoCKUI8UfhcZ4sQ0ZGKePYwy8ax6n8qRvlO5rDwJZuad4yETrG3b1sqT0zRxXNrTE7zwHBw Cy/xyxOLNsJUFxhq3DEasF3l5mJ3n1zqMd96tdrsDHfIfc6iJkdtXvzw1g5DwfiF9d9G0etF TfyLG80PTcsEXXvu6lMaRleFpXFZ8tUr7+Tv8QQv8sOeVa7s6rib9DTp/L4cqf1933Ltlymy 5OgTXffXefr12pe1Ubk/sg+IG3Wnr8y/pfQpn7W594LWsv5a9biGxJPpStfbGz/c/ve+HRmv zfnUsvVjTma4qJtlurmPM6IqNmwxlmxbIMheQWbJpEU+EyQlRvexw/LMeqck6tLAhjs76eTb 8unPvq7rFL7fEiQtut7tmismNAcVshBcrVH8AmNSxplZAwAA X-CMS-MailID: 20230327084322epcas5p12f01e676e47d3c8ba880f3f5d58999b4 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230327084322epcas5p12f01e676e47d3c8ba880f3f5d58999b4 References: <20230327084103.21601-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Nitesh Shetty Setting copy_offload_supported flag to enable offload. Signed-off-by: Nitesh Shetty --- drivers/md/dm-linear.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c index 3e622dcc9dbd..785a6822d06c 100644 --- a/drivers/md/dm-linear.c +++ b/drivers/md/dm-linear.c @@ -62,6 +62,7 @@ static int linear_ctr(struct dm_target *ti, unsigned int argc, char **argv) ti->num_discard_bios = 1; ti->num_secure_erase_bios = 1; ti->num_write_zeroes_bios = 1; + ti->copy_offload_supported = 1; ti->private = lc; return 0; From patchwork Mon Mar 27 08:40:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13192083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C696C761A6 for ; Wed, 29 Mar 2023 08:18:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230522AbjC2ISA (ORCPT ); Wed, 29 Mar 2023 04:18:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230413AbjC2IQ7 (ORCPT ); Wed, 29 Mar 2023 04:16:59 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 883084ED4 for ; Wed, 29 Mar 2023 01:16:36 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230329081634epoutp01128babd425079dd8f2c2445c08590272~Q1dpZou8h1483014830epoutp01D for ; Wed, 29 Mar 2023 08:16:34 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230329081634epoutp01128babd425079dd8f2c2445c08590272~Q1dpZou8h1483014830epoutp01D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1680077794; bh=lm0WP6O4CL5/qIdjNGTZX/T8lPRCo7+74BXwP7Ula+E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=apw/ZAAvoyBAzNqib27S9mMdwX1Dtba9K5LYX2p7RI5fFUVB0Lg1jkJ525/0FnSMn LSatC9SgJoTfkkewhjj3vAXF5z8sRhbXnjkhXFHq3MT6yQLms9qVXUtjIr/tAAM8Jo M5p4/VmXWv88oW6BR9vMqsLmDefQ8HgwEPKtM3Xs= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230329081634epcas5p384a60fc6e935d5677d4fd6fdae0febc5~Q1do1vqoW1882518825epcas5p3E; Wed, 29 Mar 2023 08:16:34 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.174]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4PmfXJ2dzfz4x9Q9; Wed, 29 Mar 2023 08:16:32 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id C1.20.06765.0E3F3246; Wed, 29 Mar 2023 17:16:32 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230327084331epcas5p2510ed79d04fe3432c2ec84ce528745c6~QOimo1dPU0968909689epcas5p21; Mon, 27 Mar 2023 08:43:31 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230327084331epsmtrp2fb8c9870f1fe27704e0328bf8f28bf73~QOimnxreA2690426904epsmtrp2Q; Mon, 27 Mar 2023 08:43:31 +0000 (GMT) X-AuditID: b6c32a4b-46dfa70000011a6d-23-6423f3e00c8b Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id A9.4C.18071.33751246; Mon, 27 Mar 2023 17:43:31 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230327084327epsmtip160db9f1d1db73f5d72d3f78f49ac1595~QOijROTfK3005130051epsmtip15; Mon, 27 Mar 2023 08:43:27 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Vincent Fu , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v8 9/9] null_blk: add support for copy offload Date: Mon, 27 Mar 2023 14:10:57 +0530 Message-Id: <20230327084103.21601-10-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230327084103.21601-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTdxTH87u3vRRI3eVh9pNlrrkbYTh5KZQfaHUPt90EXRqXSOJmSkdv CoM+1tuObbJZxlAeARSQuIIylZGAmUBhrDxqWBFYOx6aCgiZwrSIHfIQzESYspbC5n+fc873 5Jzz/eXHw/3PeAXxUpVaRqOUplOED6elKzQ0bGLxVVnk0qIvqrf14Ojbk09xdOlWMYGmuxYA Kp9/gqOV/kEcmWcruGi0sxVDHRdKMFR7qRtD7ecfYqh7dYZAJZZhgCaHDBgyj72BOsxWDrK3 VRKoqmbSC1lKszFkcmQB1LJShaPL03Mc9NvYS+huQS5Ag097uW9uoe03EmjDeD9BtxpuedGD txs5tL1fRxvr8gi6qfoY3T6qJ+jC7FmXIGecS89dGSLoouY6QDf9fpReNG6ljY4ZTPzC4bTd KYxUxmgEjDJZJUtVykVUwoeSdyQxwsiosKg4FEsJlFIFI6L27ReHvZea7jKDEnwuTde5UmIp y1IRe3ZrVDotI0hRsVoRxahl6epodTgrVbA6pTxcyWjjoyIjd8S4hElpKTd7sgh1VvAXlpZO oAc3t+YDbx4ko6EpX++VD3x4/mQ7gI+WywhPsACg80wx7gn+BvBGcy53o2X0bCvHUzAD2FLk XA+yMVjzpBe4VQQZAq9O5QB3IZA8jsOHk3lrKpzMwaGj0+TlVgWQe+C9yuK1Dg4ZDB8s5HLc zCfj4Yy92qXhueZFwOJxP3fa25WuNNm4HokftH7vWJPj5Csw++eKtV0hWe8Ns8obOZ5d98Hl Cifu4QD4V2+zl4eD4OKsmfCwHC7ZJzEPq2F2zxXg4b0wx+Y2gOcaEArr2yI86ZfhadtlzDN3 Eyxccay38qHp3AZT8ERt5TpDaB7QrzMNa0uG1g0uBPBi6SpxEggMz91jeO4ew/+jfwB4HdjC qFmFnGFj1DuVTMZ/75ysUhjB2hfZlmACdybmwy0A4wELgDycCuSvDFMyf75M+uVXjEYl0ejS GdYCYlx+n8KDNierXH9MqZVERcdFRguFwui4ncIo6kV+iMia7E/KpVomjWHUjGajD+N5B+kx MpXutB/elbHaydrbdlivNpKiB9gH1WXL796BpyXbq0JNzpkfhwt+3XV8WmG1HvLdzt178TMM tzYcva6IOzD3S9LrbWevOQcO+Y4smI0ZIwESWH9BCrKLngXmKaR9iY63BhwnQpfePhCXlsn/ mq2YbeqoKDvvVNret7ApMzF+xoa6ubGPk3jlR1qmM5uonxqmDROncq6F8CLvHxxZnkr+VCSA /lN5jzfdTqqfbPVXPy44do9rDG40xOYNqkvrz3X/08XVfhKf2KfiBXFe436XKH7EyJdtfqN4 7p99YtGR+Tbhviy8SVyz3+dZpk2XFrt6v/EjPX4w/Y+7Dd9cZzeHUhw2RRq1Ddew0n8BLZKS 8asEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02RXUhTcRjG+59zPDtKq9NM+qtlOVyR0aYk9q+p1YV1TBCjiyAJPbTDFOca W2bZhZpFaphiaDQtrURwluY0dX5kbM38QK1Mc5UOvyl11srIrMwpgXfP+/ye93kvXgoX9BAe VJzyPKdWsgoh6ULUmYRee/ed8pb5WeY8UVVnG46u5P7BUcVQDommTXaACr4s4GixuxdHLbZC J2R5bsBQ84M8DJVXmDHUdP8rhsxLsyTKMw4ANNGvxVDL+z2ouaWDQH2NRSQqLpvgIeOtdAw1 jKcBVLdYjKPK6TkCtb/3RGM3MgDq/fPS6bA70/c2nNFau0nGoB3iMb3D1QTT153I6HWZJFNT msI0WVJJJjvdthy4ZnVi5p71k8zNWh1garouM9/0Xox+fBaL3HjaJUjGKeIucGpJSIxL7GBb GqlKE1001j0HqWDQKws4U5AOgJa7BiILuFACugnAYesgbxVA2PmpDKxqV1j+d4q3GkrD4HD9 B9IBSHoXfDF1DTjAZjoXh2+sqaRjwOl8HM43zaykXOkQOFmUs1JF0CI4Y88gHJpPH4SzfaXL tdTyCQnMsW5y2M7LdlFDp5NDC+gDMOd6I7Ya3wQ77oyvrOL0dpj+tBDPBbR2DdKuQSUA0wF3 TqVJkCdo/FX+Si5JrGETNIlKufjsuQQ9WHm57+4GUK/7IjYCjAJGAClcuJlfE+4tE/Bl7KVk Tn0uWp2o4DRG4EkRwi38V1kd0QJazp7n4jlOxan/U4xy9kjFfN1c3UoOKddlB2dazPkege1s qymmIyoyv2ubfGS96d1o9IawHb9TQntCjmbunxjpnFL4iVKyg1UvfJZih47n9e+sPKHXLurm I+JLXonrJFXxrDnF0h7m7i7de+90cfXV2xHeW1WJvTLFR1FrHvXWZrvbKOVL9/+OSBaN5i6x twbehc5vYUft/d+NlqSBRw/fFMRUdr3OL7vsE1JwhCf80Rr0aPikMPSMZN5kLRz5Fbh0TDnX fHAGuXlMTk9Mnj159adVabQ9wb3k5YZZXpv08RCK66GlmF4XXiEB3+szx0oDxFFhBrvZ78T1 NkXSgn477RwVGfvZPr4QXR9VKyQ0say/L67WsP8A07UlKmEDAAA= X-CMS-MailID: 20230327084331epcas5p2510ed79d04fe3432c2ec84ce528745c6 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230327084331epcas5p2510ed79d04fe3432c2ec84ce528745c6 References: <20230327084103.21601-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org From: Nitesh Shetty Implementaion is based on existing read and write infrastructure. Suggested-by: Damien Le Moal Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu --- drivers/block/null_blk/main.c | 94 +++++++++++++++++++++++++++++++ drivers/block/null_blk/null_blk.h | 7 +++ 2 files changed, 101 insertions(+) diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index 9e6b032c8ecc..84c5fbcd67a5 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -1257,6 +1257,81 @@ static int null_transfer(struct nullb *nullb, struct page *page, return err; } +static inline int nullb_setup_copy_read(struct nullb *nullb, + struct bio *bio) +{ + struct nullb_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + + memcpy(token->subsys, "nullb", 5); + token->sector_in = bio->bi_iter.bi_sector; + token->nullb = nullb; + token->sectors = bio->bi_iter.bi_size >> SECTOR_SHIFT; + + return 0; +} + +static inline int nullb_setup_copy_write(struct nullb *nullb, + struct bio *bio, bool is_fua) +{ + struct nullb_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + sector_t sector_in, sector_out; + void *in, *out; + size_t rem, temp; + unsigned long offset_in, offset_out; + struct nullb_page *t_page_in, *t_page_out; + int ret = -EIO; + + if (unlikely(memcmp(token->subsys, "nullb", 5))) + return -EOPNOTSUPP; + if (unlikely(token->nullb != nullb)) + return -EOPNOTSUPP; + if (WARN_ON(token->sectors != bio->bi_iter.bi_size >> SECTOR_SHIFT)) + return -EOPNOTSUPP; + + sector_in = token->sector_in; + sector_out = bio->bi_iter.bi_sector; + rem = token->sectors << SECTOR_SHIFT; + + spin_lock_irq(&nullb->lock); + while (rem > 0) { + temp = min_t(size_t, nullb->dev->blocksize, rem); + offset_in = (sector_in & SECTOR_MASK) << SECTOR_SHIFT; + offset_out = (sector_out & SECTOR_MASK) << SECTOR_SHIFT; + + if (null_cache_active(nullb) && !is_fua) + null_make_cache_space(nullb, PAGE_SIZE); + + t_page_in = null_lookup_page(nullb, sector_in, false, + !null_cache_active(nullb)); + if (!t_page_in) + goto err; + t_page_out = null_insert_page(nullb, sector_out, + !null_cache_active(nullb) || is_fua); + if (!t_page_out) + goto err; + + in = kmap_local_page(t_page_in->page); + out = kmap_local_page(t_page_out->page); + + memcpy(out + offset_out, in + offset_in, temp); + kunmap_local(out); + kunmap_local(in); + __set_bit(sector_out & SECTOR_MASK, t_page_out->bitmap); + + if (is_fua) + null_free_sector(nullb, sector_out, true); + + rem -= temp; + sector_in += temp >> SECTOR_SHIFT; + sector_out += temp >> SECTOR_SHIFT; + } + + ret = 0; +err: + spin_unlock_irq(&nullb->lock); + return ret; +} + static int null_handle_rq(struct nullb_cmd *cmd) { struct request *rq = cmd->rq; @@ -1267,6 +1342,14 @@ static int null_handle_rq(struct nullb_cmd *cmd) struct req_iterator iter; struct bio_vec bvec; + if (rq->cmd_flags & REQ_COPY) { + if (op_is_write(req_op(rq))) + return nullb_setup_copy_write(nullb, rq->bio, + rq->cmd_flags & REQ_FUA); + else + return nullb_setup_copy_read(nullb, rq->bio); + } + spin_lock_irq(&nullb->lock); rq_for_each_segment(bvec, rq, iter) { len = bvec.bv_len; @@ -1294,6 +1377,14 @@ static int null_handle_bio(struct nullb_cmd *cmd) struct bio_vec bvec; struct bvec_iter iter; + if (bio->bi_opf & REQ_COPY) { + if (op_is_write(bio_op(bio))) + return nullb_setup_copy_write(nullb, bio, + bio->bi_opf & REQ_FUA); + else + return nullb_setup_copy_read(nullb, bio); + } + spin_lock_irq(&nullb->lock); bio_for_each_segment(bvec, bio, iter) { len = bvec.bv_len; @@ -2146,6 +2237,9 @@ static int null_add_dev(struct nullb_device *dev) list_add_tail(&nullb->list, &nullb_list); mutex_unlock(&lock); + blk_queue_max_copy_sectors_hw(nullb->disk->queue, 1024); + blk_queue_flag_set(QUEUE_FLAG_COPY, nullb->disk->queue); + pr_info("disk %s created\n", nullb->disk_name); return 0; diff --git a/drivers/block/null_blk/null_blk.h b/drivers/block/null_blk/null_blk.h index eb5972c50be8..94e524e7306a 100644 --- a/drivers/block/null_blk/null_blk.h +++ b/drivers/block/null_blk/null_blk.h @@ -67,6 +67,13 @@ enum { NULL_Q_MQ = 2, }; +struct nullb_copy_token { + char subsys[5]; + struct nullb *nullb; + u64 sector_in; + u64 sectors; +}; + struct nullb_device { struct nullb *nullb; struct config_item item;