From patchwork Mon Jun 5 12:17:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13267300 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FC08C7EE2F for ; Mon, 5 Jun 2023 12:29:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233424AbjFEM3k (ORCPT ); Mon, 5 Jun 2023 08:29:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232203AbjFEM3e (ORCPT ); Mon, 5 Jun 2023 08:29:34 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2E36EC for ; Mon, 5 Jun 2023 05:29:30 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230605122929epoutp03de7dbd752e705b77fa56c950c2df50a2~lwx39p9_R1131011310epoutp03C for ; Mon, 5 Jun 2023 12:29:29 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230605122929epoutp03de7dbd752e705b77fa56c950c2df50a2~lwx39p9_R1131011310epoutp03C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1685968169; bh=XtAXhvnJKkZLthC90pQwMUapDiLrIj5s2XGYljXFy3I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fQ/7mf1JbbFspT7Ngh2SeB7IRihc9hG+tJrspQllaq+gFBDuBv/iI+Y9GUal7JbNN FnwTungbrscVzhu1ft9u6jB++8PVwQtuwZFe42YQymrHXfcJwagRKnq/giVx11ojzh Gq2bBwOjw/fAD5qAakWy9TNx0dn/DFsDiobT7IUs= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230605122928epcas5p4b692346fb9b641b4c9b3e8a20b72a4be~lwx2_KwUy0962809628epcas5p4T; Mon, 5 Jun 2023 12:29:28 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.175]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4QZXwk1xfBz4x9Pt; Mon, 5 Jun 2023 12:29:26 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id F1.49.04567.625DD746; Mon, 5 Jun 2023 21:29:26 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230605122122epcas5p27d70bd972c1063ecb543ea4489870fd7~lwqy3owFb3040930409epcas5p2o; Mon, 5 Jun 2023 12:21:22 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230605122122epsmtrp1e2d7fa0ca0d16df66ef92d953df766b4~lwqy2XyPz1437514375epsmtrp1u; Mon, 5 Jun 2023 12:21:22 +0000 (GMT) X-AuditID: b6c32a49-943ff700000011d7-4a-647dd5260903 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 1A.A5.28392.243DD746; Mon, 5 Jun 2023 21:21:22 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230605122114epsmtip224958e7fecb35e637355c81ec48af8a0~lwqrYRYqT2526625266epsmtip2U; Mon, 5 Jun 2023 12:21:14 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, willy@infradead.org, hare@suse.de, djwong@kernel.org, bvanassche@acm.org, ming.lei@redhat.com, dlemoal@kernel.org, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Kanchan Joshi , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v12 1/9] block: Introduce queue limits for copy-offload support Date: Mon, 5 Jun 2023 17:47:17 +0530 Message-Id: <20230605121732.28468-2-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230605121732.28468-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TbUxTZxT2vbdcignsAjJeYUItwQUcH92gviggEbNcBBJ0Rjb9wTp6BQa0 TVtQwWEZgzkVisA2VlBk6mjBgICw8infX8oQOzCQISSjbgiCwCJfAdbSsvnvOc97nnPOc94c Jm5VaWrHjBFIabGAF8cmdjJq211c3fYNfcX3zM81QxV9XTj6OnsdR2VjcgJNty8A9MPrFRxN tnwLkGbSAk08PIyaZgtM0EhLHYYaf87BkKqsE0MNxfMY6tx8RaCctmGAtEMKDDWN7kfFGXcY qLGpl4E09YUEKvpFa4quPlMTqKR7A0NtuWkYUk+mAlS7VoSj8uk5BuoZtUcD690maG25kAhw oDS/B1OK8X6CqlOMmVIDzysZVLXSldL0J1BVpd8RVPWdS1TDiIygbmflmlCZabMENa8dZVBz zUMElfWgFFDVj5KpxSqHMMvTsb7RNI9Pi1m0IFLIjxFE+bGDP4kIjPDmenLcOD7oAJsl4MXT fuyjIWFuH8fE6fbEZiXy4hJ0VBhPImF7+PuKhQlSmhUtlEj92LSIHyfyErlLePGSBEGUu4CW HuR4en7orUv8PDZ65O96XFQedv7G5R5MBnoCrwAzJiS9oGxpnnEF7GRakQ0AZs7LjcECgEPq MtwQLAJ4r3vBZFuS19FIGB7qAZxrWTIG6Rj8MVujkzCZBLkfPtpk6vldpBKHqUuPGXo1Ts7i UH0zRJ9jTR6H+b179TSDdIYvVDJcj83Jg7CxtRfTp0DSA8rHLfW0GXkITvW3GVMsYe9Pk8aK jjCtpmBrUEiOmsFrHb8Bw6BHoWqmGTNga/iy+4GpAdvBKXmGEZ+DqjwlYRB/A6DimcIoPgzT ++RbXnDSBVbUexjoPfD7vnLM0NgCZq5NGuub61xtYyd4r+IWYcC74fBSqhFTMGfzqalhV1kA TjW/ZGQDluItQ4q3DCn+b30L4KVgNy2SxEfREm8RR0Cf+++XI4XxVWDrdlyD1GBs4rV7G8CY oA1AJs7eZV5/LJlvZc7nXUiixcIIcUIcLWkD3rqFX8ftbCKFuuMTSCM4Xj6eXlwu18vnIy6H bWv+vl9vpBUZxZPSsTQtosXbOoxpZifDxBb+J5bFNU7au+3jr4r3TAw9KawfXE0czN1BZpwx V9VQQRaXsnyO3GxwGQgOLe1KkdIpTloW+2lBJdfhpFnwyJ9n9raUDL5oSj518W6oLd21IZOv 9l1YPawU+hdF2XbYRMPTnylbC35l2a6818hkbZ78B3ofmHb+g62QXc+PD2JtBGoijwVpwx0D l9xXBAF/Jd7/wvH40HlrwqG0wi1ccy24dnNf0qk3MW9uLLtQn9Y+TvlyfXxHXS7zyDvv5i30 pwyGL+OtZ0PXoi9qYrLzO7FhZ3HdoQ8Kff0KR4QDt1G6+nmIUpsxP2qjGseuVj25f/byiaSZ CfuAhzNy5xJH+0UTNkMSzeO44mIJ719D7dsKxAQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02SaUxTWRzFvfe9Ph516jwK0QtmJFZcAooLfLjjMA6D280QFUkMxkShyAsu pdYWxm0MaIcRMUhDcasLgojSuiCIUtqiVhCqAppSTEkEIwU1CFjgg1tBC5nEb+d/fud/Ph2W Er+hg9jt8nReKZfKJIyQvvNQErwgxn4wZVHeV4RvPn5E4cMaD4UNL/MZ3PdwCOCTHz5R2HX/ CMB21xT86t4f2DJwVoCd940Qm0sKIC43NEBsKnZD3DDWz+ACazvAPQ4dxJaOMFz8XymNzRYb je215xhcVNbjg4+9qGHwlcZRiK1aNcQ1rkMA3/lSROEbfYM0buqYjls9jQL85eM5JnoGsbfF El1XM0OMupc+pLXzFk2qroYSe3MGqdQfZUhVaSYxObMYcum4VkDy1AMMcfd00GSwzsGQ47f1 gFQ9OUCGK2fE+W0SRqXwsu1/88qFy5KE25xvaynFjbi953OaYBZoWp4LfFnERaLCejOTC4Ss mKsBqL5II5gAgajMU09NaH9UPvrGZyKkhmho5PF3wLIMF4aejLFeP4C7TaHmz1qB96C4wzR6 WtYNvd/+3Dp0pLvJx6tpbjbqLc8abxVxS5H5gQ16ixC3EOV3+XltX+439K7ZOt4v/h5xDEZN pP2Q7YyL9mqKC0bq6rOUBnC6H5DuB3QRQD0I5BWqtNQ01WLFEjm/J1wlTVNlyFPDt+5KqwTj WwgNrQFm/YdwK4AssALEUpIAUe1fB1LEohTpvv28cleiMkPGq6xgOktLpome5doSxVyqNJ3f yfMKXvk/haxvUBbcN+knXm9hZ/mFDx0dLv4louR9I+mOaI/d3CdgZZw5cvUS2rH7xOCOnOFq 9Dn7a3TCeY9wjWa0n+rXf4STCnLuloXkbxjpzE7KL9h4fU6F4cSVqPhVvcceuOTJoC3+lPJP 68wuexIwh+i0dZ6YsXt7tXNh4emqsOr11otzLytKmUfu+ucxr7KT5+3xb7l6yuCMlQXYemdy /nHunYFBCZr213UXXkytaOk1Zvy6X2VKMNaaSGtfYUl8ZEnDuslcwibLz44WnjfOEqTn6UK2 rA1+/j5uxT85hSJn9Eb3/JWSNktEpuna79ejEgciKsStl3VqalnP7MwYQ6fYuTl55F8Jrdom XRxKKVXSbz8IBUF6AwAA X-CMS-MailID: 20230605122122epcas5p27d70bd972c1063ecb543ea4489870fd7 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230605122122epcas5p27d70bd972c1063ecb543ea4489870fd7 References: <20230605121732.28468-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Add device limits as sysfs entries, - copy_offload (RW) - copy_max_bytes (RW) - copy_max_bytes_hw (RO) Above limits help to split the copy payload in block layer. copy_offload: used for setting copy offload(1) or emulation(0). copy_max_bytes: maximum total length of copy in single payload. copy_max_bytes_hw: Reflects the device supported maximum limit. Reviewed-by: Hannes Reinecke Signed-off-by: Nitesh Shetty Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- Documentation/ABI/stable/sysfs-block | 33 +++++++++++++++ block/blk-settings.c | 24 +++++++++++ block/blk-sysfs.c | 63 ++++++++++++++++++++++++++++ include/linux/blkdev.h | 12 ++++++ include/uapi/linux/fs.h | 3 ++ 5 files changed, 135 insertions(+) diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block index c57e5b7cb532..3c97303f658b 100644 --- a/Documentation/ABI/stable/sysfs-block +++ b/Documentation/ABI/stable/sysfs-block @@ -155,6 +155,39 @@ Description: last zone of the device which may be smaller. +What: /sys/block//queue/copy_offload +Date: June 2023 +Contact: linux-block@vger.kernel.org +Description: + [RW] When read, this file shows whether offloading copy to a + device is enabled (1) or disabled (0). Writing '0' to this + file will disable offloading copies for this device. + Writing any '1' value will enable this feature. If the device + does not support offloading, then writing 1, will result in an + error. + + +What: /sys/block//queue/copy_max_bytes +Date: June 2023 +Contact: linux-block@vger.kernel.org +Description: + [RW] This is the maximum number of bytes that the block layer + will allow for a copy request. This will is always smaller or + equal to the maximum size allowed by the hardware, indicated by + 'copy_max_bytes_hw'. An attempt to set a value higher than + 'copy_max_bytes_hw' will truncate this to 'copy_max_bytes_hw'. + + +What: /sys/block//queue/copy_max_bytes_hw +Date: June 2023 +Contact: linux-block@vger.kernel.org +Description: + [RO] This is the maximum number of bytes that the hardware + will allow for single data copy request. + A value of 0 means that the device does not support + copy offload. + + What: /sys/block//queue/crypto/ Date: February 2022 Contact: linux-block@vger.kernel.org diff --git a/block/blk-settings.c b/block/blk-settings.c index 4dd59059b788..738cd3f21259 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -59,6 +59,8 @@ void blk_set_default_limits(struct queue_limits *lim) lim->zoned = BLK_ZONED_NONE; lim->zone_write_granularity = 0; lim->dma_alignment = 511; + lim->max_copy_sectors_hw = 0; + lim->max_copy_sectors = 0; } /** @@ -82,6 +84,8 @@ void blk_set_stacking_limits(struct queue_limits *lim) lim->max_dev_sectors = UINT_MAX; lim->max_write_zeroes_sectors = UINT_MAX; lim->max_zone_append_sectors = UINT_MAX; + lim->max_copy_sectors_hw = UINT_MAX; + lim->max_copy_sectors = UINT_MAX; } EXPORT_SYMBOL(blk_set_stacking_limits); @@ -183,6 +187,22 @@ void blk_queue_max_discard_sectors(struct request_queue *q, } EXPORT_SYMBOL(blk_queue_max_discard_sectors); +/** + * blk_queue_max_copy_sectors_hw - set max sectors for a single copy payload + * @q: the request queue for the device + * @max_copy_sectors: maximum number of sectors to copy + **/ +void blk_queue_max_copy_sectors_hw(struct request_queue *q, + unsigned int max_copy_sectors) +{ + if (max_copy_sectors > (COPY_MAX_BYTES >> SECTOR_SHIFT)) + max_copy_sectors = COPY_MAX_BYTES >> SECTOR_SHIFT; + + q->limits.max_copy_sectors_hw = max_copy_sectors; + q->limits.max_copy_sectors = max_copy_sectors; +} +EXPORT_SYMBOL_GPL(blk_queue_max_copy_sectors_hw); + /** * blk_queue_max_secure_erase_sectors - set max sectors for a secure erase * @q: the request queue for the device @@ -578,6 +598,10 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, t->max_segment_size = min_not_zero(t->max_segment_size, b->max_segment_size); + t->max_copy_sectors = min(t->max_copy_sectors, b->max_copy_sectors); + t->max_copy_sectors_hw = min(t->max_copy_sectors_hw, + b->max_copy_sectors_hw); + t->misaligned |= b->misaligned; alignment = queue_limit_alignment_offset(b, start); diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index a64208583853..94e90a877cec 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -212,6 +212,62 @@ static ssize_t queue_discard_zeroes_data_show(struct request_queue *q, char *pag return queue_var_show(0, page); } +static ssize_t queue_copy_offload_show(struct request_queue *q, char *page) +{ + return queue_var_show(blk_queue_copy(q), page); +} + +static ssize_t queue_copy_offload_store(struct request_queue *q, + const char *page, size_t count) +{ + s64 copy_offload; + ssize_t ret = queue_var_store64(©_offload, page); + + if (ret < 0) + return ret; + + if (copy_offload && !q->limits.max_copy_sectors_hw) + return -EINVAL; + + if (copy_offload) + blk_queue_flag_set(QUEUE_FLAG_COPY, q); + else + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + + return count; +} + +static ssize_t queue_copy_max_hw_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors_hw << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_store(struct request_queue *q, + const char *page, size_t count) +{ + unsigned long max_copy; + ssize_t ret = queue_var_store(&max_copy, page, count); + + if (ret < 0) + return ret; + + if (max_copy & (queue_logical_block_size(q) - 1)) + return -EINVAL; + + max_copy >>= SECTOR_SHIFT; + q->limits.max_copy_sectors = min_t(unsigned int, max_copy, + q->limits.max_copy_sectors_hw); + + return count; +} + static ssize_t queue_write_same_max_show(struct request_queue *q, char *page) { return queue_var_show(0, page); @@ -590,6 +646,10 @@ QUEUE_RO_ENTRY(queue_nr_zones, "nr_zones"); QUEUE_RO_ENTRY(queue_max_open_zones, "max_open_zones"); QUEUE_RO_ENTRY(queue_max_active_zones, "max_active_zones"); +QUEUE_RW_ENTRY(queue_copy_offload, "copy_offload"); +QUEUE_RO_ENTRY(queue_copy_max_hw, "copy_max_bytes_hw"); +QUEUE_RW_ENTRY(queue_copy_max, "copy_max_bytes"); + QUEUE_RW_ENTRY(queue_nomerges, "nomerges"); QUEUE_RW_ENTRY(queue_rq_affinity, "rq_affinity"); QUEUE_RW_ENTRY(queue_poll, "io_poll"); @@ -637,6 +697,9 @@ static struct attribute *queue_attrs[] = { &queue_discard_max_entry.attr, &queue_discard_max_hw_entry.attr, &queue_discard_zeroes_data_entry.attr, + &queue_copy_offload_entry.attr, + &queue_copy_max_hw_entry.attr, + &queue_copy_max_entry.attr, &queue_write_same_max_entry.attr, &queue_write_zeroes_max_entry.attr, &queue_zone_append_max_entry.attr, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 04a133d32c67..5ff161c18ae8 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -295,6 +295,9 @@ struct queue_limits { unsigned int discard_alignment; unsigned int zone_write_granularity; + unsigned int max_copy_sectors_hw; + unsigned int max_copy_sectors; + unsigned short max_segments; unsigned short max_integrity_segments; unsigned short max_discard_segments; @@ -563,6 +566,7 @@ struct request_queue { #define QUEUE_FLAG_NOWAIT 29 /* device supports NOWAIT */ #define QUEUE_FLAG_SQ_SCHED 30 /* single queue style io dispatch */ #define QUEUE_FLAG_SKIP_TAGSET_QUIESCE 31 /* quiesce_tagset skip the queue*/ +#define QUEUE_FLAG_COPY 32 /* enable/disable device copy offload */ #define QUEUE_FLAG_MQ_DEFAULT ((1UL << QUEUE_FLAG_IO_STAT) | \ (1UL << QUEUE_FLAG_SAME_COMP) | \ @@ -583,6 +587,7 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q); test_bit(QUEUE_FLAG_STABLE_WRITES, &(q)->queue_flags) #define blk_queue_io_stat(q) test_bit(QUEUE_FLAG_IO_STAT, &(q)->queue_flags) #define blk_queue_add_random(q) test_bit(QUEUE_FLAG_ADD_RANDOM, &(q)->queue_flags) +#define blk_queue_copy(q) test_bit(QUEUE_FLAG_COPY, &(q)->queue_flags) #define blk_queue_zone_resetall(q) \ test_bit(QUEUE_FLAG_ZONE_RESETALL, &(q)->queue_flags) #define blk_queue_dax(q) test_bit(QUEUE_FLAG_DAX, &(q)->queue_flags) @@ -901,6 +906,8 @@ extern void blk_queue_chunk_sectors(struct request_queue *, unsigned int); extern void blk_queue_max_segments(struct request_queue *, unsigned short); extern void blk_queue_max_discard_segments(struct request_queue *, unsigned short); +extern void blk_queue_max_copy_sectors_hw(struct request_queue *q, + unsigned int max_copy_sectors); void blk_queue_max_secure_erase_sectors(struct request_queue *q, unsigned int max_sectors); extern void blk_queue_max_segment_size(struct request_queue *, unsigned int); @@ -1220,6 +1227,11 @@ static inline unsigned int bdev_discard_granularity(struct block_device *bdev) return bdev_get_queue(bdev)->limits.discard_granularity; } +static inline unsigned int bdev_max_copy_sectors(struct block_device *bdev) +{ + return bdev_get_queue(bdev)->limits.max_copy_sectors; +} + static inline unsigned int bdev_max_secure_erase_sectors(struct block_device *bdev) { diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index b7b56871029c..a16bafada09d 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -64,6 +64,9 @@ struct fstrim_range { __u64 minlen; }; +/* maximum total copy length, this is set to 128 MB based on current testing */ +#define COPY_MAX_BYTES (1 << 27) + /* extent-same (dedupe) ioctls; these MUST match the btrfs ioctl definitions */ #define FILE_DEDUPE_RANGE_SAME 0 #define FILE_DEDUPE_RANGE_DIFFERS 1 From patchwork Mon Jun 5 12:17:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13267302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91063C7EE2F for ; Mon, 5 Jun 2023 12:30:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233461AbjFEM3p (ORCPT ); Mon, 5 Jun 2023 08:29:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55636 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230229AbjFEM3i (ORCPT ); Mon, 5 Jun 2023 08:29:38 -0400 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F739DA for ; Mon, 5 Jun 2023 05:29:35 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20230605122933epoutp02a9b7cd06cc5b2e7805ecb5590f7c721d~lwx75Zifz0753307533epoutp02Q for ; Mon, 5 Jun 2023 12:29:33 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20230605122933epoutp02a9b7cd06cc5b2e7805ecb5590f7c721d~lwx75Zifz0753307533epoutp02Q DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1685968173; bh=1y7JV8IkqYpeDgQv38tdyIZC5pub23kWz9W5MNLaQSI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qX8OA5v4kpqn+/1h/aR0HK6FBmLMFZ5Zj13zmL+QbM6jUr9ZD33SSzoYtdC/Giizn wwcmND4oZKrUoJAM2OPOmnoOFAgZxJP4prDmqwB5OY7NqH0gf7Ol/zmnIX6WPWHhNK BJsRtruojB0eNyG9WlCN58AaiPgxfdUBuB5KoDaw= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230605122932epcas5p411eee49c2bdc0c26f007b4f8c85c1465~lwx69wDEM1407614076epcas5p46; Mon, 5 Jun 2023 12:29:32 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.182]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4QZXwp47n0z4x9Pq; Mon, 5 Jun 2023 12:29:30 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id B5.49.04567.A25DD746; Mon, 5 Jun 2023 21:29:30 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230605122141epcas5p256df105da840c937758de63e67516f32~lwrEi2xrA0039600396epcas5p2v; Mon, 5 Jun 2023 12:21:41 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230605122141epsmtrp1e8fd82e63b03d16158bfa3e069db6896~lwrEhrG9W1437514375epsmtrp1B; Mon, 5 Jun 2023 12:21:41 +0000 (GMT) X-AuditID: b6c32a49-db3fe700000011d7-57-647dd52ad418 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id A9.9C.27706.553DD746; Mon, 5 Jun 2023 21:21:41 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230605122135epsmtip29ea7bccfcc6947623ad8181d7fe93c03~lwq_4aJnE2248722487epsmtip2O; Mon, 5 Jun 2023 12:21:35 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, willy@infradead.org, hare@suse.de, djwong@kernel.org, bvanassche@acm.org, ming.lei@redhat.com, dlemoal@kernel.org, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v12 2/9] block: Add copy offload support infrastructure Date: Mon, 5 Jun 2023 17:47:18 +0530 Message-Id: <20230605121732.28468-3-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230605121732.28468-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTVxzHc+4tt4VZvOWxHV6zqTGOMh5lwA5MJk7E62vCiC7ZNKyhN8CA tmlhzhHCuygKRWATSxFhTHkoFRDGU6EEO1DmIitMFgYsQEQ2XmOgAm6UK5v/fb7f83uc3+/k cHCrG2x7TpQ0jlZIxTECwoLV2OX8lqvQmCjxaDM6Il3vXRyl5q7hqHpYTaDprgWAvpl7hqPx jkyA+sct0eid3ah9psgMPepoxlBbWR6GKqu7MdRaOo+hPP0AQBNGDYbah1xQqaqchdrae1io v0VLoJKrE2x0brCJQNcMLzCkz0/DUNN4CkCNKyU4qpmeZaEfhhzQgzWDGVp5qiUCnKj+nw9R mpE+gmrWDLOpB7/Vsqj6CiHV3xdP1VWdJaj68iSq9VEyQX2bk29GZafNENT8xBCLmr1tJKic W1WAqr+XQP1V92Yw75PoXZG0WEIr+LQ0XCaJkkb4Cw6Fhu0N8/bxELmKfNG7Ar5UHEv7CwIP B7sGRcWsr0jA/0IcE79uBYuVSoH7+7sUsvg4mh8pU8b5C2i5JEbuJXdTimOV8dIINykd5yfy 8PD0Xg/8LDqyY/K8mbwz7Mv+2b9ZySDvwyxgzoGkF0xW1+JZwIJjRbYC2KE6TzBiAUBDV6YZ I5YArJ5pwzdTrl99+DKqHcC6kiLAiAwMqsoXsSzA4RCkC7z3D8fk25AVOExZvs8yCZw04PDu fMpGKWtyP9QvrwATs8gdMK+sgzAxl/SD05mDhKkQJN2heoRnss3J9+BUnx5nQniw59I4y8Q4 uQ2mNRRtDAHJh+aw53stzuQGwpp6b+bW1vCJ4RabYXs4pVa95FOwsqCCYHLTAdQMagBzsBtm 9Ko36uCkM9S1uDO2E/y6twZj+lrC7JVxjPG5sOnyJm+H13VXCIbt4MByykumoLHxR4xZVg6A o+kZWC7ga16ZR/PKPJr/W18BeBWwo+XK2Aha6S0XSelT/z1zuCy2Dmz8G+GBJjA8OuemBxgH 6AHk4AIbbsvBBIkVVyI+/RWtkIUp4mNopR54r+/7Am5vGy5b/3jSuDCRl6+Hl4+Pj5fvOz4i wRvcnf494VZkhDiOjqZpOa3YzMM45vbJWOx4wFrhd6VC7uoCv7NHZ3BOlAg+dx9o7nb7veHs sUKLy+Ge9wM9TxTPjHHZuo+D9zuW1aWe7hXqDKnPhav6bSGP2TMnxVN9j6NXXW8Tma87vjAe 0006JHwadNiy+MC+sblemd2ZscSlI4qPth4sgLaLyGPt19Bze4+udYb+YXm8UvHTiIMTr//k UUfzhuqgPU+zrf+8KVAfCfEvX7QNv9i2hffLc2F3stWArau2bWkr+21V1LRN0oRWl+tce6l4 bHvNiZtnQnZCmzHR/I7FPZO8D/bduFYof6bKj5m0LHjiHa1ydj+enVIU65frcsfd+NoW64AW jjC1L109EN3eob2YpC4TsJSRYpEQVyjF/wJVA0JrwAQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrIIsWRmVeSWpSXmKPExsWy7bCSvG7o5doUgwXnxC3WnzrGbNE04S+z xeq7/WwWrw9/YrSY9uEns8WTA+2MFpef8Fk82G9vsffdbFaLmwd2MlnsWTSJyWLl6qNMFrsX fmSymHToGqPF06uzmCz23tK2WNi2hMViz96TLBaXd81hs5i/7Cm7Rff1HWwWy4//Y7I4NLmZ yWLHk0ZGi22/5zNbrHv9nsXixC1pi/N/j7Na/P4xh81B1uPyFW+PWffPsnnsnHWX3eP8vY0s HptXaHlcPlvqsWlVJ5vH5iX1HrtvNrB5LO6bzOrR2/yOzePj01ssHu/3XWXz6NuyitFj8+lq j8+b5AIEo7hsUlJzMstSi/TtErgyDjzrYS04GF9x+f1XlgbGSX5djJwcEgImEmuWXWLrYuTi EBLYzSjx4cRVdoiEpMSyv0eYIWxhiZX/nrNDFDUzSfQ/2A7UwcHBJqAtcfo/B0hcRGALs8TZ X5NZQRqYBW4zS8w8KwNiCwu4Sxz6/psRxGYRUJWYtOgAG4jNK2Al8br9OtgcCQF9if77giBh TgFriZdnDzGDhIWASq6+t4GoFpQ4OfMJC8R0eYnmrbOZJzAKzEKSmoUktYCRaRWjZGpBcW56 brFhgWFearlecWJucWleul5yfu4mRnAq0NLcwbh91Qe9Q4xMHIyHGCU4mJVEeHd5VacI8aYk VlalFuXHF5XmpBYfYpTmYFES573QdTJeSCA9sSQ1OzW1ILUIJsvEwSnVwDSj5bP4vQlzpt/h PqZv2N7vFKb+o3h/XI0Nd3uC182n4dWO7QLrLM1ZUgpdHgt85//pc46j/MkOm6rVZzvtZtTI Pg50dDLLu3HHxOHMghadv0wly4O2fUwIl0x31a5TWl/+8knSI7nimXcWzrtdb329v1Lwx947 P//VzDYv3G+ZuFX7B3/Vp/hVkxJVrx8T+lPJaJ/wS69E3TJ91mzVxM9Vcw71ZWf457q0aIba 9K1tbNdfrn8vNUVRXTCHUc4q8swZrUiJIicPAbVq++XZ7JPLbrzir/ZaIPW+MZglJnButcOi U5cqNxTnHZjcftXgHr9rxb6yy//ZzG1Wz3MzLSwumZGzS+/sh4lGUS+VWIozEg21mIuKEwFW c+R5dAMAAA== X-CMS-MailID: 20230605122141epcas5p256df105da840c937758de63e67516f32 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230605122141epcas5p256df105da840c937758de63e67516f32 References: <20230605121732.28468-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Introduce blkdev_issue_copy which takes similar arguments as copy_file_range and performs copy offload between two bdevs. Introduce REQ_COPY copy offload operation flag. Create a read-write bio pair with a token as payload and submitted to the device in order. Read request populates token with source specific information which is then passed with write request. This design is courtesy Mikulas Patocka's token based copy Larger copy will be divided, based on max_copy_sectors limit. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- block/blk-lib.c | 243 ++++++++++++++++++++++++++++++++++++++ block/blk.h | 2 + include/linux/blk_types.h | 25 ++++ include/linux/blkdev.h | 4 + include/uapi/linux/fs.h | 5 +- 5 files changed, 278 insertions(+), 1 deletion(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index e59c3069e835..b8e11997b5bf 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -115,6 +115,249 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, } EXPORT_SYMBOL(blkdev_issue_discard); +/* + * For synchronous copy offload/emulation, wait and process all in-flight BIOs. + * This must only be called once all bios have been issued so that the refcount + * can only decrease. This just waits for all bios to make it through + * blkdev_copy_(offload/emulate)_write_endio. + */ +static ssize_t blkdev_copy_wait_completion(struct cio *cio) +{ + ssize_t ret; + + if (cio->endio) + return 0; + + if (atomic_read(&cio->refcount)) { + __set_current_state(TASK_UNINTERRUPTIBLE); + blk_io_schedule(); + } + + ret = cio->comp_len; + kfree(cio); + + return ret; +} + +static void blkdev_copy_offload_write_endio(struct bio *bio) +{ + struct copy_ctx *ctx = bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (bio->bi_status) { + clen = (bio->bi_iter.bi_sector << SECTOR_SHIFT) - cio->pos_out; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + } + kfree(bvec_virt(&bio->bi_io_vec[0])); + bio_put(bio); + + kfree(ctx); + if (!atomic_dec_and_test(&cio->refcount)) + return; + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); +} + +static void blkdev_copy_offload_read_endio(struct bio *read_bio) +{ + struct copy_ctx *ctx = read_bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (read_bio->bi_status) { + clen = (read_bio->bi_iter.bi_sector << SECTOR_SHIFT) + - cio->pos_in; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + kfree(bvec_virt(&read_bio->bi_io_vec[0])); + bio_put(ctx->write_bio); + bio_put(read_bio); + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } + return; + } + + schedule_work(&ctx->dispatch_work); + bio_put(read_bio); +} + +static void blkdev_copy_dispatch_work(struct work_struct *work) +{ + struct copy_ctx *ctx = container_of(work, struct copy_ctx, + dispatch_work); + + submit_bio(ctx->write_bio); +} + +/* + * __blkdev_copy_offload - Use device's native copy offload feature. + * we perform copy operation by sending 2 bio. + * 1. First we send a read bio with REQ_COPY flag along with a token and source + * and length. Once read bio reaches driver layer, device driver adds all the + * source info to token and does a fake completion. + * 2. Once read operation completes, we issue write with REQ_COPY flag with same + * token. In driver layer, token info is used to form a copy offload command. + * + * Returns the length of bytes copied or error if encountered + */ +static ssize_t __blkdev_copy_offload( + struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, + size_t len, cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct cio *cio; + struct copy_ctx *ctx; + struct bio *read_bio, *write_bio; + void *token; + sector_t copy_len; + sector_t rem, max_copy_len; + + cio = kzalloc(sizeof(struct cio), GFP_KERNEL); + if (!cio) + return -ENOMEM; + atomic_set(&cio->refcount, 0); + cio->waiter = current; + cio->endio = endio; + cio->private = private; + + max_copy_len = min(bdev_max_copy_sectors(bdev_in), + bdev_max_copy_sectors(bdev_out)) << SECTOR_SHIFT; + + cio->pos_in = pos_in; + cio->pos_out = pos_out; + /* If there is a error, comp_len will be set to least successfully + * completed copied length + */ + cio->comp_len = len; + for (rem = len; rem > 0; rem -= copy_len) { + copy_len = min(rem, max_copy_len); + + token = kmalloc(COPY_TOKEN_SIZE, gfp_mask); + if (unlikely(!token)) + goto err_token; + + ctx = kzalloc(sizeof(struct copy_ctx), gfp_mask); + if (!ctx) + goto err_ctx; + read_bio = bio_alloc(bdev_in, 1, REQ_OP_READ | REQ_COPY + | REQ_SYNC | REQ_NOMERGE, gfp_mask); + if (!read_bio) + goto err_read_bio; + write_bio = bio_alloc(bdev_out, 1, REQ_OP_WRITE + | REQ_COPY | REQ_SYNC | REQ_NOMERGE, gfp_mask); + if (!write_bio) + goto err_write_bio; + + ctx->cio = cio; + ctx->write_bio = write_bio; + INIT_WORK(&ctx->dispatch_work, blkdev_copy_dispatch_work); + + __bio_add_page(read_bio, virt_to_page(token), COPY_TOKEN_SIZE, + offset_in_page(token)); + read_bio->bi_iter.bi_size = copy_len; + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_end_io = blkdev_copy_offload_read_endio; + read_bio->bi_private = ctx; + + __bio_add_page(write_bio, virt_to_page(token), COPY_TOKEN_SIZE, + offset_in_page(token)); + write_bio->bi_iter.bi_size = copy_len; + write_bio->bi_end_io = blkdev_copy_offload_write_endio; + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_private = ctx; + + atomic_inc(&cio->refcount); + submit_bio(read_bio); + pos_in += copy_len; + pos_out += copy_len; + } + + /* Wait for completion of all IO's*/ + return blkdev_copy_wait_completion(cio); + +err_write_bio: + bio_put(read_bio); +err_read_bio: + kfree(ctx); +err_ctx: + kfree(token); +err_token: + cio->comp_len = min_t(sector_t, cio->comp_len, (len - rem)); + if (!atomic_read(&cio->refcount)) { + kfree(cio); + return -ENOMEM; + } + /* Wait for submitted IOs to complete */ + return blkdev_copy_wait_completion(cio); +} + +static inline ssize_t blkdev_copy_sanity_check( + struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, + size_t len) +{ + unsigned int align = max(bdev_logical_block_size(bdev_out), + bdev_logical_block_size(bdev_in)) - 1; + + if (bdev_read_only(bdev_out)) + return -EPERM; + + if ((pos_in & align) || (pos_out & align) || (len & align) || !len || + len >= COPY_MAX_BYTES) + return -EINVAL; + + return 0; +} + +/* + * @bdev_in: source block device + * @pos_in: source offset + * @bdev_out: destination block device + * @pos_out: destination offset + * @len: length in bytes to be copied + * @endio: endio function to be called on completion of copy operation, + * for synchronous operation this should be NULL + * @private: endio function will be called with this private data, should be + * NULL, if operation is synchronous in nature + * @gfp_mask: memory allocation flags (for bio_alloc) + * + * Returns the length of bytes copied or error if encountered + * + * Description: + * Copy source offset from source block device to destination block + * device. If copy offload is not supported or fails, fallback to + * emulation. Max total length of copy is limited to COPY_MAX_BYTES + */ +ssize_t blkdev_copy_offload( + struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, + size_t len, cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct request_queue *q_in = bdev_get_queue(bdev_in); + struct request_queue *q_out = bdev_get_queue(bdev_out); + ssize_t ret; + + ret = blkdev_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len); + if (ret) + return ret; + + if (blk_queue_copy(q_in) && blk_queue_copy(q_out)) + ret = __blkdev_copy_offload(bdev_in, pos_in, bdev_out, pos_out, + len, endio, private, gfp_mask); + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_copy_offload); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/block/blk.h b/block/blk.h index 7ad7cb6ffa01..f7593396c637 100644 --- a/block/blk.h +++ b/block/blk.h @@ -303,6 +303,8 @@ static inline bool bio_may_exceed_limits(struct bio *bio, break; } + if (unlikely(op_is_copy(bio->bi_opf))) + return false; /* * All drivers must accept single-segments bios that are <= PAGE_SIZE. * This is a quick and dirty check that relies on the fact that diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index d7eeaedddd16..1d54a1648adf 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -427,6 +427,7 @@ enum req_flag_bits { */ /* for REQ_OP_WRITE_ZEROES: */ __REQ_NOUNMAP, /* do not free blocks when zeroing */ + __REQ_COPY, /* copy request */ __REQ_NR_BITS, /* stops here */ }; @@ -451,6 +452,7 @@ enum req_flag_bits { #define REQ_POLLED (__force blk_opf_t)(1ULL << __REQ_POLLED) #define REQ_ALLOC_CACHE (__force blk_opf_t)(1ULL << __REQ_ALLOC_CACHE) #define REQ_SWAP (__force blk_opf_t)(1ULL << __REQ_SWAP) +#define REQ_COPY ((__force blk_opf_t)(1ULL << __REQ_COPY)) #define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV) #define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE) @@ -481,6 +483,11 @@ static inline bool op_is_write(blk_opf_t op) return !!(op & (__force blk_opf_t)1); } +static inline bool op_is_copy(blk_opf_t op) +{ + return op & REQ_COPY; +} + /* * Check if the bio or request is one that needs special treatment in the * flush state machine. @@ -540,4 +547,22 @@ struct blk_rq_stat { u64 batch; }; +typedef void (cio_iodone_t)(void *private, int comp_len); + +struct cio { + struct task_struct *waiter; /* waiting task (NULL if none) */ + atomic_t refcount; + loff_t pos_in; + loff_t pos_out; + ssize_t comp_len; + cio_iodone_t *endio; /* applicable for async operation */ + void *private; /* applicable for async operation */ +}; + +struct copy_ctx { + struct cio *cio; + struct work_struct dispatch_work; + struct bio *write_bio; +}; + #endif /* __LINUX_BLK_TYPES_H */ diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 5ff161c18ae8..96e986b37a29 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1053,6 +1053,10 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop); int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp); +ssize_t blkdev_copy_offload( + struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, + size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index a16bafada09d..3c8224a2ad85 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -64,9 +64,12 @@ struct fstrim_range { __u64 minlen; }; -/* maximum total copy length, this is set to 128 MB based on current testing */ +/* maximum copy offload length, this is set to 128MB based on current testing */ #define COPY_MAX_BYTES (1 << 27) +/* copy offload token size */ +#define COPY_TOKEN_SIZE SECTOR_SIZE + /* extent-same (dedupe) ioctls; these MUST match the btrfs ioctl definitions */ #define FILE_DEDUPE_RANGE_SAME 0 #define FILE_DEDUPE_RANGE_DIFFERS 1 From patchwork Mon Jun 5 12:17:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13267301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 866C6C77B73 for ; Mon, 5 Jun 2023 12:29:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233448AbjFEM3o (ORCPT ); Mon, 5 Jun 2023 08:29:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233438AbjFEM3l (ORCPT ); Mon, 5 Jun 2023 08:29:41 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C11FFA for ; Mon, 5 Jun 2023 05:29:38 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230605122936epoutp0137f18a5f3cb58ea213bf8668fbad5840~lwx-Dn9K41442314423epoutp01M for ; Mon, 5 Jun 2023 12:29:36 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230605122936epoutp0137f18a5f3cb58ea213bf8668fbad5840~lwx-Dn9K41442314423epoutp01M DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1685968176; bh=FZjAen5BLaL48oB6pQ6fLBXdXNFuo0Xeex4cPJyfepw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f9l2XqAgDQZGZN8/SjYBVxb0q1uHwOXmN1LsCXKb51nPHTRDGILK0zIBshgiV5TXW KWygpMswL4oTGnS2dI56v6T5g3k8fBWtE5p5q1Or5LJz7k0OIdrRXqipha6BadzfWw XHW+SMaNfuJbMRgiPvL1uEQe1GB3ZPViWAT/FlCU= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230605122935epcas5p4fa7c841a0718ed12c7422ced406345f0~lwx_XJsvk0962809628epcas5p4i; Mon, 5 Jun 2023 12:29:35 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.180]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4QZXws5zh0z4x9Pv; Mon, 5 Jun 2023 12:29:33 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 65.F6.44881.D25DD746; Mon, 5 Jun 2023 21:29:33 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230605122208epcas5p297068ea23b0aff43e40751219574ff9f~lwreFrU9P2830028300epcas5p2Q; Mon, 5 Jun 2023 12:22:08 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230605122208epsmtrp228fa4e823273dbca931ac9e52f1381c3~lwreEen5A0879508795epsmtrp2F; Mon, 5 Jun 2023 12:22:08 +0000 (GMT) X-AuditID: b6c32a4a-c47ff7000001af51-66-647dd52d779d Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 4A.B5.28392.073DD746; Mon, 5 Jun 2023 21:22:08 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230605122203epsmtip2ef3933d0ab30729821518ffb0b7096f2~lwrZHH17B2137821378epsmtip2Q; Mon, 5 Jun 2023 12:22:03 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, willy@infradead.org, hare@suse.de, djwong@kernel.org, bvanassche@acm.org, ming.lei@redhat.com, dlemoal@kernel.org, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Vincent Fu , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v12 3/9] block: add emulation for copy Date: Mon, 5 Jun 2023 17:47:19 +0530 Message-Id: <20230605121732.28468-4-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230605121732.28468-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta0xbZRjOd057ekpkOcAM37hsXdEs3MutfEwu011yhMURmYHIFBt6BATa 2hZQ0QkDhEG4DaauXMYAuTPCVaAwseg6bmEZUDLmBmpxYQu3YQJaGQIF3b/ne973eS/Pl5fE zVs5VmSMRMnIJaI4PmHC6hq0P+bsPPWFWDBf7IRahm/j6FLBJo4aH+YT6OngM4C+XvkLR/qB TIAm9AfQ3A8BqH+phI3uD/RgqK/yCobqG3/GkPrGKoauaHQAzU+pMNQ/44hufFXNQn39Qyw0 0VtKoOs18xyUM91NoFrtcwxpitIw1K1PBajLcB1HN58us9CdGWv0e04WQOObWjYybJQSJw7T E5NBtGp2jKB7VA859PijVhbdXudAT4wl0G0Nlwm6vfpLWn0/haCr8orYdG7aEkGvzs+w6OVb UwSd19EA6PaRZHqt7XCw2buxvtGMSMzIeYwkUiqOkUT58YNCIk5GeAkFbs5uPsibz5OI4hk/ /qmzwc5nYuK2feLzEkVxCdtUsEih4Lv6+8qlCUqGFy1VKP34jEwcJ/OUuShE8YoESZSLhFEe dxMI3L22Ez+IjV4eGwCyEv9P7v24zkkBaR7ZgEtCyhNmqueIbGBCmlNqAKsWpzHj4xmAORsp e5E1AK9u1WD7knJV3y42p3oBfKIXGJMyMDg98w2eDUiSoBzhyBa5wx+k6nCYuj7K2hHg1BIO qwq5O9iCEsL6u6u7PIt6FQ486AE72JQ6DisN2t06kHKF+bNmOzSXeg0ujGlwY4oZHLqm3yt5 BKZ1luA7vSB1jws7NrLYxkFPwWltPsuILeATbQfHiK3g2lI/YcRJsL64jjCK0wFUTauAMRAA M4bzd4fAKXvY0utqpG3h1eGbmLHxAZhr0O+ZYgq7y/exHWxqqdirfwjq1lP3MA3Lm+4Ao1l5 ALYX/8MuADzVCwupXlhI9X/rCoA3gEOMTBEfxSi8ZO4SJum/X46UxreB3dtxCOwGv86tuGgA RgINgCTOP2jaG5gsNjcViz79jJFLI+QJcYxCA7y2DS/ErV6OlG4fn0QZ4ebpI/AUCoWePh5C N76l6TG/oUhzKkqkZGIZRsbI93UYybVKwY7wTr8Xd05XZJNYOvi8tvxWmMEjai3okZ+/oP5y 2QX6XBmusP7Yu9TEVnTGweW7up/CinxjfNV+0lonFO5gwXmTelwxHnZ+Mvad239nmnDqtrIm We9Xv2XzeF347WJqiICLWnS/nS9bSJq1T76olNjFvtR8tHkxhNdZ83o+qrEOtMzo2vQy6Gz5 k0e9RzK3Pncys/HhBKSLNOJKJ3Wae7mJ5fjsR7ywE5FJiQtnR//oe6P2z9YHb4eTXDJ3o/Fa WsYvFwq7ThZo3bPxuQ9D7TphycRAWWpoUj1pFe5/Oj145BU2TvlsjWpmW3TWK9pLHsWEY+hQ k5WJ2RwWKrh78ftmPksRLXJzwOUK0b9/gZioxAQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02SXVBMYRzG5z3n7NlTM8tpq/GyY2Nn+hAiJa80Psd46aa4KYzasWdS2m1n V5LGSOuzSLYmdoskgzLjYwvFKWv7kiw7tqSSol2EKXEhJbE1Ztw9z/N7/s/VnyHFH6hZTKJq D6dRyZNltDt1t17ms1BtP6BYrH8Xim4+aSJRVt44ia73nKbR5/pvABV+/Ukih/kYQHbHNNT3 cBWqHSwSoE5zDYH4S3oClV9vJNCD0mEC6S0vAXK2GwlU2zUflR69TCG+toVC9vvFNCq54hSi nI5qGl1t/k0gS76OQNWOQwDdHSsh0Y3PQxR63CVB/TnHAXo+3ixAYyPF9GoptrdFYmOvlcY1 xh4hfv7mNoUrrwViuzUVmypO0Ljy8kH8oDOTxmW5+QJ8SjdI42FnF4WH6tppnFtVAXBlawb+ bpJGeWx1j1BwyYl7Oc2ilfHuu4asZqAuWrnvxaMfwkygC8kGbgxkQ+EFI09kA3dGzFYDWFlw lZ4CM+GV8QZySnvC8t8fhFMlHQGfvOr5axiGZufD1gnGlXuxVSS0juYLXIZksyjY+frT5LUn GwbLbcOUS1OsLzR31wCXFrHh8NJYM+kaguwieLrXwxW7sSvggNUyGYv/VtqHIqbaHrDF4Jhc IVkfqLtTROYB1vgfMv6HLgKiAszk1FplglIbrF6i4tKCtHKlNlWVELQzRWkCk78QGFgN+Iqv QRZAMMACIEPKvET3N2UoxCKFPH0/p0mJ06Qmc1oLkDCUbIbIlt0SJ2YT5Hu43Ryn5jT/KMG4 zcokkrJh//SRmNmDTsMhi8SfM1zbrLmory+L/7im0XTjrGhC0rp2XnJkWMa58KYX8XrzoPRM 31P7cb+QZ30lAum6ez4308Pj+v3luTir1MSHxnp6lNZNU+VEbesQp/EXvKPJQlrULamPYbcE RJ6wp0W3NVUVzlnxOL2B2WIb9W/8FmLzovi1ecJmNLd2jt+qxJPOuF/eyxTOl3t9y7rL1v16 uuGIdPtDi3dQDRkcayuIuXUr/mzH9tjgiOmPGgLW955P8uV3pvRqzEsNE7e/OEZn7CDMeYff N4alJ/XvOCLcuMAtOor3ff82NmVJ3ciZAFoZ3jAwMFBw4MdybQDXVu5nkFHaXfLgQFKjlf8B pK5sbHoDAAA= X-CMS-MailID: 20230605122208epcas5p297068ea23b0aff43e40751219574ff9f X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230605122208epcas5p297068ea23b0aff43e40751219574ff9f References: <20230605121732.28468-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org For the devices which does not support copy, copy emulation is added. It is required for in-kernel users like fabrics, where file descriptor is not available and hence they can't use copy_file_range. Copy-emulation is implemented by reading from source into memory and writing to the corresponding destination asynchronously. Also emulation is used, if copy offload fails or partially completes. Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu Signed-off-by: Anuj Gupta --- block/blk-lib.c | 178 ++++++++++++++++++++++++++++++++++++++++- block/blk-map.c | 4 +- include/linux/blkdev.h | 3 + 3 files changed, 182 insertions(+), 3 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index b8e11997b5bf..99b65af8bfc1 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -300,6 +300,175 @@ static ssize_t __blkdev_copy_offload( return blkdev_copy_wait_completion(cio); } +static void *blkdev_copy_alloc_buf(sector_t req_size, sector_t *alloc_size, + gfp_t gfp_mask) +{ + int min_size = PAGE_SIZE; + void *buf; + + while (req_size >= min_size) { + buf = kvmalloc(req_size, gfp_mask); + if (buf) { + *alloc_size = req_size; + return buf; + } + /* retry half the requested size */ + req_size >>= 1; + } + + return NULL; +} + +static void blkdev_copy_emulate_write_endio(struct bio *bio) +{ + struct copy_ctx *ctx = bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (bio->bi_status) { + clen = (bio->bi_iter.bi_sector << SECTOR_SHIFT) - cio->pos_out; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + } + kfree(bvec_virt(&bio->bi_io_vec[0])); + bio_map_kern_endio(bio); + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } +} + +static void blkdev_copy_emulate_read_endio(struct bio *read_bio) +{ + struct copy_ctx *ctx = read_bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (read_bio->bi_status) { + clen = (read_bio->bi_iter.bi_sector << SECTOR_SHIFT) - + cio->pos_in; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + kfree(bvec_virt(&read_bio->bi_io_vec[0])); + bio_map_kern_endio(read_bio); + kfree(ctx); + + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } + } + schedule_work(&ctx->dispatch_work); + kfree(read_bio); +} + +/* + * If native copy offload feature is absent, this function tries to emulate, + * by copying data from source to a temporary buffer and from buffer to + * destination device. + * Returns the length of bytes copied or error if encountered + */ +static ssize_t __blkdev_copy_emulate( + struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, + size_t len, cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct request_queue *in = bdev_get_queue(bdev_in); + struct request_queue *out = bdev_get_queue(bdev_out); + struct bio *read_bio, *write_bio; + void *buf = NULL; + struct copy_ctx *ctx; + struct cio *cio; + sector_t buf_len, req_len, rem = 0; + sector_t max_src_hw_len = min_t(unsigned int, + queue_max_hw_sectors(in), + queue_max_segments(in) << (PAGE_SHIFT - SECTOR_SHIFT)) + << SECTOR_SHIFT; + sector_t max_dst_hw_len = min_t(unsigned int, + queue_max_hw_sectors(out), + queue_max_segments(out) << (PAGE_SHIFT - SECTOR_SHIFT)) + << SECTOR_SHIFT; + sector_t max_hw_len = min_t(unsigned int, + max_src_hw_len, max_dst_hw_len); + + cio = kzalloc(sizeof(struct cio), GFP_KERNEL); + if (!cio) + return -ENOMEM; + atomic_set(&cio->refcount, 0); + cio->pos_in = pos_in; + cio->pos_out = pos_out; + cio->waiter = current; + cio->endio = endio; + cio->private = private; + + for (rem = len; rem > 0; rem -= buf_len) { + req_len = min_t(int, max_hw_len, rem); + + buf = blkdev_copy_alloc_buf(req_len, &buf_len, gfp_mask); + if (!buf) + goto err_alloc_buf; + + ctx = kzalloc(sizeof(struct copy_ctx), gfp_mask); + if (!ctx) + goto err_ctx; + + read_bio = bio_map_kern(in, buf, buf_len, gfp_mask); + if (IS_ERR(read_bio)) + goto err_read_bio; + + write_bio = bio_map_kern(out, buf, buf_len, gfp_mask); + if (IS_ERR(write_bio)) + goto err_write_bio; + + ctx->cio = cio; + ctx->write_bio = write_bio; + INIT_WORK(&ctx->dispatch_work, blkdev_copy_dispatch_work); + + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_iter.bi_size = buf_len; + read_bio->bi_opf = REQ_OP_READ | REQ_SYNC; + bio_set_dev(read_bio, bdev_in); + read_bio->bi_end_io = blkdev_copy_emulate_read_endio; + read_bio->bi_private = ctx; + + write_bio->bi_iter.bi_size = buf_len; + write_bio->bi_opf = REQ_OP_WRITE | REQ_SYNC; + bio_set_dev(write_bio, bdev_out); + write_bio->bi_end_io = blkdev_copy_emulate_write_endio; + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_private = ctx; + + atomic_inc(&cio->refcount); + submit_bio(read_bio); + + pos_in += buf_len; + pos_out += buf_len; + } + + /* Wait for completion of all IO's*/ + return blkdev_copy_wait_completion(cio); + +err_write_bio: + bio_put(read_bio); +err_read_bio: + kfree(ctx); +err_ctx: + kvfree(buf); +err_alloc_buf: + cio->comp_len -= min_t(sector_t, cio->comp_len, len - rem); + if (!atomic_read(&cio->refcount)) { + kfree(cio); + return -ENOMEM; + } + /* Wait for submitted IOs to complete */ + return blkdev_copy_wait_completion(cio); +} + static inline ssize_t blkdev_copy_sanity_check( struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, @@ -350,9 +519,16 @@ ssize_t blkdev_copy_offload( if (ret) return ret; - if (blk_queue_copy(q_in) && blk_queue_copy(q_out)) + if (blk_queue_copy(q_in) && blk_queue_copy(q_out)) { ret = __blkdev_copy_offload(bdev_in, pos_in, bdev_out, pos_out, len, endio, private, gfp_mask); + if (ret < 0) + ret = 0; + } + + if (ret != len) + ret = __blkdev_copy_emulate(bdev_in, pos_in + ret, bdev_out, + pos_out + ret, len - ret, endio, private, gfp_mask); return ret; } diff --git a/block/blk-map.c b/block/blk-map.c index 44d74a30ddac..ceeb70a95fd1 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -363,7 +363,7 @@ static void bio_invalidate_vmalloc_pages(struct bio *bio) #endif } -static void bio_map_kern_endio(struct bio *bio) +void bio_map_kern_endio(struct bio *bio) { bio_invalidate_vmalloc_pages(bio); bio_uninit(bio); @@ -380,7 +380,7 @@ static void bio_map_kern_endio(struct bio *bio) * Map the kernel address into a bio suitable for io to a block * device. Returns an error pointer in case of error. */ -static struct bio *bio_map_kern(struct request_queue *q, void *data, +struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, gfp_t gfp_mask) { unsigned long kaddr = (unsigned long)data; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 96e986b37a29..69fe977afdc9 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1057,6 +1057,9 @@ ssize_t blkdev_copy_offload( struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); +struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, + gfp_t gfp_mask); +void bio_map_kern_endio(struct bio *bio); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Mon Jun 5 12:17:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13267303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E10AEC7EE2A for ; Mon, 5 Jun 2023 12:30:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233680AbjFEMaG (ORCPT ); Mon, 5 Jun 2023 08:30:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232894AbjFEM3o (ORCPT ); Mon, 5 Jun 2023 08:29:44 -0400 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0D67DB for ; Mon, 5 Jun 2023 05:29:41 -0700 (PDT) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20230605122939epoutp04fccc5341372cf4c511704c25766ac1d5~lwyCChi1J1942419424epoutp04E for ; Mon, 5 Jun 2023 12:29:39 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20230605122939epoutp04fccc5341372cf4c511704c25766ac1d5~lwyCChi1J1942419424epoutp04E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1685968179; bh=zUsRGAnFK4MYn0CIcdQ1QIVt7eT1JTxOky0uRhgN1RI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Wjawjtwu0p02VDfEH5pb9Evg72QC9c4JnldUlMpi4rduWz7RwH/GH2gKTDnKSYZJz cOp6cxGCUrZIqoVGzFgCjdq4A6kJnIJXc5HripHvsSQrFKdPKZb9ricKBQLS5sRvXq 9+VIArf03chUe3NLIq2nGW8b0V0egpAJHYADFhQk= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230605122938epcas5p44ce472b062119d34b73bea811b4495fa~lwyAwaV4G1407614076epcas5p4F; Mon, 5 Jun 2023 12:29:38 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.182]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4QZXww5kP3z4x9Pr; Mon, 5 Jun 2023 12:29:36 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 4B.F6.44881.035DD746; Mon, 5 Jun 2023 21:29:36 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230605122239epcas5p236a716d16fa1d1a2dfff856158deb5af~lwr6aRzdQ2830028300epcas5p2a; Mon, 5 Jun 2023 12:22:39 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230605122239epsmtrp2f271b064dcc3e26955698db8682ab4eb~lwr6X1iFE0879508795epsmtrp2W; Mon, 5 Jun 2023 12:22:39 +0000 (GMT) X-AuditID: b6c32a4a-c47ff7000001af51-78-647dd5307c53 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id D3.C5.28392.F83DD746; Mon, 5 Jun 2023 21:22:39 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230605122234epsmtip228b80cbb7dc1de7f7d1b0755231f14a3~lwr1f4M562245922459epsmtip2J; Mon, 5 Jun 2023 12:22:34 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, willy@infradead.org, hare@suse.de, djwong@kernel.org, bvanassche@acm.org, ming.lei@redhat.com, dlemoal@kernel.org, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v12 4/9] fs, block: copy_file_range for def_blk_ops for direct block device Date: Mon, 5 Jun 2023 17:47:20 +0530 Message-Id: <20230605121732.28468-5-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230605121732.28468-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TazBcZxjud846dmOY4xL5rFy221GDuKygnyBMaswZ2ozEn1TT6rLHpdjd 7K5KKm13o2EisSSadLqUxErdWluXGtdWVhC3iiIZWkGsGNnUdTJ0RNRa2vx73ud9n+e9fPMx casqUzYzQSijJUJ+EpfYw6hvd3J09Rj5QuAxo3dEmp5OHF3M3cBR5XgOgfTtywDdXPwHR7q2 TICGdBZo8rdA1Dqfb4JG2xox1FJ8HUPllR0Yar69hKHr2ocAzYyoMNQ65oJuZ5QwUEtrNwMN NRUQqOiHGVN05VEDgUq7XmFIm5eOoQadAqD69SIcVekXGOj+mD0a2OgyQetrBUTQAWpoOIxS TfQTVKNq3JQaeFzNoGrLnKmh/hSqpuIyQdWWfEU1j8oJSq3MM6Gy0+cJamlmjEEt/DpCUMq6 CkDV9qZRKzUHwy0jE/3jab6AlnBoYYxIkCCMC+CGRUS9G+Xt48Fz5fmid7gcIT+ZDuAGvxfu GpKQtHUiLuczflLKFhXOl0q57sf8JaIUGc2JF0llAVxaLEgSe4ndpPxkaYowzk1Iy47yPDw8 vbcKP0mMz9bcwcWvHM5Va0oJORg9lAVYTEh6QeWDGiIL7GFakc0A3nk2yzAGywA+v6E2MQYr ABZtTjF2JZMZP+0kmgCsyNjY0V/C4MDLKdMswGQSpAvs3WQaeBuyDIeK1b5tX5zswmHnkgI3 WFmTH8GnbZ3AgBmkAyxZM/RjMc3Jo/De2JVtI0i6w5wJSwPNIv3gXL8WN5ZYwu7vdNsT4eQh mP5LPm7wh+QfLDh5d4EwjhoMu6aLTYzYGj7rqjM1YjZcmW/dqUmF5d+UEUbx1wCqHqmAMREI L/Xk4IYhcNIJaprcjfQBeKOnCjM2toDZ6zrMyJvDhsJd/Bb8UXNrx98OPlxV7GAK1r3QmRqv pQRQrlBjuYCjem0h1WsLqf5vfQvgFcCOFkuT42ipt9hTSKf+984xouQasP1xnEMbwNTkopsW YEygBZCJc23Mm0LTBFbmAv75z2mJKEqSkkRLtcB76+DXcPbeGNHWzxPKonhevh5ePj4+Xr5H fHjcfeaOAd0xVmQcX0Yn0rSYluzqMCaLLcea9SH2/QNvaPShsZ3Mw3YFdX+ymvoPMjYGn0S8 +bQus9GWnZ8pyw88k+ueSh7+nROJ8vZfveYiP3vm+JH7eYknzGZ00fyPI02OdduWJPWGfdtc +NfQqWELQtHB+tK3peJ0sdOp8I6ILNGL9g8uZqYx5hQs0pNjM9HnNzvmHJYZjav19m0Bk5/y jkuDKwdL1e9PR3MWTs6dK33sGXAiRyl7uWoer9xU4E9az374/G5DSZ9ZdQytcYDhF/auX1ZF 7Fv7OcReMYDl2y1O4/vTrVdjN80KtRH++VcvNNmyBX1/DwapU8frL7sEve2n8vv+HjwZu9Q5 e5M8P6Itly9HodPDD7gMaTyf54xLpPx/AQAthoPBBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02Se0hTYRjG+845OzsK2nHdPlMzRpoYaa3bp86KwDpkVlCIFJajnaY1dW5Z 2QXNlaVimpbgpl103iYlatk0Z7YsM1u3peW6WbpCoja7WqbWHEH/Pfx+z/v89VI47w0xk4pL 2MPKE0RSPulMNN7ke8/PNR0WL/h4dTKqvXsbR+l5oziqeZlLog83PwNUaPuJo4G24wCZBlxR 3/UVSP9JzUG9bU0YainNx1B1zS0MXbswhKF8Qw9Alm4VhvTmeehChoZALfpOApmai0l0rsLC RdlPdSSq7BjDkKFAiSHdwBGAGkfO4ejSByuB7pg90IPRDg4aGS4mV3oxpifhjOq1kWSaVC+5 zINXdQTTUOXPmIzJTL02k2QaNKnMtd40kik7WcBhcpSfSGbIYiYYa2s3yZy8rAVMQ9dB5kv9 rI1uW5yFYlYat5eVBy6PcY7NqS3HZWM+++tqK8k00OudBZwoSC+GfRkXOVnAmeLROgDLuzq4 DuEOK0bbcUeeAqvH3nMdJSUG085YsSxAUSQ9D3aNU3Y+lb6MQ+OvAo79AKef47DI6GnPU+it sDBfOcEJ2gdqhssmsgsdDNvN2Vz7DqQDYe5rNzt2okPgoNGA2zHvb6XbKnS03WBn0QDhWPeG yitqPA/Qqv+U6j91HmBa4M7KFPGSeMVCmSCB3RegEMUrkhMkATsS4+vBxBf4++tAi9YWYAAY BQwAUjh/qkvz2oNinotYlHKAlSdulydLWYUBeFAEf4bLw6zO7TxaItrD7mZZGSv/ZzHKaWYa Ftl/1/uGdH2w8NQbT7NuUVDMlaMRukZJ+9hqkLohpOwhPsgpzMuWC1tiXqzhnJ62Ifrez225 O0w9KX2U+JCgu0cT8EiwJmewaqdNWLFkWP3tV2ytxJLU2vquuOGspu7U7nF3imd1C2t+rzuu r7y1NrD3u8+SE5lRcdPFNX3pJavMR8fv19m8jCfa3s2OijbYHheJ1EOaj0GZSaEjzVJ1yNy5 P/iTEzfLuMcEtz301zdFWHxb5Y9dO7g1kybNkYY1gYr+4HE/vwMZHqVeg56l/Snpm79UBQX2 JEt+L4h8ti/sa7jWN3Ra+a63XkvXLS86Q0WOLktNCnUNni4W8JW+ESX5fEIRK1roj8sVoj92 6+OjdAMAAA== X-CMS-MailID: 20230605122239epcas5p236a716d16fa1d1a2dfff856158deb5af X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230605122239epcas5p236a716d16fa1d1a2dfff856158deb5af References: <20230605121732.28468-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org For direct block device opened with O_DIRECT, use copy_file_range to issue device copy offload, and fallback to generic_copy_file_range incase device copy offload capability is absent. Modify checks to allow bdevs to use copy_file_range. Suggested-by: Ming Lei Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty --- block/blk-lib.c | 26 ++++++++++++++++++++++++++ block/fops.c | 20 ++++++++++++++++++++ fs/read_write.c | 7 +++++-- include/linux/blkdev.h | 4 ++++ 4 files changed, 55 insertions(+), 2 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index 99b65af8bfc1..31cfd5026367 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -534,6 +534,32 @@ ssize_t blkdev_copy_offload( } EXPORT_SYMBOL_GPL(blkdev_copy_offload); +/* Copy source offset from source block device to destination block + * device. Returns the length of bytes copied. + */ +ssize_t blkdev_copy_offload_failfast( + struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, + size_t len, gfp_t gfp_mask) +{ + struct request_queue *in_q = bdev_get_queue(bdev_in); + struct request_queue *out_q = bdev_get_queue(bdev_out); + ssize_t ret = 0; + + if (blkdev_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len)) + return 0; + + if (blk_queue_copy(in_q) && blk_queue_copy(out_q)) { + ret = __blkdev_copy_offload(bdev_in, pos_in, bdev_out, pos_out, + len, NULL, NULL, gfp_mask); + if (ret < 0) + return 0; + } + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_copy_offload_failfast); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/block/fops.c b/block/fops.c index f56811a925a0..9189f3239c9c 100644 --- a/block/fops.c +++ b/block/fops.c @@ -599,6 +599,25 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to) return ret; } +static ssize_t blkdev_copy_file_range(struct file *file_in, loff_t pos_in, + struct file *file_out, loff_t pos_out, + size_t len, unsigned int flags) +{ + struct block_device *in_bdev = I_BDEV(bdev_file_inode(file_in)); + struct block_device *out_bdev = I_BDEV(bdev_file_inode(file_out)); + ssize_t comp_len = 0; + + if ((file_in->f_iocb_flags & IOCB_DIRECT) && + (file_out->f_iocb_flags & IOCB_DIRECT)) + comp_len = blkdev_copy_offload_failfast(in_bdev, pos_in, + out_bdev, pos_out, len, GFP_KERNEL); + if (comp_len != len) + comp_len = generic_copy_file_range(file_in, pos_in + comp_len, + file_out, pos_out + comp_len, len - comp_len, flags); + + return comp_len; +} + #define BLKDEV_FALLOC_FL_SUPPORTED \ (FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE | \ FALLOC_FL_ZERO_RANGE | FALLOC_FL_NO_HIDE_STALE) @@ -692,6 +711,7 @@ const struct file_operations def_blk_fops = { .splice_read = filemap_splice_read, .splice_write = iter_file_splice_write, .fallocate = blkdev_fallocate, + .copy_file_range = blkdev_copy_file_range, }; static __init int blkdev_init(void) diff --git a/fs/read_write.c b/fs/read_write.c index b07de77ef126..d27148a2543f 100644 --- a/fs/read_write.c +++ b/fs/read_write.c @@ -1447,7 +1447,8 @@ static int generic_copy_file_checks(struct file *file_in, loff_t pos_in, return -EOVERFLOW; /* Shorten the copy to EOF */ - size_in = i_size_read(inode_in); + size_in = i_size_read(file_in->f_mapping->host); + if (pos_in >= size_in) count = 0; else @@ -1708,7 +1709,9 @@ int generic_file_rw_checks(struct file *file_in, struct file *file_out) /* Don't copy dirs, pipes, sockets... */ if (S_ISDIR(inode_in->i_mode) || S_ISDIR(inode_out->i_mode)) return -EISDIR; - if (!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) + + if ((!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) && + (!S_ISBLK(inode_in->i_mode) || !S_ISBLK(inode_out->i_mode))) return -EINVAL; if (!(file_in->f_mode & FMODE_READ) || diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 69fe977afdc9..a634768a2318 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1057,6 +1057,10 @@ ssize_t blkdev_copy_offload( struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); +ssize_t blkdev_copy_offload_failfast( + struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, + size_t len, gfp_t gfp_mask); struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, gfp_t gfp_mask); void bio_map_kern_endio(struct bio *bio); From patchwork Mon Jun 5 12:17:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13267304 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7F55C8300C for ; Mon, 5 Jun 2023 12:30:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232948AbjFEMaJ (ORCPT ); Mon, 5 Jun 2023 08:30:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233502AbjFEM3t (ORCPT ); Mon, 5 Jun 2023 08:29:49 -0400 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CF97EFA for ; Mon, 5 Jun 2023 05:29:44 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20230605122943epoutp02253eb990af5226b176c0fcfb08ffbcfe~lwyE8_cEs0553505535epoutp02f for ; Mon, 5 Jun 2023 12:29:43 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20230605122943epoutp02253eb990af5226b176c0fcfb08ffbcfe~lwyE8_cEs0553505535epoutp02f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1685968183; bh=oy1X2h64u1n5X0mEoxDQWm1StM8EqkhjUrEzZtW2Yo0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AqJC/qPmbDYM8jlQhrtMhvFXfHEItVPa7U9EkSB9QkRECGCZ5Ao2XfvmmN3FsKqUG FqfqWyskb86vaWBUsL4JyFROMfqK6leni5Q69pe42JmFe4sMFKfkYzBeHWH/mqLIjj 6aHs2GqjZ58YNMVrLTgb/9FOpQzDdOup9Q9BZUHA= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230605122941epcas5p43bbe345fc5ef80aa688da9862f9d2715~lwyDkFktU1407614076epcas5p4K; Mon, 5 Jun 2023 12:29:41 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.176]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4QZXx01WlSz4x9Pv; Mon, 5 Jun 2023 12:29:40 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id DF.F6.44881.435DD746; Mon, 5 Jun 2023 21:29:40 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20230605122310epcas5p4aaebfc26fe5377613a36fe50423cf494~lwsXrmMe23059130591epcas5p45; Mon, 5 Jun 2023 12:23:10 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230605122310epsmtrp232f613e843c1168fd0664cfa5fbb3d11~lwsXqVU8p0879508795epsmtrp2g; Mon, 5 Jun 2023 12:23:10 +0000 (GMT) X-AuditID: b6c32a4a-ea9fa7000001af51-85-647dd53474a3 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 69.C5.28392.EA3DD746; Mon, 5 Jun 2023 21:23:10 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230605122302epsmtip22383636a4f2611013bf69e260f4ff717~lwsPwmiL42527625276epsmtip2X; Mon, 5 Jun 2023 12:23:02 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, willy@infradead.org, hare@suse.de, djwong@kernel.org, bvanassche@acm.org, ming.lei@redhat.com, dlemoal@kernel.org, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Kanchan Joshi , =?utf-8?q?Javier_Gonz=C3=A1lez?= , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v12 5/9] nvme: add copy offload support Date: Mon, 5 Jun 2023 17:47:21 +0530 Message-Id: <20230605121732.28468-6-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230605121732.28468-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta1BUZRj2O2f37IItc7iUH5jBrDIkxGVhwQ8SKbXmhDExUj/MSVzhxG7A 7rbLeqOSi0BQXBcYXSggIW4OxEVcQIQWEUG5GIJJchthrLgvM0IB2S4Hyn/P+7zv817n5eIW 9RwbrkQaRSukogg+YcpqaN/r4Cwc/DLUreISgaq7b+MoLmMdR5Uj6QSabtcDlLvwF44m25IA Gpg0Q+OtfqhlLo+NHrU1YujGD1kYKq/swFBz0SKGnjxe4qCO57MEytINATQ1qMFQy7ATKkos ZqEbLV0sNNCUT6CCH6c46JuHWgKVdv6DIZ06HkPayViAGlYLcFQ1Pc9Cd4Z3or71TjZaXckn 3rKlBh4coTRjPQTVqBnhUH2jNSyqrsyRGuhRUbUVyQRVV3yBan4UQ1BX0tRsKjV+jqAWp4ZZ 1PzNQYJKq68AVN3daGqp9rVA84/D94tpUSitsKOlIbJQiTTMl38kKPhQsKeXm8BZ4I328e2k okjal3/4/UDndyURhmXx7U6LIlQGKlCkVPJdD+xXyFRRtJ1Ypozy5dPy0Ai5UO6iFEUqVdIw Fykd5SNwc3P3NASeDBfXD3whf3D6bNJ8DDsGrAanABMuJIVwueYqYcQWZDOA6kXXFGBqwHoA 66//ymaMJQDHMhvwLcWd2zqccTQBOJPZQzBGAgZj+0s4KYDLJUgnePc518hbkWU4jF2+xzIa OFnFgtmla2xjKktyHywY/Q0zClikPXzWvUHzSB/YMtZIGGlIusL0MXMjbUK+Cf/o0eFMiDns ujzJMmKctIXx1/I2GoKk3gTq256xGe1hWFqpYpq2hH921nMYbAOX5loIBp+B5dllBKO9CKDm oQYwDj+Y0J2OG/Pg5F5Y3eTK0LtgTncVxtQ1g6mrkxjD86D2+y28G16tLtzMbw2HlmM3MQVr +tc2d5UGYFzDMpYB7DQvzKN5YR7N/6ULAV4BrGm5MjKMVnrK3aX0mf9uHCKLrAUb7+PorwUT 4wsuOoBxgQ5ALs634jX5R4da8EJF587TClmwQhVBK3XA07DuTNzm5RCZ4f+kUcECobeb0MvL S+jt4SXg7+A5+HaFWJBhoig6nKbltGJLh3FNbGKwS7/Un4o5QCVva3xpQOM/+5PFbMkr6qle 3dKJCXsr96BbufafqUbnb76e+N0Tu8Y9T+/huSPOWanDM2/IShd8Pj1kynN+b0jd34Hf97I2 sb31d7H4eHFGkejr7WqTVm1IQGW7sunbfJ6fXlC4ZmZ70GNXlf+2wZQCeuKC09up9i4r3J3l saeOyc4G9eb5bD9WEv17588FeZKZc2T7+lHhRxenQ6o1fW4HA5aSQfXjz69JOOYjzRxJftXx zEB94FetO3xLqA/qxGKPwndWrkd+4q5/tfRD64jMzj3ns09eScyJO5o4rN6dUzp+P8Aj60Tg oEMrFpab4CnpHeZ2Wl7WJnGetk9P8llKsUjgiCuUon8B5e2cnMcEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpgleLIzCtJLcpLzFFi42LZdlhJXnfd5doUg9VTdSzWnzrGbNE04S+z xeq7/WwWrw9/YrSY9uEns8WTA+2MFpef8Fk82G9vsffdbFaLmwd2MlnsWTSJyWLl6qNMFrsX fmSyeHznM7vF0f9v2SwmHbrGaPH06iwmi723tC0Wti1hsdiz9ySLxeVdc9gs5i97ym7RfX0H m8Xy4/+YLA5Nbmay2PGkkdFi2+/5zBbrXr9nsThxS9ri/N/jrBa/f8xhc5D3uHzF22PW/bNs Hjtn3WX3OH9vI4vH5hVaHpfPlnpsWtXJ5rF5Sb3H7psNbB6L+yazevQ2v2Pz+Pj0FovH+31X 2Tz6tqxi9Nh8utrj8ya5AMEoLpuU1JzMstQifbsErowtl2sKrpRVtL9vYG1g/B3fxcjJISFg InHi2CHmLkYuDiGBHYwSj07/YYJISEos+3uEGcIWllj57zk7RFEzk8SWM4vZuhg5ONgEtCVO /+cAiYsIbGGWOPtrMiuIwyxwiEXizseFjCDdwgLmEvPv3WYCaWARUJX4dooVJMwrYCWx9/5O sDkSAvoS/fcFQcKcAtYSL8+CHMQBtMtK4up7G4hqQYmTM5+wgISZBdQl1s8TAgkzC8hLNG+d zTyBUXAWkqpZCFWzkFQtYGRexSiZWlCcm55bbFhglJdarlecmFtcmpeul5yfu4kRnEi0tHYw 7ln1Qe8QIxMH4yFGCQ5mJRHeXV7VKUK8KYmVValF+fFFpTmpxYcYpTlYlMR5L3SdjBcSSE8s Sc1OTS1ILYLJMnFwSjUw8Rdad8syNddu7/R6ofbP13zXnITvPev3VDMULDp+5oa9+NItq03T 3rYz6Gkx3pafz7tMMUi8qat1hdeaFwzJvkqFUUqdIjqe1beiTzXcZCw3nHaHf863Oq7ZGgZe 3xcqSvT6ScoFL4z497/iToH1xNoy/k2lxtddcnJiDHMc57n0nm6Q9y+qL5z0duu7py23PJgN frMI/J+9z/TzoadKFV6vsl/KCUofjf3jtFpT9aRP8MxTz9hvv2OLUjtzUHhVT0KW2vTbEk+0 fCxD7A8vcjhSdrn43+KHfWqckt9T+rSbLj3b4OzhE9L0wiWnNSyxSVdoVRLfxU8q1Ynv8puW Ksi+lvbr1LaRTF2RqcRSnJFoqMVcVJwIAEBOgpuTAwAA X-CMS-MailID: 20230605122310epcas5p4aaebfc26fe5377613a36fe50423cf494 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230605122310epcas5p4aaebfc26fe5377613a36fe50423cf494 References: <20230605121732.28468-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org For device supporting native copy, nvme driver receives read and write request with BLK_COPY op flags. For read request the nvme driver populates the payload with source information. For write request the driver converts it to nvme copy command using the source information in the payload and submits to the device. current design only supports single source range. This design is courtesy Mikulas Patocka's token based copy trace event support for nvme_copy_cmd. Set the device copy limits to queue limits. Signed-off-by: Kanchan Joshi Signed-off-by: Nitesh Shetty Signed-off-by: Javier González Signed-off-by: Anuj Gupta --- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 103 +++++++++++++++++++++++++++++++++- drivers/nvme/host/fc.c | 5 ++ drivers/nvme/host/nvme.h | 7 +++ drivers/nvme/host/pci.c | 27 ++++++++- drivers/nvme/host/rdma.c | 7 +++ drivers/nvme/host/tcp.c | 16 ++++++ drivers/nvme/host/trace.c | 19 +++++++ include/linux/nvme.h | 43 +++++++++++++- 9 files changed, 220 insertions(+), 8 deletions(-) diff --git a/drivers/nvme/host/constants.c b/drivers/nvme/host/constants.c index 5e4f8848dce0..311ad67e9cf3 100644 --- a/drivers/nvme/host/constants.c +++ b/drivers/nvme/host/constants.c @@ -19,6 +19,7 @@ static const char * const nvme_ops[] = { [nvme_cmd_resv_report] = "Reservation Report", [nvme_cmd_resv_acquire] = "Reservation Acquire", [nvme_cmd_resv_release] = "Reservation Release", + [nvme_cmd_copy] = "Copy Offload", [nvme_cmd_zone_mgmt_send] = "Zone Management Send", [nvme_cmd_zone_mgmt_recv] = "Zone Management Receive", [nvme_cmd_zone_append] = "Zone Append", diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 1715a508496c..ce1fec07dda6 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -763,6 +763,77 @@ static inline void nvme_setup_flush(struct nvme_ns *ns, cmnd->common.nsid = cpu_to_le32(ns->head->ns_id); } +static inline void nvme_setup_copy_read(struct nvme_ns *ns, struct request *req) +{ + struct bio *bio = req->bio; + struct nvme_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + + token->subsys = "nvme"; + token->ns = ns; + token->src_sector = bio->bi_iter.bi_sector; + token->sectors = bio->bi_iter.bi_size >> 9; +} + +static inline blk_status_t nvme_setup_copy_write(struct nvme_ns *ns, + struct request *req, struct nvme_command *cmnd) +{ + struct nvme_copy_range *range = NULL; + struct bio *bio = req->bio; + struct nvme_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + sector_t src_sector, dst_sector, n_sectors; + u64 src_lba, dst_lba, n_lba; + unsigned short nr_range = 1; + u16 control = 0; + + if (unlikely(memcmp(token->subsys, "nvme", 4))) + return BLK_STS_NOTSUPP; + if (unlikely(token->ns != ns)) + return BLK_STS_NOTSUPP; + + src_sector = token->src_sector; + dst_sector = bio->bi_iter.bi_sector; + n_sectors = token->sectors; + if (WARN_ON(n_sectors != bio->bi_iter.bi_size >> 9)) + return BLK_STS_NOTSUPP; + + src_lba = nvme_sect_to_lba(ns, src_sector); + dst_lba = nvme_sect_to_lba(ns, dst_sector); + n_lba = nvme_sect_to_lba(ns, n_sectors); + + if (WARN_ON(!n_lba)) + return BLK_STS_NOTSUPP; + + if (req->cmd_flags & REQ_FUA) + control |= NVME_RW_FUA; + + if (req->cmd_flags & REQ_FAILFAST_DEV) + control |= NVME_RW_LR; + + memset(cmnd, 0, sizeof(*cmnd)); + cmnd->copy.opcode = nvme_cmd_copy; + cmnd->copy.nsid = cpu_to_le32(ns->head->ns_id); + cmnd->copy.sdlba = cpu_to_le64(dst_lba); + + range = kmalloc_array(nr_range, sizeof(*range), + GFP_ATOMIC | __GFP_NOWARN); + if (!range) + return BLK_STS_RESOURCE; + + range[0].slba = cpu_to_le64(src_lba); + range[0].nlb = cpu_to_le16(n_lba - 1); + + cmnd->copy.nr_range = 0; + + req->special_vec.bv_page = virt_to_page(range); + req->special_vec.bv_offset = offset_in_page(range); + req->special_vec.bv_len = sizeof(*range) * nr_range; + req->rq_flags |= RQF_SPECIAL_PAYLOAD; + + cmnd->copy.control = cpu_to_le16(control); + + return BLK_STS_OK; +} + static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req, struct nvme_command *cmnd) { @@ -997,10 +1068,16 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req) ret = nvme_setup_discard(ns, req, cmd); break; case REQ_OP_READ: - ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_read); + if (unlikely(req->cmd_flags & REQ_COPY)) + nvme_setup_copy_read(ns, req); + else + ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_read); break; case REQ_OP_WRITE: - ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_write); + if (unlikely(req->cmd_flags & REQ_COPY)) + ret = nvme_setup_copy_write(ns, req, cmd); + else + ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_write); break; case REQ_OP_ZONE_APPEND: ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_zone_append); @@ -1742,6 +1819,26 @@ static void nvme_config_discard(struct gendisk *disk, struct nvme_ns *ns) blk_queue_max_write_zeroes_sectors(queue, UINT_MAX); } +static void nvme_config_copy(struct gendisk *disk, struct nvme_ns *ns, + struct nvme_id_ns *id) +{ + struct nvme_ctrl *ctrl = ns->ctrl; + struct request_queue *q = disk->queue; + + if (!(ctrl->oncs & NVME_CTRL_ONCS_COPY)) { + blk_queue_max_copy_sectors_hw(q, 0); + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + return; + } + + /* setting copy limits */ + if (blk_queue_flag_test_and_set(QUEUE_FLAG_COPY, q)) + return; + + blk_queue_max_copy_sectors_hw(q, + nvme_lba_to_sect(ns, le16_to_cpu(id->mssrl))); +} + static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b) { return uuid_equal(&a->uuid, &b->uuid) && @@ -1941,6 +2038,7 @@ static void nvme_update_disk_info(struct gendisk *disk, set_capacity_and_notify(disk, capacity); nvme_config_discard(disk, ns); + nvme_config_copy(disk, ns, id); blk_queue_max_write_zeroes_sectors(disk->queue, ns->ctrl->max_zeroes_sectors); } @@ -5244,6 +5342,7 @@ static inline void _nvme_check_size(void) BUILD_BUG_ON(sizeof(struct nvme_download_firmware) != 64); BUILD_BUG_ON(sizeof(struct nvme_format_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_dsm_cmd) != 64); + BUILD_BUG_ON(sizeof(struct nvme_copy_command) != 64); BUILD_BUG_ON(sizeof(struct nvme_write_zeroes_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_abort_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_get_log_page_command) != 64); diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 691f2df574ce..702965f5047f 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -2807,6 +2807,11 @@ nvme_fc_queue_rq(struct blk_mq_hw_ctx *hctx, if (ret) return ret; + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_end_request(rq, BLK_STS_OK); + return BLK_STS_OK; + } /* * nvme core doesn't quite treat the rq opaquely. Commands such * as WRITE ZEROES will return a non-zero rq payload_bytes yet diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 8f07aee68d59..a10bb80faa9c 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -508,6 +508,13 @@ struct nvme_ns { }; +struct nvme_copy_token { + char *subsys; + struct nvme_ns *ns; + sector_t src_sector; + sector_t sectors; +}; + /* NVMe ns supports metadata actions by the controller (generate/strip) */ static inline bool nvme_ns_has_pi(struct nvme_ns *ns) { diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 492f319ebdf3..c231565b69cf 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -495,16 +495,19 @@ static inline void nvme_sq_copy_cmd(struct nvme_queue *nvmeq, nvmeq->sq_tail = 0; } -static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +static inline void nvme_commit_sq_db(struct nvme_queue *nvmeq) { - struct nvme_queue *nvmeq = hctx->driver_data; - spin_lock(&nvmeq->sq_lock); if (nvmeq->sq_tail != nvmeq->last_sq_tail) nvme_write_sq_db(nvmeq, true); spin_unlock(&nvmeq->sq_lock); } +static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +{ + nvme_commit_sq_db(hctx->driver_data); +} + static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req, int nseg) { @@ -848,6 +851,12 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, struct request *req) if (ret) return ret; + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_start_request(req); + return BLK_STS_OK; + } + if (blk_rq_nr_phys_segments(req)) { ret = nvme_map_data(dev, req, &iod->cmd); if (ret) @@ -894,6 +903,18 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, ret = nvme_prep_rq(dev, req); if (unlikely(ret)) return ret; + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_set_request_complete(req); + blk_mq_end_request(req, BLK_STS_OK); + /* Commit the sq if copy read was the last req in the list, + * as copy read deoesn't update sq db + */ + if (bd->last) + nvme_commit_sq_db(nvmeq); + return ret; + } + spin_lock(&nvmeq->sq_lock); nvme_sq_copy_cmd(nvmeq, &iod->cmd); nvme_write_sq_db(nvmeq, bd->last); diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 0eb79696fb73..be1d20ac8bb0 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -2038,6 +2038,13 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, nvme_start_request(rq); + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_end_request(rq, BLK_STS_OK); + ret = BLK_STS_OK; + goto unmap_qe; + } + if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) && queue->pi_support && (c->common.opcode == nvme_cmd_write || diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index bf0230442d57..5ba1bb35c557 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -2373,6 +2373,11 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns, if (ret) return ret; + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + return BLK_STS_OK; + } + req->state = NVME_TCP_SEND_CMD_PDU; req->status = cpu_to_le16(NVME_SC_SUCCESS); req->offset = 0; @@ -2441,6 +2446,17 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx, nvme_start_request(rq); + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_set_request_complete(rq); + blk_mq_end_request(rq, BLK_STS_OK); + /* if copy read is the last req queue tcp reqs */ + if (bd->last && nvme_tcp_queue_more(queue)) + queue_work_on(queue->io_cpu, nvme_tcp_wq, + &queue->io_work); + return ret; + } + nvme_tcp_queue_request(req, true, bd->last); return BLK_STS_OK; diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c index 1c36fcedea20..da4a7494e5a7 100644 --- a/drivers/nvme/host/trace.c +++ b/drivers/nvme/host/trace.c @@ -150,6 +150,23 @@ static const char *nvme_trace_read_write(struct trace_seq *p, u8 *cdw10) return ret; } +static const char *nvme_trace_copy(struct trace_seq *p, u8 *cdw10) +{ + const char *ret = trace_seq_buffer_ptr(p); + u64 slba = get_unaligned_le64(cdw10); + u8 nr_range = get_unaligned_le16(cdw10 + 8); + u16 control = get_unaligned_le16(cdw10 + 10); + u32 dsmgmt = get_unaligned_le32(cdw10 + 12); + u32 reftag = get_unaligned_le32(cdw10 + 16); + + trace_seq_printf(p, + "slba=%llu, nr_range=%u, ctrl=0x%x, dsmgmt=%u, reftag=%u", + slba, nr_range, control, dsmgmt, reftag); + trace_seq_putc(p, 0); + + return ret; +} + static const char *nvme_trace_dsm(struct trace_seq *p, u8 *cdw10) { const char *ret = trace_seq_buffer_ptr(p); @@ -243,6 +260,8 @@ const char *nvme_trace_parse_nvm_cmd(struct trace_seq *p, return nvme_trace_zone_mgmt_send(p, cdw10); case nvme_cmd_zone_mgmt_recv: return nvme_trace_zone_mgmt_recv(p, cdw10); + case nvme_cmd_copy: + return nvme_trace_copy(p, cdw10); default: return nvme_trace_common(p, cdw10); } diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 182b6d614eb1..bbd877111b57 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -337,7 +337,7 @@ struct nvme_id_ctrl { __u8 nvscc; __u8 nwpc; __le16 acwu; - __u8 rsvd534[2]; + __le16 ocfs; __le32 sgls; __le32 mnan; __u8 rsvd544[224]; @@ -365,6 +365,7 @@ enum { NVME_CTRL_ONCS_WRITE_ZEROES = 1 << 3, NVME_CTRL_ONCS_RESERVATIONS = 1 << 5, NVME_CTRL_ONCS_TIMESTAMP = 1 << 6, + NVME_CTRL_ONCS_COPY = 1 << 8, NVME_CTRL_VWC_PRESENT = 1 << 0, NVME_CTRL_OACS_SEC_SUPP = 1 << 0, NVME_CTRL_OACS_NS_MNGT_SUPP = 1 << 3, @@ -414,7 +415,10 @@ struct nvme_id_ns { __le16 npdg; __le16 npda; __le16 nows; - __u8 rsvd74[18]; + __le16 mssrl; + __le32 mcl; + __u8 msrc; + __u8 rsvd91[11]; __le32 anagrpid; __u8 rsvd96[3]; __u8 nsattr; @@ -831,6 +835,7 @@ enum nvme_opcode { nvme_cmd_resv_report = 0x0e, nvme_cmd_resv_acquire = 0x11, nvme_cmd_resv_release = 0x15, + nvme_cmd_copy = 0x19, nvme_cmd_zone_mgmt_send = 0x79, nvme_cmd_zone_mgmt_recv = 0x7a, nvme_cmd_zone_append = 0x7d, @@ -854,7 +859,8 @@ enum nvme_opcode { nvme_opcode_name(nvme_cmd_resv_release), \ nvme_opcode_name(nvme_cmd_zone_mgmt_send), \ nvme_opcode_name(nvme_cmd_zone_mgmt_recv), \ - nvme_opcode_name(nvme_cmd_zone_append)) + nvme_opcode_name(nvme_cmd_zone_append), \ + nvme_opcode_name(nvme_cmd_copy)) @@ -1031,6 +1037,36 @@ struct nvme_dsm_range { __le64 slba; }; +struct nvme_copy_command { + __u8 opcode; + __u8 flags; + __u16 command_id; + __le32 nsid; + __u64 rsvd2; + __le64 metadata; + union nvme_data_ptr dptr; + __le64 sdlba; + __u8 nr_range; + __u8 rsvd12; + __le16 control; + __le16 rsvd13; + __le16 dspec; + __le32 ilbrt; + __le16 lbat; + __le16 lbatm; +}; + +struct nvme_copy_range { + __le64 rsvd0; + __le64 slba; + __le16 nlb; + __le16 rsvd18; + __le32 rsvd20; + __le32 eilbrt; + __le16 elbat; + __le16 elbatm; +}; + struct nvme_write_zeroes_cmd { __u8 opcode; __u8 flags; @@ -1792,6 +1828,7 @@ struct nvme_command { struct nvme_download_firmware dlfw; struct nvme_format_cmd format; struct nvme_dsm_cmd dsm; + struct nvme_copy_command copy; struct nvme_write_zeroes_cmd write_zeroes; struct nvme_zone_mgmt_send_cmd zms; struct nvme_zone_mgmt_recv_cmd zmr; From patchwork Mon Jun 5 12:17:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13267305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10C92C87FDC for ; Mon, 5 Jun 2023 12:30:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233148AbjFEMaL (ORCPT ); Mon, 5 Jun 2023 08:30:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233612AbjFEM35 (ORCPT ); Mon, 5 Jun 2023 08:29:57 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 708C2187 for ; Mon, 5 Jun 2023 05:29:47 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230605122945epoutp0197c4703819b8e8653f293f27b3ad67a2~lwyHVPlJM1683116831epoutp01I for ; Mon, 5 Jun 2023 12:29:45 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230605122945epoutp0197c4703819b8e8653f293f27b3ad67a2~lwyHVPlJM1683116831epoutp01I DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1685968185; bh=TBNd/dnNsAHe6X0oTWIBGxw9WAVD6NR0t+TrHecNZK0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uKMdLDoOo+lJd7GtbnZ03O+7Rc5YKDjiPorKd8BjZbeuCHDRAk7YoBlXRbcoNxao3 PbVJzgfpnJJLbFH4J/bYWzwqvFUgn48R48gzfWZNKo5nJzf4ukvVwzYuTK7tLDgXmJ ITv1tKvna3Zjausswket4K3CvqMfBKzaDZhkRqT4= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230605122944epcas5p3f477ea1077cb5e5f3b040e4d898986b9~lwyGrZo892911529115epcas5p3C; Mon, 5 Jun 2023 12:29:44 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.182]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4QZXx34c2cz4x9Pp; Mon, 5 Jun 2023 12:29:43 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 98.5D.16380.735DD746; Mon, 5 Jun 2023 21:29:43 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230605122330epcas5p18a452356d5bc39b852b85304bac5b0cc~lwsqKi6O71025410254epcas5p1H; Mon, 5 Jun 2023 12:23:30 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230605122330epsmtrp14a9fa7dbfc34f25df40badb6d08bbe6c~lwsqJfPPq1611816118epsmtrp1E; Mon, 5 Jun 2023 12:23:30 +0000 (GMT) X-AuditID: b6c32a4b-7dffd70000013ffc-fd-647dd5371884 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 72.D5.28392.2C3DD746; Mon, 5 Jun 2023 21:23:30 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230605122324epsmtip24cff4d84328f230681a4d89417d1d39f~lwskbp6ph2245922459epsmtip2P; Mon, 5 Jun 2023 12:23:24 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, willy@infradead.org, hare@suse.de, djwong@kernel.org, bvanassche@acm.org, ming.lei@redhat.com, dlemoal@kernel.org, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Anuj Gupta , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v12 6/9] nvmet: add copy command support for bdev and file ns Date: Mon, 5 Jun 2023 17:47:22 +0530 Message-Id: <20230605121732.28468-7-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230605121732.28468-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTdxTH/d3bXgpb3QU0/oDgSJljQngplB9ifQTi7lYWkS1mc1uwo1dg lLZpCyqOyEsFlKdzcQV5CPIeZOBYec1a3tVGoQMiG48xyEQ2ngt1EHWUi5v/fc/5nc85Od9f Dge3qbKw50RJVbRCKpLwCCtWU8duF3e/wQSxV16xA6rXd+MoOecZjmpGswk027EE0DcL/+Bo SnsJIOPUVjRx5yBqn8tno0faZgy13czDUFVNF4ZaSxYxlKcbAmh6UI2h9hE3VHKxjIXa2vtY yNhSQKCi8mkLdHlYQ6CKnucY0l1NwZBmKgmgprUiHNXNzrNQ74gDevCsh43WnhYQhxwp489C Sj1uIKhm9agF9WDsexbVWOlKGQ2xVEN1OkE1lp2nWh8lElRp1lU2lZkyR1CL0yMsav6nQYLK ul0NqMZ756jlhp0h1iei90fSIjGtcKKl4TJxlDRCwBN+GBYY5sv38nb39kd+PCepKIYW8IKC Q9yPREnWLeI5xYkkseupEJFSyfM8sF8hi1XRTpEypUrAo+ViidxH7qEUxShjpREeUlq1z9vL a4/veuHJ6MiBH+fZ8uXDZzonM4hEYOJnAEsOJH3gr2mrWAaw4tiQrQDeLf1tM1gCMKfiIZsJ lgFczR0jMgBnAxkuOGqmbcgWAP/q3MHUXMBg/ZW0jRqCdIP3XnDM+W1kJQ6TTPdZ5gAne3DY vZiEm2lb8ihc0ZazzJpF7oIrpSsbMJfcB7sr32FmecLscWtzhSUZAGcMug2SS1rDvm+nNkic fBOm/JCPm9tDcsgStlbPWjBsECy5wWO2tIVPem5bMNoezmRf3NSnYdXXlQTDpgKoHlYD5uEg vKDPxs19cHI3rG/xZNKO8Jq+DmPmboWZa1MYk+dCTeFL7Qxr64sJRtvBIVPSpm0ULDB9yliV BWBeew0rBzipX1lH/co66v8nFwO8GtjRcmVMBK30le+V0qf/++JwWUwD2LgZV6EGTE4seOgA xgE6ADk4bxu35f1zYhuuWHQ2nlbIwhSxElqpA77rbufi9tvDZetHJ1WFefv4e/nw+Xwf/718 b94OrougL9yGjBCp6GialtOKlxzGsbRPxCa9svsH5hLTVaSzvrGov9lP8Li+2OVPQVshlUDj x6XpUtMl4Red8vbar14/HveZJt31QOYhcMPxpiatBhuxbqECftEGVPvn1OkfGyqO/WH70Rsp M7LQs2rVJ3euV1S994SfKkx+Lhm7264qd2zWbZk+tfPagLjD9oguqNk5scKQtH2LsWhi7WR3 2/Uup4fC5UDn7zTsv1d7x1N3SZra5qdD+yNH3w0tW0nwyxjRenygSe7NZ8dn3NrzO3dK/bTw zJLhlFg81KMNaqBO3DKej7OK91sLzLUKNpVKHWyj7ruVWJV+Ht34IllfO3T4Ne6XXXYDbseE huDUtxbsZmcE3Lc/vsxjKSNF3q64Qin6Fz4g3ma8BAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrPIsWRmVeSWpSXmKPExsWy7bCSvO6hy7UpBv+/KFqsP3WM2aJpwl9m i9V3+9ksXh/+xGgx7cNPZosnB9oZLS4/4bN4sN/eYu+72awWNw/sZLLYs2gSk8XK1UeZLHYv /MhkMenQNUaLp1dnMVnsvaVtsbBtCYvFnr0nWSwu75rDZjF/2VN2i+7rO9gslh//x2RxaHIz k8WOJ42MFtt+z2e2WPf6PYvFiVvSFuf/Hme1+P1jDpuDrMflK94es+6fZfPYOesuu8f5extZ PDav0PK4fLbUY9OqTjaPzUvqPXbfbGDzWNw3mdWjt/kdm8fHp7dYPN7vu8rm0bdlFaPH5tPV Hp83yQUIRnHZpKTmZJalFunbJXBlXNr+nrXgs2PFkUddbA2M3826GDk4JARMJK7P8e9i5OIQ EtjBKHHg2EqWLkZOoLikxLK/R5ghbGGJlf+es0MUNTNJnHs+gQmkmU1AW+L0fw6QuIjAFmaJ s78ms4I0MAvcZpaYeVYGxBYW8JXYcKkHbCiLgKrEt8Xf2EB6eQWsJI6t0IC4QV+i/74gSAWn gLXEy7OHmEHCQkAVV9/bgIR5BQQlTs58wgIxXF6ieets5gmMArOQpGYhSS1gZFrFKJlaUJyb nltsWGCUl1quV5yYW1yal66XnJ+7iRGcBrS0djDuWfVB7xAjEwfjIUYJDmYlEd5dXtUpQrwp iZVVqUX58UWlOanFhxilOViUxHkvdJ2MFxJITyxJzU5NLUgtgskycXBKNTA1Ooc1vbZnZZ97 5FNaTnzGzXNnLh5YHHZ9/UH5c4e/rKuefvUdg5x/gDDv/Nl3cmTe+apP+3NQW2YTZ+DqRJvn dY7P3GclmZ/T5N7/Nl+4PGjnotaH60Nu/Fe+5VRxXuqa/drw2ONhzmtP7a2YmfvOXGhKdZLx s4u5t8w3sl4MPvFX6ssi1dyUx19Slrcv02aRXn3z1Bm/Z9975jZpS+3Z5T5DyiyP/555d4HN udfmCkJ/+pVchHa9YZ8v5tdSdfanx4Pq/ddC5bdW7Kmf2mC0/5VZxM9QZXZ115ig/qx1Mzz+ V9ZJl/VN18mcc3WzT+z5zskVP2KFVV7/Tsrtin3TJ/O5ME/xjszLF5beAbuUWIozEg21mIuK EwED+IwNcgMAAA== X-CMS-MailID: 20230605122330epcas5p18a452356d5bc39b852b85304bac5b0cc X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230605122330epcas5p18a452356d5bc39b852b85304bac5b0cc References: <20230605121732.28468-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Add support for handling nvme_cmd_copy command on target. For bdev-ns we call into blkdev_issue_copy, which the block layer completes by a offloaded copy request to backend bdev or by emulating the request. For file-ns we call vfs_copy_file_range to service our request. Currently target always shows copy capability by setting NVME_CTRL_ONCS_COPY in controller ONCS. loop target has copy support, which can be used to test copy offload. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- drivers/nvme/target/admin-cmd.c | 9 ++++- drivers/nvme/target/io-cmd-bdev.c | 62 +++++++++++++++++++++++++++++++ drivers/nvme/target/io-cmd-file.c | 52 ++++++++++++++++++++++++++ drivers/nvme/target/loop.c | 6 +++ drivers/nvme/target/nvmet.h | 1 + 5 files changed, 128 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c index 39cb570f833d..8e644b8ec0fd 100644 --- a/drivers/nvme/target/admin-cmd.c +++ b/drivers/nvme/target/admin-cmd.c @@ -433,8 +433,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) id->nn = cpu_to_le32(NVMET_MAX_NAMESPACES); id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES); id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM | - NVME_CTRL_ONCS_WRITE_ZEROES); - + NVME_CTRL_ONCS_WRITE_ZEROES | NVME_CTRL_ONCS_COPY); /* XXX: don't report vwc if the underlying device is write through */ id->vwc = NVME_CTRL_VWC_PRESENT; @@ -536,6 +535,12 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req) if (req->ns->bdev) nvmet_bdev_set_limits(req->ns->bdev, id); + else { + id->msrc = (__force u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16(BIO_MAX_VECS << + (PAGE_SHIFT - SECTOR_SHIFT)); + id->mcl = cpu_to_le32(le16_to_cpu(id->mssrl)); + } /* * We just provide a single LBA format that matches what the diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c index c2d6cea0236b..92b5accf0743 100644 --- a/drivers/nvme/target/io-cmd-bdev.c +++ b/drivers/nvme/target/io-cmd-bdev.c @@ -46,6 +46,18 @@ void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id) id->npda = id->npdg; /* NOWS = Namespace Optimal Write Size */ id->nows = to0based(bdev_io_opt(bdev) / bdev_logical_block_size(bdev)); + + if (bdev_max_copy_sectors(bdev)) { + id->msrc = id->msrc; + id->mssrl = cpu_to_le16((bdev_max_copy_sectors(bdev) << + SECTOR_SHIFT) / bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32((__force u32)id->mssrl); + } else { + id->msrc = (__force u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16((BIO_MAX_VECS << PAGE_SHIFT) / + bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32((__force u32)id->mssrl); + } } void nvmet_bdev_ns_disable(struct nvmet_ns *ns) @@ -184,6 +196,21 @@ static void nvmet_bio_done(struct bio *bio) nvmet_req_bio_put(req, bio); } +static void nvmet_bdev_copy_end_io(void *private, int comp_len) +{ + struct nvmet_req *req = (struct nvmet_req *)private; + u16 status; + + if (comp_len == req->copy_len) { + req->cqe->result.u32 = cpu_to_le32(1); + status = errno_to_nvme_status(req, 0); + } else { + req->cqe->result.u32 = cpu_to_le32(0); + status = errno_to_nvme_status(req, (__force u16)BLK_STS_IOERR); + } + nvmet_req_complete(req, status); +} + #ifdef CONFIG_BLK_DEV_INTEGRITY static int nvmet_bdev_alloc_bip(struct nvmet_req *req, struct bio *bio, struct sg_mapping_iter *miter) @@ -450,6 +477,37 @@ static void nvmet_bdev_execute_write_zeroes(struct nvmet_req *req) } } +/* At present we handle only one range entry, since copy offload is aligned with + * copy_file_range, only one entry is passed from block layer. + */ +static void nvmet_bdev_execute_copy(struct nvmet_req *req) +{ + struct nvme_copy_range range; + struct nvme_command *cmd = req->cmd; + ssize_t ret; + u16 status; + + status = nvmet_copy_from_sgl(req, 0, &range, sizeof(range)); + if (status) + goto out; + + ret = blkdev_copy_offload(req->ns->bdev, + le64_to_cpu(cmd->copy.sdlba) << req->ns->blksize_shift, + req->ns->bdev, + le64_to_cpu(range.slba) << req->ns->blksize_shift, + (le16_to_cpu(range.nlb) + 1) << req->ns->blksize_shift, + nvmet_bdev_copy_end_io, (void *)req, GFP_KERNEL); + if (ret) { + req->cqe->result.u32 = cpu_to_le32(0); + status = blk_to_nvme_status(req, BLK_STS_IOERR); + goto out; + } + + return; +out: + nvmet_req_complete(req, errno_to_nvme_status(req, status)); +} + u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) { switch (req->cmd->common.opcode) { @@ -468,6 +526,10 @@ u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_bdev_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_bdev_execute_copy; + return 0; + default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c index 2d068439b129..f61aa834f7a5 100644 --- a/drivers/nvme/target/io-cmd-file.c +++ b/drivers/nvme/target/io-cmd-file.c @@ -322,6 +322,49 @@ static void nvmet_file_dsm_work(struct work_struct *w) } } +static void nvmet_file_copy_work(struct work_struct *w) +{ + struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); + int nr_range = req->cmd->copy.nr_range + 1; + u16 status = 0; + int src, id; + ssize_t len, ret; + loff_t pos; + + pos = le64_to_cpu(req->cmd->copy.sdlba) << req->ns->blksize_shift; + if (unlikely(pos + req->transfer_len > req->ns->size)) { + nvmet_req_complete(req, errno_to_nvme_status(req, -ENOSPC)); + return; + } + + for (id = 0 ; id < nr_range; id++) { + struct nvme_copy_range range; + + status = nvmet_copy_from_sgl(req, id * sizeof(range), &range, + sizeof(range)); + if (status) + goto out; + + src = (le64_to_cpu(range.slba) << (req->ns->blksize_shift)); + len = (le16_to_cpu(range.nlb) + 1) << (req->ns->blksize_shift); + ret = vfs_copy_file_range(req->ns->file, src, req->ns->file, + pos, len, 0); + if (ret != len) { + pos += ret; + req->cqe->result.u32 = cpu_to_le32(id); + if (ret < 0) + status = errno_to_nvme_status(req, ret); + else + status = errno_to_nvme_status(req, -EIO); + goto out; + } else + pos += len; + } + +out: + nvmet_req_complete(req, status); +} + static void nvmet_file_execute_dsm(struct nvmet_req *req) { if (!nvmet_check_data_len_lte(req, nvmet_dsm_len(req))) @@ -330,6 +373,12 @@ static void nvmet_file_execute_dsm(struct nvmet_req *req) queue_work(nvmet_wq, &req->f.work); } +static void nvmet_file_execute_copy(struct nvmet_req *req) +{ + INIT_WORK(&req->f.work, nvmet_file_copy_work); + queue_work(nvmet_wq, &req->f.work); +} + static void nvmet_file_write_zeroes_work(struct work_struct *w) { struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); @@ -376,6 +425,9 @@ u16 nvmet_file_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_file_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_file_execute_copy; + return 0; default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c index f2d24b2d992f..d18ed8067a15 100644 --- a/drivers/nvme/target/loop.c +++ b/drivers/nvme/target/loop.c @@ -146,6 +146,12 @@ static blk_status_t nvme_loop_queue_rq(struct blk_mq_hw_ctx *hctx, return ret; nvme_start_request(req); + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_set_request_complete(req); + blk_mq_end_request(req, BLK_STS_OK); + return BLK_STS_OK; + } iod->cmd.common.flags |= NVME_CMD_SGL_METABUF; iod->req.port = queue->ctrl->port; if (!nvmet_req_init(&iod->req, &queue->nvme_cq, diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h index dc60a22646f7..1615dc9194ba 100644 --- a/drivers/nvme/target/nvmet.h +++ b/drivers/nvme/target/nvmet.h @@ -393,6 +393,7 @@ struct nvmet_req { struct device *p2p_client; u16 error_loc; u64 error_slba; + size_t copy_len; }; #define NVMET_MAX_MPOOL_BVEC 16 From patchwork Mon Jun 5 12:17:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13267306 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45B80C7EE37 for ; Mon, 5 Jun 2023 12:30:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233494AbjFEMaN (ORCPT ); Mon, 5 Jun 2023 08:30:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55966 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233620AbjFEM36 (ORCPT ); Mon, 5 Jun 2023 08:29:58 -0400 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB5EBDF for ; Mon, 5 Jun 2023 05:29:52 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20230605122951epoutp0257996b0e740342af1e0e00ae71f63751~lwyMaRxCY0552705527epoutp02L for ; Mon, 5 Jun 2023 12:29:51 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20230605122951epoutp0257996b0e740342af1e0e00ae71f63751~lwyMaRxCY0552705527epoutp02L DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1685968191; bh=H1PdiFvQZD1R853saHzyRCE8capZ0U3aPv5cQi8i5WE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lCuHr4EKm8alYf4gZ75wfkT7o9JhJG88K/v06hB7cvw9R1Mz8rVxWfA9VvN4jCW/k hrmBCbCDRZojjdnWTqHXh8+jGF0sKdagpSZmZyf555Whu3y6LvAGbz3e+OhWjoCChJ ydnbxwtYi0dwArlwODWI4llgqMcksiB+cz3G05tU= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230605122950epcas5p136d93b014386a849cfd5869aaec685f3~lwyLul6So0215902159epcas5p1h; Mon, 5 Jun 2023 12:29:50 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.174]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4QZXx91CgYz4x9Pv; Mon, 5 Jun 2023 12:29:49 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 8C.5D.16380.D35DD746; Mon, 5 Jun 2023 21:29:49 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230605122349epcas5p2de6c758bc38194ebfb1fd8c700e45608~lws7aUZgZ3244532445epcas5p2G; Mon, 5 Jun 2023 12:23:49 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230605122349epsmtrp29f7a21e8ed8ea1907e7bb327a40f384c~lws7ZGsem1056310563epsmtrp2Z; Mon, 5 Jun 2023 12:23:49 +0000 (GMT) X-AuditID: b6c32a4b-7dffd70000013ffc-11-647dd53dac52 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 09.D5.28392.5D3DD746; Mon, 5 Jun 2023 21:23:49 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230605122343epsmtip245dddb1bc0bcc26599c570b39b4b9fcb~lws2FYJfX2527325273epsmtip2l; Mon, 5 Jun 2023 12:23:43 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, willy@infradead.org, hare@suse.de, djwong@kernel.org, bvanassche@acm.org, ming.lei@redhat.com, dlemoal@kernel.org, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v12 7/9] dm: Add support for copy offload Date: Mon, 5 Jun 2023 17:47:23 +0530 Message-Id: <20230605121732.28468-8-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230605121732.28468-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTVxzHPfe2t8Ws7AISDw+32k0IGB6dpTtVGWQjcjPcVrdlEXErHVwe AdqutwwHWYCBOjSCwJihOB5KeG4ghUEBYVJRxnuOh5MBGkLjJlsBGQEkwqgtm/99zvf8vr/X yeHi9hUcZ26sQkOrFfJ4AbGT1XzTw93Lf+zLSN+H/WxU33cbR7VTOQSau/kYoG8X1nA0e+Ms QCOztujBTwGow1TERvdutGLo+pU8DFXX3sJQe9kihvIM4wAZx7QY6pjYj8rOlLPQ9Y5eFhpp u0ygkgojB52/qydQZc8Ghgz5GRjSz6YD1LxegqO6uXkW+nnCBQ0/7WGj9dXLRKArNTIaQmnv DxJUq3aKQw1PN7CoxipPamQwkdLVZBFUY3kq1X4vjaCuZuezqQsZJoJaNE6wqPnOMYLKbqoB VGN/CrWke0lqdyLucAwtj6TVfFoRoYyMVUT7C0I+kL0l8xP7Cr2EEvS6gK+QJ9D+gqCjUq8j sfFb2xHwP5fHJ25JUjnDCHzeOKxWJmpofoyS0fgLaFVkvEqk8mbkCUyiItpbQWsOCn19X/Pb CgyPi/m7c5mlanI+9SQ/Pg20OZ4DNlxIimBOazvrHNjJtSfbAZwbXSUsh8cAGv8xAXOUPbkE 4GDL7m3H4m8twBLUBmBrXr7VcRqDZ550cc4BLpcg98P+Ta5Z30VW4TB9ZeBZDZwsx2H1Xwts cyoHUgK75hsxM7PIfVCf/hVmNvPIg3Cg6SMzQtIH5ty3M0fYkIfgn4MG3Mw80g72Fs6yzIyT L8OMH4twc3pIDtnAlj+mCEunQfBK7pCVHeCjniaOhZ3hkqnDqifB6m+qCIs5E0DtXS2wXATA 0305uLkJnPSA9W0+FnkPLOirwyyFbeGF9VnMovOgvnibX4Hf15da8zvB8ZV0K1PQmGGw7job wFHdGOci4GufG0j73EDa/0uXArwGONEqJiGaZvxUBxR00n+PHKFM0IFnH8YzRA9mHix4GwDG BQYAubhgF6/t7ZRIe16k/ItkWq2UqRPjacYA/Lb2nYs7O0Yot36cQiMTiiS+IrFYLJIcEAsF u3nu/r0R9mS0XEPH0bSKVm/7MK6NcxoGiAr/ZF7oypzXd8FRyYpEt0+iujJ3iEMq9bLNfTMc terksvvTcN0d+4LR5l/6J7tXz+aWc08uf33+UtkIf3hgr0/WcePRzXBhlKT8qnL5jtR0S8kZ y607XsSQoS7GydrxUwPN0hTTh06Btx8Vn+g6psceagaQg9uLqQEz3dOZP5Q21BcUKd50Ueo2 dpjePRQ+JG73zArLW7tYqcv+9FVPVYJgOkktwYIL63yPBI2WdhtySukJ6aUpR9m13KlfM6Tv e78QcY39XtWUm2vYrGtwQtbehtDAzD1B74RVh9EeMtu1jsacjzfaaGqO7i1hFx5jOidTlaGf FXvxf28nFkWlAhYTIxd64mpG/i8/msbEuQQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrNIsWRmVeSWpSXmKPExsWy7bCSvO7Vy7UpBu/7WS3WnzrGbLH6bj+b xevDnxgtpn34yWzx5EA7o8XlJ3wWD/bbW+x9N5vV4uaBnUwWexZNYrJYufook8XuhR+ZLCYd usZo8fTqLCaLvbe0LRa2LWGx2LP3JIvF5V1z2CzmL3vKbtF9fQebxfLj/5gsDk1uZrLY8aSR 0WLb7/nMFutev2exOHFL2uL83+OsFr9/zGFzkPG4fMXbY9b9s2weO2fdZfc4f28ji8fmFVoe l8+Wemxa1cnmsXlJvcfumw1sHov7JrN69Da/Y/P4+PQWi8f7fVfZPPq2rGL02Hy62uPzJrkA wSgum5TUnMyy1CJ9uwSujLf7vrIUbJGq+DU5p4Fxl2gXIyeHhICJxMcb2xm7GLk4hAR2MEp0 bmtnhkhISiz7ewTKFpZY+e85O0RRM5PE843NQB0cHGwC2hKn/3OAxEUEtjBLnP01mRXEYRbY xizx4cMXdpBuYQFLiYPvNzOB2CwCqhI7GpuYQJp5BawkzmwJAzElBPQl+u8LglRwClhLvDx7 iBkkLARUcfW9DUiYV0BQ4uTMJywgNrOAvETz1tnMExgFZiFJzUKSWsDItIpRMrWgODc9t9iw wCgvtVyvODG3uDQvXS85P3cTIzj+tbR2MO5Z9UHvECMTB+MhRgkOZiUR3l1e1SlCvCmJlVWp RfnxRaU5qcWHGKU5WJTEeS90nYwXEkhPLEnNTk0tSC2CyTJxcEo1MIkbP1hnzBXvn/7r08Hs PyxMm8T0AyI23dS9++JZ7gGm9Vou2mt3WV/4t0Xnzc0qfZO5U75qesgEzp1RVXd5s8gBF/ue E2dnPb+9KEssXMjrbVOVpf0XPpf1tY01RdtKTqXZmlfo5r8T3P5DMs3GXit3qd7XtfwxG1NL LM5apX/9sf3Xq1+iKkLzmb0/Rnav+9Z5bt+8xkuLDbnEnphtylaPdPI9utGco8uFZ/eC2Q9C 7qh9Ut46pVeJ5WfBtpyjupVT2mPU7sTLu2Umz74jdd1D79Ck73dj8jn+Hbq6Lif6ftcRbSZP /ozlp38HKzDbsnrO3byYd9bUevMbLE59sssr51+XOd9npt3b991mthJLcUaioRZzUXEiAC14 gQpuAwAA X-CMS-MailID: 20230605122349epcas5p2de6c758bc38194ebfb1fd8c700e45608 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230605122349epcas5p2de6c758bc38194ebfb1fd8c700e45608 References: <20230605121732.28468-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Before enabling copy for dm target, check if underlying devices and dm target support copy. Avoid split happening inside dm target. Fail early if the request needs split, currently splitting copy request is not supported. Signed-off-by: Nitesh Shetty --- drivers/md/dm-table.c | 41 +++++++++++++++++++++++++++++++++++ drivers/md/dm.c | 7 ++++++ include/linux/device-mapper.h | 5 +++++ 3 files changed, 53 insertions(+) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 1398f1d6e83e..b3269271e761 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1867,6 +1867,39 @@ static bool dm_table_supports_nowait(struct dm_table *t) return true; } +static int device_not_copy_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + struct request_queue *q = bdev_get_queue(dev->bdev); + + return !blk_queue_copy(q); +} + +static bool dm_table_supports_copy(struct dm_table *t) +{ + struct dm_target *ti; + unsigned int i; + + for (i = 0; i < t->num_targets; i++) { + ti = dm_table_get_target(t, i); + + if (!ti->copy_offload_supported) + return false; + + /* + * target provides copy support (as implied by setting + * 'copy_offload_supported') + * and it relies on _all_ data devices having copy support. + */ + if (!ti->type->iterate_devices || + ti->type->iterate_devices(ti, + device_not_copy_capable, NULL)) + return false; + } + + return true; +} + static int device_not_discard_capable(struct dm_target *ti, struct dm_dev *dev, sector_t start, sector_t len, void *data) { @@ -1949,6 +1982,14 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, q->limits.discard_misaligned = 0; } + if (!dm_table_supports_copy(t)) { + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + q->limits.max_copy_sectors = 0; + q->limits.max_copy_sectors_hw = 0; + } else { + blk_queue_flag_set(QUEUE_FLAG_COPY, q); + } + if (!dm_table_supports_secure_erase(t)) q->limits.max_secure_erase_sectors = 0; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 4361a01bff3a..d9f45a1f0a77 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1720,6 +1720,13 @@ static blk_status_t __split_and_process_bio(struct clone_info *ci) if (unlikely(ci->is_abnormal_io)) return __process_abnormal_io(ci, ti); + if ((unlikely(op_is_copy(ci->bio->bi_opf)) && + max_io_len(ti, ci->sector) < ci->sector_count)) { + DMERR("Error, IO size(%u) > max target size(%llu)\n", + ci->sector_count, max_io_len(ti, ci->sector)); + return BLK_STS_IOERR; + } + /* * Only support bio polling for normal IO, and the target io is * exactly inside the dm_io instance (verified in dm_poll_dm_io) diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index a52d2b9a6846..04016bd76e73 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -398,6 +398,11 @@ struct dm_target { * bio_set_dev(). NOTE: ideally a target should _not_ need this. */ bool needs_bio_set_dev:1; + + /* + * copy offload is supported + */ + bool copy_offload_supported:1; }; void *dm_per_bio_data(struct bio *bio, size_t data_size); From patchwork Mon Jun 5 12:17:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13267307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4F0BC7EE24 for ; Mon, 5 Jun 2023 12:31:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233004AbjFEMbM (ORCPT ); Mon, 5 Jun 2023 08:31:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233673AbjFEMaG (ORCPT ); Mon, 5 Jun 2023 08:30:06 -0400 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2626710C for ; Mon, 5 Jun 2023 05:29:57 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20230605122955epoutp04854029c9d9a59ee7bc341a0dea864efa~lwyQgG4Od1771217712epoutp04U for ; Mon, 5 Jun 2023 12:29:55 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20230605122955epoutp04854029c9d9a59ee7bc341a0dea864efa~lwyQgG4Od1771217712epoutp04U DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1685968195; bh=cp2ikyfpBdIhGRYkH19yhVPh/sP+Xdj/kjqikt163DM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WZd2sKjTNiucvGt243+3gh20yK9G3bPvIfM4Y/lg77gXFXbY9yCUIucN4EZQVzz1w 86/GFU1Q9zdvAXqRe3yTupd4fsOJFmQY+HxT6VVJItRGImcjjmZjcMXjYIxuRxO3ml 6IsqiEo9E91DWoyRozQKKvB/RTxgCXDbgCHt+mRU= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230605122954epcas5p229827afd5b605331722ca1efe39ab864~lwyP7jP3I1471314713epcas5p22; Mon, 5 Jun 2023 12:29:54 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.176]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4QZXxF4ZS0z4x9Pv; Mon, 5 Jun 2023 12:29:53 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 10.6D.16380.145DD746; Mon, 5 Jun 2023 21:29:53 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230605122402epcas5p2d2fc7493b04aaa70363324429411e689~lwtIQAFJl3244432444epcas5p2Y; Mon, 5 Jun 2023 12:24:02 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230605122402epsmtrp2ba078ddca865e8855cd175a22e7a9086~lwtIO99cp1056210562epsmtrp2j; Mon, 5 Jun 2023 12:24:02 +0000 (GMT) X-AuditID: b6c32a4b-56fff70000013ffc-24-647dd541d8d3 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 1B.BC.27706.2E3DD746; Mon, 5 Jun 2023 21:24:02 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230605122357epsmtip2c0fc194d4469fec3c69fc0db88c10697~lwtDg78hu2527325273epsmtip2o; Mon, 5 Jun 2023 12:23:57 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, willy@infradead.org, hare@suse.de, djwong@kernel.org, bvanassche@acm.org, ming.lei@redhat.com, dlemoal@kernel.org, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v12 8/9] dm: Enable copy offload for dm-linear target Date: Mon, 5 Jun 2023 17:47:24 +0530 Message-Id: <20230605121732.28468-9-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230605121732.28468-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Te0xTVxzeube9FCbkUtk8wobsOuYAgXZAPSD4GGhugE2MS+ZGFmjam0Io bdcHzhEzGqYbD4sgLwtBECZQpjgeXSl207LJQBk6XoHMiVi2AAsvcWgI2yiFzf++33e+3+s7 +XFwbr2TJydVpmaUMqGUIlxYxi4/v8CDQ6fEvLZhCjX33sJR0/0CAs10LQJUOv8MR7YbXwA0 YHND49/vR5bZCjYavdGBoeuXijDU2PQjhjprFjBUZB0GaHJIjyHLWACqOVPHQtctPSw0YK4k 0MXLk04ob8REoPruvzFkPZ+NIZNNC5Bx5SKOrs7MsdBPY16of7WbjVaeVhIHXqEHBuNo/YM+ gu7Q33ei+3/7hkW3NvjTA30ausWQQ9CtdZ/RnaNZBF2rO8+mz2bPEvTC5BiLnvtuiKB1bQZA t97OpB+3eCe4f5gWmcIIxYzSh5GJ5OJUmSSKijuWFJ0UJuDxA/nhaA/lIxOmM1FUTHxC4OFU 6Zo7lE+GUKpZoxKEKhUVvC9SKdeoGZ8UuUodRTEKsVQRqghSCdNVGpkkSMaoI/g83ltha8Lk tJQyQxlboWV/8vvqTSILlLNygTMHkqHQbC7Gc4ELh0t2AninvxI4gkUAy6cfbbz8BaAxpwjb TKksyQZ2zCUtAOa3fewQncag9YnJKRdwOAQZAG//w7HzHmQDDrXLd1j2ACfrcNj45zzbLtpK HoIjC4fshVikL5yoXWLbsSsZAQef1q7XgWQwLHjgbqedyb1wqs+KOyTusOeCbX0FnNwBs9sr 1geFZK8z1JZPAMegMTD7h7kNvBVOd7c5ObAnfDxrIRz4BGwsbiAcyZ8DqB/RbyTsh6d7C3D7 EDjpB5vNwQ76VVjSexVzNHaDZ1dsG6a4QlPVJt4Jv26u3qi/HQ4vazcwDScGpzGHWToAu5a1 2Dngo39uIf1zC+n/b10NcAPYzihU6RJGFaYIkTEn/vtlkTy9BaxfjH+cCUyMzwdZAcYBVgA5 OOXhao7NFHNdxcKTnzJKeZJSI2VUVhC2Zngh7vmSSL52cjJ1Ej80nBcqEAhCw0MEfGqb666o HhGXlAjVTBrDKBjlZh7GcfbMwvIqMrYtst59mOjdHhGgTqZee8PjhZrqX0tjBpO73lve83bp y1dg1wXnAmaLdHS3r9PUsdwkE394OSe+fWJqxyIjEbiOyu95+M1X3qztQE8Oqhd0ZT+nrRCt q7VGlzboFvO+WR0XO976Zu5EZM7l8EpFptVblr/zGbAZX/fNC6kWuR817L5iOe7WlUiVa2yr 19yD6r+sj9dc27XUTVa5/XGySHcv2mcmuOAdrx7jmG9m3N2jL36bUbKyL79plBR9FKtO5FJH brXnX+KNFicWBX+QLz6iOKU//LDQWdxT1bS3iDgwKZDMVBdGZy1ovzJ4HY+8a1U+2uLSpOvR /XKGXqoZpliqFCHfH1eqhP8CtvkgJboEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02Sa1BMYRjHveecPXtqpNPGeCu5nFkZSyWMeYUtTDNnoqZcxuVLlo4y2m1n Twk7xmaJFmU0iW1da4qVS8UqabFyyc4KK03RUjaXot0w7oWtMePb//n/fvPM8+GhcNFzIpDa oMjgVApZGkN6E6ZbzLjQTvu25Gnl5unowv07ODrbnk+inlsfATrk/o4j543dANmdI9DL61Go vrdYgFpv1GLo2qmDGDpz9jaG6k72Yeig5SlAXc16DNW3TUEnc0oJdK2+kUD2qwYSHS/rEqK9 LTUkKr87gCFLgRZDNc5sgEw/j+PofI+LQPfaglBT/10B+vnNQEaPYe1PFrH6FzaSrdW3C9km RyXBVp+WsHZbJltlzCXZ6tLtbF2rhmRL8goE7H5tL8n2dbURrMvcTLJ5l4yArbaq2U9VYxP8 VnvPTebSNmziVOHSNd6pRcYigTJbsPl1/01SAw4TOuBFQXomNBRqgQ54UyK6DsB25zPhEAiA Zf0N+FD2h2cG3giHJC0GW8tcAh2gKJKeAq2/KU8/kr6EQ9uPAoFnwGkTDt3uz0KP5E/HwJa+ GM8igp4IO0s+CzzZh46ET76VDCqQDof5L/w8tRc9B76zWXBPLfqrNLvmDtl+sPGIc/BmnB4H tZeL8QOA1v+H9P+hEwAzggBOyctT5HyEMkLBZYXxMjmfqUgJW5curwKDDyCZXAOuGN1hFoBR wAIghTMjfa7GqpNFPsmyLVs5VXqSKjON4y0giCKY0T4PdY1JIjpFlsFt5Dglp/pHMcorUIOt DPvi4B/9sGYERE/taBoeFxpkqFf3Pt1MpK51xSd0DyzQjKrYL5nXtcNyWlk7kGedlSscVhlp HsOIXf67EtQitzTJLI6ZFvD2w4qWQ0tmGczRzKpIuyN2wZrxo4Ivcl95e/yz+Uf5sTayZ6VG 0R1evc8gyT3q7M6u2NjQLH6VI9WXV3zsEM6b8Wjp99i49cuCXNaQmUCtdO8ypiVm9WVN/iAN 7A7NfjVaaF+YJbtQ/CDEt6G4srGi09ERvFy8I14n3unY5upMYuZMYkIem/oDj01I7CnKT19c rpIuySkM/oUvBXWZs2XRe1Zv+XpvZy8RZXoojzrn9d7XN4405MkZgk+VRUhwFS/7A62CYfhv AwAA X-CMS-MailID: 20230605122402epcas5p2d2fc7493b04aaa70363324429411e689 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230605122402epcas5p2d2fc7493b04aaa70363324429411e689 References: <20230605121732.28468-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Setting copy_offload_supported flag to enable offload. Signed-off-by: Nitesh Shetty --- drivers/md/dm-linear.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c index f4448d520ee9..1d1ee30bbefb 100644 --- a/drivers/md/dm-linear.c +++ b/drivers/md/dm-linear.c @@ -62,6 +62,7 @@ static int linear_ctr(struct dm_target *ti, unsigned int argc, char **argv) ti->num_discard_bios = 1; ti->num_secure_erase_bios = 1; ti->num_write_zeroes_bios = 1; + ti->copy_offload_supported = 1; ti->private = lc; return 0; From patchwork Mon Jun 5 12:17:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13267318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43CE5C7EE2A for ; Mon, 5 Jun 2023 12:31:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233528AbjFEMbs (ORCPT ); Mon, 5 Jun 2023 08:31:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56574 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233053AbjFEMaa (ORCPT ); Mon, 5 Jun 2023 08:30:30 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F22B6DB for ; Mon, 5 Jun 2023 05:30:01 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230605123000epoutp01f64a14a3a7694583b8785863b7555184~lwyU-AkyE1545015450epoutp015 for ; Mon, 5 Jun 2023 12:30:00 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230605123000epoutp01f64a14a3a7694583b8785863b7555184~lwyU-AkyE1545015450epoutp015 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1685968200; bh=8Y/BlYOox2MFz3ATsHjfRRZ6YEWMtN00Y8t1o+Qp/wI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gGnQAMI8r5gU2OKI4xPe570GZnexR6UQaRcJxvcJOJ5DrDssllm1ULj1mbeclbRtP 2thFVTahNXQQjfL/xblsbdZ23s5SUSXvCG8bmCXJGU+UVX+uXNUhD877IEgMIoc9q0 VlvRDxaHVOZHmPf550dRrGN36xuruKX56U1C2rrQ= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230605122959epcas5p19650c5e46b41292eb1c31b9f6432d755~lwyUPcDrR2755927559epcas5p1X; Mon, 5 Jun 2023 12:29:59 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.174]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4QZXxL1NZgz4x9Pw; Mon, 5 Jun 2023 12:29:58 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 09.07.44881.645DD746; Mon, 5 Jun 2023 21:29:58 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230605122416epcas5p3b37c08a7bf7ad57361b22be0f30d3bf7~lwtUX_CAR0800608006epcas5p33; Mon, 5 Jun 2023 12:24:16 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230605122415epsmtrp11bd9a11fb9f4c12a0751ff4cdb618775~lwtUW6YFY1611816118epsmtrp1r; Mon, 5 Jun 2023 12:24:15 +0000 (GMT) X-AuditID: b6c32a4a-ea9fa7000001af51-bb-647dd546e72a Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 3F.BC.27706.FE3DD746; Mon, 5 Jun 2023 21:24:15 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230605122411epsmtip20cac9b9b910a1f71d950a511e1a00c94~lwtQSajIv2526925269epsmtip2i; Mon, 5 Jun 2023 12:24:11 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Jonathan Corbet , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, willy@infradead.org, hare@suse.de, djwong@kernel.org, bvanassche@acm.org, ming.lei@redhat.com, dlemoal@kernel.org, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Damien Le Moal , Anuj Gupta , Vincent Fu , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v12 9/9] null_blk: add support for copy offload Date: Mon, 5 Jun 2023 17:47:25 +0530 Message-Id: <20230605121732.28468-10-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230605121732.28468-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Te0xTZxT3u7e9FLa6y2t8QphNJ25AeBSBfaAMzFCvsEwCLpuSDCu9AUJp m7bopjLAjsdgtAhjbNWJCCqvSAZKShEwRXkjc4gPGMON4iI4Xk5gVGAthc3/fr9zfuc75/y+ HBZuo7FwZCWI5LRUxBdyCStGQ5urq8fewRSBd2snA9V2t+PodP4yjqpHVASabJsD6LuZf3Ck v5kFkKGvH0cD+s3ocWswap46y0SPbjZi6MbFAgxVVt/GUFPpLIYKdPcBGh9UY6h5yB2VZpYz 0I3mLgYa0J4jUMnlcQuU+0BDoCsdKxjSFSowpNGnA9RgKMHR1clpBuocckJjudkA9S93MJFh 8RwRspUauBdOqUf7CKpRPWJB9f/2E4Oqr3CjBvqSqbqqrwmqvjyVanqURlBlykImlaeYMioz RpnU7PgQg5puGSQo5bUqQNX3nIywOZy4K57mC2gphxbFigUJorggbnhUzAcxfv7ePA9eAHqP yxHxk+ggbuiHER57E4RGt7icY3xhsjEUwZfJuF7v75KKk+U0J14skwdxaYlAKPGVeMr4SbJk UZyniJYH8ry9ffyMwiOJ8cWna5iSidDPx6sqQBpoC8gBlixI+sKf035l5gArlg3ZBOAtQysw kzkApwvqLMxkHsCsxovYRklLaw9hTjQDmPnLM9xMMjBYc2fOqGKxCNId9qyyTHE7sgKH6Qu9 DBPByVIGHH7yDTA9ZUsGw5HnStyEGaQLvDSWRpiK2eRO2Pt9lAlC0guqRq1NCktj9Gmfbk3N Jq1h1w96hgnj5FaouH52bQZITljC1VuXCfOkofCPphqmGdvCiY5rFmbsCJ+qMtfxcVj5bQVh Lv4KQPUDNTAngmFGtwo3DYGTrrBW62UOO8Oi7quYufFmmGfQr7vChprzG/htWFN7YX2GLfD+ Qvo6pmD53F3MbJbS2Ku/hZkPOOpXFlK/spD6/9YXAF4FttASWVIcLfOT+Ijo4/99c6w4qQ6s nZBbmAb8/njGUwcwFtAByMK5dmxt2EmBDVvA/+IELRXHSJOFtEwH/Ix+n8Ed7WPFxhsUyWN4 vgHevv7+/r4BO/x5XAf2O0FdsTZkHF9OJ9K0hJZu1GEsS8c0bL9D4CeNXxrsh7vkpYt7xos/ fuboHr2kEAonU10IJ69PF1/6WB0DVs050YNaWX7q0dlDR4vpfWHS8KmUwsLOity0ot0h8pKC h0V1VMsY6n234c6OwwtHBJPV1dkPh18G8ZTN+5VO00kc1fwb2rdWrrjM/RVkbKHofz3zniyy /qPC7Q1v9v+dZZWQmnvXp8xzKc+zbzAKBp9hupC9T3LnBaJWzWtl27vGDmisX2xz7oj8M6Ry Hz916PlydvWLyEMrA+kxbPxg3u4T6dtUe0ZX4w/mBvPCfQKvK2Y2DdvZtq/sDEupvmSoLzgl PqU94Lz02dCPDnm3O2tSVmbs26P14TPnN3EZsng+zw2Xyvj/AnmzEAjLBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrDIsWRmVeSWpSXmKPExsWy7bCSvO77y7UpBie/GVmsP3WM2aJpwl9m i9V3+9ksXh/+xGgx7cNPZosnB9oZLX6fPc9scfkJn8WD/fYWe9/NZrW4eWAnk8WeRZOYLFau PspksXvhRyaLSYeuMVo8vTqLyWLvLW2LhW1LWCz27D3JYnF51xw2i/nLnrJbdF/fwWax/Pg/ JotDk5uZLHY8aWS02PZ7PrPFutfvWSxO3JK2eNzdwWhx/u9xVovfP+awOch7XL7i7THr/lk2 j52z7rJ7nL+3kcVj8wotj8tnSz02repk89i8pN5j980GNo/FfZNZPXqb3wFVtt5n9fj49BaL x/t9V9k8+rasYvTYfLo6QCiKyyYlNSezLLVI3y6BK2N60xrWglcuFU9XrWBsYDxs2cXIySEh YCKxb/9pti5GLg4hgd2MEv/PPGSHSEhKLPt7hBnCFpZY+e85O0RRM5PEs5tHWbsYOTjYBLQl Tv/nAImLCGxhljj7azIriMMssJ1FYsn6pawg3cIC9hJ3P/eBTWIRUJVY+riBDaSZV8Ba4syM YBBTQkBfov++IEgFJ1D05dlDzCBhIQEriavvbUDCvAKCEidnPmEBsZkF5CWat85mnsAoMAtJ ahaS1AJGplWMkqkFxbnpucWGBYZ5qeV6xYm5xaV56XrJ+bmbGMEJQktzB+P2VR/0DjEycTAe YpTgYFYS4d3lVZ0ixJuSWFmVWpQfX1Sak1p8iFGag0VJnPdC18l4IYH0xJLU7NTUgtQimCwT B6dUA9PO2Bkd5eIck7INnlrP6bnyleE0l0RO8JOfn/Z8NHP5HPjj1O5gZhUTH8XXJhkRf/Tf lCR9KLlexq1Y5fmObc9s5cblHt9y5iSaz55v9GzFooyLfKvtI+9nXch+I+1uKbre+q1R2803 d8+v74usFP69IvpluAnLcmOFqoPBX+zvvDnBFGrX++U422xOP2+P07WBF/K/vlWeOWvt3POP Nhh9kPK6e+Z+R/DUhjlWHSlMv48aCMetKhNJZvGan5l3IvDetq+Ghaq3mpUlrO7sPJy28F2L s8DDzWdV6xecPS2ot0uh+e0jq+UsofeNza9ckCqLbsgJUVpWM9dJ0Hj+cttGEeETH5dF+HS8 SK5RWa3EUpyRaKjFXFScCAD58AP4fwMAAA== X-CMS-MailID: 20230605122416epcas5p3b37c08a7bf7ad57361b22be0f30d3bf7 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230605122416epcas5p3b37c08a7bf7ad57361b22be0f30d3bf7 References: <20230605121732.28468-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org Implementaion is based on existing read and write infrastructure. copy_max_bytes: A new configfs and module parameter is introduced, which can be used to set hardware/driver supported maximum copy limit. Suggested-by: Damien Le Moal Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu --- Documentation/block/null_blk.rst | 5 ++ drivers/block/null_blk/main.c | 108 ++++++++++++++++++++++++++++-- drivers/block/null_blk/null_blk.h | 8 +++ 3 files changed, 116 insertions(+), 5 deletions(-) diff --git a/Documentation/block/null_blk.rst b/Documentation/block/null_blk.rst index 4dd78f24d10a..6153e02fcf13 100644 --- a/Documentation/block/null_blk.rst +++ b/Documentation/block/null_blk.rst @@ -149,3 +149,8 @@ zone_size=[MB]: Default: 256 zone_nr_conv=[nr_conv]: Default: 0 The number of conventional zones to create when block device is zoned. If zone_nr_conv >= nr_zones, it will be reduced to nr_zones - 1. + +copy_max_bytes=[size in bytes]: Default: COPY_MAX_BYTES + A module and configfs parameter which can be used to set hardware/driver + supported maximum copy offload limit. + COPY_MAX_BYTES(=128MB at present) is defined in fs.h diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index b3fedafe301e..34e009b3ebd5 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -157,6 +157,10 @@ static int g_max_sectors; module_param_named(max_sectors, g_max_sectors, int, 0444); MODULE_PARM_DESC(max_sectors, "Maximum size of a command (in 512B sectors)"); +static unsigned long g_copy_max_bytes = COPY_MAX_BYTES; +module_param_named(copy_max_bytes, g_copy_max_bytes, ulong, 0444); +MODULE_PARM_DESC(copy_max_bytes, "Maximum size of a copy command (in bytes)"); + static unsigned int nr_devices = 1; module_param(nr_devices, uint, 0444); MODULE_PARM_DESC(nr_devices, "Number of devices to register"); @@ -409,6 +413,7 @@ NULLB_DEVICE_ATTR(home_node, uint, NULL); NULLB_DEVICE_ATTR(queue_mode, uint, NULL); NULLB_DEVICE_ATTR(blocksize, uint, NULL); NULLB_DEVICE_ATTR(max_sectors, uint, NULL); +NULLB_DEVICE_ATTR(copy_max_bytes, uint, NULL); NULLB_DEVICE_ATTR(irqmode, uint, NULL); NULLB_DEVICE_ATTR(hw_queue_depth, uint, NULL); NULLB_DEVICE_ATTR(index, uint, NULL); @@ -550,6 +555,7 @@ static struct configfs_attribute *nullb_device_attrs[] = { &nullb_device_attr_queue_mode, &nullb_device_attr_blocksize, &nullb_device_attr_max_sectors, + &nullb_device_attr_copy_max_bytes, &nullb_device_attr_irqmode, &nullb_device_attr_hw_queue_depth, &nullb_device_attr_index, @@ -656,7 +662,8 @@ static ssize_t memb_group_features_show(struct config_item *item, char *page) "poll_queues,power,queue_mode,shared_tag_bitmap,size," "submit_queues,use_per_node_hctx,virt_boundary,zoned," "zone_capacity,zone_max_active,zone_max_open," - "zone_nr_conv,zone_offline,zone_readonly,zone_size\n"); + "zone_nr_conv,zone_offline,zone_readonly,zone_size," + "copy_max_bytes\n"); } CONFIGFS_ATTR_RO(memb_group_, features); @@ -722,6 +729,7 @@ static struct nullb_device *null_alloc_dev(void) dev->queue_mode = g_queue_mode; dev->blocksize = g_bs; dev->max_sectors = g_max_sectors; + dev->copy_max_bytes = g_copy_max_bytes; dev->irqmode = g_irqmode; dev->hw_queue_depth = g_hw_queue_depth; dev->blocking = g_blocking; @@ -1271,6 +1279,78 @@ static int null_transfer(struct nullb *nullb, struct page *page, return err; } +static inline void nullb_setup_copy_read(struct nullb *nullb, struct bio *bio) +{ + struct nullb_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + + token->subsys = "nullb"; + token->sector_in = bio->bi_iter.bi_sector; + token->nullb = nullb; + token->sectors = bio->bi_iter.bi_size >> SECTOR_SHIFT; +} + +static inline int nullb_setup_copy_write(struct nullb *nullb, + struct bio *bio, bool is_fua) +{ + struct nullb_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + sector_t sector_in, sector_out; + void *in, *out; + size_t rem, temp; + unsigned long offset_in, offset_out; + struct nullb_page *t_page_in, *t_page_out; + int ret = -EIO; + + if (unlikely(memcmp(token->subsys, "nullb", 5))) + return -EINVAL; + if (unlikely(token->nullb != nullb)) + return -EINVAL; + if (WARN_ON(token->sectors != bio->bi_iter.bi_size >> SECTOR_SHIFT)) + return -EINVAL; + + sector_in = token->sector_in; + sector_out = bio->bi_iter.bi_sector; + rem = token->sectors << SECTOR_SHIFT; + + spin_lock_irq(&nullb->lock); + while (rem > 0) { + temp = min_t(size_t, nullb->dev->blocksize, rem); + offset_in = (sector_in & SECTOR_MASK) << SECTOR_SHIFT; + offset_out = (sector_out & SECTOR_MASK) << SECTOR_SHIFT; + + if (null_cache_active(nullb) && !is_fua) + null_make_cache_space(nullb, PAGE_SIZE); + + t_page_in = null_lookup_page(nullb, sector_in, false, + !null_cache_active(nullb)); + if (!t_page_in) + goto err; + t_page_out = null_insert_page(nullb, sector_out, + !null_cache_active(nullb) || is_fua); + if (!t_page_out) + goto err; + + in = kmap_local_page(t_page_in->page); + out = kmap_local_page(t_page_out->page); + + memcpy(out + offset_out, in + offset_in, temp); + kunmap_local(out); + kunmap_local(in); + __set_bit(sector_out & SECTOR_MASK, t_page_out->bitmap); + + if (is_fua) + null_free_sector(nullb, sector_out, true); + + rem -= temp; + sector_in += temp >> SECTOR_SHIFT; + sector_out += temp >> SECTOR_SHIFT; + } + + ret = 0; +err: + spin_unlock_irq(&nullb->lock); + return ret; +} + static int null_handle_rq(struct nullb_cmd *cmd) { struct request *rq = cmd->rq; @@ -1280,13 +1360,20 @@ static int null_handle_rq(struct nullb_cmd *cmd) sector_t sector = blk_rq_pos(rq); struct req_iterator iter; struct bio_vec bvec; + bool fua = rq->cmd_flags & REQ_FUA; + + if (rq->cmd_flags & REQ_COPY) { + if (op_is_write(req_op(rq))) + return nullb_setup_copy_write(nullb, rq->bio, fua); + nullb_setup_copy_read(nullb, rq->bio); + return 0; + } spin_lock_irq(&nullb->lock); rq_for_each_segment(bvec, rq, iter) { len = bvec.bv_len; err = null_transfer(nullb, bvec.bv_page, len, bvec.bv_offset, - op_is_write(req_op(rq)), sector, - rq->cmd_flags & REQ_FUA); + op_is_write(req_op(rq)), sector, fua); if (err) { spin_unlock_irq(&nullb->lock); return err; @@ -1307,13 +1394,20 @@ static int null_handle_bio(struct nullb_cmd *cmd) sector_t sector = bio->bi_iter.bi_sector; struct bio_vec bvec; struct bvec_iter iter; + bool fua = bio->bi_opf & REQ_FUA; + + if (bio->bi_opf & REQ_COPY) { + if (op_is_write(bio_op(bio))) + return nullb_setup_copy_write(nullb, bio, fua); + nullb_setup_copy_read(nullb, bio); + return 0; + } spin_lock_irq(&nullb->lock); bio_for_each_segment(bvec, bio, iter) { len = bvec.bv_len; err = null_transfer(nullb, bvec.bv_page, len, bvec.bv_offset, - op_is_write(bio_op(bio)), sector, - bio->bi_opf & REQ_FUA); + op_is_write(bio_op(bio)), sector, fua); if (err) { spin_unlock_irq(&nullb->lock); return err; @@ -2161,6 +2255,10 @@ static int null_add_dev(struct nullb_device *dev) dev->max_sectors = queue_max_hw_sectors(nullb->q); dev->max_sectors = min(dev->max_sectors, BLK_DEF_MAX_SECTORS); blk_queue_max_hw_sectors(nullb->q, dev->max_sectors); + blk_queue_max_copy_sectors_hw(nullb->q, + dev->copy_max_bytes >> SECTOR_SHIFT); + if (dev->copy_max_bytes) + blk_queue_flag_set(QUEUE_FLAG_COPY, nullb->disk->queue); if (dev->virt_boundary) blk_queue_virt_boundary(nullb->q, PAGE_SIZE - 1); diff --git a/drivers/block/null_blk/null_blk.h b/drivers/block/null_blk/null_blk.h index 929f659dd255..3dda593b0747 100644 --- a/drivers/block/null_blk/null_blk.h +++ b/drivers/block/null_blk/null_blk.h @@ -67,6 +67,13 @@ enum { NULL_Q_MQ = 2, }; +struct nullb_copy_token { + char *subsys; + struct nullb *nullb; + sector_t sector_in; + sector_t sectors; +}; + struct nullb_device { struct nullb *nullb; struct config_group group; @@ -107,6 +114,7 @@ struct nullb_device { unsigned int queue_mode; /* block interface */ unsigned int blocksize; /* block size */ unsigned int max_sectors; /* Max sectors per command */ + unsigned long copy_max_bytes; /* Max copy offload length in bytes */ unsigned int irqmode; /* IRQ completion handler */ unsigned int hw_queue_depth; /* queue depth */ unsigned int index; /* index of the disk, only valid with a disk */