From patchwork Mon Feb 20 10:53:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13146392 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D23DC64EC7 for ; Mon, 20 Feb 2023 12:48:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229451AbjBTMsZ (ORCPT ); Mon, 20 Feb 2023 07:48:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39074 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231696AbjBTMsX (ORCPT ); Mon, 20 Feb 2023 07:48:23 -0500 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 594E8186 for ; Mon, 20 Feb 2023 04:48:15 -0800 (PST) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230220124813epoutp038762fa3cfa69bec6c2ad40557e1f6a86~FiTQ-rgho2305723057epoutp03H for ; Mon, 20 Feb 2023 12:48:13 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230220124813epoutp038762fa3cfa69bec6c2ad40557e1f6a86~FiTQ-rgho2305723057epoutp03H DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1676897293; bh=1hrkgfW4fuWprZ+hdImzV02kYbdcSvOZkXpC5k82hno=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RbaXnPOAMcZi/juQa/vp1WKh/1qQY9ooa7Coy1kNAaCxf7bEORqKrae38BPU/XR7n QpOuMAWeLhjEqpgZXkGEbMgKIqc0mK4+jzZ4txWzQRkRcr1nFBV9EpZkze/mDJ0sC7 +4ZIlUOBDX/rKBhsHEnhk+0jcyLaOg++7kzfBhK0= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230220124813epcas5p3e3a2a70e83cc481d02efcddd16dd3d33~FiTQY3-PX1435814358epcas5p3K; Mon, 20 Feb 2023 12:48:13 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.174]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4PL2Jp14Bvz4x9Pv; Mon, 20 Feb 2023 12:48:10 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id C4.49.06765.90C63F36; Mon, 20 Feb 2023 21:48:10 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230220105432epcas5p2dc91dbe3588d8f45bf312bde2683791a~FgwAF5_N00586305863epcas5p25; Mon, 20 Feb 2023 10:54:32 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230220105432epsmtrp23630d9e2a122f5d05c92d4bcbcc7dd65~FgwADYODD1664116641epsmtrp20; Mon, 20 Feb 2023 10:54:32 +0000 (GMT) X-AuditID: b6c32a4b-20fff70000011a6d-8e-63f36c09bff5 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 81.8C.05839.86153F36; Mon, 20 Feb 2023 19:54:32 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230220105428epsmtip227dfcc5ac704f641e0f590c825bd91cd~Fgv8kVbEk0747407474epsmtip2u; Mon, 20 Feb 2023 10:54:28 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v7 1/8] block: Introduce queue limits for copy-offload support Date: Mon, 20 Feb 2023 16:23:24 +0530 Message-Id: <20230220105336.3810-2-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230220105336.3810-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta0xbZRjOd045tCSdh8uyb9UxLDqgk0tHWw4bbCYycjJGgu6HxsxAaY+F UNrS0w5QxnXEMMNgDBfpcAVsUKjAVi5y64LVCZQxXBgoRS5iiVHkDqJhBFsKun/P+zzPe/3y MVGvj905zFSFhlIrxHIu5sHo+DaIF+whX5eE6X4FRIv1e5QoLN9BCeNUGUbcXvkHJbaHR1DC vHTHjZjo60KI3roKhGgwPkSIntpVhHi4u4gRFZZxQMyP6RDCbDtJ9JoHGcRodzVG6Ovn3QnL rSKE6LQXAKJjW48SzQvLDGLA9iIxstPv9voRcvRpHKmbGcbILt2UOzkyfZ9Bjg5rSVNjCUa2 GvLInol8jCwtWnIYimfcyOUHYxh5o60RkK1DH5LrJl/SZF9EEl54Ny0qhRJLKbUfpZAopakK WTQ37lLiG4lCURg/mB9JRHD9FOJ0KpobczEhODZV7jgB1++KWK51UAlimuaGno1SK7Uayi9F SWuiuZRKKlcJVCG0OJ3WKmQhCkpzmh8WdkroMCalpdgMfe6qqjezCgoeY/ngi5jrgMWEuAAO VhvBdeDB9MJ7ALy3uIO5gjUAhwwWN1fwF4B1AwXuBynPVs37ghnApjkDwxUUIXB+e8WhMJkY fhIO7TKdCT74zwjsGvJ3elB8FoH6QgNwCt54Atz86Ke9qgz8VfiDIX+PZ+ORsFm3iTrrQDwU ls14OiELPw23rmW7HJ5wsMrOcGIUPw6L2u+grtk+Z8GWmjMuHAMXWmb3eW/4R3/b/vwcuL5k xlw4EzZUfrm3McSvAaj7UQdcwjlYbC3bGwHFg2BLd6iLPgY/sTYjrr6HYOm2HXHxbNh59wD7 w69aavbrH4XjWwX7mISTjypR16lKATQZK93KgZ/uuX10z+2j+791DUAbwVFKRafLKFqoCldQ mf89skSZbgJ7v4IX1wnmZldCLABhAguATJTrw95lr0u82FJx9geUWpmo1sop2gKEjnPfRDmH JUrHt1JoEvmCyDCBSCQSRIaL+Nwj7IDoQYkXLhNrqDSKUlHqgzyEyeLkI6l3a2MTA33r86Ro luSJ9beN6eVvjLZb3hqPWD6oLT/kOd0xIktItUs5q0njn9qactdyOMb2Z9G51Un3Vx738bOz rD5psq+rugZfiahPNobOpXXnRfz9Xo5mY8lnIiNj5vzO1fl4ldaWOBk8gphCTvTKeSWxyXEV Nz67OqBsqGlb1rPwm7y5hXNPTSY0IPO7wNap+MDcoGNrWy/V0vfevtQeNNDUwfY8O3ah98LG Ey97f/zUiZSFUzmgV1sY8FqI4eUm+S8qvWiiZO3Pyw8sdfpg/1rho+F0ofww6/1W3lvdk8nD 1Kbv5SvKcElm3sXi43brO2eG2PT07963BVsZUclcBp0i5vNQNS3+F7mmEUueBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02RfVCLcRzA7/c8z56e5nY9LXf9Kq9DVKSdnF/OOZzjSYpdDueczHqu0jbZ FOVtM687Lwm5FoqSDItFRY0MZWqqS14inbVyx9UkebnKeMSd/z7f+3y+9/3jS+HCasKfSlJu YVVKqVxE8omyB6Kx0xIlfbKw6y4+KnlSg6M9mUM4utJ2jETZn37gaMDegCNLTy4Pvaq+jaGq C1kYunzlEYYqz/di6JG7m0RZ1ucAdbYYMGRpDUFVFhuBmu+cIVFeUacHsp7QYajCqQWobCAP R6aPLgI9bg1ADUO1vHm+TPOzKMbQbieZ24Y2D6bh7Q2CabanMmbjIZIpLdzNVL7SkMwRXc/v YF87j3HdbSGZozeNgCmt2870mccwZmc3ttxrDX9OPCtPSmNV0+eu5ye2FlZ7pORItmm1T0kN uLRQDzwpSIfDwV4LTw/4lJCuBLA2r5wcFn6waOghPsw+8PLP9x4cC2ktBgf3iPWAokg6BNa5 KW53JN2FQftbB84NON2DQeO7Ih4X+dAxUHdTxO0S9CTYWKgBHAvoCGgy9ONcAunp8Fi7N4ee 9Gz4bW/68KUI+Dir9G/tDW05ToJjnB4Ldbdy8UxAG/5Thv9UPsCMwI9NUSsSFGpxiljJbg1V SxXqVGVCqGyTwgz+PDk4qAKUGz+FWgFGASuAFC4aKXAL+mRCQbw0PYNVbYpTpcpZtRUEUITI V9Cot8UJ6QTpFjaZZVNY1T+LUZ7+Guya1AR4+bvtE1mX8Vq96r6i/+W9KaclgcV2Kx2btmzp 5ojXnc9fjP6s2+YV+LTg6kmf4h1Bi+clh22cMDNpZcjB2sEScZfJusJZZbj1uuCSQjK/Yvy6 qM1RXhOaDid8lVcu4QHNm6acMGdr2vFZbu/PU8869PE7s4sZ00Ltm1yZ+ei+jGpjAGoPci0O HxQucu58KPMdtbEm8pk70i+6e7lX0Y6amuACH2JDWfaInILjLYHNpV2jlDM2ODK+9EU6VjnO 1ctj328/0BaHpBbjqRc/9mfa30VPwTrKGtf1dzR9Hwg9Y4tZNnlBVke++eLL8l0hH0hJ+lpZ xvnYcbYY5lyJ72oRoU6UioNxlVr6C5Z+Bt5TAwAA X-CMS-MailID: 20230220105432epcas5p2dc91dbe3588d8f45bf312bde2683791a X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230220105432epcas5p2dc91dbe3588d8f45bf312bde2683791a References: <20230220105336.3810-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Add device limits as sysfs entries, - copy_offload (RW) - copy_max_bytes (RW) - copy_max_bytes_hw (RO) Above limits help to split the copy payload in block layer. copy_offload: used for setting copy offload(1) or emulation(0). copy_max_bytes: maximum total length of copy in single payload. copy_max_bytes_hw: Reflects the device supported maximum limit. Reviewed-by: Hannes Reinecke Signed-off-by: Nitesh Shetty Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- Documentation/ABI/stable/sysfs-block | 36 ++++++++++++++++ block/blk-settings.c | 24 +++++++++++ block/blk-sysfs.c | 64 ++++++++++++++++++++++++++++ include/linux/blkdev.h | 12 ++++++ include/uapi/linux/fs.h | 3 ++ 5 files changed, 139 insertions(+) diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block index cd14ecb3c9a5..e0c9be009706 100644 --- a/Documentation/ABI/stable/sysfs-block +++ b/Documentation/ABI/stable/sysfs-block @@ -155,6 +155,42 @@ Description: last zone of the device which may be smaller. +What: /sys/block//queue/copy_offload +Date: November 2022 +Contact: linux-block@vger.kernel.org +Description: + [RW] When read, this file shows whether offloading copy to + device is enabled (1) or disabled (0). Writing '0' to this + file will disable offloading copies for this device. + Writing any '1' value will enable this feature. If device + does not support offloading, then writing 1, will result in + error. + + +What: /sys/block//queue/copy_max_bytes +Date: November 2022 +Contact: linux-block@vger.kernel.org +Description: + [RW] While 'copy_max_bytes_hw' is the hardware limit for the + device, 'copy_max_bytes' setting is the software limit. + Setting this value lower will make Linux issue smaller size + copies from block layer. + + +What: /sys/block//queue/copy_max_bytes_hw +Date: November 2022 +Contact: linux-block@vger.kernel.org +Description: + [RO] Devices that support offloading copy functionality may have + internal limits on the number of bytes that can be offloaded + in a single operation. The `copy_max_bytes_hw` + parameter is set by the device driver to the maximum number of + bytes that can be copied in a single operation. Copy + requests issued to the device must not exceed this limit. + A value of 0 means that the device does not + support copy offload. + + What: /sys/block//queue/crypto/ Date: February 2022 Contact: linux-block@vger.kernel.org diff --git a/block/blk-settings.c b/block/blk-settings.c index 0477c4d527fe..ca6f15a70fdc 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -58,6 +58,8 @@ void blk_set_default_limits(struct queue_limits *lim) lim->zoned = BLK_ZONED_NONE; lim->zone_write_granularity = 0; lim->dma_alignment = 511; + lim->max_copy_sectors_hw = 0; + lim->max_copy_sectors = 0; } /** @@ -81,6 +83,8 @@ void blk_set_stacking_limits(struct queue_limits *lim) lim->max_dev_sectors = UINT_MAX; lim->max_write_zeroes_sectors = UINT_MAX; lim->max_zone_append_sectors = UINT_MAX; + lim->max_copy_sectors_hw = ULONG_MAX; + lim->max_copy_sectors = ULONG_MAX; } EXPORT_SYMBOL(blk_set_stacking_limits); @@ -177,6 +181,22 @@ void blk_queue_max_discard_sectors(struct request_queue *q, } EXPORT_SYMBOL(blk_queue_max_discard_sectors); +/** + * blk_queue_max_copy_sectors_hw - set max sectors for a single copy payload + * @q: the request queue for the device + * @max_copy_sectors: maximum number of sectors to copy + **/ +void blk_queue_max_copy_sectors_hw(struct request_queue *q, + unsigned int max_copy_sectors) +{ + if (max_copy_sectors >= MAX_COPY_TOTAL_LENGTH) + max_copy_sectors = MAX_COPY_TOTAL_LENGTH; + + q->limits.max_copy_sectors_hw = max_copy_sectors; + q->limits.max_copy_sectors = max_copy_sectors; +} +EXPORT_SYMBOL_GPL(blk_queue_max_copy_sectors_hw); + /** * blk_queue_max_secure_erase_sectors - set max sectors for a secure erase * @q: the request queue for the device @@ -572,6 +592,10 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, t->max_segment_size = min_not_zero(t->max_segment_size, b->max_segment_size); + t->max_copy_sectors = min(t->max_copy_sectors, b->max_copy_sectors); + t->max_copy_sectors_hw = min(t->max_copy_sectors_hw, + b->max_copy_sectors_hw); + t->misaligned |= b->misaligned; alignment = queue_limit_alignment_offset(b, start); diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index 93d9e9c9a6ea..82a28a6c2e8a 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -212,6 +212,63 @@ static ssize_t queue_discard_zeroes_data_show(struct request_queue *q, char *pag return queue_var_show(0, page); } +static ssize_t queue_copy_offload_show(struct request_queue *q, char *page) +{ + return queue_var_show(blk_queue_copy(q), page); +} + +static ssize_t queue_copy_offload_store(struct request_queue *q, + const char *page, size_t count) +{ + s64 copy_offload; + ssize_t ret = queue_var_store64(©_offload, page); + + if (ret < 0) + return ret; + + if (copy_offload && !q->limits.max_copy_sectors_hw) + return -EINVAL; + + if (copy_offload) + blk_queue_flag_set(QUEUE_FLAG_COPY, q); + else + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + + return count; +} + +static ssize_t queue_copy_max_hw_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors_hw << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_store(struct request_queue *q, + const char *page, size_t count) +{ + s64 max_copy; + ssize_t ret = queue_var_store64(&max_copy, page); + + if (ret < 0) + return ret; + + if (max_copy & (queue_logical_block_size(q) - 1)) + return -EINVAL; + + max_copy >>= SECTOR_SHIFT; + if (max_copy > q->limits.max_copy_sectors_hw) + max_copy = q->limits.max_copy_sectors_hw; + + q->limits.max_copy_sectors = max_copy; + return count; +} + static ssize_t queue_write_same_max_show(struct request_queue *q, char *page) { return queue_var_show(0, page); @@ -604,6 +661,10 @@ QUEUE_RO_ENTRY(queue_nr_zones, "nr_zones"); QUEUE_RO_ENTRY(queue_max_open_zones, "max_open_zones"); QUEUE_RO_ENTRY(queue_max_active_zones, "max_active_zones"); +QUEUE_RW_ENTRY(queue_copy_offload, "copy_offload"); +QUEUE_RO_ENTRY(queue_copy_max_hw, "copy_max_bytes_hw"); +QUEUE_RW_ENTRY(queue_copy_max, "copy_max_bytes"); + QUEUE_RW_ENTRY(queue_nomerges, "nomerges"); QUEUE_RW_ENTRY(queue_rq_affinity, "rq_affinity"); QUEUE_RW_ENTRY(queue_poll, "io_poll"); @@ -651,6 +712,9 @@ static struct attribute *queue_attrs[] = { &queue_discard_max_entry.attr, &queue_discard_max_hw_entry.attr, &queue_discard_zeroes_data_entry.attr, + &queue_copy_offload_entry.attr, + &queue_copy_max_hw_entry.attr, + &queue_copy_max_entry.attr, &queue_write_same_max_entry.attr, &queue_write_zeroes_max_entry.attr, &queue_zone_append_max_entry.attr, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 43d4e073b111..807ffb5f715d 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -303,6 +303,9 @@ struct queue_limits { unsigned int discard_alignment; unsigned int zone_write_granularity; + unsigned long max_copy_sectors_hw; + unsigned long max_copy_sectors; + unsigned short max_segments; unsigned short max_integrity_segments; unsigned short max_discard_segments; @@ -571,6 +574,7 @@ struct request_queue { #define QUEUE_FLAG_NOWAIT 29 /* device supports NOWAIT */ #define QUEUE_FLAG_SQ_SCHED 30 /* single queue style io dispatch */ #define QUEUE_FLAG_SKIP_TAGSET_QUIESCE 31 /* quiesce_tagset skip the queue*/ +#define QUEUE_FLAG_COPY 32 /* supports copy offload */ #define QUEUE_FLAG_MQ_DEFAULT ((1UL << QUEUE_FLAG_IO_STAT) | \ (1UL << QUEUE_FLAG_SAME_COMP) | \ @@ -591,6 +595,7 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q); test_bit(QUEUE_FLAG_STABLE_WRITES, &(q)->queue_flags) #define blk_queue_io_stat(q) test_bit(QUEUE_FLAG_IO_STAT, &(q)->queue_flags) #define blk_queue_add_random(q) test_bit(QUEUE_FLAG_ADD_RANDOM, &(q)->queue_flags) +#define blk_queue_copy(q) test_bit(QUEUE_FLAG_COPY, &(q)->queue_flags) #define blk_queue_zone_resetall(q) \ test_bit(QUEUE_FLAG_ZONE_RESETALL, &(q)->queue_flags) #define blk_queue_dax(q) test_bit(QUEUE_FLAG_DAX, &(q)->queue_flags) @@ -911,6 +916,8 @@ extern void blk_queue_chunk_sectors(struct request_queue *, unsigned int); extern void blk_queue_max_segments(struct request_queue *, unsigned short); extern void blk_queue_max_discard_segments(struct request_queue *, unsigned short); +extern void blk_queue_max_copy_sectors_hw(struct request_queue *q, + unsigned int max_copy_sectors); void blk_queue_max_secure_erase_sectors(struct request_queue *q, unsigned int max_sectors); extern void blk_queue_max_segment_size(struct request_queue *, unsigned int); @@ -1229,6 +1236,11 @@ static inline unsigned int bdev_discard_granularity(struct block_device *bdev) return bdev_get_queue(bdev)->limits.discard_granularity; } +static inline unsigned int bdev_max_copy_sectors(struct block_device *bdev) +{ + return bdev_get_queue(bdev)->limits.max_copy_sectors; +} + static inline unsigned int bdev_max_secure_erase_sectors(struct block_device *bdev) { diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index b7b56871029c..b3ad173f619c 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -64,6 +64,9 @@ struct fstrim_range { __u64 minlen; }; +/* maximum total copy length */ +#define MAX_COPY_TOTAL_LENGTH (1 << 27) + /* extent-same (dedupe) ioctls; these MUST match the btrfs ioctl definitions */ #define FILE_DEDUPE_RANGE_SAME 0 #define FILE_DEDUPE_RANGE_DIFFERS 1 From patchwork Mon Feb 20 10:53:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13146393 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E796BC678DB for ; Mon, 20 Feb 2023 12:48:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231775AbjBTMs1 (ORCPT ); Mon, 20 Feb 2023 07:48:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39016 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229679AbjBTMsZ (ORCPT ); Mon, 20 Feb 2023 07:48:25 -0500 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C88AE211E for ; Mon, 20 Feb 2023 04:48:21 -0800 (PST) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230220124820epoutp03a5db7d54408863a2b97cc6ddb8140a5c~FiTXJ-yH02305723057epoutp03M for ; Mon, 20 Feb 2023 12:48:20 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230220124820epoutp03a5db7d54408863a2b97cc6ddb8140a5c~FiTXJ-yH02305723057epoutp03M DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1676897300; bh=/o+aP9QRf4FZoRaT/LySJ+iHj8DEObsLDRxvnsfVIzw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FE/XNINBTNGBTI1mnZ9e5zgv8MVVqxNS94h0+ttsn4o8k0uY8HYdHWnOBaKF/qDQy +WjNP2vD23aqovlcFrONg3J076uMkb9RQWQO6Mc3Z8Ww+yKKwZd/SIV6P3KWhwxM8n SuYh0k/C/Fd+x9IPZolzfI1mCwBwhDkzMfcj+bs8= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230220124819epcas5p11f5208a9a418e9c136580224f2c472a8~FiTWPTEAL2968429684epcas5p1j; Mon, 20 Feb 2023 12:48:19 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.181]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4PL2Jx5sj9z4x9Pt; Mon, 20 Feb 2023 12:48:17 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 78.49.06765.11C63F36; Mon, 20 Feb 2023 21:48:17 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20230220105441epcas5p49ffde763aae06db301804175e85f9472~FgwIgKeem2130321303epcas5p4Z; Mon, 20 Feb 2023 10:54:41 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230220105441epsmtrp1b08882d9caa63d273a8b7cc07ca397e5~FgwIdq2h_2429224292epsmtrp1S; Mon, 20 Feb 2023 10:54:41 +0000 (GMT) X-AuditID: b6c32a4b-46dfa70000011a6d-9d-63f36c11e513 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 60.0A.17995.17153F36; Mon, 20 Feb 2023 19:54:41 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230220105437epsmtip2b289a6411a9f285e7bb255dd7d5f15fc~FgwFFbyHG0727007270epsmtip2g; Mon, 20 Feb 2023 10:54:37 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v7 2/8] block: Add copy offload support infrastructure Date: Mon, 20 Feb 2023 16:23:25 +0530 Message-Id: <20230220105336.3810-3-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230220105336.3810-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Tf0xTVxTHc98rr61L5+PHxoWNrT7C+OH4US31YkDUYX0bRsn2h8lGZLW8 FKS0XVsUXJaBRQbNEHAOY3EiQoaCk1iQARWjZVoLY0z5Tcb4EWDTiYB1cfxQ1vJg87/Pued8 77nfc3N4uIeR68tLVekZrUqmpIh1nMa24MBQd6VDHjFgCEJ17XdwdKz4OY5qh4sIVDo7j6PF zi4ctT4uc0ODN5sxdP3CSQxdqr2NIUvFHIZuL08T6KS1D6DJXhOGWoc2ouutdg7qbjlLoPLv J7nI+o0BQ00TOQA1Lpbj6MpfMxx0d+gN1PXc5rbdm+7uiadNI50E3Wwa5tJdv1/l0N2dGbS5 poCg66u+pC2D2QRdaHjsLDg+4kbP3Ogl6BMNNYCu7/icdpjfos0T01jC+o/TolMYWTKjFTIq uTo5VaWIoeI/SnovKVISIQoVRaEtlFAlS2diqLg9CaHSVKVzBJTwsEyZ4TxKkOl0VPi2aK06 Q88IU9Q6fQzFaJKVGrEmTCdL12WoFGEqRr9VFBGxKdJZ+GlayuUKI66p2p851lIIssHp3UbA 50FSDB9NNuBGsI7nQVoALM1v47LBEwD/rl1wYwMHgHNTudiapKdtEbCJFgCXGkowNjBg0Nz+ wHkZj0eQG2HHMs8l8CJ/w2Bzh7+rBidHMVh+rAq4Ep6kFPZXODBXPYcMgPX9CS4UkFGwq8bT hZAMh0Uj7i7kk1vhs9wsl05AukP7mQmOi3HybWi4VrZiAJKVfPjLVwUc9plxMGeqnMuyJ3xo a1hlX/igKG+Vj8BLpy4SrDgXQFO/CbCJWHi8vWjFCU4Gw7qWcPbYD37bfgVjG78KCxcnVkci gE3n1tgfXq47T7DsA/ue5awyDevyRznspAoBvFjaThQDoeklQ6aXDJn+b30e4DXAh9Ho0hWM LlKzWcUc+e+P5ep0M1hZipD4JjA+OhtmBRgPWAHk4ZSXYFngkHsIkmVZRxmtOkmboWR0VhDp nHYJ7vuaXO3cKpU+SSSOihBLJBJx1GaJiPIWBMbY5R6kQqZn0hhGw2jXdBiP75uNdcZ+8cFh j6VT428GnsueLL0n7at0+N9suC9bmu+c8fmwdsGOKrF739nrqw9NJ1FujX4wrv7nLRLF1wV3 +Rv6d1iyenMHA6wVzadtZ9rujP05/08IMT5fljh4QLfdt3lucC/Xe/sn6fsOMVyv3r29UfYb 23gDr+x+/4A0wLLeP69H6lOYeC05Vfx6im2Yr7//5MVo5q4Xhkz4TnQkmmJG/4gxfPbTw50H D24Y2vR0p2SurDrbHvxj4j73mVvBSpuN6JkuOXpLmpdvCTJ2KHb5tMVyGYl5ZsCv/MJY19X+ E0Gzfjsepf0QbUzM83y6f8/ZqXeXAn71ClqocRQbq5eEIWTWCEZxdCkyUQiu1cn+BcI0MPGd BAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrAIsWRmVeSWpSXmKPExsWy7bCSvG5h4Odkg9PzeC3WnzrGbNE04S+z xeq7/WwW0z78ZLb4ffY8s8Xed7NZLW4e2MlksWfRJCaLlauPMlnsXviRyeLo/7dsFpMOXWO0 eHp1FpPF3lvaFnv2nmSxuLxrDpvF/GVP2S0OTW5mstjxpJHRYtvv+cwW616/Z7E4cUva4vzf 46wO4h6Xr3h7zLp/ls1j56y77B7n721k8bh8ttRj06pONo/NS+o9dt9sYPPobX4HVNB6n9Xj /b6rbB59W1Yxemw+Xe3xeZOcx6Ynb5kC+KO4bFJSczLLUov07RK4MtYs7GIuWBJe8XBXL2MD 43T3LkZODgkBE4krh38zdjFycQgJ7GCUeLukgxkiISmx7O8RKFtYYuW/5+wQRY1MEjNmLwXq 4OBgE9CWOP2fAyQuIvCMSeLsvUfMIA6zwDsmiVUPl7GCdAsLuElcX/iZCaSBRUBVYvP1ABCT V8BS4vwqYRBTQkBfov++IIjJKWAl8b2lEqRPCKjgxKTNjCA2r4CgxMmZT1hAbGYBeYnmrbOZ JzAKzEKSmoUktYCRaRWjZGpBcW56brFhgVFearlecWJucWleul5yfu4mRnAka2ntYNyz6oPe IUYmDsZDjBIczEoivP95PycL8aYkVlalFuXHF5XmpBYfYpTmYFES573QdTJeSCA9sSQ1OzW1 ILUIJsvEwSnVwDT1vI64eolr9ALWsCfBApvXdv5iW7/t/MuiDxVzP+3YpDCpqv760/74TfG5 tRG7rni7injwvzdIq2fV1a52M1nX/pGPvyz+/tUHLgyT5mXs/ifz07H2ZvikxIK80gAr0Si/ jizjb3MaFpXY3rtXN3O26bSZUZ1uksL79xsv1jJclh2UfmDp9Ae39+xc/4Zpj6iGabj1fs2S FUtEL6TsXp+YfM3/4bmqiVs1ZqdMjl28yle/caoKW1P82ddiGc4fhMKe9exM1Vbk+VMcHrrj r/efHunbdxX497fIr72T0JGhNdEruuvM47lxu3xLRbRYdvoE98dsrE6N673xMVPozVKLrTqT Dj29taAu11A4UomlOCPRUIu5qDgRACDThsdTAwAA X-CMS-MailID: 20230220105441epcas5p49ffde763aae06db301804175e85f9472 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230220105441epcas5p49ffde763aae06db301804175e85f9472 References: <20230220105336.3810-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Introduce blkdev_issue_copy which takes similar arguments as copy_file_range and performs copy offload between two bdevs. Introduce REQ_COPY copy offload operation flag. Create a read-write bio pair with a token as payload and submitted to the device in order. Read request populates token with source specific information which is then passed with write request. This design is courtesy Mikulas Patocka's token based copy Larger copy will be divided, based on max_copy_sectors limit. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- block/blk-lib.c | 235 ++++++++++++++++++++++++++++++++++++++ block/blk.h | 2 + include/linux/blk_types.h | 25 ++++ include/linux/blkdev.h | 3 + 4 files changed, 265 insertions(+) diff --git a/block/blk-lib.c b/block/blk-lib.c index e59c3069e835..c48cee5b6c98 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -115,6 +115,241 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, } EXPORT_SYMBOL(blkdev_issue_discard); +/* + * For synchronous copy offload/emulation, wait and process all in-flight BIOs. + * This must only be called once all bios have been issued so that the refcount + * can only decrease. This just waits for all bios to make it through + * bio_copy_*_write_end_io. IO errors are propagated through cio->io_error. + */ +static int cio_await_completion(struct cio *cio) +{ + int ret = 0; + + if (cio->endio) + return 0; + + if (atomic_read(&cio->refcount)) { + __set_current_state(TASK_UNINTERRUPTIBLE); + blk_io_schedule(); + } + + ret = cio->comp_len; + kfree(cio); + + return ret; +} + +static void blk_copy_offload_write_end_io(struct bio *bio) +{ + struct copy_ctx *ctx = bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (bio->bi_status) { + clen = (bio->bi_iter.bi_sector << SECTOR_SHIFT) - cio->pos_out; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + } + __free_page(bio->bi_io_vec[0].bv_page); + bio_put(bio); + + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } +} + +static void blk_copy_offload_read_end_io(struct bio *read_bio) +{ + struct copy_ctx *ctx = read_bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (read_bio->bi_status) { + clen = (read_bio->bi_iter.bi_sector << SECTOR_SHIFT) + - cio->pos_in; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + __free_page(read_bio->bi_io_vec[0].bv_page); + bio_put(ctx->write_bio); + bio_put(read_bio); + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } + return; + } + + schedule_work(&ctx->dispatch_work); + bio_put(read_bio); +} + +static void blk_copy_dispatch_work_fn(struct work_struct *work) +{ + struct copy_ctx *ctx = container_of(work, struct copy_ctx, + dispatch_work); + + submit_bio(ctx->write_bio); +} + +/* + * __blk_copy_offload - Use device's native copy offload feature. + * we perform copy operation by sending 2 bio. + * 1. First we send a read bio with REQ_COPY flag along with a token and source + * and length. Once read bio reaches driver layer, device driver adds all the + * source info to token and does a fake completion. + * 2. Once read opration completes, we issue write with REQ_COPY flag with same + * token. In driver layer, token info is used to form a copy offload command. + * + * returns the length of bytes copied or negative error value + */ +static int __blk_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask) +{ + struct cio *cio; + struct copy_ctx *ctx; + struct bio *read_bio, *write_bio; + struct page *token; + sector_t copy_len; + sector_t rem, max_copy_len; + + cio = kzalloc(sizeof(struct cio), GFP_KERNEL); + if (!cio) + return -ENOMEM; + atomic_set(&cio->refcount, 0); + cio->waiter = current; + cio->endio = end_io; + cio->private = private; + + max_copy_len = min(bdev_max_copy_sectors(bdev_in), + bdev_max_copy_sectors(bdev_out)) << SECTOR_SHIFT; + + cio->pos_in = pos_in; + cio->pos_out = pos_out; + cio->comp_len = len; + for (rem = len; rem > 0; rem -= copy_len) { + copy_len = min(rem, max_copy_len); + + token = alloc_page(gfp_mask); + if (unlikely(!token)) + goto err_token; + + ctx = kzalloc(sizeof(struct copy_ctx), gfp_mask); + if (!ctx) + goto err_ctx; + read_bio = bio_alloc(bdev_in, 1, REQ_OP_READ | REQ_COPY + | REQ_SYNC | REQ_NOMERGE, gfp_mask); + if (!read_bio) + goto err_read_bio; + write_bio = bio_alloc(bdev_out, 1, REQ_OP_WRITE + | REQ_COPY | REQ_SYNC | REQ_NOMERGE, gfp_mask); + if (!write_bio) + goto err_write_bio; + + ctx->cio = cio; + ctx->write_bio = write_bio; + INIT_WORK(&ctx->dispatch_work, blk_copy_dispatch_work_fn); + + __bio_add_page(read_bio, token, PAGE_SIZE, 0); + read_bio->bi_iter.bi_size = copy_len; + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_end_io = blk_copy_offload_read_end_io; + read_bio->bi_private = ctx; + + __bio_add_page(write_bio, token, PAGE_SIZE, 0); + write_bio->bi_iter.bi_size = copy_len; + write_bio->bi_end_io = blk_copy_offload_write_end_io; + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_private = ctx; + + atomic_inc(&cio->refcount); + submit_bio(read_bio); + pos_in += copy_len; + pos_out += copy_len; + } + + /* Wait for completion of all IO's*/ + return cio_await_completion(cio); + +err_write_bio: + bio_put(read_bio); +err_read_bio: + kfree(ctx); +err_ctx: + __free_page(token); +err_token: + cio->comp_len = min_t(sector_t, cio->comp_len, (len - rem)); + return cio_await_completion(cio); +} + +static inline int blk_copy_sanity_check(struct block_device *bdev_in, + loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, + size_t len) +{ + unsigned int align = max(bdev_logical_block_size(bdev_out), + bdev_logical_block_size(bdev_in)) - 1; + + if (bdev_read_only(bdev_out)) + return -EPERM; + + if ((pos_in & align) || (pos_out & align) || (len & align) || !len || + len >= MAX_COPY_TOTAL_LENGTH) + return -EINVAL; + + return 0; +} + +static inline bool blk_check_copy_offload(struct request_queue *q_in, + struct request_queue *q_out) +{ + return blk_queue_copy(q_in) && blk_queue_copy(q_out); +} + +/* + * @bdev_in: source block device + * @pos_in: source offset + * @bdev_out: destination block device + * @pos_out: destination offset + * @end_io: end_io function to be called on completion of copy operation, + * for synchronous operation this should be NULL + * @private: end_io function will be called with this private data, should be + * NULL, if operation is synchronous in nature + * @gfp_mask: memory allocation flags (for bio_alloc) + * + * Returns the length of bytes copied or a negative error value + * + * Description: + * Copy source offset from source block device to destination block + * device. length of a source range cannot be zero. Max total length of + * copy is limited to MAX_COPY_TOTAL_LENGTH + */ +int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask) +{ + struct request_queue *q_in = bdev_get_queue(bdev_in); + struct request_queue *q_out = bdev_get_queue(bdev_out); + int ret = -EINVAL; + + ret = blk_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len); + if (ret) + return ret; + + if (blk_check_copy_offload(q_in, q_out)) + ret = __blk_copy_offload(bdev_in, pos_in, bdev_out, pos_out, + len, end_io, private, gfp_mask); + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_issue_copy); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/block/blk.h b/block/blk.h index 4c3b3325219a..6d9924a7d559 100644 --- a/block/blk.h +++ b/block/blk.h @@ -304,6 +304,8 @@ static inline bool bio_may_exceed_limits(struct bio *bio, break; } + if (unlikely(op_is_copy(bio->bi_opf))) + return false; /* * All drivers must accept single-segments bios that are <= PAGE_SIZE. * This is a quick and dirty check that relies on the fact that diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 99be590f952f..bab0ed9c767e 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -422,6 +422,7 @@ enum req_flag_bits { */ /* for REQ_OP_WRITE_ZEROES: */ __REQ_NOUNMAP, /* do not free blocks when zeroing */ + __REQ_COPY, /* copy request */ __REQ_NR_BITS, /* stops here */ }; @@ -451,6 +452,7 @@ enum req_flag_bits { #define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV) #define REQ_SWAP (__force blk_opf_t)(1ULL << __REQ_SWAP) +#define REQ_COPY ((__force blk_opf_t)(1ULL << __REQ_COPY)) #define REQ_FAILFAST_MASK \ (REQ_FAILFAST_DEV | REQ_FAILFAST_TRANSPORT | REQ_FAILFAST_DRIVER) @@ -477,6 +479,11 @@ static inline bool op_is_write(blk_opf_t op) return !!(op & (__force blk_opf_t)1); } +static inline bool op_is_copy(blk_opf_t op) +{ + return (op & REQ_COPY); +} + /* * Check if the bio or request is one that needs special treatment in the * flush state machine. @@ -536,4 +543,22 @@ struct blk_rq_stat { u64 batch; }; +typedef void (cio_iodone_t)(void *private, int comp_len); + +struct cio { + struct task_struct *waiter; /* waiting task (NULL if none) */ + atomic_t refcount; + loff_t pos_in; + loff_t pos_out; + size_t comp_len; + cio_iodone_t *endio; /* applicable for async operation */ + void *private; /* applicable for async operation */ +}; + +struct copy_ctx { + struct cio *cio; + struct work_struct dispatch_work; + struct bio *write_bio; +}; + #endif /* __LINUX_BLK_TYPES_H */ diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 807ffb5f715d..18d5bd7fc3bf 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1063,6 +1063,9 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop); int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp); +int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Mon Feb 20 10:53:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13146396 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9195CC05027 for ; Mon, 20 Feb 2023 12:49:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231841AbjBTMtD (ORCPT ); Mon, 20 Feb 2023 07:49:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40216 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231840AbjBTMs5 (ORCPT ); Mon, 20 Feb 2023 07:48:57 -0500 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71FE483F5 for ; Mon, 20 Feb 2023 04:48:29 -0800 (PST) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230220124827epoutp034ba8af899d1676cc69127e49a368d942~FiTePKG8v2510525105epoutp03Z for ; Mon, 20 Feb 2023 12:48:27 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230220124827epoutp034ba8af899d1676cc69127e49a368d942~FiTePKG8v2510525105epoutp03Z DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1676897308; bh=8/F80kzEGLQrWJdvDBY+tqHCsIARuwfbkHI8rsmvefo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SMFuzZevh3j1k1GR9YfqIjpNSZDhOFNAgGBI1Ix05Ait6G0kT2CllTBYNowAPUVPO dB4GwF+6DTZNe3xn57DkoaKjeuKZYITkungqiKrlOqpZRu6NR8C51XYThMcrWKSJ3+ 5M5BQLzo4X2SPqHUBE/2Xkd4PU0aC3wiB3m3/+Do= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230220124827epcas5p10c02fcbfc8ad7c1c8406a09cbcaf92d8~FiTdncesG1770117701epcas5p1W; Mon, 20 Feb 2023 12:48:27 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.180]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4PL2K54hf2z4x9Pv; Mon, 20 Feb 2023 12:48:25 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id DA.16.10528.91C63F36; Mon, 20 Feb 2023 21:48:25 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230220105453epcas5p32b30be5c5b4b21fe0bd4500a560b2b9b~FgwUJrj5W0346703467epcas5p3f; Mon, 20 Feb 2023 10:54:53 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230220105453epsmtrp2e436f7f567238314bcf78ceaf15ef1b0~FgwUHPt5j1664116641epsmtrp2F; Mon, 20 Feb 2023 10:54:53 +0000 (GMT) X-AuditID: b6c32a49-c17ff70000012920-79-63f36c19eb85 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 13.0A.17995.D7153F36; Mon, 20 Feb 2023 19:54:53 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230220105446epsmtip2a5a77a55c14a7a033feaba4e4c350199~FgwNNaPcD0727007270epsmtip2i; Mon, 20 Feb 2023 10:54:46 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Vincent Fu , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v7 3/8] block: add emulation for copy Date: Mon, 20 Feb 2023 16:23:26 +0530 Message-Id: <20230220105336.3810-4-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230220105336.3810-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTVxzHd+4tt8XY7QounOKGWCAMkEddYaeMTrK55RpxIVFZMnHYlDtA Slv7EGFj4x3AYFWGY2UqTOMQGEhFUh51pgx5WXAwmBB5GOr+EHm4LrSG11oLm/99zu/3/T1P fizcTcP0ZKVIVbRCKpJwiS2Mls6Ad4I5Eos4bLHECzX23cNR7rlVHNVNaAh0cfEFjpZNgzgy zFe6oLG7rRjq+OkChm7UdWGovfo5hrrW5wh0wTgK0JMRLYYM40Gow9DLQMNtPxLoyvUnTGQs y8OQ3pwDUMvyFRw1zC4wUM/4DjRzpgigwdVul2hIDf9xgNJOmQiqVTvBpAYnmxjUsElN6WqL CerWtW+p9rFsgirNm7cLCqZcqIU7IwR1trkWULf6v6IsOi9KZ57DYt/4PDUqmRYl0gpvWiqW JaZIk4TcA4cSPkoIjwjjBfME6D2ut1SURgu5+2Jigz9Jkdj3wPU+JZKo7aZYkVLJDf0gSiFT q2jvZJlSJeTS8kSJnC8PUYrSlGppUoiUVkXywsL2hNuFx1OTr96xAXll9Onxgd+Z2WCSXwJc WZDkw9LnhaAEbGG5ke0Ajl9t2Xj8DeDqowqGQ+VGWgDsuOm+GZFn07k4RW0AmmfMhFOUh8Hf zgpLAItFkEGwf53lMG8nH2Gwtd/HocfJAhya7+qZDoc7GQ6bp6cxBzNIPzhVuoI7mE0KYMfl asKRB5KhUDO1zYGuZCS05mc4Fdtg7w/ml63h5E6Yd7sSd6SHZJ0rrNWsYM4+98FK25iLk93h 0+5mppM9oWXeQDg5Hd74roZwBucDqP1TC5yOvbCgT4M7CuNkAGxsC3Wa34blfQ2Ys/DrsHTZ vFGLDfWXN9kH1jdWbeTnwFFrzgZTsKrmMebcWymAZ1ZWiXPAW/vKQNpXBtL+X7oK4LWAQ8uV aUm0MlzOk9Lp//2xWJamAy8vI3C/HkxML4YYAcYCRgBZOHc7e51tEbuxE0UZmbRClqBQS2il EYTb930e93xTLLOfllSVwOMLwvgRERF8wbsRPK4H21/YK3Yjk0QqOpWm5bRiMw5juXpmYzuS juqWV4oW2M1HX0yufvN184mhpNGeU9wVo99I/kOfiRNfBFZvjd6T3XkpzuNw12vMnV9GH7bO zsbP9AX4Hi9p9F8rfgY7XHJjcgYMHxd+Sp0OFvjXTC7u50Te51y0Hnz/w+vl+gJrrmDX3geG X7aqB+hfv9+V2Wn9zOTb31SWZasOiT8SJ8EOvVVxPlQihClrP1cIKk8uPWy7X996qbPM1RSV mSyJY+lQY2H8Xxkx0xlLN/WMxzbYz7EN9Mw8a+wbqszySDcOux07GNLzj7q2yGI7hsGmxCGj +N6aRzs9frtcc8Trqe/uoGvFD/wW8yXd9Q1NS1NZJlGaOmP3fM7JOS5DmSziBeIKpehffcLW eaIEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02RaUwTYRCG/XaX7dKkuq01foCIqVekWkRBP4kSYjw23hgxohiosAEjLU0L Kp4g4lFEEFFDMSJiiEAQKUe4Y4oVC6moIIRGaJXDAwQFUUBrtaKJ/56Z95mZH0PhggbCmToo j2aVcmmkiOQS5fUityUn/UdClxbWi1FR42McnUm14qigM4VE1z+N4+i7sRlHtYOZDqjjYSWG au6kYSivQI+h6uzPGNLbPpIoTdcGUO9LDYZqTWJUU2sgUEvVTRJl5fZykO5qAoYqeuIBKv+e haP7/UMEemJyQd1JFwBqtjY4+EGmpXUzozEbSaZS08lhmruKCabFGMNo8y+STMnd00x1RxzJ JCcM/hYSzQ7MUN1Lkrlcmg+YkqbjzIh2NqPt+YjtmLaXuzqMjTx4mFV6+IZwI3LqxoAi0++o 6elzThzo8lIDRwrSXjBhTOugBlxKQFcA+LbxBz4ZOMFc66O/PB3m/XzLmZTiMdh3pRioAUWR tBg22Sh7X0j3YdDY9Qa3Fzh9DYej1QOkfXo67Q1LLRbMzgQ9H5qTJy/w6FWw5lY2aV8EaQ+Y Yubb0ZH2gd/OxtoNwW/jSVoJmLT50JDRQ9gZp91gQlkmngpozX+R5r/oNsDygROrUMnCZSpP xTI5e0SikspUMfJwSWiUTAv+/NrdvQLU5H+S6ABGAR2AFC4S8my8kVABL0wae4xVRgUrYyJZ lQ64UIRoJu+Z2hAsoMOl0ewhllWwyn8pRjk6x2Fl2168Cmm9kRHIHQh1qdcHXlqfDhtLC1wn OIXDQn23q5qJtbYbbW1fjJbcxxvVFS2+yYOzy7ZzDXND5p1d2yV8lxodsGKgSB2/29srOKjO sO/UsC12zcqGXQ+iBopGukHY+mdTh/hJ8d79bVNMfnvavxa2jnZmVYUsbvqAnctwk3hO3I44 X1zfPTxjYf/4urlJ7118s8ab2EUHuBxxIG+TJD1gYolKk15ZbXWqIiaO2so150dXDxUZLQtk ozf4s/wDfE5vWbSQUIgP/ThxoOPppnvtW+cs3YDJ97fvXCN6HZT4Yti15K4h0yI2Lef7SDzL TIl6oUdfbs5Y9jKzV2N4kohQRUg93XGlSvoL8FFAo1oDAAA= X-CMS-MailID: 20230220105453epcas5p32b30be5c5b4b21fe0bd4500a560b2b9b X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230220105453epcas5p32b30be5c5b4b21fe0bd4500a560b2b9b References: <20230220105336.3810-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org For the devices which does not support copy, copy emulation is added. It is required for in-kernel users like fabrics, where file descriptor is not available and hence they can't use copy_file_range. Copy-emulation is implemented by reading from source into memory and writing to the corresponding destination asynchronously. Also emulation is used, if copy offload fails or partially completes. Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu Signed-off-by: Anuj Gupta --- block/blk-lib.c | 171 ++++++++++++++++++++++++++++++++++++++++- block/blk-map.c | 4 +- include/linux/blkdev.h | 3 + 3 files changed, 175 insertions(+), 3 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index c48cee5b6c98..74f58faf82d8 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -289,6 +289,169 @@ static int __blk_copy_offload(struct block_device *bdev_in, loff_t pos_in, return cio_await_completion(cio); } +static void *blk_alloc_buf(sector_t req_size, sector_t *alloc_size, + gfp_t gfp_mask) +{ + int min_size = PAGE_SIZE; + void *buf; + + while (req_size >= min_size) { + buf = kvmalloc(req_size, gfp_mask); + if (buf) { + *alloc_size = req_size; + return buf; + } + /* retry half the requested size */ + req_size >>= 1; + } + + return NULL; +} + +static void blk_copy_emulate_write_end_io(struct bio *bio) +{ + struct copy_ctx *ctx = bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (bio->bi_status) { + clen = (bio->bi_iter.bi_sector << SECTOR_SHIFT) - cio->pos_out; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + } + kvfree(page_address(bio->bi_io_vec[0].bv_page)); + bio_map_kern_endio(bio); + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } +} + +static void blk_copy_emulate_read_end_io(struct bio *read_bio) +{ + struct copy_ctx *ctx = read_bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (read_bio->bi_status) { + clen = (read_bio->bi_iter.bi_sector << SECTOR_SHIFT) - + cio->pos_in; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + __free_page(read_bio->bi_io_vec[0].bv_page); + bio_map_kern_endio(read_bio); + kfree(ctx); + + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } + } + schedule_work(&ctx->dispatch_work); + kfree(read_bio); +} + +/* + * If native copy offload feature is absent, this function tries to emulate, + * by copying data from source to a temporary buffer and from buffer to + * destination device. + * returns the length of bytes copied or negative error value + */ +static int __blk_copy_emulate(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask) +{ + struct request_queue *in = bdev_get_queue(bdev_in); + struct request_queue *out = bdev_get_queue(bdev_out); + struct bio *read_bio, *write_bio; + void *buf = NULL; + struct copy_ctx *ctx; + struct cio *cio; + sector_t buf_len, req_len, rem = 0; + sector_t max_src_hw_len = min_t(unsigned int, + queue_max_hw_sectors(in), + queue_max_segments(in) << (PAGE_SHIFT - SECTOR_SHIFT)) + << SECTOR_SHIFT; + sector_t max_dst_hw_len = min_t(unsigned int, + queue_max_hw_sectors(out), + queue_max_segments(out) << (PAGE_SHIFT - SECTOR_SHIFT)) + << SECTOR_SHIFT; + sector_t max_hw_len = min_t(unsigned int, + max_src_hw_len, max_dst_hw_len); + + cio = kzalloc(sizeof(struct cio), GFP_KERNEL); + if (!cio) + return -ENOMEM; + atomic_set(&cio->refcount, 0); + cio->pos_in = pos_in; + cio->pos_out = pos_out; + cio->waiter = current; + cio->endio = end_io; + cio->private = private; + + for (rem = len; rem > 0; rem -= buf_len) { + req_len = min_t(int, max_hw_len, rem); + + buf = blk_alloc_buf(req_len, &buf_len, gfp_mask); + if (!buf) + goto err_alloc_buf; + + ctx = kzalloc(sizeof(struct copy_ctx), gfp_mask); + if (!ctx) + goto err_ctx; + + read_bio = bio_map_kern(in, buf, buf_len, gfp_mask); + if (IS_ERR(read_bio)) + goto err_read_bio; + + write_bio = bio_map_kern(out, buf, buf_len, gfp_mask); + if (IS_ERR(write_bio)) + goto err_write_bio; + + ctx->cio = cio; + ctx->write_bio = write_bio; + INIT_WORK(&ctx->dispatch_work, blk_copy_dispatch_work_fn); + + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_iter.bi_size = buf_len; + read_bio->bi_opf = REQ_OP_READ | REQ_SYNC; + bio_set_dev(read_bio, bdev_in); + read_bio->bi_end_io = blk_copy_emulate_read_end_io; + read_bio->bi_private = ctx; + + write_bio->bi_iter.bi_size = buf_len; + write_bio->bi_opf = REQ_OP_WRITE | REQ_SYNC; + bio_set_dev(write_bio, bdev_out); + write_bio->bi_end_io = blk_copy_emulate_write_end_io; + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_private = ctx; + + atomic_inc(&cio->refcount); + submit_bio(read_bio); + + pos_in += buf_len; + pos_out += buf_len; + } + + /* Wait for completion of all IO's*/ + return cio_await_completion(cio); + +err_write_bio: + bio_put(read_bio); +err_read_bio: + kfree(ctx); +err_ctx: + kvfree(buf); +err_alloc_buf: + cio->comp_len -= min_t(sector_t, cio->comp_len, len - rem); + return cio_await_completion(cio); +} + static inline int blk_copy_sanity_check(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len) @@ -337,15 +500,21 @@ int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, struct request_queue *q_in = bdev_get_queue(bdev_in); struct request_queue *q_out = bdev_get_queue(bdev_out); int ret = -EINVAL; + bool offload = false; ret = blk_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len); if (ret) return ret; - if (blk_check_copy_offload(q_in, q_out)) + offload = blk_check_copy_offload(q_in, q_out); + if (offload) ret = __blk_copy_offload(bdev_in, pos_in, bdev_out, pos_out, len, end_io, private, gfp_mask); + if ((ret != len) || !offload) + ret = __blk_copy_emulate(bdev_in, pos_in + ret, bdev_out, + pos_out + ret, len - ret, end_io, private, gfp_mask); + return ret; } EXPORT_SYMBOL_GPL(blkdev_issue_copy); diff --git a/block/blk-map.c b/block/blk-map.c index 19940c978c73..bcf8db2b75f1 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -363,7 +363,7 @@ static void bio_invalidate_vmalloc_pages(struct bio *bio) #endif } -static void bio_map_kern_endio(struct bio *bio) +void bio_map_kern_endio(struct bio *bio) { bio_invalidate_vmalloc_pages(bio); bio_uninit(bio); @@ -380,7 +380,7 @@ static void bio_map_kern_endio(struct bio *bio) * Map the kernel address into a bio suitable for io to a block * device. Returns an error pointer in case of error. */ -static struct bio *bio_map_kern(struct request_queue *q, void *data, +struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, gfp_t gfp_mask) { unsigned long kaddr = (unsigned long)data; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 18d5bd7fc3bf..766761911190 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1066,6 +1066,9 @@ int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); +struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, + gfp_t gfp_mask); +void bio_map_kern_endio(struct bio *bio); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Mon Feb 20 10:53:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13146397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1F60C64EC4 for ; Mon, 20 Feb 2023 12:49:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231982AbjBTMtR (ORCPT ); Mon, 20 Feb 2023 07:49:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231844AbjBTMtK (ORCPT ); Mon, 20 Feb 2023 07:49:10 -0500 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C82FE1CAF3 for ; Mon, 20 Feb 2023 04:48:35 -0800 (PST) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230220124834epoutp038d5869fac4b6670a69051cfec0d60bd0~FiTkDMmid2509725097epoutp03b for ; Mon, 20 Feb 2023 12:48:34 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230220124834epoutp038d5869fac4b6670a69051cfec0d60bd0~FiTkDMmid2509725097epoutp03b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1676897314; bh=s/gsPXisGnHj5x7odGOR5QtxX7QomcTNLxphM7Mn6Jo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=U+QkyB7xiR/lG0EmC76+zd2cZCF0gXRDhI4JidOMQCJgLsi8Y28XHAOSUZm10MpLY vquAL5rKSeg1vizzH58NCC49WzejBDwhd4TUoMZ/A5HWnjEtyKqHMLrjUxRz8jYwCa 6xGOO1/erjRkg4Li53HdYZ4yWaoES6LhR8RJKBBc= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230220124833epcas5p2cc50788c454811886b7df2a4aafacecd~FiTjaW5Oh1601516015epcas5p2X; Mon, 20 Feb 2023 12:48:33 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.179]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4PL2KC5KBjz4x9Pt; Mon, 20 Feb 2023 12:48:31 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 70.59.06765.F1C63F36; Mon, 20 Feb 2023 21:48:31 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230220105503epcas5p2cb05eb671de6954b7604e49ebd23ce41~Fgwc5-wb01660316603epcas5p2D; Mon, 20 Feb 2023 10:55:03 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230220105503epsmtrp23bda6ee826dd76ec1000659f955cf78d~Fgwc4wiAU1664116641epsmtrp2V; Mon, 20 Feb 2023 10:55:03 +0000 (GMT) X-AuditID: b6c32a4b-20fff70000011a6d-b6-63f36c1f3f33 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id D8.8C.05839.78153F36; Mon, 20 Feb 2023 19:55:03 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230220105458epsmtip2562bae17173ca10a9ba68190220eb74b~FgwY0Ift80747307473epsmtip2s; Mon, 20 Feb 2023 10:54:58 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v7 4/8] fs, block: copy_file_range for def_blk_ops for direct block device. Date: Mon, 20 Feb 2023 16:23:27 +0530 Message-Id: <20230220105336.3810-5-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230220105336.3810-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02TfUxTVxjGc+4tty2z2xVRDpiNWuIcGKAV6E4NOBbQ3QQXWdyWbHFiV+6A UNraliGwjO8FQVagbsy6AQoYoAsdFR1fnayICEiIImwICBgwAwOInYBD1rW7sPnf7zzv85z3 nPfkcHC302wvToJCS6sVUrmAcGVd7fT19feW22TCgjt7kKn3Bo6yi9dxZBzXEejbx89wtNY/ gCPLwnkXNNLRgqH2i6UYqjN2YajtwhKGuuzzBCq1DgM0M2TAkOXeXtRu6WGhwdbvCVRxaYaN rPocDDVPZwF0da0CRw2PFlno5r2daGC92yXcgxq8G0UZJvoJqsUwzqYG7jeyqMH+ZMpcf5qg LldnUG0jmQRVlLPgMORNuFCLvwwR1NdN9YC63JdO2cyvUebpeSz6lY8TQ+NpaSyt5tMKmTI2 QREXJog6GhMREyIWivxFEvSmgK+QJtFhgsjD0f6HEuSOEQj4n0vlyQ4pWqrRCAIPhKqVyVqa H6/UaMMEtCpWrgpWBWikSZpkRVyAgtbuFwmF+0IcxhOJ8aO5c5jK/vqpunIjngnM/ALA5UAy GD7M/wlzshvZBqDVuKUAuDr4CYB/j+sAs1gGcGZ0Cd9MlFyrwZiCxZEoXMGZeA4GO1d3FwAO hyD3wj47xym7k2MYbOnzcfpxchKDFdnVwFnYRn4CTbVjbKefRe6GX3WmOWUeKYGL3YWYU4Zk INRNbHUil9wPV3JTGcdW2HNumuVknPSGOVfO487dIVnFhc9tHYA5ZiS8PjjlwvA2ONfdxGbY C9oWLATDKbDubC3BhHMBNPxm2Ai/BfN6dbizMU76QlNrICO/Cr/pbcCYxi/DorVpjNF5sLl8 k33gj6bKjf094fBK1gZTsHGohmDGVgSg/WEHuxjwDS9cyPDChQz/t64EeD3wpFWapDhaE6IK UtAp/z2xTJlkBv/+Cb+oZvBg8nGAFWAcYAWQgwvceXaeTebGi5WmptFqZYw6WU5rrCDEMe4S 3Gu7TOn4VAptjChYIgwWi8XBkiCxSODB2xPWI3Mj46RaOpGmVbR6M4dxuF6Z2KM/j/l4V3Tf Uhhrf3Bt6tWnPxe2RCg7j1Zlt+2z/PXpFPF7xq4DSSPveJS8tMqblLybpqi8tqPmqf7M4OzK wV8/GIAjw0+K9RdbD3amB7nFJk5kf1GjW5aL2LcX8pt4n93Fl2p4J0+421q5w7vGjtusxlHD 7bddD0viClPeSCuWmWasnlOpJYl+o1H6xiuHjt04eX3nbEQperbMCw+VpNeXRU5xzgTFmD7q vfTdh3Pm41Pvp9SBmez52e3z+XeiJQ/aqyfUR5IapNHvZXQO+5zTjaStinPS87xPPV382WaX 7AhPzfpDb90SXhyyzr2Q2XUro4NbVlte9eU862ZZX0TBkftdApYmXiryw9Ua6T8u6XEBnAQA AA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrAIsWRmVeSWpSXmKPExsWy7bCSvG574Odkg/5mPov1p44xWzRN+Mts sfpuP5vFtA8/mS1+nz3PbLH33WxWi5sHdjJZ7Fk0icli5eqjTBa7F35ksjj6/y2bxaRD1xgt nl6dxWSx95a2xZ69J1ksLu+aw2Yxf9lTdotDk5uZLHY8aWS02PZ7PrPFutfvWSxO3JK2OP/3 OKuDuMflK94es+6fZfPYOesuu8f5extZPC6fLfXYtKqTzWPzknqP3Tcb2Dx6m98BFbTeZ/V4 v+8qm0ffllWMHptPV3t83iTnsenJW6YA/igum5TUnMyy1CJ9uwSujNstr5gK/qtVrJy3mrmB cZNCFyMnh4SAicTE/UuZQGwhgd2MEstXsELEJSWW/T3CDGELS6z895y9i5ELqKaRSeLEr89A DgcHm4C2xOn/HCBxEYFnTBJn7z1iBnGYBd4xSax6uAxskrBAtMT/uR9ZQRpYBFQl2g5XgYR5 BSwl3h/vZgIJSwjoS/TfFwQxOQWsJL63VEKcYylxYtJmRohqQYmTM5+wgNjMAvISzVtnM09g FJiFJDULSWoBI9MqRsnUguLc9NxiwwLDvNRyveLE3OLSvHS95PzcTYzgSNbS3MG4fdUHvUOM TByMhxglOJiVRHj/835OFuJNSaysSi3Kjy8qzUktPsQozcGiJM57oetkvJBAemJJanZqakFq EUyWiYNTqoEp0PxteVATo43V1uAVK7QtD1Rq8KisTilWu5edovdNsf1E+molh122Zw7V1FgH Xr/M9fhZJnPrI7nbwqujnkT1vDrK6HPP99/txYqXd0qraL2p1TUtdrm9IL966QopWWueQ91T Bfddlpp/k6Xu1tK3SZta9jaqdfXukij569kj8C6Xcb3/v7O+CzwqLs44+VDbx9Hz6qUz75bd mLv0/d4bIhLHu/+8XGKX039Ff5PD4j1xbj/7Deeu+6ixwEbTY7+GwcIlr6518p87sNz4hFlM I98REcPtDc1OrPn69eFT10Z9c+Q5vs5bcZoi+8pvTDabM3+5l3dd6pnWOefG/et2bJPlzmi+ V1s8e+/kPDV3JZbijERDLeai4kQAGpeiJVMDAAA= X-CMS-MailID: 20230220105503epcas5p2cb05eb671de6954b7604e49ebd23ce41 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230220105503epcas5p2cb05eb671de6954b7604e49ebd23ce41 References: <20230220105336.3810-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org For direct block device, use copy_file_range to issue device copy offload, and fallback to generic_copy_file_range incase device copy offload capability is absent. Modify checks to allow bdevs to use copy_file_range. Suggested-by: Ming Lei Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty --- block/blk-lib.c | 22 ++++++++++++++++++++++ block/fops.c | 18 ++++++++++++++++++ fs/read_write.c | 11 +++++++++-- include/linux/blkdev.h | 3 +++ 4 files changed, 52 insertions(+), 2 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index 74f58faf82d8..6593de525a26 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -475,6 +475,28 @@ static inline bool blk_check_copy_offload(struct request_queue *q_in, return blk_queue_copy(q_in) && blk_queue_copy(q_out); } +int blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask) +{ + struct request_queue *in_q = bdev_get_queue(bdev_in); + struct request_queue *out_q = bdev_get_queue(bdev_out); + int ret = -EINVAL; + bool offload = false; + + ret = blk_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len); + if (ret) + return ret; + + offload = blk_check_copy_offload(in_q, out_q); + if (offload) + ret = __blk_copy_offload(bdev_in, pos_in, bdev_out, pos_out, + len, end_io, private, gfp_mask); + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_copy_offload); + /* * @bdev_in: source block device * @pos_in: source offset diff --git a/block/fops.c b/block/fops.c index 50d245e8c913..bcb9ee6565ea 100644 --- a/block/fops.c +++ b/block/fops.c @@ -596,6 +596,23 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to) return ret; } +ssize_t blkdev_copy_file_range(struct file *file_in, loff_t pos_in, + struct file *file_out, loff_t pos_out, + size_t len, unsigned int flags) +{ + struct block_device *in_bdev = I_BDEV(bdev_file_inode(file_in)); + struct block_device *out_bdev = I_BDEV(bdev_file_inode(file_out)); + int comp_len; + + comp_len = blkdev_copy_offload(in_bdev, pos_in, out_bdev, pos_out, len, + NULL, NULL, GFP_KERNEL); + if (comp_len != len) + comp_len = generic_copy_file_range(file_in, pos_in + comp_len, + file_out, pos_out + comp_len, len - comp_len, flags); + + return comp_len; +} + #define BLKDEV_FALLOC_FL_SUPPORTED \ (FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE | \ FALLOC_FL_ZERO_RANGE | FALLOC_FL_NO_HIDE_STALE) @@ -679,6 +696,7 @@ const struct file_operations def_blk_fops = { .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, .fallocate = blkdev_fallocate, + .copy_file_range = blkdev_copy_file_range, }; static __init int blkdev_init(void) diff --git a/fs/read_write.c b/fs/read_write.c index 7a2ff6157eda..62e925e9b2f0 100644 --- a/fs/read_write.c +++ b/fs/read_write.c @@ -20,6 +20,7 @@ #include #include #include +#include #include "internal.h" #include @@ -1448,7 +1449,11 @@ static int generic_copy_file_checks(struct file *file_in, loff_t pos_in, return -EOVERFLOW; /* Shorten the copy to EOF */ - size_in = i_size_read(inode_in); + if (S_ISBLK(inode_in->i_mode)) + size_in = bdev_nr_bytes(I_BDEV(file_in->f_mapping->host)); + else + size_in = i_size_read(inode_in); + if (pos_in >= size_in) count = 0; else @@ -1709,7 +1714,9 @@ int generic_file_rw_checks(struct file *file_in, struct file *file_out) /* Don't copy dirs, pipes, sockets... */ if (S_ISDIR(inode_in->i_mode) || S_ISDIR(inode_out->i_mode)) return -EISDIR; - if (!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) + + if ((!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) && + (!S_ISBLK(inode_in->i_mode) || !S_ISBLK(inode_out->i_mode))) return -EINVAL; if (!(file_in->f_mode & FMODE_READ) || diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 766761911190..ca7828b25d90 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1066,6 +1066,9 @@ int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); +int blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask); struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, gfp_t gfp_mask); void bio_map_kern_endio(struct bio *bio); From patchwork Mon Feb 20 10:53:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13146398 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09658C64EC4 for ; Mon, 20 Feb 2023 12:49:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232183AbjBTMtd (ORCPT ); Mon, 20 Feb 2023 07:49:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232151AbjBTMtO (ORCPT ); Mon, 20 Feb 2023 07:49:14 -0500 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D7B91CF79 for ; Mon, 20 Feb 2023 04:48:41 -0800 (PST) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230220124839epoutp033afbe2d10ed2e91cdb8a9c1b0ebc44cf~FiTpIxX-12509725097epoutp03c for ; Mon, 20 Feb 2023 12:48:39 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230220124839epoutp033afbe2d10ed2e91cdb8a9c1b0ebc44cf~FiTpIxX-12509725097epoutp03c DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1676897319; bh=bVQ7kicVZIOPaeJOrYYdOYdwRoFGTReeAtGN5nn3CRU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NDfONTj1XfFVgo5RJ6h6ZCg+Vmjt8M/+HBUbZMHfqoEKXD3azqLiQXzsh6LXLpejf NFb7+91ytcZv/bQiba+tE0aqZEOnsvHV7kFjB59eixTMOYvJl+qEA2NM26l8K1Z6h5 CnM60FJ2wGUmJVbz1nP7G7AJ/NdI0lXD6Kfmau48= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230220124838epcas5p3dcf70f2b730e00dd49ee1752af5fa9c5~FiToHmIjm1791917919epcas5p3T; Mon, 20 Feb 2023 12:48:38 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.177]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4PL2KK0Qzsz4x9Pp; Mon, 20 Feb 2023 12:48:37 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 34.A6.55678.42C63F36; Mon, 20 Feb 2023 21:48:36 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230220105512epcas5p15e1171c05e5ceda614d4f196672ad678~FgwlA0KTy1670816708epcas5p1N; Mon, 20 Feb 2023 10:55:12 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230220105512epsmtrp2889425e694cd431aaa5daf1cb3c64625~Fgwk-_7pQ1664116641epsmtrp2d; Mon, 20 Feb 2023 10:55:12 +0000 (GMT) X-AuditID: b6c32a4a-909fc7000000d97e-94-63f36c247ede Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 4C.8C.05839.F8153F36; Mon, 20 Feb 2023 19:55:11 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230220105508epsmtip29ca3624ae0ed188e3d91fa3ca62445f2~FgwhyO3SW0727007270epsmtip2o; Mon, 20 Feb 2023 10:55:08 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , =?utf-8?q?Javier_Gonz=C3=A1lez?= , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v7 5/8] nvme: add copy offload support Date: Mon, 20 Feb 2023 16:23:28 +0530 Message-Id: <20230220105336.3810-6-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230220105336.3810-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Te0xbZRTfvbf0saRweUw/QBy7my4UgRahfkVwxi3zVjQi6hIXJ63lCgi0 TUsFZrKV8UYZbAqyIgUmewCuSJkEBhVWhI02DAVaHgGZE2Zk4yF1WxAbbCno/vv9zvn9zvnO +XKYmFcZw4+ZIs2gFFJxGkHfSWvvC+KE7EuzSbh3hqNgi2kAg6fK7Rhsnimjw8qVNQyuDw1j 0LBU7QYneztR2H3+LAobm/tR2FX/Jwp/m7YxYP/GIh2eNVoROG/RoNAwFQy7DYM0OHrtazqs vTjPgMYvclHYMZeDwPb1Wgzq7i3T4M0pfzhsv+H2MiBHx2JJzewQnezUzDDI4V9aaeTokIrU NxXTybaGk2TXpJpOluYuOQT5s27k8g8WOnn6ahNCtpk/JW36p0n93CIa53E0NTqZEidSikBK KpElpkiTYojYtxMOJkTyubwQngC+QARKxelUDHHo9biQwylpjj0QgZ+I01SOUJxYqSTCXopW yFQZVGCyTJkRQ1DyxDR5hDxUKU5XqqRJoVIqI4rH5YZHOoSi1OSiW7do8pLMrHq7UI3oRCUI iwnwCFBVVuxWguxkeuFdCBioXqO7yCoC2sx1qIs8RID5fim6bVm//gBxJQwIaLCatkguChom Vh2EyaTjwcC8wXQafPBpFHSa9zo1GF6HgSFLN82Z8Mb5YE1nYzgxDX8GnCtZxJyYjQuArrzV zVkH4GGgbNbTCVl4FHiUl+1SeILBc3ObVTB8N8j9vhpzlgd4FwtocyboroceAnZtDc2FvcHC jasMF/YDtiXDliYTNH55me4y5yFAM65BXIkDIN9UhjkbY3gQaLkW5goHgAqTDnU1dgel63Nb S2GDDu023gu+banbqu8LrI9ytjAJfqwxb2IvvBQB9vnj5Uig5rF5NI/No/m/cx2CNSG+lFyZ nkQpI+XhUirzvz+WyNL1yOZlcF7rQH69vRJqRFAmYkQAEyN82Btsm8SLnSjOPk4pZAkKVRql NCKRjnWfwfx2SWSO05JmJPAiBNwIPp8fIXiezyOeZO+PGZR44UniDCqVouSUYtuHMll+arSi OCrM753n4k8SU9nsAMlP+lMxI8jIs/LdT/T8fS+g9OKR2AVVJj7w16vvR3H2X6/sz5kQsT6K nDSK1g2CAmFC28GxC9WWFcmMtaLd2jP2cEXeu/C79wViYKiquUDN+Wf07vCR+GXh7cOFvZ2v DEbgb35c83mBKX+Hj7DyMw9m63f7RMd0l9w73vP944rlpjog3DuveGMlSL9H2LMc+wbBvqtl xX4wFi9S/6ydLtJWdRSOZz114itmsPWMb9/pO8f8z48oBf5vVbrPc4U7li95cvqWHuzJMued +LC2fpem7OhskeXFbxYmV01N2YWNHmHv2oUc8YHoK5fFQWvj3oz7rbPWYIKmTBbzOJhCKf4X a9tDxKIEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA03Se1BMcRQHcL/7u3v3tmN1d/P4edasIWpEmJ1fJsZ4zFzErMGM5+hO7rRo s/YWYSjWcw0lNFqP5Jliadejp8dSSbWhZdTIzlBiqDY7ZDVbXDH89z3nfM7569BQ+ZocQq+N T+AN8VycipKRtx+qAselLvTETLj5IQJff1IO8a40H8R5jakUznB7Ie6qqYW4tO2kBNffLyRw ybl0Al/JKyNwcXYHgd+99khxWU8rhdPtLwFufmEmcGlDKC4prSRxXdEpCmddapZi+1EjgQua dgJ8uysLYsundhI/bhiKa30VkumIrXPOY82uGootNDdK2do3+SRbV5PIWnMPUKztQjJbXJ9C sYeMbb/AHpeEbb/7gmIP38wFrK1qG+uxjmCtTa2Exn+5LHINH7d2E28YPy1apt3vcJB60+ak bN+cFGCJNgE/GjGTUdeDr8AEZLSSKQbIcaQO9g4Go0u+R39yALrS3SLtRTsJ5O3cR5kATVNM KKrqocV+f+Y9gWrevIViARkLRNeMFyXidgCjRl6LRypmkhmFMk2tv6/KmQhkScuXiIcQMx6l uhRi9GOmoM7dW0Sh/CUep9tAr1agyswmUiSQCUbXzyjFNmQCkfHWSZgGFOb/lPmfMv+nzgKY CwbzekEXqxPC9eHx/OYwgdMJifGxYTEbdFbw+wFCxhaAO7nuMDsgaGAHiIaq/vIeuSdGKV/D bdnKGzasNiTG8YIdDKVJ1SD5U1PlaiUTyyXw63lezxv+Tgnab0gKMfPgV+ewvCVvv3WMlm7t yJj0XKMLcMh6Qr3y2c9fhQgLqrPz+racjVQUDGjbqNa5vmcdHN2dMGZy2LWGw00TT2dFjVg/ UOgz3DglfBWnvfrlRHXydKX7sps5UfKp+of3eLflicM/pdw/w9keFFukuXd1bwcXsWLs/ejA yMaKHWou4cacZ93VzuBMv4pNpxwrFy1122bMt5qSNTky9dRzLt+SkTkf8/stnpuzvSjxhnRB Z2FBy7Ao5zpYxkxqKwd39PpMY5X8R5TtvHqhJjBoR3bSimWKmae/vbocjB3zm/X1i8FUlvZv WEl53S+TPhNnat+la4/tXzprXs7dY7sHHjdqD6lIQcuFh0CDwP0Eoub2L28DAAA= X-CMS-MailID: 20230220105512epcas5p15e1171c05e5ceda614d4f196672ad678 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230220105512epcas5p15e1171c05e5ceda614d4f196672ad678 References: <20230220105336.3810-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org For device supporting native copy, nvme driver receives read and write request with BLK_COPY op flags. For read request the nvme driver populates the payload with source information. For write request the driver converts it to nvme copy command using the source information in the payload and submits to the device. current design only supports single source range. This design is courtesy Mikulas Patocka's token based copy trace event support for nvme_copy_cmd. Set the device copy limits to queue limits. Signed-off-by: Kanchan Joshi Signed-off-by: Nitesh Shetty Signed-off-by: Javier González Signed-off-by: Anuj Gupta --- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 106 +++++++++++++++++++++++++++++++++- drivers/nvme/host/fc.c | 5 ++ drivers/nvme/host/nvme.h | 7 +++ drivers/nvme/host/pci.c | 27 ++++++++- drivers/nvme/host/rdma.c | 7 +++ drivers/nvme/host/tcp.c | 16 +++++ drivers/nvme/host/trace.c | 19 ++++++ include/linux/nvme.h | 43 +++++++++++++- 9 files changed, 223 insertions(+), 8 deletions(-) diff --git a/drivers/nvme/host/constants.c b/drivers/nvme/host/constants.c index e958d5015585..4e60946d3aa8 100644 --- a/drivers/nvme/host/constants.c +++ b/drivers/nvme/host/constants.c @@ -19,6 +19,7 @@ static const char * const nvme_ops[] = { [nvme_cmd_resv_report] = "Reservation Report", [nvme_cmd_resv_acquire] = "Reservation Acquire", [nvme_cmd_resv_release] = "Reservation Release", + [nvme_cmd_copy] = "Copy Offload", [nvme_cmd_zone_mgmt_send] = "Zone Management Send", [nvme_cmd_zone_mgmt_recv] = "Zone Management Receive", [nvme_cmd_zone_append] = "Zone Management Append", diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 8b6421141162..4b1a1a3fad83 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -753,6 +753,80 @@ static inline void nvme_setup_flush(struct nvme_ns *ns, cmnd->common.nsid = cpu_to_le32(ns->head->ns_id); } +static inline blk_status_t nvme_setup_copy_read(struct nvme_ns *ns, + struct request *req) +{ + struct bio *bio = req->bio; + struct nvme_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + + memcpy(token->subsys, "nvme", 4); + token->ns = ns; + token->src_sector = bio->bi_iter.bi_sector; + token->sectors = bio->bi_iter.bi_size >> 9; + + return BLK_STS_OK; +} + +static inline blk_status_t nvme_setup_copy_write(struct nvme_ns *ns, + struct request *req, struct nvme_command *cmnd) +{ + struct nvme_copy_range *range = NULL; + struct bio *bio = req->bio; + struct nvme_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + sector_t src_sector, dst_sector, n_sectors; + u64 src_lba, dst_lba, n_lba; + unsigned short nr_range = 1; + u16 control = 0; + + if (unlikely(memcmp(token->subsys, "nvme", 4))) + return BLK_STS_NOTSUPP; + if (unlikely(token->ns != ns)) + return BLK_STS_NOTSUPP; + + src_sector = token->src_sector; + dst_sector = bio->bi_iter.bi_sector; + n_sectors = token->sectors; + if (WARN_ON(n_sectors != bio->bi_iter.bi_size >> 9)) + return BLK_STS_NOTSUPP; + + src_lba = nvme_sect_to_lba(ns, src_sector); + dst_lba = nvme_sect_to_lba(ns, dst_sector); + n_lba = nvme_sect_to_lba(ns, n_sectors); + + if (WARN_ON(!n_lba)) + return BLK_STS_NOTSUPP; + + if (req->cmd_flags & REQ_FUA) + control |= NVME_RW_FUA; + + if (req->cmd_flags & REQ_FAILFAST_DEV) + control |= NVME_RW_LR; + + memset(cmnd, 0, sizeof(*cmnd)); + cmnd->copy.opcode = nvme_cmd_copy; + cmnd->copy.nsid = cpu_to_le32(ns->head->ns_id); + cmnd->copy.sdlba = cpu_to_le64(dst_lba); + + range = kmalloc_array(nr_range, sizeof(*range), + GFP_ATOMIC | __GFP_NOWARN); + if (!range) + return BLK_STS_RESOURCE; + + range[0].slba = cpu_to_le64(src_lba); + range[0].nlb = cpu_to_le16(n_lba - 1); + + cmnd->copy.nr_range = 0; + + req->special_vec.bv_page = virt_to_page(range); + req->special_vec.bv_offset = offset_in_page(range); + req->special_vec.bv_len = sizeof(*range) * nr_range; + req->rq_flags |= RQF_SPECIAL_PAYLOAD; + + cmnd->copy.control = cpu_to_le16(control); + + return BLK_STS_OK; +} + static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req, struct nvme_command *cmnd) { @@ -979,10 +1053,16 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req) ret = nvme_setup_discard(ns, req, cmd); break; case REQ_OP_READ: - ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_read); + if (unlikely(req->cmd_flags & REQ_COPY)) + ret = nvme_setup_copy_read(ns, req); + else + ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_read); break; case REQ_OP_WRITE: - ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_write); + if (unlikely(req->cmd_flags & REQ_COPY)) + ret = nvme_setup_copy_write(ns, req, cmd); + else + ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_write); break; case REQ_OP_ZONE_APPEND: ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_zone_append); @@ -1731,6 +1811,26 @@ static void nvme_config_discard(struct gendisk *disk, struct nvme_ns *ns) blk_queue_max_write_zeroes_sectors(queue, UINT_MAX); } +static void nvme_config_copy(struct gendisk *disk, struct nvme_ns *ns, + struct nvme_id_ns *id) +{ + struct nvme_ctrl *ctrl = ns->ctrl; + struct request_queue *q = disk->queue; + + if (!(ctrl->oncs & NVME_CTRL_ONCS_COPY)) { + blk_queue_max_copy_sectors_hw(q, 0); + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + return; + } + + /* setting copy limits */ + if (blk_queue_flag_test_and_set(QUEUE_FLAG_COPY, q)) + return; + + blk_queue_max_copy_sectors_hw(q, + nvme_lba_to_sect(ns, le16_to_cpu(id->mssrl))); +} + static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b) { return uuid_equal(&a->uuid, &b->uuid) && @@ -1930,6 +2030,7 @@ static void nvme_update_disk_info(struct gendisk *disk, set_capacity_and_notify(disk, capacity); nvme_config_discard(disk, ns); + nvme_config_copy(disk, ns, id); blk_queue_max_write_zeroes_sectors(disk->queue, ns->ctrl->max_zeroes_sectors); } @@ -5323,6 +5424,7 @@ static inline void _nvme_check_size(void) BUILD_BUG_ON(sizeof(struct nvme_download_firmware) != 64); BUILD_BUG_ON(sizeof(struct nvme_format_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_dsm_cmd) != 64); + BUILD_BUG_ON(sizeof(struct nvme_copy_command) != 64); BUILD_BUG_ON(sizeof(struct nvme_write_zeroes_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_abort_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_get_log_page_command) != 64); diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 456ee42a6133..0ec8f9cd0e09 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -2807,6 +2807,11 @@ nvme_fc_queue_rq(struct blk_mq_hw_ctx *hctx, if (ret) return ret; + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_end_request(rq, BLK_STS_OK); + return BLK_STS_OK; + } /* * nvme core doesn't quite treat the rq opaquely. Commands such * as WRITE ZEROES will return a non-zero rq payload_bytes yet diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 424c8a467a0c..6e282956deb4 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -500,6 +500,13 @@ struct nvme_ns { }; +struct nvme_copy_token { + char subsys[4]; + struct nvme_ns *ns; + u64 src_sector; + u64 sectors; +}; + /* NVMe ns supports metadata actions by the controller (generate/strip) */ static inline bool nvme_ns_has_pi(struct nvme_ns *ns) { diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 6d88ddd35565..0b1dc5c457ad 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -499,16 +499,19 @@ static inline void nvme_sq_copy_cmd(struct nvme_queue *nvmeq, nvmeq->sq_tail = 0; } -static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +static inline void nvme_commit_sq_db(struct nvme_queue *nvmeq) { - struct nvme_queue *nvmeq = hctx->driver_data; - spin_lock(&nvmeq->sq_lock); if (nvmeq->sq_tail != nvmeq->last_sq_tail) nvme_write_sq_db(nvmeq, true); spin_unlock(&nvmeq->sq_lock); } +static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +{ + nvme_commit_sq_db(hctx->driver_data); +} + static void **nvme_pci_iod_list(struct request *req) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); @@ -898,6 +901,12 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, struct request *req) if (ret) return ret; + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_start_request(req); + return BLK_STS_OK; + } + if (blk_rq_nr_phys_segments(req)) { ret = nvme_map_data(dev, req, &iod->cmd); if (ret) @@ -944,6 +953,18 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, ret = nvme_prep_rq(dev, req); if (unlikely(ret)) return ret; + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_set_request_complete(req); + blk_mq_end_request(req, BLK_STS_OK); + /* Commit the sq if copy read was the last req in the list, + * as copy read deoesn't update sq db + */ + if (bd->last) + nvme_commit_sq_db(nvmeq); + return ret; + } + spin_lock(&nvmeq->sq_lock); nvme_sq_copy_cmd(nvmeq, &iod->cmd); nvme_write_sq_db(nvmeq, bd->last); diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index bbad26b82b56..a8bf2a87f42a 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -2043,6 +2043,13 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, nvme_start_request(rq); + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_end_request(rq, BLK_STS_OK); + ret = BLK_STS_OK; + goto unmap_qe; + } + if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) && queue->pi_support && (c->common.opcode == nvme_cmd_write || diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 8cedc1ef496c..776e2ba84911 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -2346,6 +2346,11 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns, if (ret) return ret; + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + return BLK_STS_OK; + } + req->state = NVME_TCP_SEND_CMD_PDU; req->status = cpu_to_le16(NVME_SC_SUCCESS); req->offset = 0; @@ -2414,6 +2419,17 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx, nvme_start_request(rq); + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_set_request_complete(rq); + blk_mq_end_request(rq, BLK_STS_OK); + /* if copy read is the last req queue tcp reqs */ + if (bd->last && nvme_tcp_queue_more(queue)) + queue_work_on(queue->io_cpu, nvme_tcp_wq, + &queue->io_work); + return ret; + } + nvme_tcp_queue_request(req, true, bd->last); return BLK_STS_OK; diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c index 1c36fcedea20..da4a7494e5a7 100644 --- a/drivers/nvme/host/trace.c +++ b/drivers/nvme/host/trace.c @@ -150,6 +150,23 @@ static const char *nvme_trace_read_write(struct trace_seq *p, u8 *cdw10) return ret; } +static const char *nvme_trace_copy(struct trace_seq *p, u8 *cdw10) +{ + const char *ret = trace_seq_buffer_ptr(p); + u64 slba = get_unaligned_le64(cdw10); + u8 nr_range = get_unaligned_le16(cdw10 + 8); + u16 control = get_unaligned_le16(cdw10 + 10); + u32 dsmgmt = get_unaligned_le32(cdw10 + 12); + u32 reftag = get_unaligned_le32(cdw10 + 16); + + trace_seq_printf(p, + "slba=%llu, nr_range=%u, ctrl=0x%x, dsmgmt=%u, reftag=%u", + slba, nr_range, control, dsmgmt, reftag); + trace_seq_putc(p, 0); + + return ret; +} + static const char *nvme_trace_dsm(struct trace_seq *p, u8 *cdw10) { const char *ret = trace_seq_buffer_ptr(p); @@ -243,6 +260,8 @@ const char *nvme_trace_parse_nvm_cmd(struct trace_seq *p, return nvme_trace_zone_mgmt_send(p, cdw10); case nvme_cmd_zone_mgmt_recv: return nvme_trace_zone_mgmt_recv(p, cdw10); + case nvme_cmd_copy: + return nvme_trace_copy(p, cdw10); default: return nvme_trace_common(p, cdw10); } diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 4fad4aa245fb..e92dd69c745a 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -337,7 +337,7 @@ struct nvme_id_ctrl { __u8 nvscc; __u8 nwpc; __le16 acwu; - __u8 rsvd534[2]; + __le16 ocfs; __le32 sgls; __le32 mnan; __u8 rsvd544[224]; @@ -365,6 +365,7 @@ enum { NVME_CTRL_ONCS_WRITE_ZEROES = 1 << 3, NVME_CTRL_ONCS_RESERVATIONS = 1 << 5, NVME_CTRL_ONCS_TIMESTAMP = 1 << 6, + NVME_CTRL_ONCS_COPY = 1 << 8, NVME_CTRL_VWC_PRESENT = 1 << 0, NVME_CTRL_OACS_SEC_SUPP = 1 << 0, NVME_CTRL_OACS_NS_MNGT_SUPP = 1 << 3, @@ -414,7 +415,10 @@ struct nvme_id_ns { __le16 npdg; __le16 npda; __le16 nows; - __u8 rsvd74[18]; + __le16 mssrl; + __le32 mcl; + __u8 msrc; + __u8 rsvd91[11]; __le32 anagrpid; __u8 rsvd96[3]; __u8 nsattr; @@ -796,6 +800,7 @@ enum nvme_opcode { nvme_cmd_resv_report = 0x0e, nvme_cmd_resv_acquire = 0x11, nvme_cmd_resv_release = 0x15, + nvme_cmd_copy = 0x19, nvme_cmd_zone_mgmt_send = 0x79, nvme_cmd_zone_mgmt_recv = 0x7a, nvme_cmd_zone_append = 0x7d, @@ -818,7 +823,8 @@ enum nvme_opcode { nvme_opcode_name(nvme_cmd_resv_release), \ nvme_opcode_name(nvme_cmd_zone_mgmt_send), \ nvme_opcode_name(nvme_cmd_zone_mgmt_recv), \ - nvme_opcode_name(nvme_cmd_zone_append)) + nvme_opcode_name(nvme_cmd_zone_append), \ + nvme_opcode_name(nvme_cmd_copy)) @@ -995,6 +1001,36 @@ struct nvme_dsm_range { __le64 slba; }; +struct nvme_copy_command { + __u8 opcode; + __u8 flags; + __u16 command_id; + __le32 nsid; + __u64 rsvd2; + __le64 metadata; + union nvme_data_ptr dptr; + __le64 sdlba; + __u8 nr_range; + __u8 rsvd12; + __le16 control; + __le16 rsvd13; + __le16 dspec; + __le32 ilbrt; + __le16 lbat; + __le16 lbatm; +}; + +struct nvme_copy_range { + __le64 rsvd0; + __le64 slba; + __le16 nlb; + __le16 rsvd18; + __le32 rsvd20; + __le32 eilbrt; + __le16 elbat; + __le16 elbatm; +}; + struct nvme_write_zeroes_cmd { __u8 opcode; __u8 flags; @@ -1752,6 +1788,7 @@ struct nvme_command { struct nvme_download_firmware dlfw; struct nvme_format_cmd format; struct nvme_dsm_cmd dsm; + struct nvme_copy_command copy; struct nvme_write_zeroes_cmd write_zeroes; struct nvme_zone_mgmt_send_cmd zms; struct nvme_zone_mgmt_recv_cmd zmr; From patchwork Mon Feb 20 10:53:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13146399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01888C05027 for ; Mon, 20 Feb 2023 12:49:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231265AbjBTMt5 (ORCPT ); Mon, 20 Feb 2023 07:49:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232177AbjBTMtZ (ORCPT ); Mon, 20 Feb 2023 07:49:25 -0500 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D1241D913 for ; Mon, 20 Feb 2023 04:48:47 -0800 (PST) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230220124845epoutp01bff818fdbb3f264ac79d437d314738e4~FiTuu92k41416414164epoutp016 for ; Mon, 20 Feb 2023 12:48:45 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230220124845epoutp01bff818fdbb3f264ac79d437d314738e4~FiTuu92k41416414164epoutp016 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1676897325; bh=hlngMEoG/BfQdcVrdcgCbSfAL4Q2ta3nA5lHwKWrJg0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qW4/OU0DaggEAB7BRVgLTbPl0se+L6Nw4vgR7NB8iym+pXXNxEawTt52UAEr6rhOz 6ER0q6VWU6dR1r373613/rzTOW+aC1T23pbp+QMhv3WU0ywNPDYolTO933OJnbmvHa 9XN2WKjmALS3hd7PHapMiPNiETY92Ah6Rl+ijmN8= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230220124844epcas5p4e8f677be24951b53fbcd1d1a74a7926b~FiTtzMczc2866528665epcas5p4I; Mon, 20 Feb 2023 12:48:44 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.181]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4PL2KQ4WLhz4x9Pr; Mon, 20 Feb 2023 12:48:42 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 47.A6.55678.A2C63F36; Mon, 20 Feb 2023 21:48:42 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230220105520epcas5p34fe94decec6573d9b7b3df5df9286ef7~FgwtH6krJ2350123501epcas5p35; Mon, 20 Feb 2023 10:55:20 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230220105520epsmtrp127665fdad1b26414bdde90921e6badbc~FgwtGvynU2386223862epsmtrp1-; Mon, 20 Feb 2023 10:55:20 +0000 (GMT) X-AuditID: b6c32a4a-6a3ff7000000d97e-9f-63f36c2a3b30 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 4E.8C.05839.89153F36; Mon, 20 Feb 2023 19:55:20 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230220105517epsmtip2c9ff687be9db028018092cc6b781f914~Fgwp2VCrY0503205032epsmtip2f; Mon, 20 Feb 2023 10:55:17 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v7 6/8] nvmet: add copy command support for bdev and file ns Date: Mon, 20 Feb 2023 16:23:29 +0530 Message-Id: <20230220105336.3810-7-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230220105336.3810-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Te0xbdRTO795yW0iqF4bhN5ax5i7LBMaj2NbLpBvioneDOcKi0aFh13Kl SGlrH244F6C8hMk6QBZX2OiQoTAEBKx0HThRLNAQNDAQkAGRLoAZFNE9YBPbtej++84533ee ORzU7ww7kJMu1zAqOS0jMB+W6YfgkLAQ2aok0n5jO9ky8BNK6s49QsmrU3qMPO94gJLrg0Mo 2bVU5UWO3zAj5PXacoRsuNqLkJbLKwjZu3EHI8t7RgFpv2lAyK6JUPJ6Vz+LHL5WjZE19XY2 2VORh5Cdc7mANK3XoGTzH8sssm9iGzn0yOoVG0ANj8RThulBjDIbptjU0K2vWdTwoJZqayzG qPa6bMoynoNRpXlLTkLBtBe13H0To852NAKq3XaKWm0Lotrm7iCJTx/LiJEydCqj4jFyiSI1 XZ4mJuKPpryUIhRF8sP40eTzBE9OZzJi4kBCYtjL6TLnCgjeB7RM63Ql0mo1EbEvRqXQahie VKHWiAlGmSpTCpThajpTrZWnhcsZzV5+ZGSU0Ek8niFtXrGhyof7T3Yv2r1ygElYArw5EBfA 9oXfsRLgw/HDLQDOTK55jD8B/GXYgriNuwD+tdaHbUqu/Kz3BLoA1K9Y2W4jD4G1rausEsDh YHgotG1wXAJ//DcEmm07XRwUn0Fgja4OuAJb8AS4dnGR7cIsfBesaFxiuTAXj4YXBxrZrjwQ j4D6aV8X9Mb3wnv5WW6GL+y/MPeYjeI7YN43VagrPcS/8IZ3y+o8jR6A1spyxI23wEVrB9uN A+GCvtCDT8CGT7/E3OJ8AA1jBuAO7IcFA3rUVRjFg2HLtQi3ezusHGhG3IWfgqXrc578XNh5 aRPvhE0tRk8PW+HovVzMPQoFh6wS96pKAWy1ziHnAM/wxDyGJ+Yx/F/ZCNBGsJVRqjPTGLVQ GSVnTvx3ZIkisw08/oqQQ51gdsYR3gMQDugBkIMS/twN7qrEj5tKZ33IqBQpKq2MUfcAoXPd ZWjgMxKF863kmhS+IDpSIBKJBNHPifhEAHe3uF/ih6fRGiaDYZSMalOHcLwDc5DQ0NjTV4ou 2L2lTaIXb1U4xtXiSV394YkjvW+98/ex4jPvFVSPnJ5psCT3OUbqjiYIPznLvp8xchv49Ask xWnZJr/Ln6lUr+he6Hgz2BLwY1xL+0d0VKW/4/7YtspVh/Z9pDToNqUd/bw6+dd8cdK3WbuM D4IJrXV+ut4WdLLz2bE1NHa+UsErahvc4SWdzx04NDXL2HT1E/ETrXt27S6K0X23Qn+80RzI Dor7vnypatJssvcqil49ePi8rmTfKEM8jP1q9t3sJHC8e6l22Zz0ujL5bew186nC0tzs2VCJ KbPpks63Kqwsbk+Aztdox42Di8axQvaRN/6xzosWmiPWZwiWWkrzQ1CVmv4Xvlng5p4EAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02Re0hTYRjG+845Ozsaq+OU/FTSWEVl5ZSyvqKmFdWJMiqIblItO5nk5trR zPojb0Eq5dC8tHkrauisSTNveSFWs3kZWlp0dVQuKdlUVpiamVOC/nve9/k97wvvS+FCI+FL xcjjWaVcGisi3Ynap6KA1YX7nVHB1nIRqmpvxVGqahJHlR+zSZQ/PIajCUsXjpodGh56+6QB Q013cjBUUWnCUOPtEQyZpuwkyjG+Bsj2So2h5ncrUVNzG4F6HheRqFRr4yNjbhqG6vtTAKqd KMWRfnCIQOZ3fqhr8jkv3Jvp6d3NqK0WkmlQf+QzXX0PCabHksAYdBkkU333CtP4Nplkrqc5 poGrVh4z1PKKZG480gGmuuMy4zT4M4Z+O7Zv/lH3TafZ2JgLrFIsOel+Vj/SgSt+h11s+W7j JYPa0EzgRkF6LbzXnY1lAndKSDcCqDHoiVnDB2onn+Gz2hNW/Bngz0IpGDQXVYJMQFEkvRJ2 TFGuvhf9FYOWvs+4q8BpBwZ1n7Q8V9qT3gPHi7/zXZqgl8JcnWNmg4DeAIvbdXzXIEiLYbbV wyXd6I1wND3JRQinCXNONZilPWDbrf6ZJE4HwLQaDa4CtPo/S/2fVQYwHfBhFZwsWsaFKELk bGIQJ5VxCfLooKg4mQHMvDlwRT2o0w0HGQFGASOAFC7yEkwJnFFCwWlp0iVWGXdCmRDLckbg RxEib0F3ZtsJIR0tjWfPsayCVf5zMcrNNxkTh450VmWLnEsaHd/2PymSdBMl6/Xyz3nbxr8E SyWaHarhVrttRViiWum7mm1wHFx/7VBfzd5Ibe+iNf6nthR6HUvviGjPfRCnWHXbrEr/PX1O +4GSrHHfNVbT9sjRn6ZOznozPSLUmVCsOX9o7MvkvAFQHrg561dLU3DBuUudVeLB5Un8rIwP vUfaLPpUbaGy1+PlxHOwwCdgzta6+7Gpywr8dxE/TM0DXja3uWrFEvnOM4fzU6bK/caOV9R9 kuTM2ffm0ULDcH2wSbZ5Ez7+MN473EyGqY++r4lp9eRkaWWR9xZXSAoW5O1+nNxEcuKIdS+2 UQWqIt4BPOSqvSxMRHBnpSGBuJKT/gVUrOT4VQMAAA== X-CMS-MailID: 20230220105520epcas5p34fe94decec6573d9b7b3df5df9286ef7 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230220105520epcas5p34fe94decec6573d9b7b3df5df9286ef7 References: <20230220105336.3810-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Add support for handling target command on target. For bdev-ns we call into blkdev_issue_copy, which the block layer completes by a offloaded copy request to backend bdev or by emulating the request. For file-ns we call vfs_copy_file_range to service our request. Currently target always shows copy capability by setting NVME_CTRL_ONCS_COPY in controller ONCS. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- drivers/nvme/target/admin-cmd.c | 9 +++-- drivers/nvme/target/io-cmd-bdev.c | 58 +++++++++++++++++++++++++++++++ drivers/nvme/target/io-cmd-file.c | 52 +++++++++++++++++++++++++++ drivers/nvme/target/loop.c | 6 ++++ drivers/nvme/target/nvmet.h | 1 + 5 files changed, 124 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c index 6a54ed6fb121..66a45121f5a0 100644 --- a/drivers/nvme/target/admin-cmd.c +++ b/drivers/nvme/target/admin-cmd.c @@ -433,8 +433,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) id->nn = cpu_to_le32(NVMET_MAX_NAMESPACES); id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES); id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM | - NVME_CTRL_ONCS_WRITE_ZEROES); - + NVME_CTRL_ONCS_WRITE_ZEROES | NVME_CTRL_ONCS_COPY); /* XXX: don't report vwc if the underlying device is write through */ id->vwc = NVME_CTRL_VWC_PRESENT; @@ -536,6 +535,12 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req) if (req->ns->bdev) nvmet_bdev_set_limits(req->ns->bdev, id); + else { + id->msrc = (u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16(BIO_MAX_VECS << + (PAGE_SHIFT - SECTOR_SHIFT)); + id->mcl = cpu_to_le32(le16_to_cpu(id->mssrl)); + } /* * We just provide a single LBA format that matches what the diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c index c2d6cea0236b..0af273097aa4 100644 --- a/drivers/nvme/target/io-cmd-bdev.c +++ b/drivers/nvme/target/io-cmd-bdev.c @@ -46,6 +46,19 @@ void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id) id->npda = id->npdg; /* NOWS = Namespace Optimal Write Size */ id->nows = to0based(bdev_io_opt(bdev) / bdev_logical_block_size(bdev)); + + /*Copy limits*/ + if (bdev_max_copy_sectors(bdev)) { + id->msrc = id->msrc; + id->mssrl = cpu_to_le16((bdev_max_copy_sectors(bdev) << + SECTOR_SHIFT) / bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32(id->mssrl); + } else { + id->msrc = (u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16((BIO_MAX_VECS << PAGE_SHIFT) / + bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32(id->mssrl); + } } void nvmet_bdev_ns_disable(struct nvmet_ns *ns) @@ -184,6 +197,19 @@ static void nvmet_bio_done(struct bio *bio) nvmet_req_bio_put(req, bio); } +static void nvmet_bdev_copy_end_io(void *private, int comp_len) +{ + struct nvmet_req *req = (struct nvmet_req *)private; + + if (comp_len == req->copy_len) { + req->cqe->result.u32 = cpu_to_le32(1); + nvmet_req_complete(req, errno_to_nvme_status(req, 0)); + } else { + req->cqe->result.u32 = cpu_to_le32(0); + nvmet_req_complete(req, blk_to_nvme_status(req, BLK_STS_IOERR)); + } +} + #ifdef CONFIG_BLK_DEV_INTEGRITY static int nvmet_bdev_alloc_bip(struct nvmet_req *req, struct bio *bio, struct sg_mapping_iter *miter) @@ -450,6 +476,34 @@ static void nvmet_bdev_execute_write_zeroes(struct nvmet_req *req) } } +/* At present we handle only one range entry */ +static void nvmet_bdev_execute_copy(struct nvmet_req *req) +{ + struct nvme_copy_range range; + struct nvme_command *cmnd = req->cmd; + int ret; + + + ret = nvmet_copy_from_sgl(req, 0, &range, sizeof(range)); + if (ret) + goto out; + + ret = blkdev_issue_copy(req->ns->bdev, + le64_to_cpu(cmnd->copy.sdlba) << req->ns->blksize_shift, + req->ns->bdev, + le64_to_cpu(range.slba) << req->ns->blksize_shift, + (le16_to_cpu(range.nlb) + 1) << req->ns->blksize_shift, + nvmet_bdev_copy_end_io, (void *)req, GFP_KERNEL); + if (ret) { + req->cqe->result.u32 = cpu_to_le32(0); + nvmet_req_complete(req, blk_to_nvme_status(req, BLK_STS_IOERR)); + } + + return; +out: + nvmet_req_complete(req, errno_to_nvme_status(req, ret)); +} + u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) { switch (req->cmd->common.opcode) { @@ -468,6 +522,10 @@ u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_bdev_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_bdev_execute_copy; + return 0; + default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c index 871c4f32f443..76db307f5b30 100644 --- a/drivers/nvme/target/io-cmd-file.c +++ b/drivers/nvme/target/io-cmd-file.c @@ -328,6 +328,49 @@ static void nvmet_file_dsm_work(struct work_struct *w) } } +static void nvmet_file_copy_work(struct work_struct *w) +{ + struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); + int nr_range; + loff_t pos; + struct nvme_command *cmnd = req->cmd; + int ret = 0, len = 0, src, id; + + nr_range = cmnd->copy.nr_range + 1; + pos = le64_to_cpu(req->cmd->copy.sdlba) << req->ns->blksize_shift; + if (unlikely(pos + req->transfer_len > req->ns->size)) { + nvmet_req_complete(req, errno_to_nvme_status(req, -ENOSPC)); + return; + } + + for (id = 0 ; id < nr_range; id++) { + struct nvme_copy_range range; + + ret = nvmet_copy_from_sgl(req, id * sizeof(range), &range, + sizeof(range)); + if (ret) + goto out; + + len = (le16_to_cpu(range.nlb) + 1) << (req->ns->blksize_shift); + src = (le64_to_cpu(range.slba) << (req->ns->blksize_shift)); + ret = vfs_copy_file_range(req->ns->file, src, req->ns->file, + pos, len, 0); +out: + if (ret != len) { + pos += ret; + req->cqe->result.u32 = cpu_to_le32(id); + nvmet_req_complete(req, ret < 0 ? + errno_to_nvme_status(req, ret) : + errno_to_nvme_status(req, -EIO)); + return; + + } else + pos += len; + } + + nvmet_req_complete(req, 0); + +} static void nvmet_file_execute_dsm(struct nvmet_req *req) { if (!nvmet_check_data_len_lte(req, nvmet_dsm_len(req))) @@ -336,6 +379,12 @@ static void nvmet_file_execute_dsm(struct nvmet_req *req) queue_work(nvmet_wq, &req->f.work); } +static void nvmet_file_execute_copy(struct nvmet_req *req) +{ + INIT_WORK(&req->f.work, nvmet_file_copy_work); + queue_work(nvmet_wq, &req->f.work); +} + static void nvmet_file_write_zeroes_work(struct work_struct *w) { struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); @@ -382,6 +431,9 @@ u16 nvmet_file_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_file_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_file_execute_copy; + return 0; default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c index f2d24b2d992f..d18ed8067a15 100644 --- a/drivers/nvme/target/loop.c +++ b/drivers/nvme/target/loop.c @@ -146,6 +146,12 @@ static blk_status_t nvme_loop_queue_rq(struct blk_mq_hw_ctx *hctx, return ret; nvme_start_request(req); + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_set_request_complete(req); + blk_mq_end_request(req, BLK_STS_OK); + return BLK_STS_OK; + } iod->cmd.common.flags |= NVME_CMD_SGL_METABUF; iod->req.port = queue->ctrl->port; if (!nvmet_req_init(&iod->req, &queue->nvme_cq, diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h index 89bedfcd974c..69ed4c8469e5 100644 --- a/drivers/nvme/target/nvmet.h +++ b/drivers/nvme/target/nvmet.h @@ -393,6 +393,7 @@ struct nvmet_req { struct device *p2p_client; u16 error_loc; u64 error_slba; + size_t copy_len; }; #define NVMET_MAX_MPOOL_BVEC 16 From patchwork Mon Feb 20 10:53:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13146400 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8256C64EC4 for ; Mon, 20 Feb 2023 12:50:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232152AbjBTMt7 (ORCPT ); Mon, 20 Feb 2023 07:49:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40672 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232143AbjBTMtb (ORCPT ); Mon, 20 Feb 2023 07:49:31 -0500 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8361E1D922 for ; Mon, 20 Feb 2023 04:48:52 -0800 (PST) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230220124851epoutp0180e1ed909873ccc19fb369072a621f76~FiTzu21Sd1637616376epoutp01I for ; Mon, 20 Feb 2023 12:48:51 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230220124851epoutp0180e1ed909873ccc19fb369072a621f76~FiTzu21Sd1637616376epoutp01I DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1676897331; bh=xxiDP6QZKfeqkiRLoXjxlMN+FlItKe4BO652Ksxx5Yk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rvb4p4s828aNF+EMJ0SMueQ08Q8Q+Dx+98xkLJoCt9BFmxRb7qJcHwT98PT6hhWA0 YbH63omXTe4A0f/0hW42jIZlmkbo7sLINKHZIKIW0/NNB6azu9rRIy3rXdOQSBDHdJ G9mw1bvUWa08BZJU8OGtanHeGxpC4AEjDLCyBdek= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230220124850epcas5p16d10e6638f58a2515de2b776d0de93ae~FiTy_AN3w2067920679epcas5p1a; Mon, 20 Feb 2023 12:48:50 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.178]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4PL2KX2zpcz4x9Px; Mon, 20 Feb 2023 12:48:48 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id D4.59.06765.03C63F36; Mon, 20 Feb 2023 21:48:48 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20230220105529epcas5p447cd449f13da54794e391b78bd1f5956~Fgw1DkzdR2307823078epcas5p4Q; Mon, 20 Feb 2023 10:55:29 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230220105529epsmtrp1f7f3bedff31f86864840edd7f762c623~Fgw1Cf1mh2429224292epsmtrp1q; Mon, 20 Feb 2023 10:55:29 +0000 (GMT) X-AuditID: b6c32a4b-46dfa70000011a6d-cf-63f36c30aefe Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 9E.0A.17995.1A153F36; Mon, 20 Feb 2023 19:55:29 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230220105525epsmtip221cf243ac5da4c7cb295706198a8dd86~FgwyAn1a80747307473epsmtip2z; Mon, 20 Feb 2023 10:55:25 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v7 7/8] dm: Add support for copy offload. Date: Mon, 20 Feb 2023 16:23:30 +0530 Message-Id: <20230220105336.3810-8-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230220105336.3810-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Tf1CTdRzHeZ5nezbI2cMP4evUWjMsQGALWA/EJJPo6bCTOzM4rcZz44ER Y5t7NhS5DgbaBRzMUDgYXBBSMjAQEI6fZvMQGS0s0IIYCId/lAfYyEg5pI0Hy/9en8/n/fm+ 7/P53oeLeBVz+Nx0lY7SqkilEPVgdV0PCAwWKZflosk2H7zVegPB88+uIXiz3YjiFQ8eIfiq bRTBBxar2fjEtR4Y768vg3Fz8yCM9331J4wPri+geJnlDoTfu22C8YHJILx/YJiFj/XWoHjt N/c4uOVcAYx3zxsgvGu1FsFb7i+x8JuTO/DRtSH2m37E2Hg8YZqxoUSPyc4hRqfbWMSYTU+0 NxWiREdDLtE3kYcSJQWLTsGZGTaxdPU2SpReaYKIjpEcYrn9BaJ9fgFOeP5oRrSCIlMorYBS ydUp6ao0qTD+sOyALEIiEgeLI/HXhQIVmUlJhbEHE4Lj0pXOFQgFWaRS70wlkDQtDN0XrVXr dZRAoaZ1UiGlSVFqwjUhNJlJ61VpISpKFyUWiV6LcAqTMxTjJTfYGjv/ZEuVEc2DJrYVQe5c gIWDJ9UWtAjy4HphfRCoumXlMIEDAp93PoaYYBkCjqs2p4y70WJ4fJTJ90Lg/ugCzAQFMJg0 9rNcIhQLAiPrXJeFDzYFg56R3S4Ngt2FQW1+A+QqeGORoPnvetjFLMwf9DdWbDDPmTcVNcGM WSgwzni60B2LAiunsxmFJxiumme5GMFeBAWd1YjreYDVuoO2wZ/YzGixYOjnaZhhb/DH0BUO w3zwu/GzTT4BzOcbUab5NARMv5ggphADzliNiMsYwQJAa28ok94Fyq0tMGO8FZSszm++zwPd Xz7l3eBSax3K8HZwZ8WwyQQo6ru2ud0SCDy6tMY5CwlMzwxkemYg0//WdRDSBG2nNHRmGkVH aMJU1In/PlmuzmyHNq4iML4bmrv7IMQCwVzIAgEuIvThrfOW5V68FDL7FKVVy7R6JUVboAjn vr9A+NvkaudZqXQycXikKFwikYRHhknEQj/eK9JhuReWRuqoDIrSUNqnfTDXnZ8Hpx+0V1x8 aWdWGJZeeDFu55akmuP1cdZMmf/+55qzPQLDfistj63bK93zncIRQJIHluh3ZwtXg0QXkkun VorNlfq/TmpCtBHFx8w5USO5tpUfy/OPXPZ8q/KHqdQ3OqeaCxN7s/y02bNmu1l72SZJtu/x enW2TRFm/Ohw0ts1hzoKHVuOiV7u18wZKkN+zatO3fr1jkWFz6FEw9ZvHWz/96n39kcHLqU8 SVpUfSAtsE43PBywCt7pmjylTaXLbrr1DO1iycY7vRPKvh/9JNlt73E3dQd1ITEvptQXQc5X /+Mgu4LPwX2yjEbfjzUxHz7kH7m+b+JWTq7k0zr3OZ+xJF8Dv1PIohWkOBDR0uS/jRpmh54E AAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02RfUgTYRzHee5u52nMzrnq0ki5FGWWZUU8hllBxVlZKlTkH65hN62crZ1a voAzLdjCtFWWM8g3TCdoatnMrWy9aSaWU0HBl2L2gu8aWtk0Twn67/P7fj8/fg88BCp6gbkR Z+ITWFW8LI7GnbD6l7THpqLw6egtWpMAVr97g8LLuXYUVvbl4DBv4hcK59raUWgeKxDAnqYG BJqKdQisqHyNwMaiSQS+XhjFoc7SDeBQlx6B5l4/aDK3YND69B4O75cNOUDLzUwEGm0ZANbP 3Udh1fA4Bpt73WG7/a1gzxrG2nmI0Q+04UyDvs+Bae+vwRhrWyJTa9DgTF1pOtPYo8aZ7Myx ReHKgIAZf9aFM9cfGQBT15rKTNeuZ2pto0jYykinoNNs3JkkVrU5+JRTbGf2G4Gyz+1SVX4O rgY9q7SAIChyO5XxO1ILnAgRaQTUtQdmBy1wXMzXUmX2V+gyu1IV818dlqUMhBp8Ugf4ZZz0 o1oXCD4Xk18Qqq3/M8oPKDmGUIZPZQJ+25UMpCpnihGeMdKbMpXnLbFwMddrDcjyKzZTOQMu PDqSO6nZrGTeEC0azTr+FG+7UC35NoxnlPSgMh8XoLmA1P9X6f+rCgFiAGtZJaeIUXAByq3x 7EV/TqbgEuNj/KPPK2rB0i9LJEZgMkz4WwBCAAugCJQWCxeE09Ei4WlZcgqrOi9VJcaxnAW4 Exi9RvhB2yIVkTGyBPYcyypZ1b8WIRzd1Ijs5O2Pq49+po0+4RtnYiV329N9Ako1w3b8eX6T tG6+IztYU/3WnJhWYqi44XesTDpqw9+n3byArrATox3fMdnqh00hw9MbC3JLXMXNt7sTvELL C4/20l7yYGtEyHZxvVLjzGYPeW47kqDqSZotDvDMnPTacnhX9fcDpSmTTdVRmlD3gyJ5zU+f EdMMd2Jdl7foT2uYc2PEbuf+jsHhvJH8U2LfoLB+a7exT2DEOstf4ZYi+mqI/A4XBdJnzw7W /IjTNfxSGEc8drgPyqNsJRLFLvXAXnyuWBWqTl0okLfQ+30L9Q2Gs1Ld1NjxfVmpU0Ug8Fbw hpzJGjfuWzKNcbGyAAmq4mR/AS8xISlUAwAA X-CMS-MailID: 20230220105529epcas5p447cd449f13da54794e391b78bd1f5956 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230220105529epcas5p447cd449f13da54794e391b78bd1f5956 References: <20230220105336.3810-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Before enabling copy for dm target, check if underlying devices and dm target support copy. Avoid split happening inside dm target. Fail early if the request needs split, currently splitting copy request is not supported. Signed-off-by: Nitesh Shetty --- drivers/md/dm-table.c | 42 +++++++++++++++++++++++++++++++++++ drivers/md/dm.c | 7 ++++++ include/linux/device-mapper.h | 5 +++++ 3 files changed, 54 insertions(+) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 8541d5688f3a..4a1bbbb2493b 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1875,6 +1875,39 @@ static bool dm_table_supports_nowait(struct dm_table *t) return true; } +static int device_not_copy_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + struct request_queue *q = bdev_get_queue(dev->bdev); + + return !blk_queue_copy(q); +} + +static bool dm_table_supports_copy(struct dm_table *t) +{ + struct dm_target *ti; + unsigned int i; + + for (i = 0; i < t->num_targets; i++) { + ti = dm_table_get_target(t, i); + + if (!ti->copy_offload_supported) + return false; + + /* + * target provides copy support (as implied by setting + * 'copy_offload_supported') + * and it relies on _all_ data devices having copy support. + */ + if (!ti->type->iterate_devices || + ti->type->iterate_devices(ti, + device_not_copy_capable, NULL)) + return false; + } + + return true; +} + static int device_not_discard_capable(struct dm_target *ti, struct dm_dev *dev, sector_t start, sector_t len, void *data) { @@ -1957,6 +1990,15 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, q->limits.discard_misaligned = 0; } + if (!dm_table_supports_copy(t)) { + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + /* Must also clear copy limits... */ + q->limits.max_copy_sectors = 0; + q->limits.max_copy_sectors_hw = 0; + } else { + blk_queue_flag_set(QUEUE_FLAG_COPY, q); + } + if (!dm_table_supports_secure_erase(t)) q->limits.max_secure_erase_sectors = 0; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index b424a6ee27ba..0b04093fbeb2 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1690,6 +1690,13 @@ static blk_status_t __split_and_process_bio(struct clone_info *ci) if (unlikely(ci->is_abnormal_io)) return __process_abnormal_io(ci, ti); + if ((unlikely(op_is_copy(ci->bio->bi_opf)) && + max_io_len(ti, ci->sector) < ci->sector_count)) { + DMERR("Error, IO size(%u) > max target size(%llu)\n", + ci->sector_count, max_io_len(ti, ci->sector)); + return BLK_STS_IOERR; + } + /* * Only support bio polling for normal IO, and the target io is * exactly inside the dm_io instance (verified in dm_poll_dm_io) diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index 04c6acf7faaa..da4e77e81011 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -379,6 +379,11 @@ struct dm_target { * bio_set_dev(). NOTE: ideally a target should _not_ need this. */ bool needs_bio_set_dev:1; + + /* + * copy offload is supported + */ + bool copy_offload_supported:1; }; void *dm_per_bio_data(struct bio *bio, size_t data_size); From patchwork Mon Feb 20 10:53:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13146401 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD341C64ED9 for ; Mon, 20 Feb 2023 12:50:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232170AbjBTMuA (ORCPT ); Mon, 20 Feb 2023 07:50:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232186AbjBTMte (ORCPT ); Mon, 20 Feb 2023 07:49:34 -0500 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1F4A1D926 for ; Mon, 20 Feb 2023 04:48:57 -0800 (PST) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230220124856epoutp03773f388002b43173f10038a6406cbbec~FiT4yLXrE2510525105epoutp03n for ; Mon, 20 Feb 2023 12:48:56 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230220124856epoutp03773f388002b43173f10038a6406cbbec~FiT4yLXrE2510525105epoutp03n DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1676897336; bh=5O5uDT9HWnn4Ccadtd7fTCupkIwM/OnWfdBNErBUieg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JNHLaOh3lTOgYIzmSQl5ZbwHB/rdw00sfDhX2UqH1bs8saMaLXcYlSB9vyxKdNgYX hK2JKmqZ1mbqm1PFrJuGOGJ4DiS15lYjbXESLf/x+iUV9ceiDTWhCs4I3P4s7tokuv OQ1T1Wnaqh/NkBGT7oYD89PMyZFsH6DVfaMqraAw= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230220124855epcas5p11f6e306337a2a81f22a3588e359c4fb6~FiT4DRVI62749127491epcas5p1C; Mon, 20 Feb 2023 12:48:55 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.178]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4PL2Kf0Mrnz4x9Px; Mon, 20 Feb 2023 12:48:54 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id AB.26.10528.53C63F36; Mon, 20 Feb 2023 21:48:53 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230220105537epcas5p386dd5c28197f055a8c548cca109d913c~Fgw8_Lpoe2700227002epcas5p3E; Mon, 20 Feb 2023 10:55:37 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230220105537epsmtrp1985e2196bc7179a54716b408366f4246~Fgw88qAW12386223862epsmtrp1M; Mon, 20 Feb 2023 10:55:37 +0000 (GMT) X-AuditID: b6c32a49-e75fa70000012920-b6-63f36c35ca5a Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id F2.9C.05839.9A153F36; Mon, 20 Feb 2023 19:55:37 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230220105534epsmtip21c8292d59ac36a9accd9a5a1b5b20633~Fgw58o2620727007270epsmtip20; Mon, 20 Feb 2023 10:55:34 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, damien.lemoal@opensource.wdc.com, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v7 8/8] dm: Enable copy offload for dm-linear target Date: Mon, 20 Feb 2023 16:23:31 +0530 Message-Id: <20230220105336.3810-9-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230220105336.3810-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Te1BUZRTvu3df4GxeF4IPnGxbkuSxsGvs+q0jZkF6jYZhZKYBpYFtuQGx r9ldAjSV5SFi8dJkYIHwgQZYIIjEwy1nGSRgGTBUUEOowUZRHkYDEtG2y4Xyv9855/f7zjm/ bw4H533J9uQkqQ2UTi1XCljOjJZOny1CiXJOIRqdlaGG3hs4yixaxtGl0UIWKpldxNFS/wCO zNPlTHT3ehuGrp07iaHaS10Y6jj7DENdtikWOmm5A9DD2yYMme/5oWvmHgYaaq9goaqLD9nI cioLQ60TRoBalqpwVP9khoF+urcRDSx3M3e5k0O3wkjTWD+LbDONssmBB40Mcqg/hWyqy2OR V6qPkh13M1hkfta0nZAzxiRnfrjNIgua6wB5pe8QOde0iWyamMIi1u9P3pFIyeMpHZ9SKzTx SeqEYEFYZGxIrEQqEgvFMrRNwFfLVVSwIPSDCOHuJKXdAgH/M7kyxZ6KkOv1gsCdO3SaFAPF T9ToDcECShuv1AZpA/RylT5FnRCgpgzbxSLRVomdGJecaCy9AbRGZtr31kpmBihlnABOHEgE QVtzM/MEcObwiA4Arb89wejgDwCbTRcAHcwDeDm7krUm+evyjwy6YAZwOO80TgdZGDxeMmXX czgswg/22TgOgSvxCwbb+rwcHJwYx2BVZjVwFFyIUPjzqRbMgRnEZmj8xriCuYQMZjwqWHkH EoGwcGyDAzoR2+FCdjrN2AB7yiZWVsCJ12DW1fKVESBx3gkutpSx6UFD4dh8G5PGLnCyu3k1 7wkfFx5bxamw9qsaFi3OBtA0bAJ04W2Y01uIOxrjhA9saA+k06/C0731GN34ZZi/NIHReS5s /XoNe8FvG86smuUB7ywYVzEJq5+2smiv8gHMnq5jFwG+6YWFTC8sZPq/9RmA1wEPSqtXJVB6 iVasplL/+2WFRtUEVs7Cd28rGB2fDbAAjAMsAHJwgSvXxp1T8Ljx8vSDlE4Tq0tRUnoLkNj9 LsY9X1Fo7HelNsSKg2SiIKlUGiR7SyoWuHPfDO5R8IgEuYFKpigtpVvTYRwnzwys3O33alFy PePg53M9f+4Z+YQXLLE9nQ/8h9FOGEMKRsaP3Bfe4ruQUFkZHu5bse1Q9J7a6nWTX8x7uj36 O+m5ZNNWs/dCacXHNetyFh5EWT8UnGXGXVca3reWukVfzG4cfK9IHcjJS4/pn/Z9HDFcax7c 7y2s2v3RvsyZzJhBcUdxRV07OOzvfKDvJf/UxrK0q87+pG5frp/l+bm01+M+jdlydH4yzb3+ u3iPqKiR+wMH0r3FLr/uZat5775zZLj42YWh8+sLpOxOxs3NzeqQkJHU7jB2ZFincHHjzvBd A/3RuXxr082uZZXwmKvVxwuNRIZ01xw/LKueLC2xUbo3llStuQKGPlEu9sV1evm//v2hr58E AAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrEIsWRmVeSWpSXmKPExsWy7bCSvO7KwM/JBg2P1C3WnzrGbNE04S+z xeq7/WwW0z78ZLb4ffY8s8Xed7NZLW4e2MlksWfRJCaLlauPMlnsXviRyeLo/7dsFpMOXWO0 eHp1FpPF3lvaFnv2nmSxuLxrDpvF/GVP2S0OTW5mstjxpJHRYtvv+cwW616/Z7E4cUva4vzf 46wO4h6Xr3h7zLp/ls1j56y77B7n721k8bh8ttRj06pONo/NS+o9dt9sYPPobX4HVNB6n9Xj /b6rbB59W1Yxemw+Xe3xeZOcx6Ynb5kC+KO4bFJSczLLUov07RK4MhpnHGMsaGSt2H5mLmsD 4wyWLkZODgkBE4lfG/YD2VwcQgK7GSUeP33DDpGQlFj29wgzhC0ssfLfc3aIokYmiR93PrB1 MXJwsAloS5z+zwESFxF4xiRx9t4jZhCHWeAdk8Sqh8tYQbqFBVwkLk3exgRiswioSjQubwSz eQUsJRpe9DGBDJIQ0Jfovy8IYnIKWEl8b6kEqRACqjgxaTMjRLWgxMmZT8COZhaQl2jeOpt5 AqPALCSpWUhSCxiZVjFKphYU56bnFhsWGOallusVJ+YWl+al6yXn525iBEezluYOxu2rPugd YmTiYDzEKMHBrCTC+5/3c7IQb0piZVVqUX58UWlOavEhRmkOFiVx3gtdJ+OFBNITS1KzU1ML UotgskwcnFINTJ6BjLwR/TdntN5RW8RTruO+lTHrdaVca+yt5xvDrVZYHdtxgd/fg7OH6dRa l4vF5sJHt6roLYhVOPhTyjHljffq63LrE0WbhVZeefDDIaQs8WZ6zYY412teF1n5PyVbXP92 fqmHssHada/Zkvb08hw88IfjVohM5iG2jX4rz/w6PaOqjL0wIkOzYrHnM3XH2GtynKaSGWtE 70dFvF4tNNONLXfK/+MO/amxEZzhejbNn6RzYndyHrzs8s7h+Vz3PwVa8vbHmr22Re1Uvh8s yXHnmfRb2WaJugNPa3VSlKefNJfqS07PXJW9Zqv5gXdBEtrfJgos9DnHvPEHY+jUy+df2wTu fLAkR2avmpaQEktxRqKhFnNRcSIADQTqTVUDAAA= X-CMS-MailID: 20230220105537epcas5p386dd5c28197f055a8c548cca109d913c X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230220105537epcas5p386dd5c28197f055a8c548cca109d913c References: <20230220105336.3810-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Setting copy_offload_supported flag to enable offload. Signed-off-by: Nitesh Shetty --- drivers/md/dm-linear.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c index 3212ef6aa81b..b4b57bead495 100644 --- a/drivers/md/dm-linear.c +++ b/drivers/md/dm-linear.c @@ -61,6 +61,7 @@ static int linear_ctr(struct dm_target *ti, unsigned int argc, char **argv) ti->num_discard_bios = 1; ti->num_secure_erase_bios = 1; ti->num_write_zeroes_bios = 1; + ti->copy_offload_supported = 1; ti->private = lc; return 0;