From patchwork Mon May 22 10:41:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13250162 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EF0BC77B73 for ; Mon, 22 May 2023 11:16:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233245AbjEVLQE (ORCPT ); Mon, 22 May 2023 07:16:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58164 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232918AbjEVLPi (ORCPT ); Mon, 22 May 2023 07:15:38 -0400 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3B0C210D for ; Mon, 22 May 2023 04:10:31 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20230522111030epoutp04c906a9b30bf61d420064912d8ca528e0~hcq6smIVz1580715807epoutp049 for ; Mon, 22 May 2023 11:10:30 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20230522111030epoutp04c906a9b30bf61d420064912d8ca528e0~hcq6smIVz1580715807epoutp049 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1684753830; bh=YMZdsOD0UUYdd7xweh0AWYwQGUaFZAK6DIfIElX1bCk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Fabtxh5pf53TAZOH/KBAQ07mk6suHjZQg2cteLW4ZQMsSVDuDNal6Fa7pR/OR071l PBEAaFyxjfhBHbNQUJiWRdCs2IL1fy2fDnDjytAIFKMaU2bNl2Ml5EldzK7r8wCBsa 0YwnUv84E4Eu274DPq3arPGdP3AqbtnfPoFFuYdU= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230522111029epcas5p33d877412226c985b251270a67d74dbf4~hcq5-Q3D81390713907epcas5p3R; Mon, 22 May 2023 11:10:29 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.174]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4QPvr33SXNz4x9Px; Mon, 22 May 2023 11:10:27 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 9C.8E.44881.3AD4B646; Mon, 22 May 2023 20:10:27 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230522104526epcas5p30d6cb07abadb068a95ab1f90dea42d4e~hcVChV0gH0993609936epcas5p3d; Mon, 22 May 2023 10:45:26 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230522104526epsmtrp128279e1827de9e8fccaab5742f360c3e~hcVCgIBF91351813518epsmtrp1Z; Mon, 22 May 2023 10:45:26 +0000 (GMT) X-AuditID: b6c32a4a-ea9fa7000001af51-7c-646b4da372f4 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 08.13.27706.6C74B646; Mon, 22 May 2023 19:45:26 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230522104522epsmtip23379b85128ad1a00eb9a2dd413e225e6~hcU_tZeg_1590015900epsmtip2l; Mon, 22 May 2023 10:45:22 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, James.Bottomley@HansenPartnership.com, bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v11 1/9] block: Introduce queue limits for copy-offload support Date: Mon, 22 May 2023 16:11:32 +0530 Message-Id: <20230522104146.2856-2-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230522104146.2856-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TaVBTVxid+17y8sDGeSyOF0RLQzdgIIStF0bEEXXeVFoRf3Rqp8YMvAIF kkwSyvIHFGgBgdDQMhB2Qco2UBAxAlEnFFkqwwCySAWsLKVsIp0RBYESHrb+O+fcbznfd+cj cfMGnjUZJlUxCqkkQkCYcprb7R2cyj4LD3YZWLJE9T33cVRQX0ugK1mbOKoZVxNooX0VoJyV Vzh6ctcX6ZfzuejRvdsYarumwVBVTQeGGtQkai19jqGO7SUCaQzDAM0MaTGkH3NEbfpuDhps KSDQRO02FxVXzPDQ1REdgX7p3MKQITsRQ7rpywA1bxTjqG7hGQd1jR1CfZudXLTxsoA4foQe fHiG1k72EvRt7TiP7pto4NA5mh6CvlHpQA/2RtGN1akE3biq4dFduRsc+kZ5PN36KIGgMxKX Cfr5zBiHfnZniKAzm6pBgMWF8KOhjCSYUdgy0iBZcJg0xEdw5rzYT+zh6SJyEnmhTwS2Ukkk 4yM46R/gdDosYmdTAtvvJBFRO1KARKkUCI8dVciiVIxtqEyp8hEw8uAIubvcWSmJVEZJQ5yl jMpb5OLi6rETeCk8dOHeCE++dTbm4WOHBFDhlwZIElLuMCf32zRgSppTrQA2pXRjLFkFcGp0 lMuSFwCq9amcNGCym3ErvW4vSg9gSn86hyXJGNSrX/GMdQnKEf6+TRp1S2oah0sti7iR4FQn DrMqunFjKQvqHNTrruwmcKgPYP5UjFHmU16w9Nc7BOtPCNWTZkbZhPKGc/1anA0xg91507uG cOpdmHgzH2fNDZvAxEQ7Fp+Eg5m9GIst4HxnE4/F1vBv9fd7OBpW/VRJGK1BKglA7YgWsA++ MLlHjRs94JQ9rG8RsvJh+HNPHcb23Q8zNqb36vOhrugNtoO19SUEi63g8NrlPUzDobwJgt1V BoDJunJuFrDVvjWP9q15tP+3LgF4NbBi5MrIEEbpIXeVMtH//XGQLLIR7N6Ow6c68OeTFWcD wEhgAJDEBZb8c5lBweb8YElsHKOQiRVREYzSADx21v0jbn0gSLZzfFKVWOTu5eLu6enp7uXm KRIc5H/k0x1kToVIVEw4w8gZxZs8jDSxTsDSBVMugb5Lx4WFk1fDZVHN45kG2cCRVPIpeX+9 zvoW1tL6ZeG+pDhLi8NJbZPSs8d0jvwtsd1i69V/tDZr/g1mc6X9GnvPKkXXVIdrVVZRC3h8 YBSA6z4zbvtWPrb58KLYT1TkRMabvk/kp3fFC2p+qO0pnVt7HXDti+xZN/07UO10sVqEZ6wM dGoefFNn1S/UnM8pC8xc9uPb/YG993neb4F29tjYflVFvZvTS1fHv8LKqr/O7fOa92+fVXLd bQ8trT/drJ79KiU2Wrm2fuFF2okESauV76WD4d4Gmxif1yY3TxUW5y+WC+NduYPZVEklefrE A8389QGNR2yuMOtunICjDJWIHHCFUvIvR3k1zMQEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02Sa1BMYRiAfeecPXtamjm7uXzFaKxbNm3C5NO4zbTDMQwRMyyGnTpaujpb CFEqUWxNzaS2m9us2pA2rZKSZauVe5e1UaHCWFTMuEwqVsz4987zPO/766Vw0RvChdoVFsly YYoQMSkgDHfErh51K4ID5xQnilHJvToc5ZZcItHRtEEcFbenksh25zNAmX0/cPTy1lJU/SmH h6y1lRi6eS4dQ0XFJgyVplKo6mw/hkzDH0mUbmwFqKdFg6HqNnd0s9pMoKYbuSTquDTMQwXa Hj5KsVSQ6GL9EIaMGfEYquiOA8gwUICjK7ZeAjW0TUSPBut5aOB7LrlsMtPUvIrRdD4gmUpN O5951FFKMJnp90imrFDCND2IYvS6EySj/5zOZxqyBgim7MIRpsoaSzKn4j+RTH9PG8H01rSQ jPqaDvg5yQWLAtmQXXtZznPJDoHSVmvhRwyt3d/8QhILtL7JwIGC9Hx4/eQVLBkIKBFdBWBJ tw4fEc5QO3j37+wEi4be8keieAw+/WYlkgFFkbQ7bBym7Hws3YfDkmvngH0Bp5/j0GDZZJ+d 6LXQUGDB7D1BT4c5Xfvt2JFeCM9erSHtGNKeMLVTaMcOtA9890SD27Hod5KmnTtSC6E5u5sY Oe4K48tz8DRAa/5Tmv/UGYDpgDMboQoNClV5RXiFsfukKkWoKiosSBoQHqoHf15BMqsCXNf1 SY0Ao4ARQAoXj3Vcpw4IFDkGKqIPsFz4di4qhFUZwUSKEE9wfJxs3i6igxSRbDDLRrDcP4tR Di6xWJTlw2pMXqpfv2CCm9VzgeRx7YnE6OU53ovceFlTE7/pFcEx78vjuLjChPHTH371y5Pd NnFuGTPEo4MOudm672u+F3Ycyl6tyux9siH/4eFns1LUZt+thP99njcqN71NqhQo766xULIN wstXuZ/Kk0+9XGX7ZG2eVkN2zEoPtbyyKWDSce+Y/GNTWGvN7Iv4Gb8dSxNkcb7Kca2j/I0H X45u9mmZNjfyfOvRPUJpO93QWAp2m8efnrlNKZWrZ7xeF17hvPmFv6lx6yt5f0pC2Rb3kFsu hlM/si580H45nVTk8Rzs/DlmY9eQbHGXQZjgX5frXb452ufGvPdddbb6PJekTjGhUiq8JDin UvwCzGQc1nkDAAA= X-CMS-MailID: 20230522104526epcas5p30d6cb07abadb068a95ab1f90dea42d4e X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230522104526epcas5p30d6cb07abadb068a95ab1f90dea42d4e References: <20230522104146.2856-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Add device limits as sysfs entries, - copy_offload (RW) - copy_max_bytes (RW) - copy_max_bytes_hw (RO) Above limits help to split the copy payload in block layer. copy_offload: used for setting copy offload(1) or emulation(0). copy_max_bytes: maximum total length of copy in single payload. copy_max_bytes_hw: Reflects the device supported maximum limit. Reviewed-by: Hannes Reinecke Signed-off-by: Nitesh Shetty Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- Documentation/ABI/stable/sysfs-block | 33 ++++++++++++++ block/blk-settings.c | 24 +++++++++++ block/blk-sysfs.c | 64 ++++++++++++++++++++++++++++ include/linux/blkdev.h | 12 ++++++ include/uapi/linux/fs.h | 3 ++ 5 files changed, 136 insertions(+) diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block index c57e5b7cb532..e4d31132f77c 100644 --- a/Documentation/ABI/stable/sysfs-block +++ b/Documentation/ABI/stable/sysfs-block @@ -155,6 +155,39 @@ Description: last zone of the device which may be smaller. +What: /sys/block//queue/copy_offload +Date: April 2023 +Contact: linux-block@vger.kernel.org +Description: + [RW] When read, this file shows whether offloading copy to a + device is enabled (1) or disabled (0). Writing '0' to this + file will disable offloading copies for this device. + Writing any '1' value will enable this feature. If the device + does not support offloading, then writing 1, will result in + error. + + +What: /sys/block//queue/copy_max_bytes +Date: April 2023 +Contact: linux-block@vger.kernel.org +Description: + [RW] This is the maximum number of bytes, that the block layer + will allow for copy request. This will be smaller or equal to + the maximum size allowed by the hardware, indicated by + 'copy_max_bytes_hw'. Attempt to set value higher than + 'copy_max_bytes_hw' will truncate this to 'copy_max_bytes_hw'. + + +What: /sys/block//queue/copy_max_bytes_hw +Date: April 2023 +Contact: linux-block@vger.kernel.org +Description: + [RO] This is the maximum number of bytes, that the hardware + will allow in a single data copy request. + A value of 0 means that the device does not support + copy offload. + + What: /sys/block//queue/crypto/ Date: February 2022 Contact: linux-block@vger.kernel.org diff --git a/block/blk-settings.c b/block/blk-settings.c index 896b4654ab00..23aff2d4dcba 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -59,6 +59,8 @@ void blk_set_default_limits(struct queue_limits *lim) lim->zoned = BLK_ZONED_NONE; lim->zone_write_granularity = 0; lim->dma_alignment = 511; + lim->max_copy_sectors_hw = 0; + lim->max_copy_sectors = 0; } /** @@ -82,6 +84,8 @@ void blk_set_stacking_limits(struct queue_limits *lim) lim->max_dev_sectors = UINT_MAX; lim->max_write_zeroes_sectors = UINT_MAX; lim->max_zone_append_sectors = UINT_MAX; + lim->max_copy_sectors_hw = ULONG_MAX; + lim->max_copy_sectors = ULONG_MAX; } EXPORT_SYMBOL(blk_set_stacking_limits); @@ -183,6 +187,22 @@ void blk_queue_max_discard_sectors(struct request_queue *q, } EXPORT_SYMBOL(blk_queue_max_discard_sectors); +/** + * blk_queue_max_copy_sectors_hw - set max sectors for a single copy payload + * @q: the request queue for the device + * @max_copy_sectors: maximum number of sectors to copy + **/ +void blk_queue_max_copy_sectors_hw(struct request_queue *q, + unsigned int max_copy_sectors) +{ + if (max_copy_sectors > (COPY_MAX_BYTES >> SECTOR_SHIFT)) + max_copy_sectors = COPY_MAX_BYTES >> SECTOR_SHIFT; + + q->limits.max_copy_sectors_hw = max_copy_sectors; + q->limits.max_copy_sectors = max_copy_sectors; +} +EXPORT_SYMBOL_GPL(blk_queue_max_copy_sectors_hw); + /** * blk_queue_max_secure_erase_sectors - set max sectors for a secure erase * @q: the request queue for the device @@ -578,6 +598,10 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, t->max_segment_size = min_not_zero(t->max_segment_size, b->max_segment_size); + t->max_copy_sectors = min(t->max_copy_sectors, b->max_copy_sectors); + t->max_copy_sectors_hw = min(t->max_copy_sectors_hw, + b->max_copy_sectors_hw); + t->misaligned |= b->misaligned; alignment = queue_limit_alignment_offset(b, start); diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index a64208583853..826ab29beba3 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -212,6 +212,63 @@ static ssize_t queue_discard_zeroes_data_show(struct request_queue *q, char *pag return queue_var_show(0, page); } +static ssize_t queue_copy_offload_show(struct request_queue *q, char *page) +{ + return queue_var_show(blk_queue_copy(q), page); +} + +static ssize_t queue_copy_offload_store(struct request_queue *q, + const char *page, size_t count) +{ + s64 copy_offload; + ssize_t ret = queue_var_store64(©_offload, page); + + if (ret < 0) + return ret; + + if (copy_offload && !q->limits.max_copy_sectors_hw) + return -EINVAL; + + if (copy_offload) + blk_queue_flag_set(QUEUE_FLAG_COPY, q); + else + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + + return count; +} + +static ssize_t queue_copy_max_hw_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors_hw << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_store(struct request_queue *q, + const char *page, size_t count) +{ + s64 max_copy; + ssize_t ret = queue_var_store64(&max_copy, page); + + if (ret < 0) + return ret; + + if (max_copy & (queue_logical_block_size(q) - 1)) + return -EINVAL; + + max_copy >>= SECTOR_SHIFT; + if (max_copy > q->limits.max_copy_sectors_hw) + max_copy = q->limits.max_copy_sectors_hw; + + q->limits.max_copy_sectors = max_copy; + return count; +} + static ssize_t queue_write_same_max_show(struct request_queue *q, char *page) { return queue_var_show(0, page); @@ -590,6 +647,10 @@ QUEUE_RO_ENTRY(queue_nr_zones, "nr_zones"); QUEUE_RO_ENTRY(queue_max_open_zones, "max_open_zones"); QUEUE_RO_ENTRY(queue_max_active_zones, "max_active_zones"); +QUEUE_RW_ENTRY(queue_copy_offload, "copy_offload"); +QUEUE_RO_ENTRY(queue_copy_max_hw, "copy_max_bytes_hw"); +QUEUE_RW_ENTRY(queue_copy_max, "copy_max_bytes"); + QUEUE_RW_ENTRY(queue_nomerges, "nomerges"); QUEUE_RW_ENTRY(queue_rq_affinity, "rq_affinity"); QUEUE_RW_ENTRY(queue_poll, "io_poll"); @@ -637,6 +698,9 @@ static struct attribute *queue_attrs[] = { &queue_discard_max_entry.attr, &queue_discard_max_hw_entry.attr, &queue_discard_zeroes_data_entry.attr, + &queue_copy_offload_entry.attr, + &queue_copy_max_hw_entry.attr, + &queue_copy_max_entry.attr, &queue_write_same_max_entry.attr, &queue_write_zeroes_max_entry.attr, &queue_zone_append_max_entry.attr, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index b441e633f4dd..c9bf11adccb3 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -295,6 +295,9 @@ struct queue_limits { unsigned int discard_alignment; unsigned int zone_write_granularity; + unsigned long max_copy_sectors_hw; + unsigned long max_copy_sectors; + unsigned short max_segments; unsigned short max_integrity_segments; unsigned short max_discard_segments; @@ -561,6 +564,7 @@ struct request_queue { #define QUEUE_FLAG_NOWAIT 29 /* device supports NOWAIT */ #define QUEUE_FLAG_SQ_SCHED 30 /* single queue style io dispatch */ #define QUEUE_FLAG_SKIP_TAGSET_QUIESCE 31 /* quiesce_tagset skip the queue*/ +#define QUEUE_FLAG_COPY 32 /* supports copy offload */ #define QUEUE_FLAG_MQ_DEFAULT ((1UL << QUEUE_FLAG_IO_STAT) | \ (1UL << QUEUE_FLAG_SAME_COMP) | \ @@ -581,6 +585,7 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q); test_bit(QUEUE_FLAG_STABLE_WRITES, &(q)->queue_flags) #define blk_queue_io_stat(q) test_bit(QUEUE_FLAG_IO_STAT, &(q)->queue_flags) #define blk_queue_add_random(q) test_bit(QUEUE_FLAG_ADD_RANDOM, &(q)->queue_flags) +#define blk_queue_copy(q) test_bit(QUEUE_FLAG_COPY, &(q)->queue_flags) #define blk_queue_zone_resetall(q) \ test_bit(QUEUE_FLAG_ZONE_RESETALL, &(q)->queue_flags) #define blk_queue_dax(q) test_bit(QUEUE_FLAG_DAX, &(q)->queue_flags) @@ -899,6 +904,8 @@ extern void blk_queue_chunk_sectors(struct request_queue *, unsigned int); extern void blk_queue_max_segments(struct request_queue *, unsigned short); extern void blk_queue_max_discard_segments(struct request_queue *, unsigned short); +extern void blk_queue_max_copy_sectors_hw(struct request_queue *q, + unsigned int max_copy_sectors); void blk_queue_max_secure_erase_sectors(struct request_queue *q, unsigned int max_sectors); extern void blk_queue_max_segment_size(struct request_queue *, unsigned int); @@ -1218,6 +1225,11 @@ static inline unsigned int bdev_discard_granularity(struct block_device *bdev) return bdev_get_queue(bdev)->limits.discard_granularity; } +static inline unsigned int bdev_max_copy_sectors(struct block_device *bdev) +{ + return bdev_get_queue(bdev)->limits.max_copy_sectors; +} + static inline unsigned int bdev_max_secure_erase_sectors(struct block_device *bdev) { diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index b7b56871029c..8879567791fa 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -64,6 +64,9 @@ struct fstrim_range { __u64 minlen; }; +/* maximum total copy length */ +#define COPY_MAX_BYTES (1 << 27) + /* extent-same (dedupe) ioctls; these MUST match the btrfs ioctl definitions */ #define FILE_DEDUPE_RANGE_SAME 0 #define FILE_DEDUPE_RANGE_DIFFERS 1 From patchwork Mon May 22 10:41:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13250163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5554BC7EE2A for ; Mon, 22 May 2023 11:16:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233265AbjEVLQc (ORCPT ); Mon, 22 May 2023 07:16:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231883AbjEVLPk (ORCPT ); Mon, 22 May 2023 07:15:40 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 298F12112 for ; Mon, 22 May 2023 04:10:35 -0700 (PDT) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230522111033epoutp034b6404fcc5c699ac42d0566a3c423155~hcq97oEze1227912279epoutp03i for ; Mon, 22 May 2023 11:10:33 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230522111033epoutp034b6404fcc5c699ac42d0566a3c423155~hcq97oEze1227912279epoutp03i DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1684753833; bh=R+AbMKu00GBkKoqkZeg+RhLivR8lcxL0S6iien9dIBg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LEmKCvd1cgj5c75DiMYsFDzom5XvLhNda7eBLEuw0GBKNx+D2qYCE2XDo0qwzjMij 9/KGYfwXKCp9GcMbrK2nUI+qdCxV/Xu0k1IeWHZbiCSUsVJs/Q/v+4GS4SLzAu6CVp 7JThgSewAU0rPDL73Vj756l8vhgW0C9QxgDkr6BY= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230522111032epcas5p316964144d58f4375f55a2da2a8222391~hcq9VsDLX2983029830epcas5p3I; Mon, 22 May 2023 11:10:32 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.180]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4QPvr71LKzz4x9Pt; Mon, 22 May 2023 11:10:31 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 92.9E.44881.6AD4B646; Mon, 22 May 2023 20:10:31 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230522104536epcas5p23dd8108dd267ec588e5c36e8f9eb9fe8~hcVLq1bVn0618606186epcas5p2K; Mon, 22 May 2023 10:45:36 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230522104536epsmtrp191a88594e0902b23cc508ff2e23402c8~hcVLoljLz1351813518epsmtrp1h; Mon, 22 May 2023 10:45:36 +0000 (GMT) X-AuditID: b6c32a4a-ea9fa7000001af51-90-646b4da60e8d Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id AE.DF.28392.0D74B646; Mon, 22 May 2023 19:45:36 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230522104532epsmtip2839b424711300915eee77a57ffa0f158~hcVH0JBQe1645416454epsmtip22; Mon, 22 May 2023 10:45:32 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, James.Bottomley@HansenPartnership.com, bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v11 2/9] block: Add copy offload support infrastructure Date: Mon, 22 May 2023 16:11:33 +0530 Message-Id: <20230522104146.2856-3-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230522104146.2856-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TaUxUVxjlvje8GdCxD9B4WVTyxLAJzCBMLyJihZZX1BRD05D+oSM8GYRZ OosUEsOmtEW2ohAciCJt2URGAQlrWxbLVjJStjANS1NoZMeBWpFQyjDY+u98555zz/2+m4+D W9aybTjREiUjlwhjKcKcVd/h7OxWdiEmkjdehSFN7884KtJUESglZxNHD8azCTTfoQcof2Ud R1M/+qPWpUJTNPZTI4ZaSnIxVPHgKYYeZ3NQ8/0XGHq6tUig3PYRgGaG1Rhq1bmiltYeFhps KiLQRNWWKbpXOsNGN0cbCFTW9Q+G2m+lYqhhOhmg+o17OKqeX2ahbp0t0m52maKNV0XEmcP0 4NA5Wj3ZT9CN6nE2rZ14zKLzc3sJurbchR7sV9E1lV8TdI0+l013F2yw6NrvEunmsSSCzkxd IugXMzoWvfzDMEFn1VWCEKtPY06JGGEkI7dnJBHSyGhJlB91LjQ8INxbwOO78X3Qu5S9RChm /KjA8yFuH0THbk+Ksr8qjFVtUyFChYLyOH1KLlUpGXuRVKH0oxhZZKzMS+auEIoVKkmUu4RR nuTzeJ7e28LPYkRPFtowWW7YF5OVnawksB6UDsw4kPSCw1+WsdKBOceSbAZwQf/XbqEHUPMq jTAWqwAWaPrZ6YCzY7n9+qiRbwKwo3wWNxY3MFhYXs8yiAjSFfZtcQz8fnIah4tNCzsinOzC YU5pD24ItyKD4K/aR8CAWeQxuKYb3Engkj6woemYMcwDZk9aGBRm5En4fEC94+SSFrDnzjTL gHHyCEx9UrhzPSQnzODi/U3M2FsgbHy5xDZiKzjXVbeLbeBsdtoujoMVt8sJo/k6gOpRNTAe +MMbvdm44RE46Qw1TR5G+hDM663GjMH7YObG9G4WFzbcfYOPwipNMWHE1nDk7+RdTMO1tBa2 cViZAM7lz5jmAHv1Ww2p32pI/X90McArgTUjU4ijGIW3zFPCxP33yxFScQ3Y2R6X4Abw+9SK ezvAOKAdQA5O7edezIqItORGCuMTGLk0XK6KZRTtwHt73t/gNgcipNvrJ1GG8718eF4CgcDL 54SATx3kOvr1RFiSUUIlE8MwMkb+xodxzGySMKlrD+blOevY8nLkj7IPf9sz5Hk+rOBbdsys dC/zidjHMnqqd/N4vcrSNsW8Ku90zjMuUnVWUctrRz63ZXyvVCtT3z/4UNuX6HTNmr6Cd2sP e7+XoI+/k+wQsq/7KqcgMOjycMbYqtOmxbNUrEBgM7Ua2/lOtl3cgctYnR2ZuMfBNSGg4oxE bFLef7FPccFbP+HcVohNfl8PQgdQ29CjRbPy8Yx0342g4i3HoI98F+4WHTf5ZU0XcC3jrF+3 z+uvSp/H7OWm8LUm1y9t+U6ulxzh1eU5rPBDu0o+zim5dUIUHP/noTp/Ox09IhI6bS1fujka nkVFDwjmgh+GxdfwRS3zFEshEvJdcLlC+C8EdyfRxgQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrKIsWRmVeSWpSXmKPExsWy7bCSvO4F9+wUg0n3RS3WnzrGbDFn/Ro2 i6YJf5ktVt/tZ7N4ffgTo8W0Dz+ZLR7st7fY+242q8XNAzuZLPYsmsRksXL1USaLjf0cFrsX fmSyOPr/LZvFpEPXGC2eXp3FZLH3lrbFnr0nWSwu75rDZnFvzX9Wi/nLnrJbdF/fwWax/Pg/ JotDk5uZLHY8aWS02PZ7PrPFutfvWSxO3JK2OP/3OKvF7x9z2BzkPC5f8faYdf8sm8fOWXfZ Pc7f28jiMW3SKTaPzSu0PC6fLfXYtKqTzWPTp0nsHidm/Gbx2Lyk3mP3zQY2j97md2weH5/e YvF4v+8qm0ffllWMAcJRXDYpqTmZZalF+nYJXBlb3xxkKpgUUXF/1RGWBsaf7l2MHBwSAiYS U34pdzFycggJ7GCU+HFCHcSWEJCUWPb3CDOELSyx8t9z9i5GLqCaZiaJjZ+nsoD0sgloS5z+ zwESFxH4wCyxfssiRpAGZoHbzBLbrkeA2MIC7hKXzm8Ai7MIqEp8uXWZHaSXV8BSYscuVYgT 9CX67wuCVHAKWEm8uDiLGSQsBFQxYZkRSJhXQFDi5MwnLBDD5SWat85mnsAoMAtJahaS1AJG plWMkqkFxbnpucWGBUZ5qeV6xYm5xaV56XrJ+bmbGMEpQUtrB+OeVR/0DjEycTAeYpTgYFYS 4Q3sS04R4k1JrKxKLcqPLyrNSS0+xCjNwaIkznuh62S8kEB6YklqdmpqQWoRTJaJg1OqgUlL 1n/6yXROxbMh/j2P3k/ecs00P2tFpz/rhuaIGQ17r84pP79hn0x1RQfP4aolz/aYtlruWDXz eG1v8s7IM9sNYpLMnk7csljGKvm0+PfL4lqrfu9Yc3fdN6Gt0vLWf4PckwTTKwtdFsZsn1De Nm3rv5gmQznGCVfkjKJ2i6mHpOzsZriYpfs6m3eiZXYX74/Vf2vb0zOmeloelZ522V73xadg VVvn1KWRdy4lHc9ILPjk0XzLtaGuNsXQqORnoby4lt35j198L+byKMtzJ4c+5on5Y9fwcu7t rOVz1muZJaR/jHpxL/gAc0+QZaPkjK8pG7my/lef0jyfbBEnUDpX01+uQcBvjeOc65N0lFiK MxINtZiLihMB06tcTXgDAAA= X-CMS-MailID: 20230522104536epcas5p23dd8108dd267ec588e5c36e8f9eb9fe8 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230522104536epcas5p23dd8108dd267ec588e5c36e8f9eb9fe8 References: <20230522104146.2856-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Introduce blkdev_issue_copy which takes similar arguments as copy_file_range and performs copy offload between two bdevs. Introduce REQ_COPY copy offload operation flag. Create a read-write bio pair with a token as payload and submitted to the device in order. Read request populates token with source specific information which is then passed with write request. This design is courtesy Mikulas Patocka's token based copy Larger copy will be divided, based on max_copy_sectors limit. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- block/blk-lib.c | 235 ++++++++++++++++++++++++++++++++++++++ block/blk.h | 2 + include/linux/blk_types.h | 25 ++++ include/linux/blkdev.h | 3 + 4 files changed, 265 insertions(+) diff --git a/block/blk-lib.c b/block/blk-lib.c index e59c3069e835..ed089e703cb1 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -115,6 +115,241 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, } EXPORT_SYMBOL(blkdev_issue_discard); +/* + * For synchronous copy offload/emulation, wait and process all in-flight BIOs. + * This must only be called once all bios have been issued so that the refcount + * can only decrease. This just waits for all bios to make it through + * blkdev_copy_write_endio. + */ +static int blkdev_copy_wait_completion(struct cio *cio) +{ + int ret; + + if (cio->endio) + return 0; + + if (atomic_read(&cio->refcount)) { + __set_current_state(TASK_UNINTERRUPTIBLE); + blk_io_schedule(); + } + + ret = cio->comp_len; + kfree(cio); + + return ret; +} + +static void blkdev_copy_offload_write_endio(struct bio *bio) +{ + struct copy_ctx *ctx = bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (bio->bi_status) { + clen = (bio->bi_iter.bi_sector << SECTOR_SHIFT) - cio->pos_out; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + } + __free_page(bio->bi_io_vec[0].bv_page); + bio_put(bio); + + kfree(ctx); + if (!atomic_dec_and_test(&cio->refcount)) + return; + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); +} + +static void blkdev_copy_offload_read_endio(struct bio *read_bio) +{ + struct copy_ctx *ctx = read_bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (read_bio->bi_status) { + clen = (read_bio->bi_iter.bi_sector << SECTOR_SHIFT) + - cio->pos_in; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + __free_page(read_bio->bi_io_vec[0].bv_page); + bio_put(ctx->write_bio); + bio_put(read_bio); + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } + return; + } + + schedule_work(&ctx->dispatch_work); + bio_put(read_bio); +} + +static void blkdev_copy_dispatch_work(struct work_struct *work) +{ + struct copy_ctx *ctx = container_of(work, struct copy_ctx, + dispatch_work); + + submit_bio(ctx->write_bio); +} + +/* + * __blkdev_copy_offload - Use device's native copy offload feature. + * we perform copy operation by sending 2 bio. + * 1. First we send a read bio with REQ_COPY flag along with a token and source + * and length. Once read bio reaches driver layer, device driver adds all the + * source info to token and does a fake completion. + * 2. Once read operation completes, we issue write with REQ_COPY flag with same + * token. In driver layer, token info is used to form a copy offload command. + * + * Returns the length of bytes copied or error if encountered + */ +static int __blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct cio *cio; + struct copy_ctx *ctx; + struct bio *read_bio, *write_bio; + struct page *token; + sector_t copy_len; + sector_t rem, max_copy_len; + + cio = kzalloc(sizeof(struct cio), GFP_KERNEL); + if (!cio) + return -ENOMEM; + atomic_set(&cio->refcount, 0); + cio->waiter = current; + cio->endio = endio; + cio->private = private; + + max_copy_len = min(bdev_max_copy_sectors(bdev_in), + bdev_max_copy_sectors(bdev_out)) << SECTOR_SHIFT; + + cio->pos_in = pos_in; + cio->pos_out = pos_out; + /* If there is a error, comp_len will be set to least successfully + * completed copied length + */ + cio->comp_len = len; + for (rem = len; rem > 0; rem -= copy_len) { + copy_len = min(rem, max_copy_len); + + token = alloc_page(gfp_mask); + if (unlikely(!token)) + goto err_token; + + ctx = kzalloc(sizeof(struct copy_ctx), gfp_mask); + if (!ctx) + goto err_ctx; + read_bio = bio_alloc(bdev_in, 1, REQ_OP_READ | REQ_COPY + | REQ_SYNC | REQ_NOMERGE, gfp_mask); + if (!read_bio) + goto err_read_bio; + write_bio = bio_alloc(bdev_out, 1, REQ_OP_WRITE + | REQ_COPY | REQ_SYNC | REQ_NOMERGE, gfp_mask); + if (!write_bio) + goto err_write_bio; + + ctx->cio = cio; + ctx->write_bio = write_bio; + INIT_WORK(&ctx->dispatch_work, blkdev_copy_dispatch_work); + + __bio_add_page(read_bio, token, PAGE_SIZE, 0); + read_bio->bi_iter.bi_size = copy_len; + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_end_io = blkdev_copy_offload_read_endio; + read_bio->bi_private = ctx; + + __bio_add_page(write_bio, token, PAGE_SIZE, 0); + write_bio->bi_iter.bi_size = copy_len; + write_bio->bi_end_io = blkdev_copy_offload_write_endio; + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_private = ctx; + + atomic_inc(&cio->refcount); + submit_bio(read_bio); + pos_in += copy_len; + pos_out += copy_len; + } + + /* Wait for completion of all IO's*/ + return blkdev_copy_wait_completion(cio); + +err_write_bio: + bio_put(read_bio); +err_read_bio: + kfree(ctx); +err_ctx: + __free_page(token); +err_token: + cio->comp_len = min_t(sector_t, cio->comp_len, (len - rem)); + if (!atomic_read(&cio->refcount)) + return -ENOMEM; + /* Wait for submitted IOs to complete */ + return blkdev_copy_wait_completion(cio); +} + +static inline int blkdev_copy_sanity_check(struct block_device *bdev_in, + loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, + size_t len) +{ + unsigned int align = max(bdev_logical_block_size(bdev_out), + bdev_logical_block_size(bdev_in)) - 1; + + if (bdev_read_only(bdev_out)) + return -EPERM; + + if ((pos_in & align) || (pos_out & align) || (len & align) || !len || + len >= COPY_MAX_BYTES) + return -EINVAL; + + return 0; +} + +/* + * @bdev_in: source block device + * @pos_in: source offset + * @bdev_out: destination block device + * @pos_out: destination offset + * @len: length in bytes to be copied + * @endio: endio function to be called on completion of copy operation, + * for synchronous operation this should be NULL + * @private: endio function will be called with this private data, should be + * NULL, if operation is synchronous in nature + * @gfp_mask: memory allocation flags (for bio_alloc) + * + * Returns the length of bytes copied or error if encountered + * + * Description: + * Copy source offset from source block device to destination block + * device. Max total length of copy is limited to MAX_COPY_TOTAL_LENGTH + */ +int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct request_queue *q_in = bdev_get_queue(bdev_in); + struct request_queue *q_out = bdev_get_queue(bdev_out); + int ret; + + ret = blkdev_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len); + if (ret) + return ret; + + if (blk_queue_copy(q_in) && blk_queue_copy(q_out)) + ret = __blkdev_copy_offload(bdev_in, pos_in, bdev_out, pos_out, + len, endio, private, gfp_mask); + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_issue_copy); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/block/blk.h b/block/blk.h index 45547bcf1119..ec48a237fe12 100644 --- a/block/blk.h +++ b/block/blk.h @@ -303,6 +303,8 @@ static inline bool bio_may_exceed_limits(struct bio *bio, break; } + if (unlikely(op_is_copy(bio->bi_opf))) + return false; /* * All drivers must accept single-segments bios that are <= PAGE_SIZE. * This is a quick and dirty check that relies on the fact that diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 740afe80f297..0117e33087e1 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -420,6 +420,7 @@ enum req_flag_bits { */ /* for REQ_OP_WRITE_ZEROES: */ __REQ_NOUNMAP, /* do not free blocks when zeroing */ + __REQ_COPY, /* copy request */ __REQ_NR_BITS, /* stops here */ }; @@ -444,6 +445,7 @@ enum req_flag_bits { #define REQ_POLLED (__force blk_opf_t)(1ULL << __REQ_POLLED) #define REQ_ALLOC_CACHE (__force blk_opf_t)(1ULL << __REQ_ALLOC_CACHE) #define REQ_SWAP (__force blk_opf_t)(1ULL << __REQ_SWAP) +#define REQ_COPY ((__force blk_opf_t)(1ULL << __REQ_COPY)) #define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV) #define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE) @@ -474,6 +476,11 @@ static inline bool op_is_write(blk_opf_t op) return !!(op & (__force blk_opf_t)1); } +static inline bool op_is_copy(blk_opf_t op) +{ + return op & REQ_COPY; +} + /* * Check if the bio or request is one that needs special treatment in the * flush state machine. @@ -533,4 +540,22 @@ struct blk_rq_stat { u64 batch; }; +typedef void (cio_iodone_t)(void *private, int comp_len); + +struct cio { + struct task_struct *waiter; /* waiting task (NULL if none) */ + atomic_t refcount; + loff_t pos_in; + loff_t pos_out; + size_t comp_len; + cio_iodone_t *endio; /* applicable for async operation */ + void *private; /* applicable for async operation */ +}; + +struct copy_ctx { + struct cio *cio; + struct work_struct dispatch_work; + struct bio *write_bio; +}; + #endif /* __LINUX_BLK_TYPES_H */ diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index c9bf11adccb3..6f2814ab4741 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1051,6 +1051,9 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop); int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp); +int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Mon May 22 10:41:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13250164 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2184C7EE31 for ; Mon, 22 May 2023 11:16:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232930AbjEVLQe (ORCPT ); Mon, 22 May 2023 07:16:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233017AbjEVLPm (ORCPT ); Mon, 22 May 2023 07:15:42 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BFEB12118 for ; Mon, 22 May 2023 04:10:38 -0700 (PDT) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230522111037epoutp03f0450e87de1e356747705d73622214eb~hcrBUwkQl1243712437epoutp03Z for ; Mon, 22 May 2023 11:10:37 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230522111037epoutp03f0450e87de1e356747705d73622214eb~hcrBUwkQl1243712437epoutp03Z DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1684753837; bh=KwICTTElyy2GV/L0ha5pErdJKJEIbobgZ8uRqAtM6V0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DM9RHfDGjuVPRUYF2Ez5+EGwimEhezmeD7lELqmk9A3KwCqZSqvTJ/UtdP4lDvWIF h0xLYDmxN4nl6U3SCQp7m1uznvoXN//0Q3OBGfJIVqBt+2Rdmg0skWvDBjpHZx5BdZ wJTp2X64TcL/9ZJBBf+RWCgSGEV0Z4O2qfMha7tE= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230522111036epcas5p31976ea6f82d59aff10da72023e4f959e~hcrAu4Lxa1165311653epcas5p3u; Mon, 22 May 2023 11:10:36 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.175]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4QPvrB4ymmz4x9Pv; Mon, 22 May 2023 11:10:34 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id BE.29.16380.AAD4B646; Mon, 22 May 2023 20:10:34 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230522104607epcas5p11b718b8ed5006e92eca0e628b8196f08~hcVowuHWW1779017790epcas5p1I; Mon, 22 May 2023 10:46:07 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230522104607epsmtrp228a1c1b0e7d220836a3c8ca6c072bc51~hcVoqAutM0041200412epsmtrp2Y; Mon, 22 May 2023 10:46:07 +0000 (GMT) X-AuditID: b6c32a4b-56fff70000013ffc-df-646b4daa3a6c Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 95.EF.28392.FE74B646; Mon, 22 May 2023 19:46:07 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230522104603epsmtip2c628a826770c4b1d5efde2819688ccea~hcVkviBrY1645716457epsmtip2s; Mon, 22 May 2023 10:46:03 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, James.Bottomley@HansenPartnership.com, bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Vincent Fu , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v11 3/9] block: add emulation for copy Date: Mon, 22 May 2023 16:11:34 +0530 Message-Id: <20230522104146.2856-4-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230522104146.2856-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta0xbZRjed045LTPgAYZ8gIRaMWQQKHUFvi4gRgYexSnO+yB2tT0WArRN W7ZpdDIum9zGRQR3ioVtRC5jEC5DVkAmC+O+ZeE6DIOwkimVcpvZlAyktOj+Pc/zvu/zXr58 HNzZwPbgJCq0tFohSeYRe1ltN/b7BtQdTpIFbT32QY2DN3FU3lhPoPTCJzi6PFNAINONNYBK V/7G0Vx3BOoy6+zQ3evXMNR5sRhDtZd7MdRUwEEdF1Yx1Lu1RKDingmAFsYZDHVN+6POrgEW GjWUE+he/ZYdqvhpgY1yJ9sJVN23iaGe7zIw1G48DVDbRgWOGkzLLNQ/7Ynu534L0O0nfXZo 43E58ao3NToWQzGzIwR1jZlhU7fvNbGo0uJBgmqp8aNGR1Kp5rpsgmpeK2ZT/T9ssKiWqm+o jrtpBJWfYSao1YVpFrX8yzhBnWutA7EuR5PCEmiJjFZzaYVUKUtUyMN5Me+JI8XBIUGCAIEI hfK4CkkKHc479FZsQHRi8va5eNzjkuTUbSlWotHw+K+EqZWpWpqboNRow3m0SpasEqoCNZIU TapCHqigtQcFQUEvB28nHktKuKVLt1O1RZwsq3lApIEzwhxgz4GkED4aqGLlgL0cZ7IDQFNn i42sAZg28cBG1gHU/XwT2y1pma0mrAEDgFPNOsxKsjB4ZXRlm3A4BOkPh7Y4Fn0facThkuFP 3EJw0ozD9E0TsFi5kCGwKj93B7PIl2BZ7uwOdiBF8FIdg1uMIMmHBbNOFtmePAh/v2ORLSlO cOC8kWXBOOkNM67qdvwhOW8Pl/tr2dZRD0FTUbdtbBe42Ndq0z3gurmLsOITsLakhrAWZwLI TDLAGoiAWYMFO0Pg5H7YaOBbZS/4/WADZm3sCPM3jDZ/B9iu38UvwvrGSpu/O5x4dNqGKVja XoJbr5UP4NjgP3gh4DJPLcQ8tRDzf+tKgNcBd1qlSZHTmmDVAQV94r93lipTmsHOJ/KLaQfz cyuBPQDjgB4AOThvn8O756QyZweZ5IsvabVSrE5NpjU9IHj74EW4h6tUuf0LFVqxQCgKEoaE hAhFB0IEPDcH3/ABqTMpl2jpJJpW0erdOoxj75GG+YBxz89bGK83xvOy71QsLT4f85nYVObo OyJLkGb2Rcqj9ay6+DxXQ9aY65TKy/3qdGikvl4iZ0dxf5vLLHwGlzt+IFwret/rOdYLBa4f 5r99Bn1VyXx6YaQzXe+27pVXnOcpuBRwPj3Ge/VUVNPZzaHI1xrMYv7wvOiPqabjboxPYeeV OP/EYUNmvC78lrJsz0O+cdxvTz2KbzUWOWnctzy1gQkdWpHRg2dSZptzfnwY+vrUqTljoJIn PXpxoCEmak6ZFx0c13tf+fUxrd1M7skxZuidT6ozPp4sWecX+B6ZG/21IcsXnA07olx8Nvgj fa7e9XDHsFtOdbd88fpfb8bxWJoEicAPV2sk/wL30b77zQQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02SbUxTZxzFfe69vb0wOi+FxAd1I9QtJpW3GrI8bMtcosZnc0s0JsL4IHT0 phpKqb2AoglD0ImQUlYMSGFDsfJSHATQpkCBWV4EDCKiIGQFoxDMGC2QmLRj0FmaJX475/zO /3z6M6TYSe1kzqgzOa1arpLQgZSlTxIe5TqSpoidvx2BWkYGSVTdcpdG+aUbJGpy6Gm01LcG UPmKh0Qvew+gbmeVAE3/0UEgW62BQI1NAwRq1TOo69YqgQa8yzQy2CcBWnhuJFD3zD5k6x6m 0ERnNY1m73oFqKZuQYiKp6w0qn+4SSB7WQGBrPOXALKs15CoeclFoaGZXeh1cSFAYxsPBWjd XU1/HY4nnh3FxrlRGncYHUI8NttK4XLDCI3bG6R4YjQLt5mv0bhtzSDEQzfWKdxu+gl3TefR WFfgpPHqwgyFXT3PaVxyzwyOhSQFfqngVGeyOW3MVymBpx9X5Qs0lgPnKxoW6Tzwc1wRCGAg Gwfb5+rpIhDIiFkrgHV95YQfhMG6jX7Sr0Ng4+ai0F8qIODVzWfvDMPQ7D74yMv48lB2hYQt 92qBz5BsPgULLz+gfNch7GfQpCsGPk2xn8KK4rktLWLj4W2zkfQNQTYG6ueCfXEA+zl8M+6P xe8qpXX7/e1gOFw5v7VIsuGw4H4VWQpY43vI+B66CQgzCOM0fLoynZdp9qu5c9G8PJ3PUiuj UzPS28DWS0ilVmAzr0TbAcEAO4AMKQkVHS9JVYhFCnnOBU6bkazNUnG8HexiKMkO0ZOi4WQx q5Rncmkcp+G0/1OCCdiZR9T3VSm8H51dPPqJPGRH2cidKw4mvqf14rLXddC7959s4yNnTO21 fyUnXwW9yn3TpjWlgFG98xcLGRWU67qemIAqKt0541Nq/Jdue9jQYc/1yfLB7w5/HHnsbcpA eCLnjsjoL31sVeHsVOGPe5XVb52V7iPJexoPWdAyH6ytOS9N9ET92Tyc0Gz/bSoiyGYSXc5W fB+kTOLHJLrQtHHT0oUbUbIrOoP796beUyWNzTX9gy93f9Pe6qj0ZCmXar/9Iu3Dbdvm18dd sjtlJ88mqDYaOq++iLfIYp+O5PzaFXwp+cTuxfjOlExN7sWknLW/HcuTsXGF+g8if7AJSBas JnkkFH9aLpOSWl7+H6pWFG2BAwAA X-CMS-MailID: 20230522104607epcas5p11b718b8ed5006e92eca0e628b8196f08 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230522104607epcas5p11b718b8ed5006e92eca0e628b8196f08 References: <20230522104146.2856-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org For the devices which does not support copy, copy emulation is added. It is required for in-kernel users like fabrics, where file descriptor is not available and hence they can't use copy_file_range. Copy-emulation is implemented by reading from source into memory and writing to the corresponding destination asynchronously. Also emulation is used, if copy offload fails or partially completes. Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu Signed-off-by: Anuj Gupta --- block/blk-lib.c | 175 ++++++++++++++++++++++++++++++++++++++++- block/blk-map.c | 4 +- include/linux/blkdev.h | 3 + 3 files changed, 179 insertions(+), 3 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index ed089e703cb1..ba32545eb8d5 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -295,6 +295,172 @@ static int __blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, return blkdev_copy_wait_completion(cio); } +static void *blkdev_copy_alloc_buf(sector_t req_size, sector_t *alloc_size, + gfp_t gfp_mask) +{ + int min_size = PAGE_SIZE; + void *buf; + + while (req_size >= min_size) { + buf = kvmalloc(req_size, gfp_mask); + if (buf) { + *alloc_size = req_size; + return buf; + } + /* retry half the requested size */ + req_size >>= 1; + } + + return NULL; +} + +static void blkdev_copy_emulate_write_endio(struct bio *bio) +{ + struct copy_ctx *ctx = bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (bio->bi_status) { + clen = (bio->bi_iter.bi_sector << SECTOR_SHIFT) - cio->pos_out; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + } + kvfree(page_address(bio->bi_io_vec[0].bv_page)); + bio_map_kern_endio(bio); + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } +} + +static void blkdev_copy_emulate_read_endio(struct bio *read_bio) +{ + struct copy_ctx *ctx = read_bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (read_bio->bi_status) { + clen = (read_bio->bi_iter.bi_sector << SECTOR_SHIFT) - + cio->pos_in; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + __free_page(read_bio->bi_io_vec[0].bv_page); + bio_map_kern_endio(read_bio); + kfree(ctx); + + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } + } + schedule_work(&ctx->dispatch_work); + kfree(read_bio); +} + +/* + * If native copy offload feature is absent, this function tries to emulate, + * by copying data from source to a temporary buffer and from buffer to + * destination device. + * Returns the length of bytes copied or error if encountered + */ +static int __blkdev_copy_emulate(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct request_queue *in = bdev_get_queue(bdev_in); + struct request_queue *out = bdev_get_queue(bdev_out); + struct bio *read_bio, *write_bio; + void *buf = NULL; + struct copy_ctx *ctx; + struct cio *cio; + sector_t buf_len, req_len, rem = 0; + sector_t max_src_hw_len = min_t(unsigned int, + queue_max_hw_sectors(in), + queue_max_segments(in) << (PAGE_SHIFT - SECTOR_SHIFT)) + << SECTOR_SHIFT; + sector_t max_dst_hw_len = min_t(unsigned int, + queue_max_hw_sectors(out), + queue_max_segments(out) << (PAGE_SHIFT - SECTOR_SHIFT)) + << SECTOR_SHIFT; + sector_t max_hw_len = min_t(unsigned int, + max_src_hw_len, max_dst_hw_len); + + cio = kzalloc(sizeof(struct cio), GFP_KERNEL); + if (!cio) + return -ENOMEM; + atomic_set(&cio->refcount, 0); + cio->pos_in = pos_in; + cio->pos_out = pos_out; + cio->waiter = current; + cio->endio = endio; + cio->private = private; + + for (rem = len; rem > 0; rem -= buf_len) { + req_len = min_t(int, max_hw_len, rem); + + buf = blkdev_copy_alloc_buf(req_len, &buf_len, gfp_mask); + if (!buf) + goto err_alloc_buf; + + ctx = kzalloc(sizeof(struct copy_ctx), gfp_mask); + if (!ctx) + goto err_ctx; + + read_bio = bio_map_kern(in, buf, buf_len, gfp_mask); + if (IS_ERR(read_bio)) + goto err_read_bio; + + write_bio = bio_map_kern(out, buf, buf_len, gfp_mask); + if (IS_ERR(write_bio)) + goto err_write_bio; + + ctx->cio = cio; + ctx->write_bio = write_bio; + INIT_WORK(&ctx->dispatch_work, blkdev_copy_dispatch_work); + + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_iter.bi_size = buf_len; + read_bio->bi_opf = REQ_OP_READ | REQ_SYNC; + bio_set_dev(read_bio, bdev_in); + read_bio->bi_end_io = blkdev_copy_emulate_read_endio; + read_bio->bi_private = ctx; + + write_bio->bi_iter.bi_size = buf_len; + write_bio->bi_opf = REQ_OP_WRITE | REQ_SYNC; + bio_set_dev(write_bio, bdev_out); + write_bio->bi_end_io = blkdev_copy_emulate_write_endio; + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_private = ctx; + + atomic_inc(&cio->refcount); + submit_bio(read_bio); + + pos_in += buf_len; + pos_out += buf_len; + } + + /* Wait for completion of all IO's*/ + return blkdev_copy_wait_completion(cio); + +err_write_bio: + bio_put(read_bio); +err_read_bio: + kfree(ctx); +err_ctx: + kvfree(buf); +err_alloc_buf: + cio->comp_len -= min_t(sector_t, cio->comp_len, len - rem); + if (!atomic_read(&cio->refcount)) + return -ENOMEM; + /* Wait for submitted IOs to complete */ + return blkdev_copy_wait_completion(cio); +} + static inline int blkdev_copy_sanity_check(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len) @@ -342,9 +508,16 @@ int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, if (ret) return ret; - if (blk_queue_copy(q_in) && blk_queue_copy(q_out)) + if (blk_queue_copy(q_in) && blk_queue_copy(q_out)) { ret = __blkdev_copy_offload(bdev_in, pos_in, bdev_out, pos_out, len, endio, private, gfp_mask); + if (ret < 0) + ret = 0; + } + + if (ret != len) + ret = __blkdev_copy_emulate(bdev_in, pos_in + ret, bdev_out, + pos_out + ret, len - ret, endio, private, gfp_mask); return ret; } diff --git a/block/blk-map.c b/block/blk-map.c index 04c55f1c492e..e79eb4d2e545 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -363,7 +363,7 @@ static void bio_invalidate_vmalloc_pages(struct bio *bio) #endif } -static void bio_map_kern_endio(struct bio *bio) +void bio_map_kern_endio(struct bio *bio) { bio_invalidate_vmalloc_pages(bio); bio_uninit(bio); @@ -380,7 +380,7 @@ static void bio_map_kern_endio(struct bio *bio) * Map the kernel address into a bio suitable for io to a block * device. Returns an error pointer in case of error. */ -static struct bio *bio_map_kern(struct request_queue *q, void *data, +struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, gfp_t gfp_mask) { unsigned long kaddr = (unsigned long)data; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 6f2814ab4741..a95c26faa8b6 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1054,6 +1054,9 @@ int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); +struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, + gfp_t gfp_mask); +void bio_map_kern_endio(struct bio *bio); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Mon May 22 10:41:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13250165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BB40C77B73 for ; Mon, 22 May 2023 11:16:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233092AbjEVLQ4 (ORCPT ); Mon, 22 May 2023 07:16:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233111AbjEVLPn (ORCPT ); Mon, 22 May 2023 07:15:43 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0C78E211F for ; Mon, 22 May 2023 04:10:43 -0700 (PDT) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230522111041epoutp0322002560590b12ad4160fa2ac081e00d~hcrFK3pD71243312433epoutp03c for ; Mon, 22 May 2023 11:10:41 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230522111041epoutp0322002560590b12ad4160fa2ac081e00d~hcrFK3pD71243312433epoutp03c DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1684753841; bh=3AnFoe+19/5j5GvwnjpyvmPBq9a3SDPvQ6hFuEAN/NY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LtxlWUvQ9JArfMu+krSp/qiXigCH914uBLHZe1Ak/IiTSIdyjy6Hj07esZatJyPzL rnmfdNXae2zawxNN+DtUdvIpGpGFGexnLD11a5OwxqTUXh9JzGekNtqEthrq6iz8Qq BKXgYRu0ihPPsF6igEXyM7YLNsklFuYjh8psdw2k= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230522111040epcas5p398b8e98bf0206ff5cd5e529783610c34~hcrEbIqZc1165311653epcas5p35; Mon, 22 May 2023 11:10:40 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.183]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4QPvrG2BVwz4x9Px; Mon, 22 May 2023 11:10:38 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id E3.39.16380.EAD4B646; Mon, 22 May 2023 20:10:38 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230522104617epcas5p25ef0cfacfb4e2d4e8d7a0661f7181e7d~hcVyGEL0Q0438604386epcas5p2d; Mon, 22 May 2023 10:46:17 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230522104617epsmtrp28412315ea425dc05b2486615125c4a03~hcVyE4GGj3249732497epsmtrp2G; Mon, 22 May 2023 10:46:17 +0000 (GMT) X-AuditID: b6c32a4b-56fff70000013ffc-f1-646b4daeecd1 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id F9.EF.28392.9F74B646; Mon, 22 May 2023 19:46:17 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230522104613epsmtip2850c4d13d3f6f140a1f07f20cf27405b~hcVuSFeXT1863318633epsmtip2D; Mon, 22 May 2023 10:46:13 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, James.Bottomley@HansenPartnership.com, bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v11 4/9] fs, block: copy_file_range for def_blk_ops for direct block device Date: Mon, 22 May 2023 16:11:35 +0530 Message-Id: <20230522104146.2856-5-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230522104146.2856-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TazBcZxj2nbN7dtFNT5Dx0SjZjCbu1q2f1CaUdk5HTbWZTttMRLe7Jyh2 t7skojONS9WILLopE8tI0mTcVlyWqttiFnWpUJQ0Oi7NrE41LGEiySiKpc2/53ne933eyzcf G7doYNmyY8QJtEwsiOMSZozGLqdjbtVhsSJPVRsH1Qz8hKPimioCpeVt4Eg9lUugR10rABUs P8fRbMcppDUUMdGDzmYMtX2vxFCFugdDdbls1HrrMYZ6thYJpNRNADQ3rsKQdtIFtWn7GWis pZhA01VbTHSjdI6Fsu83EaisdxNDumvpGGrSpwLUuH4DR9WPlhiob/IVNLzRy0Trz4qJwFep sV9DKdXMPYJqVk2xqOHpOgZVoBwgqPpyZ2rsXiKlqcwiKM2KkkX1XV9nUPV3LlOtD1IISpFu IKjHc5MMaql9nKByGipBuOWZ2IBoWiCiZQ60WCgRxYij+NzQ05HBkb5+njw3nj96nesgFsTT fG7Iu+Fub8fEbV+K63BBEJe4LYUL5HKux8kAmSQxgXaIlsgT+FxaKoqT+kjd5YJ4eaI4yl1M J5zgeXp6+W4nfhobPZBazJCO+yZ1tv/BTAGFrleAKRuSPvC3glvYFWDGtiBbAczZULKMZAVA /WYZYSSrABanrDP2S9b+adsLtACouXp3j2RgcEnRuJ3FZhOkC/x5i72jW5F6HC62LOA7BCd7 cZhX2o/vWFmSEXB0MQPsYAbpCHsKZnd1DukPR7uXdo0g6QFzZw7uyKbkCfjXiGov5SDsL9Tv ToST9jD9h6Jdf0hOmsLV9hVgrA2BPfX2xqkt4d+9DSwjtoWrBi1hxBdhxXflhLH2awBV91XA GDgFMwZy8R0fnHSCNS0eRtkO5g9UY8a+B6BiXY8ZdQ5sKtnHR2FVzc09fxs48TR1D1OwTzPN MB5LAWDmjIKVBxxUL+yjemEf1f+tbwK8EtjQUnl8FC33lXqL6Yv/PbNQEq8Bu9/HObQJPJxd dtcBjA10ALJxrhXn/RyhyIIjElxKpmWSSFliHC3XAd/te3+L2x4SSrb/nzghkufj7+nj5+fn 4+/tx+Nac47x+4UWZJQggY6laSkt26/D2Ka2KRj/rWtpa3/KDGan1UFDJ5M2ztt6lj//+ID2 sy8GDWGtgxqt48JHfKVJ/ptrNszjrz1VWrdwYq1yz3RFdGd/MzIhPJxp/mH32epu8fKqfd0H 591DM1wM0meKhyUhwUcj3nC+/V6R6fGFvPzmzaagQr5W+KTWrjl0mDZYumbVmphYfJVcMmdt opkPTlo59wkl+GXmrvmR4frBw+pDPXfsGlwvq/svRKQVgMyRso4jT8TNl9S+AV63O+xTVySu Q0Ffcjp/f6eDFzhvEzrkbXK2OXmxvyT25XSPWn2gpc6cee76tNZJ2RT2+UulNd1TPwZ7F2Uj x9EsnrVSh+b9wr0q7LwFRJs5lyGPFvCccZlc8C8ddNtQxwQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02Se0xTZxjG951zOD00NjmWJv2QRJI6JVblUjv3bXGyJUv2kc0pJsuILtMG joBQLqeA1ynSZJcSC2sjWUsN0jhWi4EA2oCUyiqlhYZ0rAPXouAWGF5iCy4ZGARmYUv875ff 87zPXy9DimepTUxhSQXHl6iKZbSQctyVJe968VFRXvrjv+SofXiQRJb2GzSqqV8mUeuDOho9 vfscoIa5FyR6eCcT9UUa41Cov4dATquBQNdbPQTqqGNQb/M8gTyrz2hkcI8DNDNmJlBfeAdy 9g1RKHjbQqPJG6txqKllRoBq73XT6CfvCoHcRi2BuqcvAuRYaiJR29MohXzhJBRY9sahpUUL /f5mHPztY2yeGqFxj/mBAAcmOyjcYBimcZdNjoMjlbjT/h2NO58bBNj3wxKFu65dwL2hahpf 0kZoPD8TpnDUNUZj/U07OJhwWLg3jysurOL4tH3HhAXDFy1U2dhbp/pdf8RVA9NOHYhnIKuE /7x00jEWs90ADujIdZ8IW5YH/uMEeH1lVqADwlcdLQGtrUFKBxiGZndA/yoT8xJ2joTtN60g dkCyEyR03MuJcQJ7BPrDfwtiTLFboafh4dqoiH0H/joQXduBbBqsm9oY0/Hsu/DRqJmMafGr Sn2LYr29EQ6Zpqn19WSovdVI1gPW/Fpkfi26Cgg7SOTKNOp8tSajTFHCnUzVqNSaypL81NxS dSdY+wW5vBs47XOpbkAwwA0gQ8okomx9bp5YlKc6fYbjS4/ylcWcxg2SGEomFf2iGzoqZvNV FVwRx5Vx/P8pwcRvqiYO9Ewabf4N58vn+dqklS3SmjesjjOskl7I6mFMRc2PqJlGsVZ+bdW1 +6uXad+/Nz34pPCT1IqmrqWOzy7nBEb3R7K+JU9OLIo+lIx+uWhDy3p+5EQmilwgoumB8wJT RnRXq14ycdVn2/NnVQ7l+kD0xW1925vjxwz54UsCn10sDVUVKL2GbYla3jvuwSm9Fn63Ybz6 5wXhlrZtpb+n79++/f6ROhuQHDgVUO/NjViFYt/btSltidIhqbffmZVdvi+4efbxNyfOGkPN 5c8yj2fbHIOmK6WKz5PNY4WXPSn+Q17FcWWoyMhOuYyKuXOOGl/gx4KvN3y6oJT7t6bJKE2B KkNO8hrVv4+iC4d6AwAA X-CMS-MailID: 20230522104617epcas5p25ef0cfacfb4e2d4e8d7a0661f7181e7d X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230522104617epcas5p25ef0cfacfb4e2d4e8d7a0661f7181e7d References: <20230522104146.2856-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org For direct block device opened with O_DIRECT, use copy_file_range to issue device copy offload, and fallback to generic_copy_file_range incase device copy offload capability is absent. Modify checks to allow bdevs to use copy_file_range. Suggested-by: Ming Lei Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty --- block/blk-lib.c | 23 +++++++++++++++++++++++ block/fops.c | 20 ++++++++++++++++++++ fs/read_write.c | 11 +++++++++-- include/linux/blkdev.h | 3 +++ mm/filemap.c | 11 ++++++++--- 5 files changed, 63 insertions(+), 5 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index ba32545eb8d5..7d6ef85692a6 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -523,6 +523,29 @@ int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, } EXPORT_SYMBOL_GPL(blkdev_issue_copy); +/* Returns the length of bytes copied */ +int blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + gfp_t gfp_mask) +{ + struct request_queue *in_q = bdev_get_queue(bdev_in); + struct request_queue *out_q = bdev_get_queue(bdev_out); + int ret = 0; + + if (blkdev_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len)) + return 0; + + if (blk_queue_copy(in_q) && blk_queue_copy(out_q)) { + ret = __blkdev_copy_offload(bdev_in, pos_in, bdev_out, pos_out, + len, NULL, NULL, gfp_mask); + if (ret < 0) + return 0; + } + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_copy_offload); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/block/fops.c b/block/fops.c index ab750e8a040f..df8985675ed1 100644 --- a/block/fops.c +++ b/block/fops.c @@ -614,6 +614,25 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to) return ret; } +static ssize_t blkdev_copy_file_range(struct file *file_in, loff_t pos_in, + struct file *file_out, loff_t pos_out, + size_t len, unsigned int flags) +{ + struct block_device *in_bdev = I_BDEV(bdev_file_inode(file_in)); + struct block_device *out_bdev = I_BDEV(bdev_file_inode(file_out)); + int comp_len = 0; + + if ((file_in->f_iocb_flags & IOCB_DIRECT) && + (file_out->f_iocb_flags & IOCB_DIRECT)) + comp_len = blkdev_copy_offload(in_bdev, pos_in, out_bdev, + pos_out, len, GFP_KERNEL); + if (comp_len != len) + comp_len = generic_copy_file_range(file_in, pos_in + comp_len, + file_out, pos_out + comp_len, len - comp_len, flags); + + return comp_len; +} + #define BLKDEV_FALLOC_FL_SUPPORTED \ (FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE | \ FALLOC_FL_ZERO_RANGE | FALLOC_FL_NO_HIDE_STALE) @@ -697,6 +716,7 @@ const struct file_operations def_blk_fops = { .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, .fallocate = blkdev_fallocate, + .copy_file_range = blkdev_copy_file_range, }; static __init int blkdev_init(void) diff --git a/fs/read_write.c b/fs/read_write.c index a21ba3be7dbe..47e848fcfd42 100644 --- a/fs/read_write.c +++ b/fs/read_write.c @@ -20,6 +20,7 @@ #include #include #include +#include #include "internal.h" #include @@ -1447,7 +1448,11 @@ static int generic_copy_file_checks(struct file *file_in, loff_t pos_in, return -EOVERFLOW; /* Shorten the copy to EOF */ - size_in = i_size_read(inode_in); + if (S_ISBLK(inode_in->i_mode)) + size_in = bdev_nr_bytes(I_BDEV(file_in->f_mapping->host)); + else + size_in = i_size_read(inode_in); + if (pos_in >= size_in) count = 0; else @@ -1708,7 +1713,9 @@ int generic_file_rw_checks(struct file *file_in, struct file *file_out) /* Don't copy dirs, pipes, sockets... */ if (S_ISDIR(inode_in->i_mode) || S_ISDIR(inode_out->i_mode)) return -EISDIR; - if (!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) + + if ((!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) && + (!S_ISBLK(inode_in->i_mode) || !S_ISBLK(inode_out->i_mode))) return -EINVAL; if (!(file_in->f_mode & FMODE_READ) || diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index a95c26faa8b6..a9bb7e3a8c79 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1054,6 +1054,9 @@ int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); +int blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + gfp_t gfp_mask); struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, gfp_t gfp_mask); void bio_map_kern_endio(struct bio *bio); diff --git a/mm/filemap.c b/mm/filemap.c index 570bc8c3db87..289f0c8229ec 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -48,6 +48,7 @@ #include #include #include "internal.h" +#include #define CREATE_TRACE_POINTS #include @@ -2855,7 +2856,7 @@ ssize_t filemap_splice_read(struct file *in, loff_t *ppos, { struct folio_batch fbatch; struct kiocb iocb; - size_t total_spliced = 0, used, npages; + size_t total_spliced = 0, used, npages, size_in; loff_t isize, end_offset; bool writably_mapped; int i, error = 0; @@ -2863,6 +2864,10 @@ ssize_t filemap_splice_read(struct file *in, loff_t *ppos, init_sync_kiocb(&iocb, in); iocb.ki_pos = *ppos; + if (S_ISBLK(file_inode(in)->i_mode)) + size_in = bdev_nr_bytes(I_BDEV(in->f_mapping->host)); + else + size_in = i_size_read(file_inode(in)); /* Work out how much data we can actually add into the pipe */ used = pipe_occupancy(pipe->head, pipe->tail); npages = max_t(ssize_t, pipe->max_usage - used, 0); @@ -2873,7 +2878,7 @@ ssize_t filemap_splice_read(struct file *in, loff_t *ppos, do { cond_resched(); - if (*ppos >= i_size_read(file_inode(in))) + if (*ppos >= size_in) break; iocb.ki_pos = *ppos; @@ -2889,7 +2894,7 @@ ssize_t filemap_splice_read(struct file *in, loff_t *ppos, * part of the page is not copied back to userspace (unless * another truncate extends the file - this is desired though). */ - isize = i_size_read(file_inode(in)); + isize = size_in; if (unlikely(*ppos >= isize)) break; end_offset = min_t(loff_t, isize, *ppos + len); From patchwork Mon May 22 10:41:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13250166 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 688E9C7EE2A for ; Mon, 22 May 2023 11:17:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233296AbjEVLRA (ORCPT ); Mon, 22 May 2023 07:17:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233135AbjEVLPo (ORCPT ); Mon, 22 May 2023 07:15:44 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 696CA1B1 for ; Mon, 22 May 2023 04:10:45 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230522111043epoutp03fd6e60de23bd0187fed6670cc9281199~hcrHfDwzU1225012250epoutp03d for ; Mon, 22 May 2023 11:10:43 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230522111043epoutp03fd6e60de23bd0187fed6670cc9281199~hcrHfDwzU1225012250epoutp03d DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1684753843; bh=1WJtwnO5PvXiBgPu8IyPQ8G/PqQ97PZnyPYo9MnZUQA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vdX184PH29jsk/QU5gEcZK9qXnVnTNVNtw5k97kemp3ylnZdC5VQBH4aHNX6Dd071 P/lsge+6rzb+8S1WwXJKhmL+/MH/nahPfHIPdmKfv+w+jfVa6Dy9S7TD+javEo6uUV 06UaMvJ6Z07kHFl6shcOLTsWqi9uhOA5CtN9koiw= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230522111043epcas5p4a30dfec100c198ec352c5e3869dd178d~hcrGv25-s2858928589epcas5p4r; Mon, 22 May 2023 11:10:43 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.176]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4QPvrK5q8Kz4x9Pr; Mon, 22 May 2023 11:10:41 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 58.39.16380.1BD4B646; Mon, 22 May 2023 20:10:41 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20230522104628epcas5p4f5b3f3d7b080950955a127733d554753~hcV76Yjto0301703017epcas5p4p; Mon, 22 May 2023 10:46:28 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230522104628epsmtrp1c83ba865935ea43a74a0b42ad6ab0156~hcV73yLeM1451014510epsmtrp15; Mon, 22 May 2023 10:46:28 +0000 (GMT) X-AuditID: b6c32a4b-7dffd70000013ffc-00-646b4db1d7a6 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 19.23.27706.4084B646; Mon, 22 May 2023 19:46:28 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230522104624epsmtip29890dc3f754a2a79816cc006038bc28e~hcV38iRUA1590015900epsmtip24; Mon, 22 May 2023 10:46:23 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, James.Bottomley@HansenPartnership.com, bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , =?utf-8?q?Javier_Gonz=C3=A1lez?= , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v11 5/9] nvme: add copy offload support Date: Mon, 22 May 2023 16:11:36 +0530 Message-Id: <20230522104146.2856-6-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230522104146.2856-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te1BUVRzHO/de7i5Mi1fAOouRdNUcQB6LPM6agIyOcwMNgoZpnBzYYW9A wLLtspLlFAsoBQGCCbkwPIohXkI8WnktEIi8YigRSIxHAqEhz0pTAmK5UP73+X1/33N+jzOH j5vU8cz5YbJoViGTRNCkEaFttzpgW3UqXOqQqbZClT03cZRTWU6iuEurOCobTSPRbPsyQJmL T3A00eKBdPPZBuhOaz2Gmr7KwFBJWQeGqtL4qLFgCUOTv/zBQx3rcyTKaBsCaHpQgyHdiA1q 0nUTaKAhh0Rj5esGKK9omoeSh+tI9E3nGobaLsdjqG5KDZB2JQ9HFbMLBOoa2Y36VzsN0Mrf OeTRPczAbW9GM95HMvWaUR7TP1ZFMJkZPSRTU2zNDPSpmOrSz0imejmDx3R9uUIwNYWfMI13 YkkmJX6eZJamRwhmoXmQZFJrS4Gv6enwI6GsRMoqLFlZcJQ0TBbiRnv7Bx4LdHZxENmKxMiV tpRJIlk3+vhJX9sTYREb66Itz0oiVBuSr0SppO3djyiiVNGsZWiUMtqNZuXSCLmT3E4piVSq ZCF2Mjb6sMjBwdF5wxgUHrrU/wOQj539YGV6xCAWJAQlAUM+pJzguq4IJAEjvgnVCODVijke FywDOBFbjHPBIwDX1PkG20cqbnURXEIH4OerWVuuCxi8npW9cRmfT1I2sHedr9fNqCkczjU8 3DThVCIBtQNqQm8ypVzh18lm+lsJaj/smPwT6FlAieHgg7FNC6TsYdr4Tr1sSB2G93/S4Jxl J+y+OkXoGaf2wPjvsnGuuUeGMPOKEcfHYdPlNR7HpvD3ztotNocP0i5ucQws+aKY1LcGqQQA NcMawCU84IWeNFzfA05ZwcoGe062gFd6KjCurjFMWZnCOF0A63K3eS8sr8wnORbCocfqLWag dlRNcrtKAfDnwlGDS8BS88w8mmfm0fxfOh/gpUDIypWRIazSWX5Ixsb898rBUZHVYPMLWXvX gXsTi3ZtAOODNgD5OG0meDM1WGoikErOfcgqogIVqghW2QacN/adjpvvCo7a+IOy6ECRk9jB ycXFxUl8yEVEvyg44NYdbEKFSKLZcJaVs4rtcxjf0DwWo0sG199+6rjQV2sUGSpszfvntdHc Ne9YTdwNZZzmr/n3TeXFrRb7dzz/8H5Zfor7gqOcvE5nL73u5+k58UJzblmhT/n3p56zCHjZ YvHYRwNkVvWsn0AkTrzbLjdm5gp/FRamis/9xtzS2brv4Hn1xLy1VyD41qfP40RiT5LwncVr M+FP7VPwV1+iD0pqyALxzdrOizPO9z6e8x9/r/m0F5vidU2dbr1LhQekZrieGRavmh/1O2O8 b8gtU0vlT9qY345lVOf3LZwUlxBDxkHvTh3sfQUE1Pe2PJF6ztwIEvarZ1vOJ9tma3d3Cj9N VrfKkrzjQx8XvZF+t7AguTHwRx//hEaaUIZKRNa4Qin5F88i9NXLBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpileLIzCtJLcpLzFFi42LZdlhJXpfFIzvFYONUTov1p44xW8xZv4bN omnCX2aL1Xf72SxeH/7EaDHtw09miwf77S32vpvNanHzwE4miz2LJjFZrFx9lMliYz+Hxe6F H5ksHt/5zG5x9P9bNotJh64xWjy9OovJYu8tbYs9e0+yWFzeNYfN4t6a/6wW85c9Zbfovr6D zWL58X9MFocmNzNZ7HjSyGix7fd8Zot1r9+zWJy4JW1x/u9xVovfP+awOch7XL7i7THr/lk2 j52z7rJ7nL+3kcVj2qRTbB6bV2h5XD5b6rFpVSebx6ZPk9g9Tsz4zeKxeUm9x+6bDWwevc3v 2Dw+Pr3F4vF+31U2j74tqxgDhKO4bFJSczLLUov07RK4Mj6eP8NYcK+s4vfTW6wNjC0JXYyc HBICJhLrLp1g6WLk4hAS2M0ocWvlPDaIhKTEsr9HmCFsYYmV/56zQxQ1M0kcnvCQsYuRg4NN QFvi9H8OkLiIwAdmifVbFjGCNDALzGaR6O93A6kRFjCXWNwtAhJmEVCVOPr4C1gJr4ClxNWX 91hASiQE9CX67wuChDkFrCReXJzFDBIWAiqZsMwIolpQ4uTMJ2DVzALqEuvnCUHskZdo3jqb eQKj4CwkVbMQqmYhqVrAyLyKUTK1oDg3PbfYsMAwL7Vcrzgxt7g0L10vOT93EyM4mWhp7mDc vuqD3iFGJg7GQ4wSHMxKIryBfckpQrwpiZVVqUX58UWlOanFhxilOViUxHkvdJ2MFxJITyxJ zU5NLUgtgskycXBKNTB5Fm9WuTPN5sbU4htZN6es/rzd/EPOh31HflaWhNlt+clSGudbYleS OttE2cORfe2b7xPKNCpnKyzWuv0n9mCOzxOv+WXcrzd28EpsTWo1rHn9sle5oWBFntvLyS/m zbuo9CjzOKdQV8Mvj3tb5BMcu47dvxosOjPCZHlx6lP7+1ZbA28HiehfVS/cMPWkbUKh7Npj s/my9nrWVga5PXmi/ZKxImR59b21UnnhGZZr9l0UkfO4Umk/99FRk66nt6SOy0mnrVZOv31t XX6FxZTb01/dtWyLc7FhcLttnONyTu74724doamfNrpwiHhrWf345bzyd9yjVQtuqU9pYlEU rf9/Ra3t/MdD8XqneU4osRRnJBpqMRcVJwIAXAt8NJUDAAA= X-CMS-MailID: 20230522104628epcas5p4f5b3f3d7b080950955a127733d554753 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230522104628epcas5p4f5b3f3d7b080950955a127733d554753 References: <20230522104146.2856-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org For device supporting native copy, nvme driver receives read and write request with BLK_COPY op flags. For read request the nvme driver populates the payload with source information. For write request the driver converts it to nvme copy command using the source information in the payload and submits to the device. current design only supports single source range. This design is courtesy Mikulas Patocka's token based copy trace event support for nvme_copy_cmd. Set the device copy limits to queue limits. Signed-off-by: Kanchan Joshi Signed-off-by: Nitesh Shetty Signed-off-by: Javier González Signed-off-by: Anuj Gupta --- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 103 +++++++++++++++++++++++++++++++++- drivers/nvme/host/fc.c | 5 ++ drivers/nvme/host/nvme.h | 7 +++ drivers/nvme/host/pci.c | 27 ++++++++- drivers/nvme/host/rdma.c | 7 +++ drivers/nvme/host/tcp.c | 16 ++++++ drivers/nvme/host/trace.c | 19 +++++++ include/linux/nvme.h | 43 +++++++++++++- 9 files changed, 220 insertions(+), 8 deletions(-) diff --git a/drivers/nvme/host/constants.c b/drivers/nvme/host/constants.c index bc523ca02254..01be882b726f 100644 --- a/drivers/nvme/host/constants.c +++ b/drivers/nvme/host/constants.c @@ -19,6 +19,7 @@ static const char * const nvme_ops[] = { [nvme_cmd_resv_report] = "Reservation Report", [nvme_cmd_resv_acquire] = "Reservation Acquire", [nvme_cmd_resv_release] = "Reservation Release", + [nvme_cmd_copy] = "Copy Offload", [nvme_cmd_zone_mgmt_send] = "Zone Management Send", [nvme_cmd_zone_mgmt_recv] = "Zone Management Receive", [nvme_cmd_zone_append] = "Zone Management Append", diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index ccb6eb1282f8..aef7b59dbd61 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -754,6 +754,77 @@ static inline void nvme_setup_flush(struct nvme_ns *ns, cmnd->common.nsid = cpu_to_le32(ns->head->ns_id); } +static inline void nvme_setup_copy_read(struct nvme_ns *ns, struct request *req) +{ + struct bio *bio = req->bio; + struct nvme_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + + token->subsys = "nvme"; + token->ns = ns; + token->src_sector = bio->bi_iter.bi_sector; + token->sectors = bio->bi_iter.bi_size >> 9; +} + +static inline blk_status_t nvme_setup_copy_write(struct nvme_ns *ns, + struct request *req, struct nvme_command *cmnd) +{ + struct nvme_copy_range *range = NULL; + struct bio *bio = req->bio; + struct nvme_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + sector_t src_sector, dst_sector, n_sectors; + u64 src_lba, dst_lba, n_lba; + unsigned short nr_range = 1; + u16 control = 0; + + if (unlikely(memcmp(token->subsys, "nvme", 4))) + return BLK_STS_NOTSUPP; + if (unlikely(token->ns != ns)) + return BLK_STS_NOTSUPP; + + src_sector = token->src_sector; + dst_sector = bio->bi_iter.bi_sector; + n_sectors = token->sectors; + if (WARN_ON(n_sectors != bio->bi_iter.bi_size >> 9)) + return BLK_STS_NOTSUPP; + + src_lba = nvme_sect_to_lba(ns, src_sector); + dst_lba = nvme_sect_to_lba(ns, dst_sector); + n_lba = nvme_sect_to_lba(ns, n_sectors); + + if (WARN_ON(!n_lba)) + return BLK_STS_NOTSUPP; + + if (req->cmd_flags & REQ_FUA) + control |= NVME_RW_FUA; + + if (req->cmd_flags & REQ_FAILFAST_DEV) + control |= NVME_RW_LR; + + memset(cmnd, 0, sizeof(*cmnd)); + cmnd->copy.opcode = nvme_cmd_copy; + cmnd->copy.nsid = cpu_to_le32(ns->head->ns_id); + cmnd->copy.sdlba = cpu_to_le64(dst_lba); + + range = kmalloc_array(nr_range, sizeof(*range), + GFP_ATOMIC | __GFP_NOWARN); + if (!range) + return BLK_STS_RESOURCE; + + range[0].slba = cpu_to_le64(src_lba); + range[0].nlb = cpu_to_le16(n_lba - 1); + + cmnd->copy.nr_range = 0; + + req->special_vec.bv_page = virt_to_page(range); + req->special_vec.bv_offset = offset_in_page(range); + req->special_vec.bv_len = sizeof(*range) * nr_range; + req->rq_flags |= RQF_SPECIAL_PAYLOAD; + + cmnd->copy.control = cpu_to_le16(control); + + return BLK_STS_OK; +} + static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req, struct nvme_command *cmnd) { @@ -988,10 +1059,16 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req) ret = nvme_setup_discard(ns, req, cmd); break; case REQ_OP_READ: - ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_read); + if (unlikely(req->cmd_flags & REQ_COPY)) + nvme_setup_copy_read(ns, req); + else + ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_read); break; case REQ_OP_WRITE: - ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_write); + if (unlikely(req->cmd_flags & REQ_COPY)) + ret = nvme_setup_copy_write(ns, req, cmd); + else + ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_write); break; case REQ_OP_ZONE_APPEND: ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_zone_append); @@ -1698,6 +1775,26 @@ static void nvme_config_discard(struct gendisk *disk, struct nvme_ns *ns) blk_queue_max_write_zeroes_sectors(queue, UINT_MAX); } +static void nvme_config_copy(struct gendisk *disk, struct nvme_ns *ns, + struct nvme_id_ns *id) +{ + struct nvme_ctrl *ctrl = ns->ctrl; + struct request_queue *q = disk->queue; + + if (!(ctrl->oncs & NVME_CTRL_ONCS_COPY)) { + blk_queue_max_copy_sectors_hw(q, 0); + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + return; + } + + /* setting copy limits */ + if (blk_queue_flag_test_and_set(QUEUE_FLAG_COPY, q)) + return; + + blk_queue_max_copy_sectors_hw(q, + nvme_lba_to_sect(ns, le16_to_cpu(id->mssrl))); +} + static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b) { return uuid_equal(&a->uuid, &b->uuid) && @@ -1897,6 +1994,7 @@ static void nvme_update_disk_info(struct gendisk *disk, set_capacity_and_notify(disk, capacity); nvme_config_discard(disk, ns); + nvme_config_copy(disk, ns, id); blk_queue_max_write_zeroes_sectors(disk->queue, ns->ctrl->max_zeroes_sectors); } @@ -5343,6 +5441,7 @@ static inline void _nvme_check_size(void) BUILD_BUG_ON(sizeof(struct nvme_download_firmware) != 64); BUILD_BUG_ON(sizeof(struct nvme_format_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_dsm_cmd) != 64); + BUILD_BUG_ON(sizeof(struct nvme_copy_command) != 64); BUILD_BUG_ON(sizeof(struct nvme_write_zeroes_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_abort_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_get_log_page_command) != 64); diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 2ed75923507d..db2e22b4ca7f 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -2807,6 +2807,11 @@ nvme_fc_queue_rq(struct blk_mq_hw_ctx *hctx, if (ret) return ret; + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_end_request(rq, BLK_STS_OK); + return BLK_STS_OK; + } /* * nvme core doesn't quite treat the rq opaquely. Commands such * as WRITE ZEROES will return a non-zero rq payload_bytes yet diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index bf46f122e9e1..66af37170bff 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -500,6 +500,13 @@ struct nvme_ns { }; +struct nvme_copy_token { + char *subsys; + struct nvme_ns *ns; + sector_t src_sector; + sector_t sectors; +}; + /* NVMe ns supports metadata actions by the controller (generate/strip) */ static inline bool nvme_ns_has_pi(struct nvme_ns *ns) { diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 7f25c0fe3a0b..d5d094fa2fd1 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -495,16 +495,19 @@ static inline void nvme_sq_copy_cmd(struct nvme_queue *nvmeq, nvmeq->sq_tail = 0; } -static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +static inline void nvme_commit_sq_db(struct nvme_queue *nvmeq) { - struct nvme_queue *nvmeq = hctx->driver_data; - spin_lock(&nvmeq->sq_lock); if (nvmeq->sq_tail != nvmeq->last_sq_tail) nvme_write_sq_db(nvmeq, true); spin_unlock(&nvmeq->sq_lock); } +static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +{ + nvme_commit_sq_db(hctx->driver_data); +} + static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req, int nseg) { @@ -848,6 +851,12 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, struct request *req) if (ret) return ret; + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_start_request(req); + return BLK_STS_OK; + } + if (blk_rq_nr_phys_segments(req)) { ret = nvme_map_data(dev, req, &iod->cmd); if (ret) @@ -894,6 +903,18 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, ret = nvme_prep_rq(dev, req); if (unlikely(ret)) return ret; + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_set_request_complete(req); + blk_mq_end_request(req, BLK_STS_OK); + /* Commit the sq if copy read was the last req in the list, + * as copy read deoesn't update sq db + */ + if (bd->last) + nvme_commit_sq_db(nvmeq); + return ret; + } + spin_lock(&nvmeq->sq_lock); nvme_sq_copy_cmd(nvmeq, &iod->cmd); nvme_write_sq_db(nvmeq, bd->last); diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 0eb79696fb73..be1d20ac8bb0 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -2038,6 +2038,13 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, nvme_start_request(rq); + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_end_request(rq, BLK_STS_OK); + ret = BLK_STS_OK; + goto unmap_qe; + } + if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) && queue->pi_support && (c->common.opcode == nvme_cmd_write || diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index bf0230442d57..5ba1bb35c557 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -2373,6 +2373,11 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns, if (ret) return ret; + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + return BLK_STS_OK; + } + req->state = NVME_TCP_SEND_CMD_PDU; req->status = cpu_to_le16(NVME_SC_SUCCESS); req->offset = 0; @@ -2441,6 +2446,17 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx, nvme_start_request(rq); + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_set_request_complete(rq); + blk_mq_end_request(rq, BLK_STS_OK); + /* if copy read is the last req queue tcp reqs */ + if (bd->last && nvme_tcp_queue_more(queue)) + queue_work_on(queue->io_cpu, nvme_tcp_wq, + &queue->io_work); + return ret; + } + nvme_tcp_queue_request(req, true, bd->last); return BLK_STS_OK; diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c index 1c36fcedea20..da4a7494e5a7 100644 --- a/drivers/nvme/host/trace.c +++ b/drivers/nvme/host/trace.c @@ -150,6 +150,23 @@ static const char *nvme_trace_read_write(struct trace_seq *p, u8 *cdw10) return ret; } +static const char *nvme_trace_copy(struct trace_seq *p, u8 *cdw10) +{ + const char *ret = trace_seq_buffer_ptr(p); + u64 slba = get_unaligned_le64(cdw10); + u8 nr_range = get_unaligned_le16(cdw10 + 8); + u16 control = get_unaligned_le16(cdw10 + 10); + u32 dsmgmt = get_unaligned_le32(cdw10 + 12); + u32 reftag = get_unaligned_le32(cdw10 + 16); + + trace_seq_printf(p, + "slba=%llu, nr_range=%u, ctrl=0x%x, dsmgmt=%u, reftag=%u", + slba, nr_range, control, dsmgmt, reftag); + trace_seq_putc(p, 0); + + return ret; +} + static const char *nvme_trace_dsm(struct trace_seq *p, u8 *cdw10) { const char *ret = trace_seq_buffer_ptr(p); @@ -243,6 +260,8 @@ const char *nvme_trace_parse_nvm_cmd(struct trace_seq *p, return nvme_trace_zone_mgmt_send(p, cdw10); case nvme_cmd_zone_mgmt_recv: return nvme_trace_zone_mgmt_recv(p, cdw10); + case nvme_cmd_copy: + return nvme_trace_copy(p, cdw10); default: return nvme_trace_common(p, cdw10); } diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 779507ac750b..6582b26e532c 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -337,7 +337,7 @@ struct nvme_id_ctrl { __u8 nvscc; __u8 nwpc; __le16 acwu; - __u8 rsvd534[2]; + __le16 ocfs; __le32 sgls; __le32 mnan; __u8 rsvd544[224]; @@ -365,6 +365,7 @@ enum { NVME_CTRL_ONCS_WRITE_ZEROES = 1 << 3, NVME_CTRL_ONCS_RESERVATIONS = 1 << 5, NVME_CTRL_ONCS_TIMESTAMP = 1 << 6, + NVME_CTRL_ONCS_COPY = 1 << 8, NVME_CTRL_VWC_PRESENT = 1 << 0, NVME_CTRL_OACS_SEC_SUPP = 1 << 0, NVME_CTRL_OACS_NS_MNGT_SUPP = 1 << 3, @@ -414,7 +415,10 @@ struct nvme_id_ns { __le16 npdg; __le16 npda; __le16 nows; - __u8 rsvd74[18]; + __le16 mssrl; + __le32 mcl; + __u8 msrc; + __u8 rsvd91[11]; __le32 anagrpid; __u8 rsvd96[3]; __u8 nsattr; @@ -796,6 +800,7 @@ enum nvme_opcode { nvme_cmd_resv_report = 0x0e, nvme_cmd_resv_acquire = 0x11, nvme_cmd_resv_release = 0x15, + nvme_cmd_copy = 0x19, nvme_cmd_zone_mgmt_send = 0x79, nvme_cmd_zone_mgmt_recv = 0x7a, nvme_cmd_zone_append = 0x7d, @@ -819,7 +824,8 @@ enum nvme_opcode { nvme_opcode_name(nvme_cmd_resv_release), \ nvme_opcode_name(nvme_cmd_zone_mgmt_send), \ nvme_opcode_name(nvme_cmd_zone_mgmt_recv), \ - nvme_opcode_name(nvme_cmd_zone_append)) + nvme_opcode_name(nvme_cmd_zone_append), \ + nvme_opcode_name(nvme_cmd_copy)) @@ -996,6 +1002,36 @@ struct nvme_dsm_range { __le64 slba; }; +struct nvme_copy_command { + __u8 opcode; + __u8 flags; + __u16 command_id; + __le32 nsid; + __u64 rsvd2; + __le64 metadata; + union nvme_data_ptr dptr; + __le64 sdlba; + __u8 nr_range; + __u8 rsvd12; + __le16 control; + __le16 rsvd13; + __le16 dspec; + __le32 ilbrt; + __le16 lbat; + __le16 lbatm; +}; + +struct nvme_copy_range { + __le64 rsvd0; + __le64 slba; + __le16 nlb; + __le16 rsvd18; + __le32 rsvd20; + __le32 eilbrt; + __le16 elbat; + __le16 elbatm; +}; + struct nvme_write_zeroes_cmd { __u8 opcode; __u8 flags; @@ -1757,6 +1793,7 @@ struct nvme_command { struct nvme_download_firmware dlfw; struct nvme_format_cmd format; struct nvme_dsm_cmd dsm; + struct nvme_copy_command copy; struct nvme_write_zeroes_cmd write_zeroes; struct nvme_zone_mgmt_send_cmd zms; struct nvme_zone_mgmt_recv_cmd zmr; From patchwork Mon May 22 10:41:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13250167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36CA0C7EE2A for ; Mon, 22 May 2023 11:17:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233304AbjEVLRF (ORCPT ); Mon, 22 May 2023 07:17:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233173AbjEVLPp (ORCPT ); Mon, 22 May 2023 07:15:45 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 068A310DD for ; Mon, 22 May 2023 04:10:49 -0700 (PDT) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230522111047epoutp018853cf9c41882ffde526d4c5ffd378e8~hcrK1XJK62983329833epoutp01T for ; Mon, 22 May 2023 11:10:47 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230522111047epoutp018853cf9c41882ffde526d4c5ffd378e8~hcrK1XJK62983329833epoutp01T DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1684753847; bh=sCUBrwuMiferk03Zh8Qmlete8s6pSKJMNnA8IVPWhI4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hUfmf/dEtawJDUU0zmlZLygtayepSGV+rKyqPe96oNvAS5QRpCumh8kdAW3o8WhxV ldxwBLFV2mrHzGVm2VUrSB7P9OFCJslhc1rlk7LgTl6gtD43X9RoAfNXdDaAsm697e 9P20ybfJGLu5/HiSttxA6wWiQXq51I0srYa3Pipw= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230522111046epcas5p130438bdfeb6809a0fb2f9de869a58115~hcrJ-ArNi0198501985epcas5p1G; Mon, 22 May 2023 11:10:46 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.180]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4QPvrP34PJz4x9Py; Mon, 22 May 2023 11:10:45 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 5C.39.16380.5BD4B646; Mon, 22 May 2023 20:10:45 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230522104638epcas5p1caf2dc21c5ef7149a10a298b9baeda60~hcWFFOj3c1779017790epcas5p1O; Mon, 22 May 2023 10:46:38 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230522104638epsmtrp19306ae1a3f1187411e109137da914275~hcWFELCXc1501415014epsmtrp1F; Mon, 22 May 2023 10:46:38 +0000 (GMT) X-AuditID: b6c32a4b-56fff70000013ffc-0b-646b4db56bd0 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id AC.23.27706.D084B646; Mon, 22 May 2023 19:46:38 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230522104634epsmtip28e582cb8bc1a0c76c43316a6fe85ddb4~hcWBSeZpY1590015900epsmtip25; Mon, 22 May 2023 10:46:34 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, James.Bottomley@HansenPartnership.com, bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v11 6/9] nvmet: add copy command support for bdev and file ns Date: Mon, 22 May 2023 16:11:37 +0530 Message-Id: <20230522104146.2856-7-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230522104146.2856-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta0xTZxjOOaeclkuXw2XyAQK102zSQSmX+uFEt8jMIXUbOrNlTMMaegaE 0jZtGWpc5LoLQlsrmq1yn2EIBLByRwYrYKEoxDFhVhFEcHXIpcSMOYaspbj573me733e25eX gXno6b6MFImSkkuEYjbuQmvp3flGcPN7qaLQgXJP2GC6jsHihjocZmvWMFg7ocbhXO8yAi8s PcPgVPc+2LVw0Qne6WlH4bVKLQov1/aj8IqaATsrrCjsX5/HodYwhsDZ2zoUdpk58FrXIA2O dhTj8H7duhMsq5qlwzPjbTj80fgchYZzOShsm8lCYMtqGQbr5xZpcMDsB0fWjE5w9a9i/O0A cvRXAambvImT7boJOjly/wqNvKA14eTV6iBy9GY6qa/5Fif1y1o6OfDdKo28euk02XknEycL cxZw0jprppGLP93GSVVTDRLnGZ+6J5kSiig5i5IkSkUpkqRotuDDhP0JkfxQXjAvCu5isyTC NCqaHXMwLvhAiti2KTbrC6E43SbFCRUKNnfvHrk0XUmxkqUKZTSbkonEsghZiEKYpkiXJIVI KOVuXmhoWKQt8LPU5Of1Ytn8O8effjOEZyJWfj7CYAAiAqgGTucjLgwPohMBsyUGuoMs28iI iZaPONvInwg4X+xtx3bDSsE91BHUhYDHFaWYg+ShoGhMT7enxQkOGFpn2HUvYgYD8x1PNoIw wogBTdUgZk/lSXwANHrLRgkasQOMlbY62TGTiALLDb9jjv64QD3pbpedid3AckuHOULcweD3 MxtWjAgEOc0XN/IDwuwMpp+s4Y5WY8BQsw51YE/wh7GJ7sC+4LH6q02cAS4XVeMOcy4CdOM6 xPGwD+SZ1BtNYMRO0NDBdcj+4LypHnUUfgUUrs5s5meCttIX+DVQ11C+2YMPGFvJ2sQkmMw1 bq6uEAGtXWZEg7B0Lw2ke2kg3f+lyxGsBvGhZIq0JEoRKQuXUBn/fXKiNE2PbBxPkKANmZ5a CjEgKAMxIICBsb2Yh1SJIg+mSHjiJCWXJsjTxZTCgETaFn4W8301UWq7PokygRcRFRrB5/Mj osL5PLY38/XowUQPIkmopFIpSkbJX/hQhrNvJur289ctMmNsr+XGs75L7P7GXV+GBBz3a4rf 8o+5dUfwo0mR4q1bWU5cDefN7eOC9gXnQ6lDuTX1/oFT8aVevTGqrIKlwf2nurnx7xZsjT0a WW3SniWj76b3L/YKqhanh8M81/zdCt0XVCePyI897ZhuqpgucfK2+GyzPhxuypij1bs9yLYI XGvPndnr2gexLdLGh58eo/W4cn1rK+ddfsk++vfh98d7Ao/kndLeq+84wSlnsUIOVMYO9/lZ thepPAJG4xIKVw5qJn5TNT/46PNwq/huTLAaREY9+uE6SGMuVn3MSc7RqPo6PxEdbimrHmYs qsJS4636rtWSrZyCG42B3blsmiJZyAvC5Arhvy3onOvFBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02Sb0xTVxjGPffcXm7rai7FhaMmNmk0rphRasCcqYH5ZR4lOt0yR5o5aNo7 YNLStaBiTMTV+acg1JIZqTgnYzgr2giCVECw5V8xgqMVkIHwgQYMQQEFdQzYSrPEb0+e3+99 Pr0slIzSq9l0fRZv1KszZIyIrvHIpB+vIAe1Mb/fh9jZ0QpxibOCwT9a5yG+MVjI4HHPNMAX Jt9BPNyYgBteXBLgp00uCteX2ih8/UYLhW8Xsrju6hSFWxYnGGxz9wAceGKncEP/Rlzf4KWx 714Jg59VLArwlfJAGM7rrWXwtbYFCruLzBSuHTkBcM3cFYhvjb+kcXv/Gtw13ybAc29LmE/X Ep8/kdiHHjHEZR8MI13PbtPkgq2DIVV/RBHfo2xS6TjLkMppWxhpvzhHk6qy46TuaS5Dzplf MGQq0E+Tl/efMKTgjgPsjVCJtmn5jPRDvFERnyJKW7iVYZjYfuT1mYdMLpjabAFCFnGx6E3+ AGUBIlbC1QF0qrQRhMAqVD7fDEM5Al1fGA0LSWYKTdWXCiyAZRluI3q4yAb7ldwkRM47pUvH kPsLoprepGCO4HajvjeepSGaW496frkrCGYx9wmado7C4A7iFKhwKDxYC7ktaOxP+1It+U+x lm8K2eHIWzxCh9alyFx9CVoBZ38P2d9DvwLKAVbxBpMuVWdSGpR6/nC0Sa0zZetTozWZukqw 9AtR8lpw1zEZ7QYUC9wAsVC2UryvQKOViLXqnKO8MTPZmJ3Bm9xgDUvLIsWPLd5kCZeqzuIP 8ryBN/5PKVa4Opc69vkXDXnjJ6DzokatrGoWpQxqxfKe2XQOWr/acU1wbAPgzrz6J/Fm/tgu bU5VdXe+0iX6Pg3pz6qKDp03H3iu1607FX96a2vF5c8id6QodsoppuNLXU3eELtiJG+Us98E nSqX4d5jUKk8EBPj+Xpn72xsS5xvmab5tVc1nD7YWv3NzP6rpzc07SvJPC+1dcdlD1gCy3ep SGcyzPHINyf9VjHjUyi8iW+JpAsM/fDdmIc9skgmG0/2zZYd/0D2s6lz28yenxI8ez50/G0V RmqKY4v2v4ssHGg/Gqjzr51QfNvk978SxmdJO9v6Cj4Sxkld8uGE3ROurcj/IKm5W0ab0tTK KGg0qf8FLCrG7XoDAAA= X-CMS-MailID: 20230522104638epcas5p1caf2dc21c5ef7149a10a298b9baeda60 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230522104638epcas5p1caf2dc21c5ef7149a10a298b9baeda60 References: <20230522104146.2856-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Add support for handling nvme_cmd_copy command on target. For bdev-ns we call into blkdev_issue_copy, which the block layer completes by a offloaded copy request to backend bdev or by emulating the request. For file-ns we call vfs_copy_file_range to service our request. Currently target always shows copy capability by setting NVME_CTRL_ONCS_COPY in controller ONCS. loop target has copy support, which can be used to test copy offload. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- drivers/nvme/target/admin-cmd.c | 9 ++++- drivers/nvme/target/io-cmd-bdev.c | 62 +++++++++++++++++++++++++++++++ drivers/nvme/target/io-cmd-file.c | 52 ++++++++++++++++++++++++++ drivers/nvme/target/loop.c | 6 +++ drivers/nvme/target/nvmet.h | 1 + 5 files changed, 128 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c index 39cb570f833d..8e644b8ec0fd 100644 --- a/drivers/nvme/target/admin-cmd.c +++ b/drivers/nvme/target/admin-cmd.c @@ -433,8 +433,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) id->nn = cpu_to_le32(NVMET_MAX_NAMESPACES); id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES); id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM | - NVME_CTRL_ONCS_WRITE_ZEROES); - + NVME_CTRL_ONCS_WRITE_ZEROES | NVME_CTRL_ONCS_COPY); /* XXX: don't report vwc if the underlying device is write through */ id->vwc = NVME_CTRL_VWC_PRESENT; @@ -536,6 +535,12 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req) if (req->ns->bdev) nvmet_bdev_set_limits(req->ns->bdev, id); + else { + id->msrc = (__force u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16(BIO_MAX_VECS << + (PAGE_SHIFT - SECTOR_SHIFT)); + id->mcl = cpu_to_le32(le16_to_cpu(id->mssrl)); + } /* * We just provide a single LBA format that matches what the diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c index c2d6cea0236b..d92dfe86c647 100644 --- a/drivers/nvme/target/io-cmd-bdev.c +++ b/drivers/nvme/target/io-cmd-bdev.c @@ -46,6 +46,18 @@ void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id) id->npda = id->npdg; /* NOWS = Namespace Optimal Write Size */ id->nows = to0based(bdev_io_opt(bdev) / bdev_logical_block_size(bdev)); + + if (bdev_max_copy_sectors(bdev)) { + id->msrc = id->msrc; + id->mssrl = cpu_to_le16((bdev_max_copy_sectors(bdev) << + SECTOR_SHIFT) / bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32((__force u32)id->mssrl); + } else { + id->msrc = (__force u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16((BIO_MAX_VECS << PAGE_SHIFT) / + bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32((__force u32)id->mssrl); + } } void nvmet_bdev_ns_disable(struct nvmet_ns *ns) @@ -184,6 +196,21 @@ static void nvmet_bio_done(struct bio *bio) nvmet_req_bio_put(req, bio); } +static void nvmet_bdev_copy_end_io(void *private, int comp_len) +{ + struct nvmet_req *req = (struct nvmet_req *)private; + u16 status; + + if (comp_len == req->copy_len) { + req->cqe->result.u32 = cpu_to_le32(1); + status = errno_to_nvme_status(req, 0); + } else { + req->cqe->result.u32 = cpu_to_le32(0); + status = errno_to_nvme_status(req, (__force u16)BLK_STS_IOERR); + } + nvmet_req_complete(req, status); +} + #ifdef CONFIG_BLK_DEV_INTEGRITY static int nvmet_bdev_alloc_bip(struct nvmet_req *req, struct bio *bio, struct sg_mapping_iter *miter) @@ -450,6 +477,37 @@ static void nvmet_bdev_execute_write_zeroes(struct nvmet_req *req) } } +/* At present we handle only one range entry, since copy offload is aligned with + * copy_file_range, only one entry is passed from block layer. + */ +static void nvmet_bdev_execute_copy(struct nvmet_req *req) +{ + struct nvme_copy_range range; + struct nvme_command *cmd = req->cmd; + int ret; + u16 status; + + status = nvmet_copy_from_sgl(req, 0, &range, sizeof(range)); + if (status) + goto out; + + ret = blkdev_issue_copy(req->ns->bdev, + le64_to_cpu(cmd->copy.sdlba) << req->ns->blksize_shift, + req->ns->bdev, + le64_to_cpu(range.slba) << req->ns->blksize_shift, + (le16_to_cpu(range.nlb) + 1) << req->ns->blksize_shift, + nvmet_bdev_copy_end_io, (void *)req, GFP_KERNEL); + if (ret) { + req->cqe->result.u32 = cpu_to_le32(0); + status = blk_to_nvme_status(req, BLK_STS_IOERR); + goto out; + } + + return; +out: + nvmet_req_complete(req, errno_to_nvme_status(req, status)); +} + u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) { switch (req->cmd->common.opcode) { @@ -468,6 +526,10 @@ u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_bdev_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_bdev_execute_copy; + return 0; + default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c index 2d068439b129..f61aa834f7a5 100644 --- a/drivers/nvme/target/io-cmd-file.c +++ b/drivers/nvme/target/io-cmd-file.c @@ -322,6 +322,49 @@ static void nvmet_file_dsm_work(struct work_struct *w) } } +static void nvmet_file_copy_work(struct work_struct *w) +{ + struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); + int nr_range = req->cmd->copy.nr_range + 1; + u16 status = 0; + int src, id; + ssize_t len, ret; + loff_t pos; + + pos = le64_to_cpu(req->cmd->copy.sdlba) << req->ns->blksize_shift; + if (unlikely(pos + req->transfer_len > req->ns->size)) { + nvmet_req_complete(req, errno_to_nvme_status(req, -ENOSPC)); + return; + } + + for (id = 0 ; id < nr_range; id++) { + struct nvme_copy_range range; + + status = nvmet_copy_from_sgl(req, id * sizeof(range), &range, + sizeof(range)); + if (status) + goto out; + + src = (le64_to_cpu(range.slba) << (req->ns->blksize_shift)); + len = (le16_to_cpu(range.nlb) + 1) << (req->ns->blksize_shift); + ret = vfs_copy_file_range(req->ns->file, src, req->ns->file, + pos, len, 0); + if (ret != len) { + pos += ret; + req->cqe->result.u32 = cpu_to_le32(id); + if (ret < 0) + status = errno_to_nvme_status(req, ret); + else + status = errno_to_nvme_status(req, -EIO); + goto out; + } else + pos += len; + } + +out: + nvmet_req_complete(req, status); +} + static void nvmet_file_execute_dsm(struct nvmet_req *req) { if (!nvmet_check_data_len_lte(req, nvmet_dsm_len(req))) @@ -330,6 +373,12 @@ static void nvmet_file_execute_dsm(struct nvmet_req *req) queue_work(nvmet_wq, &req->f.work); } +static void nvmet_file_execute_copy(struct nvmet_req *req) +{ + INIT_WORK(&req->f.work, nvmet_file_copy_work); + queue_work(nvmet_wq, &req->f.work); +} + static void nvmet_file_write_zeroes_work(struct work_struct *w) { struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); @@ -376,6 +425,9 @@ u16 nvmet_file_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_file_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_file_execute_copy; + return 0; default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c index f2d24b2d992f..d18ed8067a15 100644 --- a/drivers/nvme/target/loop.c +++ b/drivers/nvme/target/loop.c @@ -146,6 +146,12 @@ static blk_status_t nvme_loop_queue_rq(struct blk_mq_hw_ctx *hctx, return ret; nvme_start_request(req); + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_set_request_complete(req); + blk_mq_end_request(req, BLK_STS_OK); + return BLK_STS_OK; + } iod->cmd.common.flags |= NVME_CMD_SGL_METABUF; iod->req.port = queue->ctrl->port; if (!nvmet_req_init(&iod->req, &queue->nvme_cq, diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h index dc60a22646f7..1615dc9194ba 100644 --- a/drivers/nvme/target/nvmet.h +++ b/drivers/nvme/target/nvmet.h @@ -393,6 +393,7 @@ struct nvmet_req { struct device *p2p_client; u16 error_loc; u64 error_slba; + size_t copy_len; }; #define NVMET_MAX_MPOOL_BVEC 16 From patchwork Mon May 22 10:41:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13250168 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19266C77B75 for ; Mon, 22 May 2023 11:17:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233317AbjEVLRd (ORCPT ); Mon, 22 May 2023 07:17:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233195AbjEVLPq (ORCPT ); Mon, 22 May 2023 07:15:46 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCBA510D4 for ; Mon, 22 May 2023 04:10:52 -0700 (PDT) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230522111051epoutp0341c2b6b1c24c9e6b3fde34f8f81ddead~hcrOMj1NJ1227912279epoutp03w for ; Mon, 22 May 2023 11:10:51 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230522111051epoutp0341c2b6b1c24c9e6b3fde34f8f81ddead~hcrOMj1NJ1227912279epoutp03w DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1684753851; bh=1P2BJZZ/eKH4o2N5bmfZBmLKhHrfzdv6MvJHTRtfiUg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jLE+yCJo8rvze8zKIR8qt3EOxKGuGd03XPUXqBHnhPs1YE3N0sLxRcSK2P5eWlrjq 4wd+Du4UGcQIEuOHc9pL7PX+sxhrubgdkZMF78VBXxodKUC5dBvyGzVjcXWjgLSjNQ 9m96fpyinsxILizzTk/Bie6+VYhmXsBYJgPpWARQ= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230522111050epcas5p21210becde80901bedfb5279ff1dc37df~hcrNhmLR22481324813epcas5p20; Mon, 22 May 2023 11:10:50 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.175]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4QPvrS5yCpz4x9Q2; Mon, 22 May 2023 11:10:48 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id F7.03.54880.8BD4B646; Mon, 22 May 2023 20:10:48 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230522104648epcas5p2286a988d89b1befdb49984ebbefb25fd~hcWON8qBh1654016540epcas5p2X; Mon, 22 May 2023 10:46:48 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230522104647epsmtrp2f54f4efe0a6063ed075b8ef1db5011c5~hcWOMewuQ3249732497epsmtrp2V; Mon, 22 May 2023 10:46:47 +0000 (GMT) X-AuditID: b6c32a49-8c5ff7000001d660-e1-646b4db84c65 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id A2.FF.28392.7184B646; Mon, 22 May 2023 19:46:47 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230522104643epsmtip2f74158f3b9d3a6a02b917f21da70e2c6~hcWKbW10-1645716457epsmtip23; Mon, 22 May 2023 10:46:43 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, James.Bottomley@HansenPartnership.com, bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v11 7/9] dm: Add support for copy offload Date: Mon, 22 May 2023 16:11:38 +0530 Message-Id: <20230522104146.2856-8-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230522104146.2856-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta0xTZxjOd045LWQdR8DsA6MrNQsD5VIs3VeFMQcxB5kRtmXJbikdPdAO elkvwkg2uekyGPdJZsVxW5Q7chnBAtPVaLnI2GSANOEyUqYTBVuygWmQFQ5u/nve532f5718 +Ti4Vxfbj6NQ6WmtSprOJzxYPTcCA4J7T6TJwizzwah9+BaOqtpbCJRbuoGj5pkSAi3dcABU +fgJjuavRaOB5QtuaPr6VQz115VjqLH5JoY6Sjior9aOoZubjwhUbp4EaHHCiKEB6wHUPzDE QuOmKgLNtmy6oepLi2xUONVLoMuWpxgyV+RhqNeWA1CPsxpHbUsrLDRo3YPGNixuyLleRbyx jxr/PZ4yzo0S1FXjDJsam+1gUZXlwwTV1RBEjY8aqM6mrwmq01HOpga/c7Korh9OU33T2QRV lLdMUPZFK4ta+WmCoIq7m0CC9wdpkXJaKqO1PFqVrJYpVKlR/Ph3JDGSCFGYIFggRq/xeSqp ko7ix76VEHxMke66FJ93SppucFEJUp2OH/p6pFZt0NM8uVqnj+LTGlm6RqgJ0UmVOoMqNURF 6w8LwsLCI1yFSWlyW3090HT7ZS7cH8SzgWl3AeBwICmEFbPHC4AHx4vsA/CPvAIWEzgAzLWv 4kywCuDI5ogr476t6OyZIJiECcC6jvkdyRkMNhSbiS1fgjzgUnC2eB/ShsNHpofbVjhpwWHp pSF8q8ibFMOLDUe2XFnkK/DsrI3YwlwX7VzKZjPzhcKSuV1btDt5GN7/zYgzJbvg0Hnb9kA4 +TLM+/HCtj0kre6w/O88wEwaC2dMcwSDveEDSzebwX5wdXlgh8+Ajd82EIw4H0DjlHFHHA3P DJdsz4mTgbDdFMrQe+G54TaMafwiLHLaMIbnwt7vn+H9sKW9ZsffF06u5exgCl4ZzQbMsYoA bB7oxEsBz/jcQsbnFjL+37oG4E3Al9bolKm0LkIjUNEZ/71yslrZCbZ/T1BcL5iZfxxiBhgH mAHk4HwfbmJxssyLK5N+nkVr1RKtIZ3WmUGE6+BluN/uZLXr+6n0EoFQHCYUiURC8SGRgP8S NyBqKNmLTJXq6TSa1tDaZzqM4+6XjbFb1XFvn1dc7MhsXKuozyJjJIX9eyYcnu8FfLGwiEWm 3Lu7Ul+zz6JcJBf//CRJVY3Nm86K162Zp0v/Wm+w/nziTjTt7xdbebtKQ7w/9+mdUVvWUYX8 BSV3qqIsUSiqqxB+lOCWlfPVUenkA3sbf/7Jl0v9q85aedX0kbZ7ilNlYebjafkHr3mghZMG n2NLI++mhPs74vsCp1Kk6zGv2mvF/p9x6/4xrG2A5F8Syk46UmMPDufg+YmHHp5T1uR+Yw5v tP+693ZnvU3PI64kZTUO1zSlf4yI2rU3PVsLUyRuyylDXdefxo2Jl1vd+i/n5qbdpTm+nhmJ CV23rCsf+vtb+CydXCoIwrU66b+C/c1ixgQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02SbUxTVxiAc+65Pb1tbHYtLhyozqXGmcAG07h4pmaQuZmzEXVsfscEG7gB AoWuBRGdUq1zW5kUixi9lLhVg1BQIiArApUUlOJHWkTU8rk5GBiiAi4ZlCCzwyX+e/M8z/v+ ejmoHGHDudSMLEGfoUlXIzlb36pe+kEoTUv6sGzkI1J96yYktuoqRI4WzkJS2W9BZKx1EpDT 49OQ/H49hjQ/K5EQf0sDQ5rsVoZUVN5gyBULRxp/nWDIjbmniFjdDwAZ7hYZ0twTSZqaO1jS dc2GyEDVnIScKxuWkvyHTkQutr9kiLvIxBDn0BFA6mfOQXJ57DlLPD0q4p1tl5CZKRuKfYd2 3Y+j4uBdRBvEfin1Dlxh6WnrLURryyNo191sWuP4CdGaSauUes7MsLT2Qh5t9BsRPWF6hujE cA9Ln7u6ES2oc4CvQnbL1ycJ6an7BH30J3vlKUPnzwNdXfj+x6MeaATX3jYDGYf51bimvhuZ gZxT8k6AS1weOC/CcNls2+s5BFe8HJHORyYGn3L+xZoBxyE+Et+e44J8ET8OcXWdHQQXIN8L cf3DncEmhP8Yl5avC2KWX46PDwyh4Kx4hWfGjNJggvlobBlcGMQyfi0e7RRhECtfJYVlq+br hbjj7BA7f3wpNl0tgYWAF99Q4hvqF8A4QJigM2iTtYaVulUZQk6UQaM1ZGckRyVmamvAf78Q EeEETY7xKDdgOOAGmIPqRYr4gsQkpSJJk3tA0Gcm6LPTBYMbqDhWHarwmTsSlHyyJktIEwSd oP/fMpws3MjkS9Y3+r++VHgk50fNFicf6MaPmh+FlbgKVNAnk8E/59r+8cZuMm1cdu/yVjuc 2FXeN+rc9eVJ3b2EkRfvxuRsPbnG8NbqS1x82zd/L3O7OzYuMG0JfV+ieLDkhbI1zlGMHw/q i/fkLn6y7kRfnmyxTdX03W3UcrMoIM93icdO7XXHBIpyI9c2lF/Pq+3foFnzaZXecfhoWq+r 9z2Zjynt82pt8Snx49/emUr94ucfVkz64zZb4wKJxb2+YZNYamy3+4ySCf/n0xd/s6dFWjot nSvaLfe37cjpjF1waEfFweh9Kteh2emsxpbM/YG6zxo8ZGr7btGrvTMt5oWaK0u//0PNGlI0 KyOg3qD5F3kRmWl6AwAA X-CMS-MailID: 20230522104648epcas5p2286a988d89b1befdb49984ebbefb25fd X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230522104648epcas5p2286a988d89b1befdb49984ebbefb25fd References: <20230522104146.2856-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Before enabling copy for dm target, check if underlying devices and dm target support copy. Avoid split happening inside dm target. Fail early if the request needs split, currently splitting copy request is not supported. Signed-off-by: Nitesh Shetty --- drivers/md/dm-table.c | 41 +++++++++++++++++++++++++++++++++++ drivers/md/dm.c | 7 ++++++ include/linux/device-mapper.h | 5 +++++ 3 files changed, 53 insertions(+) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 1398f1d6e83e..b3269271e761 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1867,6 +1867,39 @@ static bool dm_table_supports_nowait(struct dm_table *t) return true; } +static int device_not_copy_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + struct request_queue *q = bdev_get_queue(dev->bdev); + + return !blk_queue_copy(q); +} + +static bool dm_table_supports_copy(struct dm_table *t) +{ + struct dm_target *ti; + unsigned int i; + + for (i = 0; i < t->num_targets; i++) { + ti = dm_table_get_target(t, i); + + if (!ti->copy_offload_supported) + return false; + + /* + * target provides copy support (as implied by setting + * 'copy_offload_supported') + * and it relies on _all_ data devices having copy support. + */ + if (!ti->type->iterate_devices || + ti->type->iterate_devices(ti, + device_not_copy_capable, NULL)) + return false; + } + + return true; +} + static int device_not_discard_capable(struct dm_target *ti, struct dm_dev *dev, sector_t start, sector_t len, void *data) { @@ -1949,6 +1982,14 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, q->limits.discard_misaligned = 0; } + if (!dm_table_supports_copy(t)) { + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + q->limits.max_copy_sectors = 0; + q->limits.max_copy_sectors_hw = 0; + } else { + blk_queue_flag_set(QUEUE_FLAG_COPY, q); + } + if (!dm_table_supports_secure_erase(t)) q->limits.max_secure_erase_sectors = 0; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 3b694ba3a106..ab9069090a7d 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1720,6 +1720,13 @@ static blk_status_t __split_and_process_bio(struct clone_info *ci) if (unlikely(ci->is_abnormal_io)) return __process_abnormal_io(ci, ti); + if ((unlikely(op_is_copy(ci->bio->bi_opf)) && + max_io_len(ti, ci->sector) < ci->sector_count)) { + DMERR("Error, IO size(%u) > max target size(%llu)\n", + ci->sector_count, max_io_len(ti, ci->sector)); + return BLK_STS_IOERR; + } + /* * Only support bio polling for normal IO, and the target io is * exactly inside the dm_io instance (verified in dm_poll_dm_io) diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index a52d2b9a6846..04016bd76e73 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -398,6 +398,11 @@ struct dm_target { * bio_set_dev(). NOTE: ideally a target should _not_ need this. */ bool needs_bio_set_dev:1; + + /* + * copy offload is supported + */ + bool copy_offload_supported:1; }; void *dm_per_bio_data(struct bio *bio, size_t data_size); From patchwork Mon May 22 10:41:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13250170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50F62C7EE30 for ; Mon, 22 May 2023 11:17:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232192AbjEVLRk (ORCPT ); Mon, 22 May 2023 07:17:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229837AbjEVLPt (ORCPT ); Mon, 22 May 2023 07:15:49 -0400 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E0353212D for ; Mon, 22 May 2023 04:10:56 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20230522111055epoutp04cd198b57794e6879a986e964d092f5fb~hcrSPPrf01725517255epoutp04g for ; Mon, 22 May 2023 11:10:55 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20230522111055epoutp04cd198b57794e6879a986e964d092f5fb~hcrSPPrf01725517255epoutp04g DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1684753855; bh=cp2ikyfpBdIhGRYkH19yhVPh/sP+Xdj/kjqikt163DM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IzQJrA5xfmVQexqHTuv8PLDMXc1xR/Gr7ke6dXvs6vYXRavsdUbuSr0wFp6fOzn5Z zP4G78bIKvJd4ZiR574V6AwYTFshFBjPtF0tfG8rIfD/2NFhU/pQ+Z84LniB2ZlS2c qzvGud0Sn0/wU0Svzwm/er4Kgk9+pBMg1RB8EKrQ= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230522111054epcas5p33acd05785f22834d299a20230b560857~hcrRipPPd1390713907epcas5p3i; Mon, 22 May 2023 11:10:54 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.176]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4QPvrX4txBz4x9Ps; Mon, 22 May 2023 11:10:52 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id FB.03.54880.CBD4B646; Mon, 22 May 2023 20:10:52 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230522104657epcas5p19117017c9dfd3d7a4860d2f9122b1277~hcWXN8Hld1779017790epcas5p18; Mon, 22 May 2023 10:46:57 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230522104657epsmtrp151d988d04f34f0cc5a959c2d73aa022c~hcWXM9K5l1501415014epsmtrp1W; Mon, 22 May 2023 10:46:57 +0000 (GMT) X-AuditID: b6c32a49-b21fa7000001d660-f2-646b4dbc5e45 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 07.FF.28392.1284B646; Mon, 22 May 2023 19:46:57 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230522104653epsmtip2d7d619611d9ae6ee3b4a9dd3ccc0c4dd~hcWTcnaV61590015900epsmtip2B; Mon, 22 May 2023 10:46:53 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, James.Bottomley@HansenPartnership.com, bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v11 8/9] dm: Enable copy offload for dm-linear target Date: Mon, 22 May 2023 16:11:39 +0530 Message-Id: <20230522104146.2856-9-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230522104146.2856-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TazBcZxie75zdY5nqHCTjw0T1JJ2W1GUTtp97tUnmGDFxq8500pEde1hj 9+x2dyW0yWRFJaLDujVkaUPT0QRFEHWNlorQMSY2rkFIKaIukalUDbprafPveZ/3fd7L883H wy3rTGx5cayKUbBCCUWYceo7HB2dW4LjRW65Px5CVT33cVRUVUGgS1mbOCof1xBooWMVoGsr 6ziabPNHrUuFXDTycyOGWr7LwdDt8k4M3dHwUHPJcwx1bi8SKKd9EKCZAS2GWkcPo5bWbg7S NRURaKJim4tulM6YoK+GGgj0Q9cWhtpzUzDUMJ0MUP3GDRxVLixz0INRO9S32cVFG38XEe/b 07pHQbT2SS9BN2rHTei+iTsc+lpOD0HX3nKidb0JdE3ZVYKuWc0xoR8UbHDo2u8v0s0jaoLO SFki6Oczoxx6+d4AQWfWlYEQq0/ifcSMUMQoHBg2WiaKY2N9qaDwqA+jPARufGe+J3qPcmCF UsaXOnYyxPlEnETvFOVwVihJ0FMhQqWScvXzUcgSVIyDWKZU+VKMXCSRu8tdlEKpMoGNdWEZ lRffze2Ih77wTLw4vyyfK0/mJv6x+QuhBgWcdGDKg6Q7LBi4jBmwJdkMYFpyQjow0+NVACsq 87jG4AWA/feriT1FsfrZbqIJwPS0IWAMUjFYODanz/B4BHkY/rbNM/D7yGkcLjb9iRsCnOzC YVZpN25oZUUeh9Ulv+8M55BvwT7dDDBgc9IT5s9V7zSCpCvUPLEw0KakF5x7qMWNJRaw+/r0 zg04+QZMuVu40x+SE6awNq8UN656DK4XLAMjtoLPuupMjNgWzmsu7+Jz8HbeLcIo/hJA7ZB2 V+APU3s0uGEJnHSEVU2uRvoA/LqnEjMOfh1mbExjRt4cNny7hw/CiqriXbts4ODL5F1Mw1+n SnbdygDwytNqIgs4aF85SPvKQdr/RxcDvAzYMHKlNJZResj5LHPuv2eOlklrwM73cQpsAOOT Ky7tAOOBdgB5OLXPPDQzWmRpLhImfc4oZFGKBAmjbAceesOzcdv90TL9/2NVUXx3Tzd3gUDg 7nlUwKeszd/27Y62JGOFKiaeYeSMYk+H8Uxt1Vgae1ISUCm8ZBGw348fofkiidqK++t0xEJY umkklLG6RU1fr+7U9Q9aT0HtN62RmaKYjs3J+nKz0rGpm2tJ/LHOYxuDIXlBF+zPlygu3sxQ n7jS9VgXcPT82pqd9KV41Tp0q6LbP/g1aWV6meDTF2dOu/jZ+36c6KMO5siveqtj/CNxb+5C ugk1JesvfSRKpMad0w6kFsxbD4es/4MrZv3bbGwc2Y9S7/0ERrzDYLY3792Vp3ltY4HhDd6W EXZzoRes3nkoIKzm2IPhw7lpTLHtbE2V1GuiWhmt+6wxDD1uGZNIZv2yhwfy598U97NHDiHx 9NnALd8lxULM3ZbcYYqjFAv5TrhCKfwXX8ItBscEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02SXUxTZxzG855zenpoqBxaZa9gdOvYxWpAQI2vH/Ej6vImEIEtW9hEoaFn yIBCWnDDXazYaGKNtKsKpTThQ8egOj6KlGKLsCpSJMikU1KUShAUx5QCXmgIViuaePfk+f3+ z9WfIUXTVCSToyjilApZnoQWULbrknUxn+FceVxA/TlqvnWTRObmyzQ6rl8i0aUxHY1mrs8D VO5/RaLx7l2o63kVD3l7OgnkrDMQqPFSL4FadQxy1M4RqDfwjEYG1z2Apu6aCNQ1uh45u/op 5LlqppHvcoCHquun+Oj0iJ1Gf/a9JpDrrIZA9slSgGyL1SRqmpmlkHs0Cg0t9fHQ4kszvXst 9vybiE0PB2ncaRrj4yFfK4XLDbdo3NYgxZ7BYmy1nKKxdd7Ax27jIoXbLv6GHV41jc9ontN4 bmqUwrPX7tK47IoFpIh/EOyQc3k5Rznlhp2ZgiMVlgpeYSnvl8dLf9NqYKS0IISB7CZYo/6P pwUCRsTaAeypXngPVsP6pRvkchbDxtdP+MuShoDW3/20FjAMza6HAwEm2K9k/SRsvlIHggck e5+EtpG0YBaz+2FL7SMimCn2CzjkmXrnCNmtsGK6hRfcgewGqHsYHqxD2G1w+o6JDNait4q+ PmHZDof9lZPU8vo6qGmvIvWANX2ETB+hGkBYwGquUJWfna+KL0xQcD/HqmT5qmJFdmxWQb4V vHsGqdQOnBZ/rAsQDHAByJCSlcLUsiy5SCiXlRzjlAUZyuI8TuUCUQwl+UT4j7Y/Q8Rmy4q4 XI4r5JQfKMGERKqJ6Ju3XzR8ZYhOf/LguPQPz0u3+evNzj2mVWJizVhcajhaSORvDHPuajiz IrlAH+MY/7/1wrPHZ8/zB35t3Bum3+9rcRXPDib21e8rCZXqLm4M9PvW+hO6S73DvsrWqKfe sttf5oyv+BS6ztd4mrj0trQ1w8b4FxUdfRPfMsmqcz9mLhg7Zzo7xt1JJQ1mOzV47bCwRaRo qs3N2xc6eyMiK+5AxkQEq7EZfyqr3PJNe8FYVtIdddpBNy9i+6aYvw7pTuTOWT0dTYLQyMnv UpNsaNW27zOTy+dThuSjzipn0cDJC9HNTgerlobV8lO8igmt1jEs6H1wbkTsJvfEnFpsk1Cq I7J4KalUyd4AmRghvnsDAAA= X-CMS-MailID: 20230522104657epcas5p19117017c9dfd3d7a4860d2f9122b1277 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230522104657epcas5p19117017c9dfd3d7a4860d2f9122b1277 References: <20230522104146.2856-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Setting copy_offload_supported flag to enable offload. Signed-off-by: Nitesh Shetty --- drivers/md/dm-linear.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c index f4448d520ee9..1d1ee30bbefb 100644 --- a/drivers/md/dm-linear.c +++ b/drivers/md/dm-linear.c @@ -62,6 +62,7 @@ static int linear_ctr(struct dm_target *ti, unsigned int argc, char **argv) ti->num_discard_bios = 1; ti->num_secure_erase_bios = 1; ti->num_write_zeroes_bios = 1; + ti->copy_offload_supported = 1; ti->private = lc; return 0; From patchwork Mon May 22 10:41:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13250169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5632C7EE2D for ; Mon, 22 May 2023 11:17:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233330AbjEVLRh (ORCPT ); Mon, 22 May 2023 07:17:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58094 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232417AbjEVLPv (ORCPT ); Mon, 22 May 2023 07:15:51 -0400 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0567410E6 for ; Mon, 22 May 2023 04:11:00 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20230522111058epoutp0279f2785d401ff3cc6a65d32cc1a1456c~hcrVI9q3s1081810818epoutp027 for ; Mon, 22 May 2023 11:10:58 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20230522111058epoutp0279f2785d401ff3cc6a65d32cc1a1456c~hcrVI9q3s1081810818epoutp027 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1684753858; bh=83AxBeKcHPZBcp1RaGv5HmeWYA8Mwjvvo3x8XiJloJA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Xmfl7l4VsLbl8cXQcRRaFCzkBsCRsmx5rO5jLXWg0o0+Ap/oXJw+QUX8lzaYHZpu5 egOHAHyviE2oTg7tqqc3gUON0XNxwVKo/b5aQJQ0db3sDg+wgaD5HZBN6b15mKARb4 PwBprG3GBVIyAiekwunNklZKNc5YFp6W09wQztFQ= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230522111057epcas5p2bd0372a943b87bc5dfe45e25fcae9cf7~hcrUkUFmj2988629886epcas5p22; Mon, 22 May 2023 11:10:57 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.175]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4QPvrc4Nzjz4x9Pr; Mon, 22 May 2023 11:10:56 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 3F.03.54880.0CD4B646; Mon, 22 May 2023 20:10:56 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230522104708epcas5p20f182069898f9c6852826600ce6116c1~hcWheoaM90821608216epcas5p2E; Mon, 22 May 2023 10:47:08 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230522104708epsmtrp17ff4a7d88fedc98b3fa557152b70893a~hcWhdhI5V1451014510epsmtrp1X; Mon, 22 May 2023 10:47:08 +0000 (GMT) X-AuditID: b6c32a49-8c5ff7000001d660-fd-646b4dc011fb Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 0C.FF.28392.C284B646; Mon, 22 May 2023 19:47:08 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230522104703epsmtip25ab915d72944e30a8e5989b7e059b922~hcWc5i_3U1827918279epsmtip2T; Mon, 22 May 2023 10:47:03 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner , "Matthew Wilcox (Oracle)" , Andrew Morton Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, James.Bottomley@HansenPartnership.com, bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Damien Le Moal , Vincent Fu , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v11 9/9] null_blk: add support for copy offload Date: Mon, 22 May 2023 16:11:40 +0530 Message-Id: <20230522104146.2856-10-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230522104146.2856-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TaVBTVxjtfS+8BGdoH4tyAas0QqkwQMLWC4i0gM4r1JHWsds4xQx5Awwh SZNQbdUWRIrEocEUgT5oQGupLAMVxUIAhahswoAFBSMYLAEFKls7dAGlCYHWf+c795xvu/Nx cLtmtjMnSaygZWKBiEtsYF25vt3Du2VPspA32u2CarracFRcU0Wg47lPcVQ5oiLQ9PUFgPLn /sbRUk8vjkavhaPmmSIrdK+lAUNN59QYKq+8iaGLKg5qPDuPoZsrTwik1t0FaPwOg6FmvRdq au5koX5tMYEeVK1YoZKycTY6NVhPoB/bn2FI900GhuqN6QBdWSrBUfX0LAt16F3Q2KmTAPU+ bbdCS38VE2+4Uv0DMRRj6CGoBmaETfU+uMii8tVdBHXpgifV35NK1VZkE1TtgppNdRQusahL 57+kGu+lEVROxozJk2mwoubH9Sxq9uodItbho+QdibRASMtcaXG8RJgkTgjjxuyLi4wLDOLx vfnB6HWuq1iQQodxo96O9d6dJDLtjOv6qUCUaqJiBXI513fnDpkkVUG7JkrkijAuLRWKpAFS H7kgRZ4qTvAR04oQPo/nF2gSHkxOzMsrY0m/Cz9cUNaDpYHxACWw5kAyABqVp4ESbODYkY0A pjP6tWABQM3EfbYl+B3A5oob7HXLzCRDWB60AP7U2r2mysRg/T8TJj+HQ5Be8NYKx8w7kEYc PtH+hpsDnPyeBSdv5OBmkT0ZDq9lbTFnZZHusD9/gmWmbcgQONuy1wwh6QtVBluzwtrEPr7N 4GZsQ9rCzm+NLDPGya0wo65oNTskp6zhQM0Ibmk0CmbfLgAWbA+n2i+vDeAMJ1VfreFDsDzv AmExnwCQGWTWDOEws0u12iZOboc1Wl8L/TI801WNWQq/CHOWjJiFt4H1mnW8DVbVlBIW7ATv /pm+himoz6+2suwqB8DZglKQC1yZ5wZinhuI+b90KcArgBMtlack0PJAKV9MH/rvl+MlKbVg 9Y4836oHI6NzPjqAcYAOQA7OdbB55+t4oZ2NUPDZ57RMEidLFdFyHQg07fs07rwxXmI6RLEi jh8QzAsICgoKCPYP4nMdbTzCOuPtyASBgk6maSktW/dhHGvnNOxkW1Zf+CP3SdVKfKZHYEzm YMHhvhdsNZrCll17nErDRP702IHurr6kT7z+eDe6KNTN1zBo1O49ph52/GCxdWxzsDLn1RDO 0TljXVbo0JRYeNy6Ka1PV9g4vjx/hnsi82j6rP/Qx4vLm0MLS/X8FvQD47LFLS4tP1IT9gg8 jtp0v+TNkejL7ruOnH3vC+feurlC/+iJ1/Yr/SJu5WPHGnjFuQz7IS8qwuXXsVdUH0YOYLsd 7YYlvEG+YWOrxp6vvRpyxI2d0iB/yS0HHdy0uFOtnn/IGIcQqdlfoixf7ggfm3bM9tv3fqWP 07Z0efaz6HO/bA2eNLj8PHp+Zrit9EBEpSOXJU8U8D1xmVzwL1D/wD/QBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrIIsWRmVeSWpSXmKPExsWy7bCSvK6OR3aKwfK/shbrTx1jtpizfg2b RdOEv8wWq+/2s1m8PvyJ0WLah5/MFr/Pnme2eLDf3mLvu9msFjcP7GSy2LNoEpPFytVHmSw2 9nNY7F74kcni6P+3bBaTDl1jtHh6dRaTxd5b2hZ79p5ksbi8aw6bxb01/1kt5i97ym7RfX0H m8Xy4/+YLA5Nbmay2PGkkdFi2+/5zBbrXr9nsThxS9ricXcHo8X5v8dZLX7/mMPmoOBx+Yq3 x6z7Z9k8ds66y+5x/t5GFo9pk06xeWxeoeVx+Wypx6ZVnWwemz5NYvc4MeM3i8fmJfUeu282 sHn0Nr8D6mm9z+rx8ektFo/3+66yBYhEcdmkpOZklqUW6dslcGVMmbKMpWCufcX0ZWeZGhif mnQxcnJICJhIvHs5i62LkYtDSGAHo8S8n2+ZIRKSEsv+HoGyhSVW/nvODlHUzCQxvW0KUIKD g01AW+L0fw6QuIjAB2aJ9VsWMYI4zALbWSSOt55lAykSFrCX2N8uBzKIRUBV4vK0ZywgYV4B K4n3B/xBTAkBfYn++4IgFZxA0RcXZ4FNFxKwlJiwzAgkzCsgKHFy5hMWEJtZQF6ieets5gmM ArOQpGYhSS1gZFrFKJlaUJybnltsWGCUl1quV5yYW1yal66XnJ+7iRGcKrS0djDuWfVB7xAj EwfjIUYJDmYlEd7AvuQUId6UxMqq1KL8+KLSnNTiQ4zSHCxK4rwXuk7GCwmkJ5akZqemFqQW wWSZODilGpjKND/rrfvD650vucdsxa/oio7XCW7LWuezW4qXPny/fedduX9FXEpzrSx13+mf cOJsZxTmfHT0YGDixAVtS/gl9kTrPNr8VvN2QgYrp6CKXY3G0lqXVJkTPu6bo9z32dztFv94 S6R99aQIS8Y1F1fkzblzO8hO+TzXEeey7sNHetaZWHge/1V+3LrFJsgk+tT+ad56JlkzPJ/Z 16jUGIjnbXtdJub2zHRHr6vR1EIB06fLZFgsWfdca88y6t0S6c2U+cvWr7//r2ySDstOjeP2 Zlkbr6vtPnNvxd3049cvBj18xbLcYXfcdfXJfEsc60qy71RXZr04b72D3/L2UeWXSa/P6v7X LfuzdG7ueiWW4oxEQy3mouJEACpqFsCEAwAA X-CMS-MailID: 20230522104708epcas5p20f182069898f9c6852826600ce6116c1 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230522104708epcas5p20f182069898f9c6852826600ce6116c1 References: <20230522104146.2856-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Implementaion is based on existing read and write infrastructure. copy_max_bytes: A new configfs and module parameter is introduced, which can be used to set hardware/driver supported maximum copy limit. Suggested-by: Damien Le Moal Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu --- drivers/block/null_blk/main.c | 108 ++++++++++++++++++++++++++++-- drivers/block/null_blk/null_blk.h | 8 +++ 2 files changed, 111 insertions(+), 5 deletions(-) diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index b3fedafe301e..34e009b3ebd5 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -157,6 +157,10 @@ static int g_max_sectors; module_param_named(max_sectors, g_max_sectors, int, 0444); MODULE_PARM_DESC(max_sectors, "Maximum size of a command (in 512B sectors)"); +static unsigned long g_copy_max_bytes = COPY_MAX_BYTES; +module_param_named(copy_max_bytes, g_copy_max_bytes, ulong, 0444); +MODULE_PARM_DESC(copy_max_bytes, "Maximum size of a copy command (in bytes)"); + static unsigned int nr_devices = 1; module_param(nr_devices, uint, 0444); MODULE_PARM_DESC(nr_devices, "Number of devices to register"); @@ -409,6 +413,7 @@ NULLB_DEVICE_ATTR(home_node, uint, NULL); NULLB_DEVICE_ATTR(queue_mode, uint, NULL); NULLB_DEVICE_ATTR(blocksize, uint, NULL); NULLB_DEVICE_ATTR(max_sectors, uint, NULL); +NULLB_DEVICE_ATTR(copy_max_bytes, uint, NULL); NULLB_DEVICE_ATTR(irqmode, uint, NULL); NULLB_DEVICE_ATTR(hw_queue_depth, uint, NULL); NULLB_DEVICE_ATTR(index, uint, NULL); @@ -550,6 +555,7 @@ static struct configfs_attribute *nullb_device_attrs[] = { &nullb_device_attr_queue_mode, &nullb_device_attr_blocksize, &nullb_device_attr_max_sectors, + &nullb_device_attr_copy_max_bytes, &nullb_device_attr_irqmode, &nullb_device_attr_hw_queue_depth, &nullb_device_attr_index, @@ -656,7 +662,8 @@ static ssize_t memb_group_features_show(struct config_item *item, char *page) "poll_queues,power,queue_mode,shared_tag_bitmap,size," "submit_queues,use_per_node_hctx,virt_boundary,zoned," "zone_capacity,zone_max_active,zone_max_open," - "zone_nr_conv,zone_offline,zone_readonly,zone_size\n"); + "zone_nr_conv,zone_offline,zone_readonly,zone_size," + "copy_max_bytes\n"); } CONFIGFS_ATTR_RO(memb_group_, features); @@ -722,6 +729,7 @@ static struct nullb_device *null_alloc_dev(void) dev->queue_mode = g_queue_mode; dev->blocksize = g_bs; dev->max_sectors = g_max_sectors; + dev->copy_max_bytes = g_copy_max_bytes; dev->irqmode = g_irqmode; dev->hw_queue_depth = g_hw_queue_depth; dev->blocking = g_blocking; @@ -1271,6 +1279,78 @@ static int null_transfer(struct nullb *nullb, struct page *page, return err; } +static inline void nullb_setup_copy_read(struct nullb *nullb, struct bio *bio) +{ + struct nullb_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + + token->subsys = "nullb"; + token->sector_in = bio->bi_iter.bi_sector; + token->nullb = nullb; + token->sectors = bio->bi_iter.bi_size >> SECTOR_SHIFT; +} + +static inline int nullb_setup_copy_write(struct nullb *nullb, + struct bio *bio, bool is_fua) +{ + struct nullb_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + sector_t sector_in, sector_out; + void *in, *out; + size_t rem, temp; + unsigned long offset_in, offset_out; + struct nullb_page *t_page_in, *t_page_out; + int ret = -EIO; + + if (unlikely(memcmp(token->subsys, "nullb", 5))) + return -EINVAL; + if (unlikely(token->nullb != nullb)) + return -EINVAL; + if (WARN_ON(token->sectors != bio->bi_iter.bi_size >> SECTOR_SHIFT)) + return -EINVAL; + + sector_in = token->sector_in; + sector_out = bio->bi_iter.bi_sector; + rem = token->sectors << SECTOR_SHIFT; + + spin_lock_irq(&nullb->lock); + while (rem > 0) { + temp = min_t(size_t, nullb->dev->blocksize, rem); + offset_in = (sector_in & SECTOR_MASK) << SECTOR_SHIFT; + offset_out = (sector_out & SECTOR_MASK) << SECTOR_SHIFT; + + if (null_cache_active(nullb) && !is_fua) + null_make_cache_space(nullb, PAGE_SIZE); + + t_page_in = null_lookup_page(nullb, sector_in, false, + !null_cache_active(nullb)); + if (!t_page_in) + goto err; + t_page_out = null_insert_page(nullb, sector_out, + !null_cache_active(nullb) || is_fua); + if (!t_page_out) + goto err; + + in = kmap_local_page(t_page_in->page); + out = kmap_local_page(t_page_out->page); + + memcpy(out + offset_out, in + offset_in, temp); + kunmap_local(out); + kunmap_local(in); + __set_bit(sector_out & SECTOR_MASK, t_page_out->bitmap); + + if (is_fua) + null_free_sector(nullb, sector_out, true); + + rem -= temp; + sector_in += temp >> SECTOR_SHIFT; + sector_out += temp >> SECTOR_SHIFT; + } + + ret = 0; +err: + spin_unlock_irq(&nullb->lock); + return ret; +} + static int null_handle_rq(struct nullb_cmd *cmd) { struct request *rq = cmd->rq; @@ -1280,13 +1360,20 @@ static int null_handle_rq(struct nullb_cmd *cmd) sector_t sector = blk_rq_pos(rq); struct req_iterator iter; struct bio_vec bvec; + bool fua = rq->cmd_flags & REQ_FUA; + + if (rq->cmd_flags & REQ_COPY) { + if (op_is_write(req_op(rq))) + return nullb_setup_copy_write(nullb, rq->bio, fua); + nullb_setup_copy_read(nullb, rq->bio); + return 0; + } spin_lock_irq(&nullb->lock); rq_for_each_segment(bvec, rq, iter) { len = bvec.bv_len; err = null_transfer(nullb, bvec.bv_page, len, bvec.bv_offset, - op_is_write(req_op(rq)), sector, - rq->cmd_flags & REQ_FUA); + op_is_write(req_op(rq)), sector, fua); if (err) { spin_unlock_irq(&nullb->lock); return err; @@ -1307,13 +1394,20 @@ static int null_handle_bio(struct nullb_cmd *cmd) sector_t sector = bio->bi_iter.bi_sector; struct bio_vec bvec; struct bvec_iter iter; + bool fua = bio->bi_opf & REQ_FUA; + + if (bio->bi_opf & REQ_COPY) { + if (op_is_write(bio_op(bio))) + return nullb_setup_copy_write(nullb, bio, fua); + nullb_setup_copy_read(nullb, bio); + return 0; + } spin_lock_irq(&nullb->lock); bio_for_each_segment(bvec, bio, iter) { len = bvec.bv_len; err = null_transfer(nullb, bvec.bv_page, len, bvec.bv_offset, - op_is_write(bio_op(bio)), sector, - bio->bi_opf & REQ_FUA); + op_is_write(bio_op(bio)), sector, fua); if (err) { spin_unlock_irq(&nullb->lock); return err; @@ -2161,6 +2255,10 @@ static int null_add_dev(struct nullb_device *dev) dev->max_sectors = queue_max_hw_sectors(nullb->q); dev->max_sectors = min(dev->max_sectors, BLK_DEF_MAX_SECTORS); blk_queue_max_hw_sectors(nullb->q, dev->max_sectors); + blk_queue_max_copy_sectors_hw(nullb->q, + dev->copy_max_bytes >> SECTOR_SHIFT); + if (dev->copy_max_bytes) + blk_queue_flag_set(QUEUE_FLAG_COPY, nullb->disk->queue); if (dev->virt_boundary) blk_queue_virt_boundary(nullb->q, PAGE_SIZE - 1); diff --git a/drivers/block/null_blk/null_blk.h b/drivers/block/null_blk/null_blk.h index 929f659dd255..3dda593b0747 100644 --- a/drivers/block/null_blk/null_blk.h +++ b/drivers/block/null_blk/null_blk.h @@ -67,6 +67,13 @@ enum { NULL_Q_MQ = 2, }; +struct nullb_copy_token { + char *subsys; + struct nullb *nullb; + sector_t sector_in; + sector_t sectors; +}; + struct nullb_device { struct nullb *nullb; struct config_group group; @@ -107,6 +114,7 @@ struct nullb_device { unsigned int queue_mode; /* block interface */ unsigned int blocksize; /* block size */ unsigned int max_sectors; /* Max sectors per command */ + unsigned long copy_max_bytes; /* Max copy offload length in bytes */ unsigned int irqmode; /* IRQ completion handler */ unsigned int hw_queue_depth; /* queue depth */ unsigned int index; /* index of the disk, only valid with a disk */