From patchwork Wed Apr 19 11:43:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13216702 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A956C7EE20 for ; Wed, 19 Apr 2023 11:54:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233067AbjDSLyi (ORCPT ); Wed, 19 Apr 2023 07:54:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45704 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232900AbjDSLy3 (ORCPT ); Wed, 19 Apr 2023 07:54:29 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 20151B759 for ; Wed, 19 Apr 2023 04:54:26 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230419115424epoutp01b3e81158b30e0beb9e36ec7aafb174d8~XU_1OPwhu1217412174epoutp01e for ; Wed, 19 Apr 2023 11:54:24 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230419115424epoutp01b3e81158b30e0beb9e36ec7aafb174d8~XU_1OPwhu1217412174epoutp01e DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681905264; bh=LRiLMxf1Jtl4CWqtNs+ryHUB2RCNKjjIL4t4lTMZCiI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KZoJL6DFB/ISYvL6ObnvIicPPT6e13RUwk/2X7yvwn70hCPdEn9h3TPjrY9n3jp9t pI7MyOsHqAYM4VRfGls6bxzdzinBdR687ybY8yYCopmaS0gXHwM1idnFyCt7SNX+Pa 7LWdEeOuGyFONmm6vsMhn3eL6RlSOyKYrZYSWOSA= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230419115423epcas5p3d11042d1fddaa2d996a66cf0e54c249c~XU_0omp3r1237612376epcas5p3k; Wed, 19 Apr 2023 11:54:23 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.176]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4Q1fMx6y6nz4x9Pp; Wed, 19 Apr 2023 11:54:21 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 08.04.09987.D66DF346; Wed, 19 Apr 2023 20:54:21 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20230419114656epcas5p404001300c41f5bbd02362edd45d3ff45~XU4Tyt2ZZ2617926179epcas5p4l; Wed, 19 Apr 2023 11:46:56 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230419114656epsmtrp1771dce5179207bae5b16d284e5fb97f7~XU4Tw9-Ct1675116751epsmtrp1k; Wed, 19 Apr 2023 11:46:56 +0000 (GMT) X-AuditID: b6c32a4b-a67fd70000002703-02-643fd66d3513 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 84.83.08609.0B4DF346; Wed, 19 Apr 2023 20:46:56 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230419114652epsmtip18947edad37a475338f9d05db3d5bb2ea~XU4QiJJVw2332323323epsmtip1Y; Wed, 19 Apr 2023 11:46:52 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v10 1/9] block: Introduce queue limits for copy-offload support Date: Wed, 19 Apr 2023 17:13:06 +0530 Message-Id: <20230419114320.13674-2-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230419114320.13674-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTVxzHPfeW28JSd3mYHTFutYsoEKCFUg7Iy4nkbswM45gLumFDb4BQ 2qYtVHAZINMoKmWiPAoDHRsqgsTyCK/iVgZSHiHCxMGCuAUYIBSEiQPHHKVl87/P75vv7/we Jz8W7pDNdGYlSFW0QiqScAk7RkO7q5tH0lCImFc1LUQ13Z04Op27hqPboxoCPW1fBCh/YQVH T+6FIL2p2AYN/9CEodZvL2Po1u0ODLVcf4ahjldzBLpsGAJo4qEWQ/oRd9SqNzLQYHMJgcoq JpjIkJeFocbxTIAaXpbh6M7TeQbqGtmB+tfu24RCavDnCEo71kdQTdpRJtX/+C6DGuxLpnSV 5wmq9rt0qmU4g6AuZZkIar7tIUHl1FUCqrbnFLWke5vSjc9hkVujEwPjaZGYVnBoaaxMnCCN C+JGHIk5EOMr5PE9+P7Ij8uRipLoIG7Yh5Ee4QmS9Q1wOSkiSfK6FClSKrlewYEKWbKK5sTL lKogLi0XS+QCuadSlKRMlsZ5SmlVAJ/H8/ZdN55IjJ/46VeG/J+PTtaXVuAZoOJANrBlQVIA n+fX49nAjuVAtgCoHx0AlmARwNVvjNZgCcCxmSmwmbLU1840swPZDOCFB4cspjMYXLqywsgG LBZBusOeVyyz7kSexeGzifMMc4CTAxj8u24eN5scycOwoc3W/BCD3A2NNQWEWWaTATBjOcqM kPSCmjF7s8OW3AdvtOXhZmaT9tBYNM4wM06+A7PqizcmgGS5LbyZW8uw9BkGB3pnCQs7wpn7 dUwLO8NpzVkrq+GtKzcJS/JXAGofaa1DhsAz3ZqNNnHSFdY0e1nknfBq9x3MUngrvPRyHLPo bNhYusnvwqqaa9a62+HQi0wrU3B68px1oTkAlk/Wg1zA0b42kPa1gbT/l74G8EqwnZYrk+Jo pa/cR0qr//vkWFmSDmwchVtEI/j9yYKnAWAsYACQhXOd2L0HA8QObLEoNY1WyGIUyRJaaQC+ 6/v+GnfeFitbvyqpKoYv8OcJhEKhwN9HyOe+xd4TZIx1IONEKjqRpuW0YjMPY9k6Z2Dw6AfK kh9HBabqmi7183S7LaH95YHfF3+ZMulC37u4u2P5xbaIzjdXo0rpczfozvo3/prtGz0SXlVm Uh89wEyJKNDuUv8yQ7435bJXvwZ+a36/5zCxay5NsSMq3hCWp15w9z75mfiYX93+Tx+stf4J jfoLTQEDRe681WN1A49Cy3p8gneash+f9ujlVEe7cw6mmSQ2nNmoVJAuLtyrcLIp92jV9KiH 0cyprkJdoUZRsPJJjumLHPXixauex8NEATBwdSIzt9bbMdq0Zc/8xw37F6qv77OJ8Z4qHqGV xbp+/9T8xM95kj+KvGFwSfjxQp591PLdaD9XlxPhkkOTmcVz6VyGMl7Ed8MVStG/Hza31Z0E AAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02Ra0hTYRjHfc85Ox7NyWlJvWUqzRY1m3aj3tQy6cIhCUsoKCIdenCWznGm 3aXVsGh2sUyjKdnVy4pmM0Pz2tR0mRg5i1xzRq6bNS1NE1NrWtC33/P8/n+eDw+FCx4Tc6gE eQrLyaWJQtKdeFgv9JWUmMPilliHeUj/9AmOTmSO4eiO9TyJeuu/A5TTP4Kj7towVO3I5aHX dRUYqrpxEUPFdxoxVHn9G4YaJ76S6KLxJUD2Di2GqjsDUFW1iUDtj/JIlF9gd0XGLDWGynuO A/RwNB9H93r7CNTc6Y3axpp46yDTbo5gtLZWkqnQWl2Ztq77BNPemsoYdKdJpvTWMabytYpk zqodJNNX00Ey5x7oAFPacoQZMPgyhp6v2FbPXe6hcWxiwn6WC1ob4y6zN1gIxXjkwbKrBbgK FKzXADcK0ivgQGu9qwa4UwK6HMCRTzZySsyGBWMN+BTPgMXjH/6G1BgczqjANICiSDoAtkxQ zr0XnYnDFzYV6RxwuhuDbR8Gec72DDoSvnpTN8kELYIm/WXSWebTwVA1tN2JkA6C523TnQk3 OgQW1mRN3hX8SZwxjgMn8+np0HSlh3AyTvtBdVkunglo7X9K+5+6BjAdmM0qlEnxScqlimVy 9kCgUpqkTJXHB8YmJxnA5JfF4nJQpesPNAKMAkYAKVzoxX+2MThOwI+THjrMcsnRXGoiqzQC b4oQzuI/15iiBXS8NIXdx7IKlvtnMcptjgorlK2MGlpw05DReGrfyFDtoCn7TLYPx48Q+Rbp FV2l8/sWea7p59I0tVhT5ar6cEP0xiJC/z71pLalWeKxYq5IXWpefYURF4fHmshtsm0u8nCX wCBevuhA1x6/vjYrlxjT+EuUvGPTPL8NrWHbbSHv4lfvWVJnjtmc4xvuKbj74MimZDJz56jZ Mo260PDj7JvcUQ9Xo49lt9U/5ZKj5Pno59Cbu0oaFt7o/NLxSHnwWhr3rgjlJfhdL7y6ZRYf s3j07n6L63GtS9RPWZZ/pdRut0j2fmTrlqdHHDKI02PW6BzdRGTUzMhQk6TM3y645cgb7824 3Xy0mxhYLDk9USsklDLpUjHOKaW/AY/X4BpUAwAA X-CMS-MailID: 20230419114656epcas5p404001300c41f5bbd02362edd45d3ff45 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230419114656epcas5p404001300c41f5bbd02362edd45d3ff45 References: <20230419114320.13674-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Add device limits as sysfs entries, - copy_offload (RW) - copy_max_bytes (RW) - copy_max_bytes_hw (RO) Above limits help to split the copy payload in block layer. copy_offload: used for setting copy offload(1) or emulation(0). copy_max_bytes: maximum total length of copy in single payload. copy_max_bytes_hw: Reflects the device supported maximum limit. Reviewed-by: Hannes Reinecke Signed-off-by: Nitesh Shetty Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- Documentation/ABI/stable/sysfs-block | 33 ++++++++++++++ block/blk-settings.c | 24 +++++++++++ block/blk-sysfs.c | 64 ++++++++++++++++++++++++++++ include/linux/blkdev.h | 12 ++++++ include/uapi/linux/fs.h | 3 ++ 5 files changed, 136 insertions(+) diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block index c57e5b7cb532..e4d31132f77c 100644 --- a/Documentation/ABI/stable/sysfs-block +++ b/Documentation/ABI/stable/sysfs-block @@ -155,6 +155,39 @@ Description: last zone of the device which may be smaller. +What: /sys/block//queue/copy_offload +Date: April 2023 +Contact: linux-block@vger.kernel.org +Description: + [RW] When read, this file shows whether offloading copy to a + device is enabled (1) or disabled (0). Writing '0' to this + file will disable offloading copies for this device. + Writing any '1' value will enable this feature. If the device + does not support offloading, then writing 1, will result in + error. + + +What: /sys/block//queue/copy_max_bytes +Date: April 2023 +Contact: linux-block@vger.kernel.org +Description: + [RW] This is the maximum number of bytes, that the block layer + will allow for copy request. This will be smaller or equal to + the maximum size allowed by the hardware, indicated by + 'copy_max_bytes_hw'. Attempt to set value higher than + 'copy_max_bytes_hw' will truncate this to 'copy_max_bytes_hw'. + + +What: /sys/block//queue/copy_max_bytes_hw +Date: April 2023 +Contact: linux-block@vger.kernel.org +Description: + [RO] This is the maximum number of bytes, that the hardware + will allow in a single data copy request. + A value of 0 means that the device does not support + copy offload. + + What: /sys/block//queue/crypto/ Date: February 2022 Contact: linux-block@vger.kernel.org diff --git a/block/blk-settings.c b/block/blk-settings.c index 896b4654ab00..23aff2d4dcba 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -59,6 +59,8 @@ void blk_set_default_limits(struct queue_limits *lim) lim->zoned = BLK_ZONED_NONE; lim->zone_write_granularity = 0; lim->dma_alignment = 511; + lim->max_copy_sectors_hw = 0; + lim->max_copy_sectors = 0; } /** @@ -82,6 +84,8 @@ void blk_set_stacking_limits(struct queue_limits *lim) lim->max_dev_sectors = UINT_MAX; lim->max_write_zeroes_sectors = UINT_MAX; lim->max_zone_append_sectors = UINT_MAX; + lim->max_copy_sectors_hw = ULONG_MAX; + lim->max_copy_sectors = ULONG_MAX; } EXPORT_SYMBOL(blk_set_stacking_limits); @@ -183,6 +187,22 @@ void blk_queue_max_discard_sectors(struct request_queue *q, } EXPORT_SYMBOL(blk_queue_max_discard_sectors); +/** + * blk_queue_max_copy_sectors_hw - set max sectors for a single copy payload + * @q: the request queue for the device + * @max_copy_sectors: maximum number of sectors to copy + **/ +void blk_queue_max_copy_sectors_hw(struct request_queue *q, + unsigned int max_copy_sectors) +{ + if (max_copy_sectors > (COPY_MAX_BYTES >> SECTOR_SHIFT)) + max_copy_sectors = COPY_MAX_BYTES >> SECTOR_SHIFT; + + q->limits.max_copy_sectors_hw = max_copy_sectors; + q->limits.max_copy_sectors = max_copy_sectors; +} +EXPORT_SYMBOL_GPL(blk_queue_max_copy_sectors_hw); + /** * blk_queue_max_secure_erase_sectors - set max sectors for a secure erase * @q: the request queue for the device @@ -578,6 +598,10 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, t->max_segment_size = min_not_zero(t->max_segment_size, b->max_segment_size); + t->max_copy_sectors = min(t->max_copy_sectors, b->max_copy_sectors); + t->max_copy_sectors_hw = min(t->max_copy_sectors_hw, + b->max_copy_sectors_hw); + t->misaligned |= b->misaligned; alignment = queue_limit_alignment_offset(b, start); diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index a64208583853..826ab29beba3 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -212,6 +212,63 @@ static ssize_t queue_discard_zeroes_data_show(struct request_queue *q, char *pag return queue_var_show(0, page); } +static ssize_t queue_copy_offload_show(struct request_queue *q, char *page) +{ + return queue_var_show(blk_queue_copy(q), page); +} + +static ssize_t queue_copy_offload_store(struct request_queue *q, + const char *page, size_t count) +{ + s64 copy_offload; + ssize_t ret = queue_var_store64(©_offload, page); + + if (ret < 0) + return ret; + + if (copy_offload && !q->limits.max_copy_sectors_hw) + return -EINVAL; + + if (copy_offload) + blk_queue_flag_set(QUEUE_FLAG_COPY, q); + else + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + + return count; +} + +static ssize_t queue_copy_max_hw_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors_hw << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_store(struct request_queue *q, + const char *page, size_t count) +{ + s64 max_copy; + ssize_t ret = queue_var_store64(&max_copy, page); + + if (ret < 0) + return ret; + + if (max_copy & (queue_logical_block_size(q) - 1)) + return -EINVAL; + + max_copy >>= SECTOR_SHIFT; + if (max_copy > q->limits.max_copy_sectors_hw) + max_copy = q->limits.max_copy_sectors_hw; + + q->limits.max_copy_sectors = max_copy; + return count; +} + static ssize_t queue_write_same_max_show(struct request_queue *q, char *page) { return queue_var_show(0, page); @@ -590,6 +647,10 @@ QUEUE_RO_ENTRY(queue_nr_zones, "nr_zones"); QUEUE_RO_ENTRY(queue_max_open_zones, "max_open_zones"); QUEUE_RO_ENTRY(queue_max_active_zones, "max_active_zones"); +QUEUE_RW_ENTRY(queue_copy_offload, "copy_offload"); +QUEUE_RO_ENTRY(queue_copy_max_hw, "copy_max_bytes_hw"); +QUEUE_RW_ENTRY(queue_copy_max, "copy_max_bytes"); + QUEUE_RW_ENTRY(queue_nomerges, "nomerges"); QUEUE_RW_ENTRY(queue_rq_affinity, "rq_affinity"); QUEUE_RW_ENTRY(queue_poll, "io_poll"); @@ -637,6 +698,9 @@ static struct attribute *queue_attrs[] = { &queue_discard_max_entry.attr, &queue_discard_max_hw_entry.attr, &queue_discard_zeroes_data_entry.attr, + &queue_copy_offload_entry.attr, + &queue_copy_max_hw_entry.attr, + &queue_copy_max_entry.attr, &queue_write_same_max_entry.attr, &queue_write_zeroes_max_entry.attr, &queue_zone_append_max_entry.attr, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index e3242e67a8e3..200338f2ec2e 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -298,6 +298,9 @@ struct queue_limits { unsigned int discard_alignment; unsigned int zone_write_granularity; + unsigned long max_copy_sectors_hw; + unsigned long max_copy_sectors; + unsigned short max_segments; unsigned short max_integrity_segments; unsigned short max_discard_segments; @@ -564,6 +567,7 @@ struct request_queue { #define QUEUE_FLAG_NOWAIT 29 /* device supports NOWAIT */ #define QUEUE_FLAG_SQ_SCHED 30 /* single queue style io dispatch */ #define QUEUE_FLAG_SKIP_TAGSET_QUIESCE 31 /* quiesce_tagset skip the queue*/ +#define QUEUE_FLAG_COPY 32 /* supports copy offload */ #define QUEUE_FLAG_MQ_DEFAULT ((1UL << QUEUE_FLAG_IO_STAT) | \ (1UL << QUEUE_FLAG_SAME_COMP) | \ @@ -584,6 +588,7 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q); test_bit(QUEUE_FLAG_STABLE_WRITES, &(q)->queue_flags) #define blk_queue_io_stat(q) test_bit(QUEUE_FLAG_IO_STAT, &(q)->queue_flags) #define blk_queue_add_random(q) test_bit(QUEUE_FLAG_ADD_RANDOM, &(q)->queue_flags) +#define blk_queue_copy(q) test_bit(QUEUE_FLAG_COPY, &(q)->queue_flags) #define blk_queue_zone_resetall(q) \ test_bit(QUEUE_FLAG_ZONE_RESETALL, &(q)->queue_flags) #define blk_queue_dax(q) test_bit(QUEUE_FLAG_DAX, &(q)->queue_flags) @@ -902,6 +907,8 @@ extern void blk_queue_chunk_sectors(struct request_queue *, unsigned int); extern void blk_queue_max_segments(struct request_queue *, unsigned short); extern void blk_queue_max_discard_segments(struct request_queue *, unsigned short); +extern void blk_queue_max_copy_sectors_hw(struct request_queue *q, + unsigned int max_copy_sectors); void blk_queue_max_secure_erase_sectors(struct request_queue *q, unsigned int max_sectors); extern void blk_queue_max_segment_size(struct request_queue *, unsigned int); @@ -1221,6 +1228,11 @@ static inline unsigned int bdev_discard_granularity(struct block_device *bdev) return bdev_get_queue(bdev)->limits.discard_granularity; } +static inline unsigned int bdev_max_copy_sectors(struct block_device *bdev) +{ + return bdev_get_queue(bdev)->limits.max_copy_sectors; +} + static inline unsigned int bdev_max_secure_erase_sectors(struct block_device *bdev) { diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index b7b56871029c..8879567791fa 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -64,6 +64,9 @@ struct fstrim_range { __u64 minlen; }; +/* maximum total copy length */ +#define COPY_MAX_BYTES (1 << 27) + /* extent-same (dedupe) ioctls; these MUST match the btrfs ioctl definitions */ #define FILE_DEDUPE_RANGE_SAME 0 #define FILE_DEDUPE_RANGE_DIFFERS 1 From patchwork Wed Apr 19 11:43:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13216703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D2B7C7EE22 for ; Wed, 19 Apr 2023 11:54:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233075AbjDSLyj (ORCPT ); Wed, 19 Apr 2023 07:54:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45824 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233037AbjDSLyg (ORCPT ); Wed, 19 Apr 2023 07:54:36 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8494C14F5B for ; Wed, 19 Apr 2023 04:54:30 -0700 (PDT) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230419115428epoutp03e0ff4033a9cdb24cf940aec96b3f4897~XU_4eKPIh1371913719epoutp038 for ; Wed, 19 Apr 2023 11:54:28 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230419115428epoutp03e0ff4033a9cdb24cf940aec96b3f4897~XU_4eKPIh1371913719epoutp038 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681905268; bh=QlVDKHXxeSIn7ttIPNdI5Kwjrx+MA7ilbM9ui+Z3ous=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CEwNm7q62VYI0YSA2KlykHgv9dY4J55g/GhAGROM0okwwXyAa5PCbvDvGylkuUhaX m/f8dcwTObPxOUE7IleWoDgkir+uQ2TkehqcIlF+S+67KuYwbEH366VgBK0Tv5cXnq Vy1ny/iKVMtgqepdzTPdr73X829cUAOj54TsNUNc= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230419115427epcas5p4cb4a56e7e55f15087e9498b3a825bef5~XU_3oi3bu0896108961epcas5p4M; Wed, 19 Apr 2023 11:54:27 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.182]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4Q1fN12tJMz4x9Pp; Wed, 19 Apr 2023 11:54:25 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 4C.04.09987.176DF346; Wed, 19 Apr 2023 20:54:25 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230419114705epcas5p376d05f7c5f892d82590c2137650dd291~XU4cZ_fLw0477504775epcas5p33; Wed, 19 Apr 2023 11:47:05 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230419114705epsmtrp29b292ae34f628324de1701f82e39a126~XU4cYkyw22700627006epsmtrp2k; Wed, 19 Apr 2023 11:47:05 +0000 (GMT) X-AuditID: b6c32a4b-7fbff70000002703-11-643fd6718857 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id EF.87.08279.9B4DF346; Wed, 19 Apr 2023 20:47:05 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230419114701epsmtip1854900c6066e7cb09f6e4078a2838c2d~XU4Y5mivx2050920509epsmtip1K; Wed, 19 Apr 2023 11:47:01 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v10 2/9] block: Add copy offload support infrastructure Date: Wed, 19 Apr 2023 17:13:07 +0530 Message-Id: <20230419114320.13674-3-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230419114320.13674-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Te0xTVxzeubfctmZlV2DjAJtrasYEVmhnYQcUOx/oTWAJw2wYJqlNuQFG aUsfMNFlIANB5TEqRgsTgeGQMojAHIIwKamMgjHjNZCBblAzdbwXGXHgKBc3//vO9/u+3+vk x8FdTrI9OYkqPa1VyZUCYhPrWpePrzBlWBonmq9zRg22Wzg6UbSCI/N4IYEedy0AdG5uGUf3 f5Si9plSJzR68zqGblQWY+iK2Yqhtop5DFmfTROo2DIMkH3IhKH2u37oRnsPCw20lhGo/LKd jSzGLAy1TGUCdO1pOY7qH8+y0E93vdCdlW6n9yE1MBhOme7dJqjrpnE2dWfiKosauG2gGmvz CKrpmy+ottEMgsrPmiGo2Y4hgiporgVUU+8xarFxC9U4NY1FOsck7Uyg5XG0lk+rFOq4RFV8 qCD8oGyvLDBIJBaKg9F7Ar5KnkyHCvZFRAr3JyrXNiDgp8qVhjUqUq7TCQJ27dSqDXqan6DW 6UMFtCZOqZFo/HXyZJ1BFe+vovUhYpHo3cA14ZGkhKrcVUxTfOizhr6UDLB84BTgciApgaeX RlgO7EK2ATiWHX4KbFrDCwCW2cws5vEEwNWpZvYpwFl3VK6oGEM7gDl/ezOabAzaW35wcmgI 0g/2PuM4eDcyB4fz9rz1RDjZj8F/mmdxh9uVPABzLvxBODCLfAs+Kq5eN/PIEJhZE83UCoCF 9zY7FFxyB/y2w7ju5JGbYc+FqfWmcfJNmPV9Ke5ID8kqLizMLWEzk+2DqxcHcQa7wkfdzRu8 J1ycaScYnAavnK0hGPOXAJp+MQEmIIXZtkLc0QRO+sCG1gCGfgOW2OoxprAzzH86hTE8D7Zc fI63wrqGSxv5PeDwUuYGpmDlZDdgFlcAYP+irAjwTS/MY3phHtP/lS8BvBZ40BpdcjytC9Rs V9Fp//2wQp3cCNYvwje8Bfx+f87fAjAOsADIwQVuvL6wkDgXXpz8aDqtVcu0BiWts4DAtXV/ hXu+qlCvnZRKLxNLgkWSoKAgSfD2ILHAnfd2aI/ChYyX6+kkmtbQ2uc+jMP1zMD4HVGW5fqA WKPd62Hi54t15sgHvhL/aez4YEeAV2MM+aTi2M8xZdsGS0Q3Rw59F+lenfWxMbo0Raz5JKpp 6bj0aFJA1/y5BeVoIS6ILhN+PWGzlcfuvryVmPTMjKjJOG3uId129G+LMpxNz08f4XB2H77K Ixf2jL+uqKvIa/2woJMj7RTlhp2srmmTjt068xfbNbVIY431CKuYT+389U/viYixg6/sCjQ2 263c0hw9ErvX9MXmtghrjcLQInvf/rIY9/O51tc+cPfhDrgm9E7+NuR9/qXYtpYj75jDpfmH 93j7b3FSnEgzZM8MiObSRh5+ekYym/lAVmUOWf7IavB7eU7A0iXIxb64Vif/FziYlyeaBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrPIsWRmVeSWpSXmKPExsWy7bCSnO7OK/YpBrdPiFusP3WM2aJpwl9m i9V3+9ksXh/+xGgx7cNPZosH++0t9r6bzWpx88BOJos9iyYxWaxcfZTJYvfCj0wWR/+/ZbOY dOgao8XTq7OYLPbe0rbYs/cki8XlXXPYLOYve8pucWhyM5PFjieNjBbbfs9ntlj3+j2LxYlb 0hbn/x5ndZDwuHzF22PW/bNsHjtn3WX3OH9vI4vH5bOlHptWdbJ5bF5S77H7ZgObR2/zOzaP 9/uusnn0bVnF6LH5dLXH501yHpuevGUK4IvisklJzcksSy3St0vgyljc8Y+pYFJExfozhQ2M P927GDk4JARMJBb9zeti5OIQEtjNKHHx+jLGLkZOoLikxLK/R5ghbGGJlf+es0MUNTNJdN/a xwTSzCagLXH6PwdIXERgArPEpfsNbCAOs8ADJonzz7+wgnQLC7hLtM18wQZiswioSryatJQV pJlXwEqicUU4xBH6Ev33BUEqOAWsJZbvmwy2VwiooufQP7B7eAUEJU7OfMICYjMLyEs0b53N PIFRYBaS1CwkqQWMTKsYJVMLinPTc4sNCwzzUsv1ihNzi0vz0vWS83M3MYLjWEtzB+P2VR/0 DjEycTAeYpTgYFYS4XW3sUsR4k1JrKxKLcqPLyrNSS0+xCjNwaIkznuh62S8kEB6Yklqdmpq QWoRTJaJg1OqgWmCSzjXLB7RohYOdd26w9kiy0qD5QsfRTyb0Frk3MN3P5lRsn7OTU2+N0sD 0j72fbl2+F16ftmNS5mvO8VOdk7JZH3Bl6woumymWnN5Pfe/pXv5r0ebcPv9j2SuqgrhcXtg wjHxYOTue9qCv9vKzc2qfk0I29eWzll2plfM0OtW2v/n+4Nrp/Bz/331ryLFjsnXxOhc7euZ 7/+rqp/x+57+XGHdsR3Pgnul46TyDKsmpT+9dO2ol4vv9nU7FBf8OcC7a/J8X/N/nmuy0+rX G1a8vjZr8UnxPzUiDyvF/BdIGuZFL7i26ZzvubUneWzUnpxde1M7zHzpPU299r1G+f92puf8 2POrPHn5iWmtR5RYijMSDbWYi4oTAXRMuvZSAwAA X-CMS-MailID: 20230419114705epcas5p376d05f7c5f892d82590c2137650dd291 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230419114705epcas5p376d05f7c5f892d82590c2137650dd291 References: <20230419114320.13674-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Introduce blkdev_issue_copy which takes similar arguments as copy_file_range and performs copy offload between two bdevs. Introduce REQ_COPY copy offload operation flag. Create a read-write bio pair with a token as payload and submitted to the device in order. Read request populates token with source specific information which is then passed with write request. This design is courtesy Mikulas Patocka's token based copy Larger copy will be divided, based on max_copy_sectors limit. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- block/blk-lib.c | 235 ++++++++++++++++++++++++++++++++++++++ block/blk.h | 2 + include/linux/blk_types.h | 25 ++++ include/linux/blkdev.h | 3 + 4 files changed, 265 insertions(+) diff --git a/block/blk-lib.c b/block/blk-lib.c index e59c3069e835..ed089e703cb1 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -115,6 +115,241 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, } EXPORT_SYMBOL(blkdev_issue_discard); +/* + * For synchronous copy offload/emulation, wait and process all in-flight BIOs. + * This must only be called once all bios have been issued so that the refcount + * can only decrease. This just waits for all bios to make it through + * blkdev_copy_write_endio. + */ +static int blkdev_copy_wait_completion(struct cio *cio) +{ + int ret; + + if (cio->endio) + return 0; + + if (atomic_read(&cio->refcount)) { + __set_current_state(TASK_UNINTERRUPTIBLE); + blk_io_schedule(); + } + + ret = cio->comp_len; + kfree(cio); + + return ret; +} + +static void blkdev_copy_offload_write_endio(struct bio *bio) +{ + struct copy_ctx *ctx = bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (bio->bi_status) { + clen = (bio->bi_iter.bi_sector << SECTOR_SHIFT) - cio->pos_out; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + } + __free_page(bio->bi_io_vec[0].bv_page); + bio_put(bio); + + kfree(ctx); + if (!atomic_dec_and_test(&cio->refcount)) + return; + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); +} + +static void blkdev_copy_offload_read_endio(struct bio *read_bio) +{ + struct copy_ctx *ctx = read_bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (read_bio->bi_status) { + clen = (read_bio->bi_iter.bi_sector << SECTOR_SHIFT) + - cio->pos_in; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + __free_page(read_bio->bi_io_vec[0].bv_page); + bio_put(ctx->write_bio); + bio_put(read_bio); + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } + return; + } + + schedule_work(&ctx->dispatch_work); + bio_put(read_bio); +} + +static void blkdev_copy_dispatch_work(struct work_struct *work) +{ + struct copy_ctx *ctx = container_of(work, struct copy_ctx, + dispatch_work); + + submit_bio(ctx->write_bio); +} + +/* + * __blkdev_copy_offload - Use device's native copy offload feature. + * we perform copy operation by sending 2 bio. + * 1. First we send a read bio with REQ_COPY flag along with a token and source + * and length. Once read bio reaches driver layer, device driver adds all the + * source info to token and does a fake completion. + * 2. Once read operation completes, we issue write with REQ_COPY flag with same + * token. In driver layer, token info is used to form a copy offload command. + * + * Returns the length of bytes copied or error if encountered + */ +static int __blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct cio *cio; + struct copy_ctx *ctx; + struct bio *read_bio, *write_bio; + struct page *token; + sector_t copy_len; + sector_t rem, max_copy_len; + + cio = kzalloc(sizeof(struct cio), GFP_KERNEL); + if (!cio) + return -ENOMEM; + atomic_set(&cio->refcount, 0); + cio->waiter = current; + cio->endio = endio; + cio->private = private; + + max_copy_len = min(bdev_max_copy_sectors(bdev_in), + bdev_max_copy_sectors(bdev_out)) << SECTOR_SHIFT; + + cio->pos_in = pos_in; + cio->pos_out = pos_out; + /* If there is a error, comp_len will be set to least successfully + * completed copied length + */ + cio->comp_len = len; + for (rem = len; rem > 0; rem -= copy_len) { + copy_len = min(rem, max_copy_len); + + token = alloc_page(gfp_mask); + if (unlikely(!token)) + goto err_token; + + ctx = kzalloc(sizeof(struct copy_ctx), gfp_mask); + if (!ctx) + goto err_ctx; + read_bio = bio_alloc(bdev_in, 1, REQ_OP_READ | REQ_COPY + | REQ_SYNC | REQ_NOMERGE, gfp_mask); + if (!read_bio) + goto err_read_bio; + write_bio = bio_alloc(bdev_out, 1, REQ_OP_WRITE + | REQ_COPY | REQ_SYNC | REQ_NOMERGE, gfp_mask); + if (!write_bio) + goto err_write_bio; + + ctx->cio = cio; + ctx->write_bio = write_bio; + INIT_WORK(&ctx->dispatch_work, blkdev_copy_dispatch_work); + + __bio_add_page(read_bio, token, PAGE_SIZE, 0); + read_bio->bi_iter.bi_size = copy_len; + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_end_io = blkdev_copy_offload_read_endio; + read_bio->bi_private = ctx; + + __bio_add_page(write_bio, token, PAGE_SIZE, 0); + write_bio->bi_iter.bi_size = copy_len; + write_bio->bi_end_io = blkdev_copy_offload_write_endio; + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_private = ctx; + + atomic_inc(&cio->refcount); + submit_bio(read_bio); + pos_in += copy_len; + pos_out += copy_len; + } + + /* Wait for completion of all IO's*/ + return blkdev_copy_wait_completion(cio); + +err_write_bio: + bio_put(read_bio); +err_read_bio: + kfree(ctx); +err_ctx: + __free_page(token); +err_token: + cio->comp_len = min_t(sector_t, cio->comp_len, (len - rem)); + if (!atomic_read(&cio->refcount)) + return -ENOMEM; + /* Wait for submitted IOs to complete */ + return blkdev_copy_wait_completion(cio); +} + +static inline int blkdev_copy_sanity_check(struct block_device *bdev_in, + loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, + size_t len) +{ + unsigned int align = max(bdev_logical_block_size(bdev_out), + bdev_logical_block_size(bdev_in)) - 1; + + if (bdev_read_only(bdev_out)) + return -EPERM; + + if ((pos_in & align) || (pos_out & align) || (len & align) || !len || + len >= COPY_MAX_BYTES) + return -EINVAL; + + return 0; +} + +/* + * @bdev_in: source block device + * @pos_in: source offset + * @bdev_out: destination block device + * @pos_out: destination offset + * @len: length in bytes to be copied + * @endio: endio function to be called on completion of copy operation, + * for synchronous operation this should be NULL + * @private: endio function will be called with this private data, should be + * NULL, if operation is synchronous in nature + * @gfp_mask: memory allocation flags (for bio_alloc) + * + * Returns the length of bytes copied or error if encountered + * + * Description: + * Copy source offset from source block device to destination block + * device. Max total length of copy is limited to MAX_COPY_TOTAL_LENGTH + */ +int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct request_queue *q_in = bdev_get_queue(bdev_in); + struct request_queue *q_out = bdev_get_queue(bdev_out); + int ret; + + ret = blkdev_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len); + if (ret) + return ret; + + if (blk_queue_copy(q_in) && blk_queue_copy(q_out)) + ret = __blkdev_copy_offload(bdev_in, pos_in, bdev_out, pos_out, + len, endio, private, gfp_mask); + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_issue_copy); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/block/blk.h b/block/blk.h index 62fca868bc61..d245817d2242 100644 --- a/block/blk.h +++ b/block/blk.h @@ -311,6 +311,8 @@ static inline bool bio_may_exceed_limits(struct bio *bio, break; } + if (unlikely(op_is_copy(bio->bi_opf))) + return false; /* * All drivers must accept single-segments bios that are <= PAGE_SIZE. * This is a quick and dirty check that relies on the fact that diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 8ef209e3aa96..f8aa539dc4c8 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -421,6 +421,7 @@ enum req_flag_bits { */ /* for REQ_OP_WRITE_ZEROES: */ __REQ_NOUNMAP, /* do not free blocks when zeroing */ + __REQ_COPY, /* copy request */ __REQ_NR_BITS, /* stops here */ }; @@ -445,6 +446,7 @@ enum req_flag_bits { #define REQ_POLLED (__force blk_opf_t)(1ULL << __REQ_POLLED) #define REQ_ALLOC_CACHE (__force blk_opf_t)(1ULL << __REQ_ALLOC_CACHE) #define REQ_SWAP (__force blk_opf_t)(1ULL << __REQ_SWAP) +#define REQ_COPY ((__force blk_opf_t)(1ULL << __REQ_COPY)) #define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV) #define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE) @@ -475,6 +477,11 @@ static inline bool op_is_write(blk_opf_t op) return !!(op & (__force blk_opf_t)1); } +static inline bool op_is_copy(blk_opf_t op) +{ + return op & REQ_COPY; +} + /* * Check if the bio or request is one that needs special treatment in the * flush state machine. @@ -534,4 +541,22 @@ struct blk_rq_stat { u64 batch; }; +typedef void (cio_iodone_t)(void *private, int comp_len); + +struct cio { + struct task_struct *waiter; /* waiting task (NULL if none) */ + atomic_t refcount; + loff_t pos_in; + loff_t pos_out; + size_t comp_len; + cio_iodone_t *endio; /* applicable for async operation */ + void *private; /* applicable for async operation */ +}; + +struct copy_ctx { + struct cio *cio; + struct work_struct dispatch_work; + struct bio *write_bio; +}; + #endif /* __LINUX_BLK_TYPES_H */ diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 200338f2ec2e..1bb43697d43d 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1054,6 +1054,9 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop); int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp); +int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Wed Apr 19 11:43:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13216704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A5DDC77B75 for ; Wed, 19 Apr 2023 11:54:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233056AbjDSLym (ORCPT ); Wed, 19 Apr 2023 07:54:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233049AbjDSLyh (ORCPT ); Wed, 19 Apr 2023 07:54:37 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F4F914458 for ; Wed, 19 Apr 2023 04:54:33 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230419115431epoutp01b3d2ae2faa37d13b3d3b94182b1a9819~XU_71JPPT1491614916epoutp01Y for ; Wed, 19 Apr 2023 11:54:31 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230419115431epoutp01b3d2ae2faa37d13b3d3b94182b1a9819~XU_71JPPT1491614916epoutp01Y DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681905271; bh=zU+GbPVLniyzLXqnw+/kZiZDz+gADtrdaFwLreejzFg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RgtVJ3SXi7W5wDHHdqYyoVbZR1GnohaIXEc1FtQmqpJSJ0KP228mGK6hSMt9Pk63X WktqlhaxOoweYaIiWeFCOwjb8jIW6kMNQJ3bcgYUpX23f2hbL+imVYjeGzX1nxvt29 MRB4C2WlNjJG7L1HGOuTRG+HExTtoYxRYZVpQMT0= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230419115430epcas5p4c4fc875af9bfa3e1d1b7a0053d658018~XU_68lK_40259502595epcas5p4H; Wed, 19 Apr 2023 11:54:30 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.176]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4Q1fN46btRz4x9Pw; Wed, 19 Apr 2023 11:54:28 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 16.6D.09540.476DF346; Wed, 19 Apr 2023 20:54:28 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230419114714epcas5p33084dcdc06787292b46c376aa51e5ec5~XU4k0bKDE0243802438epcas5p3C; Wed, 19 Apr 2023 11:47:14 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230419114714epsmtrp287261cf1c9a9408d6e300cd342b3c5e1~XU4kyFJpe2737927379epsmtrp2E; Wed, 19 Apr 2023 11:47:14 +0000 (GMT) X-AuditID: b6c32a4a-4afff70000002544-4a-643fd674d87e Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id E2.97.08279.2C4DF346; Wed, 19 Apr 2023 20:47:14 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230419114711epsmtip11aa690ad8b0169d54a3eb3301fe130c7~XU4hjF8il2496324963epsmtip1C; Wed, 19 Apr 2023 11:47:10 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Vincent Fu , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v10 3/9] block: add emulation for copy Date: Wed, 19 Apr 2023 17:13:08 +0530 Message-Id: <20230419114320.13674-4-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230419114320.13674-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02TeVCUZRzHed53eVlo0BeEeKDBttdBBwzY5dgeUI4ptHekgxFqmnKEBd4W Yq92l0smATnkGE6zYsEQBiVuBWU4XDU2RCAiBsHEFIhFEoZDcGSSZNttofzv8/v+7t8zDxu3 LjZzYMdKlIxcIhBRhAWrXeO8z1U5HhDNLXjqjloGbuHoVPELHDU8KCLQgmYVoG9W/sLR1I0A pF4qN0X3bnZi6Fp1KYbqGnox1F31BEO9ukUClfaMAzQ7psKQemI/uqbuZ6HRrgoCVV6cNUM9 ZzIw1KFNB6h9oxJHzQvLLHR74jU0k58D0PCLPtNAe3r0TjCtmhwi6E7VAzN6+OFlFj06FE+3 1ucSdFtNKt19L42gCzKWCHr5+hhBF16pB3TbYAq91rqbbtUuYiE7Po07GMMIohk5h5FESaNj JUI/Kjg0/J1wbz6X58rzQW9RHIlAzPhRQe+FuB6OFenPQHESBKJ4vRQiUCgod/+Dcmm8kuHE SBVKP4qRRYtkXjI3hUCsiJcI3SSM0pfH5Xp46wMj4mIacwYxWXtAkjZPB9JAtlceMGdD0gvq KuYxA1uT3QCevSnPAxZ6XgVwcW3D1Gg8A3C9psRsO+OP078SRocawB+rn29FZWFwc0ajN9hs gtwPB3Vsg25DZuPwyWwuy2Dg5DoGJ+faTA2ldpF8qF34ljAwi3SCS+P3gYEtSV/YrZ0GhkKQ dIdFk1YG2Zw8AGuvn8GNIVawv0zLMjBOvg4zrpbjhvqQbDCHi79XbY0aBKvmNSwj74LzfVe2 dAe4tqQmjJwI677+gTAmZwKouqsCRkcAzBoowg1D4KQzbOlyN8qO8OxAM2ZsvAMWbGgxo24J O77f5j2wseX8Vn17OL6eThh3oWFmY5jxWIUATmobQDHgqF7aR/XSPqr/O58HeD2wZ2QKsZBR eMs8JEzif68cJRW3gn+/hsuRDjA9teLWAzA26AGQjVM2lj8f8o22towWJJ9g5NJwebyIUfQA b/29S3AH2yip/m9JlOE8Lx+uF5/P9/Lx5PMoO8t9fv1R1qRQoGTiGEbGyLfzMLa5QxrWZvXG 7ktjuJ/Y5Wjqiil2J2vip6BXjpwYyC4/+WbkYENjs3P/scJQp70JwSnpr2Yu/y3xD2vZ+eyz p+dihY/ft/Mt/a69k+ffOet2/JfLd5t0yflNObKM1Ecmp2pVHWrb1KvYh0Sl7fFRk5EvWz54 1F09H9p1oK/kYmSKsiTpof3OUZcazxubTaspFkdF+YdNyo7dDqNCAjwD55q+ilvLvmShE59L nvrt0IXI2uS6PYseXxAjM1Mwc/1djdD9Y12gZ4Qon9dRPGfHd0q0iZiXpVP344ducbifdMQ5 YpuPOzVtw+jt4I+GVIKRPxMWpCVJMc8HcjNPXvh8ek5eVrG311Z+2tGeYiliBDwXXK4Q/AOU gbQaowQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02RfUzMcRzH9/39fve7XxG/rsw3rHLWKFMa2dfDHJ3xxTz+EZnhuN8KXeWu 8xDN5TZ3rueiuDIxO3rAFFEpXCXnonRKD0vhWra6HmiMnriLzX/vvV/vvT5/fBhSUEvNYg5H xnDySEmEkHamSqqEnouM70TSxaUjEN179YJE51LHSFTQkUKj3qqvAGUO/iRR11MRqujP5qHW Z6UEenIjnUB5BTUEKr8+RKCaCRuN0o3NAHU36QlU0bYQPakwUchSlkOja4ZuPjJmqAn02BoP UMnINRLd7R2g0Mu22ehzghag+rFa3hoPbHm3Ges7X9O4VN/Bx/Uf7lPY8lqJi/Iv0Lj45llc 3qqicZK6n8YDlU00Tn6QD3Cx+TT+VuSJi6w2Yvu0Pc6rpFzE4eOcPGD1AefwQq2ZiC4RnbTq JoAKnF+qA04MZJfCT5oGWgecGQFbDuDX5wZqEnhAw1g1OZndYN54D39ypCZge3USTwcYhmYX QvMEY+/d2VQSNnaqHCaS1ZCwzzDoMLmxy6C1N4u2Z4r1gf3N7cCeXdgVsNz6EdhFkA2AKZ2u 9tqJXQlvVWY4Dgv+TBKN43/nrtB0xepQkqwXVD/MJlMBq/8P6f9DuYDIBx5ctEIWJlMERgdG cif8FRKZQhkZ5n8oSlYEHL/2830MHuUP+hsBwQAjgAwpdHfZsGq1VOAilZyK5eRR++XKCE5h BLMZSjjTpUFn2i9gwyQx3FGOi+bk/yjBOM1SEWFDbs97g8UXNv7wSe32Dr2za6frCOQ3tgSL FNUrC/xz2ytmBvXEVe3t8Z6XbfJObBRPXZDXtTl2qNjDVIN+dnpNXR/jtnv7l4vv90J6+TAV FBF36HtygPb2/MwdRrM0wZYpChkJLUsJH94kZrpim+fGqT31eN+o7/T4y5pKS9p5v7ol6aE7 sXax9tKyoeIc5bEs3yPbZOeU9ba1heJnocFb5pwZl6taPoeEnPZK61ibNZxWqCkz8FqV1+vm HJuic1clxR9Ii7cm9/QF816Obq2p6z6Y/NRye09uKb6amDAx71fQnYGxN+t+aHi1AWYsbh/N 4efuTmw7EtWwxuY34+2wkFKESwL9SLlC8huiqIjTWgMAAA== X-CMS-MailID: 20230419114714epcas5p33084dcdc06787292b46c376aa51e5ec5 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230419114714epcas5p33084dcdc06787292b46c376aa51e5ec5 References: <20230419114320.13674-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org For the devices which does not support copy, copy emulation is added. It is required for in-kernel users like fabrics, where file descriptor is not available and hence they can't use copy_file_range. Copy-emulation is implemented by reading from source into memory and writing to the corresponding destination asynchronously. Also emulation is used, if copy offload fails or partially completes. Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu Signed-off-by: Anuj Gupta --- block/blk-lib.c | 175 ++++++++++++++++++++++++++++++++++++++++- block/blk-map.c | 4 +- include/linux/blkdev.h | 3 + 3 files changed, 179 insertions(+), 3 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index ed089e703cb1..ba32545eb8d5 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -295,6 +295,172 @@ static int __blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, return blkdev_copy_wait_completion(cio); } +static void *blkdev_copy_alloc_buf(sector_t req_size, sector_t *alloc_size, + gfp_t gfp_mask) +{ + int min_size = PAGE_SIZE; + void *buf; + + while (req_size >= min_size) { + buf = kvmalloc(req_size, gfp_mask); + if (buf) { + *alloc_size = req_size; + return buf; + } + /* retry half the requested size */ + req_size >>= 1; + } + + return NULL; +} + +static void blkdev_copy_emulate_write_endio(struct bio *bio) +{ + struct copy_ctx *ctx = bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (bio->bi_status) { + clen = (bio->bi_iter.bi_sector << SECTOR_SHIFT) - cio->pos_out; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + } + kvfree(page_address(bio->bi_io_vec[0].bv_page)); + bio_map_kern_endio(bio); + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } +} + +static void blkdev_copy_emulate_read_endio(struct bio *read_bio) +{ + struct copy_ctx *ctx = read_bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (read_bio->bi_status) { + clen = (read_bio->bi_iter.bi_sector << SECTOR_SHIFT) - + cio->pos_in; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + __free_page(read_bio->bi_io_vec[0].bv_page); + bio_map_kern_endio(read_bio); + kfree(ctx); + + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } + } + schedule_work(&ctx->dispatch_work); + kfree(read_bio); +} + +/* + * If native copy offload feature is absent, this function tries to emulate, + * by copying data from source to a temporary buffer and from buffer to + * destination device. + * Returns the length of bytes copied or error if encountered + */ +static int __blkdev_copy_emulate(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct request_queue *in = bdev_get_queue(bdev_in); + struct request_queue *out = bdev_get_queue(bdev_out); + struct bio *read_bio, *write_bio; + void *buf = NULL; + struct copy_ctx *ctx; + struct cio *cio; + sector_t buf_len, req_len, rem = 0; + sector_t max_src_hw_len = min_t(unsigned int, + queue_max_hw_sectors(in), + queue_max_segments(in) << (PAGE_SHIFT - SECTOR_SHIFT)) + << SECTOR_SHIFT; + sector_t max_dst_hw_len = min_t(unsigned int, + queue_max_hw_sectors(out), + queue_max_segments(out) << (PAGE_SHIFT - SECTOR_SHIFT)) + << SECTOR_SHIFT; + sector_t max_hw_len = min_t(unsigned int, + max_src_hw_len, max_dst_hw_len); + + cio = kzalloc(sizeof(struct cio), GFP_KERNEL); + if (!cio) + return -ENOMEM; + atomic_set(&cio->refcount, 0); + cio->pos_in = pos_in; + cio->pos_out = pos_out; + cio->waiter = current; + cio->endio = endio; + cio->private = private; + + for (rem = len; rem > 0; rem -= buf_len) { + req_len = min_t(int, max_hw_len, rem); + + buf = blkdev_copy_alloc_buf(req_len, &buf_len, gfp_mask); + if (!buf) + goto err_alloc_buf; + + ctx = kzalloc(sizeof(struct copy_ctx), gfp_mask); + if (!ctx) + goto err_ctx; + + read_bio = bio_map_kern(in, buf, buf_len, gfp_mask); + if (IS_ERR(read_bio)) + goto err_read_bio; + + write_bio = bio_map_kern(out, buf, buf_len, gfp_mask); + if (IS_ERR(write_bio)) + goto err_write_bio; + + ctx->cio = cio; + ctx->write_bio = write_bio; + INIT_WORK(&ctx->dispatch_work, blkdev_copy_dispatch_work); + + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_iter.bi_size = buf_len; + read_bio->bi_opf = REQ_OP_READ | REQ_SYNC; + bio_set_dev(read_bio, bdev_in); + read_bio->bi_end_io = blkdev_copy_emulate_read_endio; + read_bio->bi_private = ctx; + + write_bio->bi_iter.bi_size = buf_len; + write_bio->bi_opf = REQ_OP_WRITE | REQ_SYNC; + bio_set_dev(write_bio, bdev_out); + write_bio->bi_end_io = blkdev_copy_emulate_write_endio; + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_private = ctx; + + atomic_inc(&cio->refcount); + submit_bio(read_bio); + + pos_in += buf_len; + pos_out += buf_len; + } + + /* Wait for completion of all IO's*/ + return blkdev_copy_wait_completion(cio); + +err_write_bio: + bio_put(read_bio); +err_read_bio: + kfree(ctx); +err_ctx: + kvfree(buf); +err_alloc_buf: + cio->comp_len -= min_t(sector_t, cio->comp_len, len - rem); + if (!atomic_read(&cio->refcount)) + return -ENOMEM; + /* Wait for submitted IOs to complete */ + return blkdev_copy_wait_completion(cio); +} + static inline int blkdev_copy_sanity_check(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len) @@ -342,9 +508,16 @@ int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, if (ret) return ret; - if (blk_queue_copy(q_in) && blk_queue_copy(q_out)) + if (blk_queue_copy(q_in) && blk_queue_copy(q_out)) { ret = __blkdev_copy_offload(bdev_in, pos_in, bdev_out, pos_out, len, endio, private, gfp_mask); + if (ret < 0) + ret = 0; + } + + if (ret != len) + ret = __blkdev_copy_emulate(bdev_in, pos_in + ret, bdev_out, + pos_out + ret, len - ret, endio, private, gfp_mask); return ret; } diff --git a/block/blk-map.c b/block/blk-map.c index 3551c3ff17cf..e75bae459cfa 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -363,7 +363,7 @@ static void bio_invalidate_vmalloc_pages(struct bio *bio) #endif } -static void bio_map_kern_endio(struct bio *bio) +void bio_map_kern_endio(struct bio *bio) { bio_invalidate_vmalloc_pages(bio); bio_uninit(bio); @@ -380,7 +380,7 @@ static void bio_map_kern_endio(struct bio *bio) * Map the kernel address into a bio suitable for io to a block * device. Returns an error pointer in case of error. */ -static struct bio *bio_map_kern(struct request_queue *q, void *data, +struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, gfp_t gfp_mask) { unsigned long kaddr = (unsigned long)data; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 1bb43697d43d..a54153610800 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1057,6 +1057,9 @@ int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); +struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, + gfp_t gfp_mask); +void bio_map_kern_endio(struct bio *bio); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Wed Apr 19 11:43:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13216705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40746C6FD18 for ; Wed, 19 Apr 2023 11:55:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233144AbjDSLy7 (ORCPT ); Wed, 19 Apr 2023 07:54:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45994 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232727AbjDSLym (ORCPT ); Wed, 19 Apr 2023 07:54:42 -0400 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD79C14F5D for ; Wed, 19 Apr 2023 04:54:39 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20230419115438epoutp0401c7f08b47deb475830919dacfa252ee~XU-CAbV091585715857epoutp04e for ; Wed, 19 Apr 2023 11:54:38 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20230419115438epoutp0401c7f08b47deb475830919dacfa252ee~XU-CAbV091585715857epoutp04e DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681905278; bh=Z2LnmpXCeKveNBOyEvE4QzoJ7tl7STCy44ODCcmiAWc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UUDjwC6i8dJU4tbPFSeIQ2+15zoQ2Whp1Na8mTJyy3o4EAxjqUwkyE+k9VJ9ymFPv LcB8o7dZPVdmyLTlAYJwL4q0BWtBlixGg5jJkTX2Evyjk5yT/NQ9GYmBgaS+8XAZDZ MYTdZ/10BU9As+gDROQTsRb8d3YSGnbAujpkKTRc= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230419115437epcas5p3c6249257f01eee32ddc9a14293f9b2e8~XU-BPcNcy1261612616epcas5p35; Wed, 19 Apr 2023 11:54:37 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.182]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4Q1fNC56FSz4x9Pt; Wed, 19 Apr 2023 11:54:35 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 0B.1D.09961.B76DF346; Wed, 19 Apr 2023 20:54:35 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p4.samsung.com (KnoxPortal) with ESMTPA id 20230419114723epcas5p461a6d54ffc6cc5c32ee9d5ab37978135~XU4tL4L9m0326203262epcas5p45; Wed, 19 Apr 2023 11:47:23 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230419114723epsmtrp2ad7279e1070694177e5d45cd57f14a30~XU4tK01uf2737927379epsmtrp2P; Wed, 19 Apr 2023 11:47:23 +0000 (GMT) X-AuditID: b6c32a49-52dfd700000026e9-67-643fd67b99e2 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id D5.97.08279.BC4DF346; Wed, 19 Apr 2023 20:47:23 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230419114720epsmtip1660107e95db765e14dfba3b279361149~XU4p7mRkg2495924959epsmtip1C; Wed, 19 Apr 2023 11:47:19 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v10 4/9] fs, block: copy_file_range for def_blk_ops for direct block device Date: Wed, 19 Apr 2023 17:13:09 +0530 Message-Id: <20230419114320.13674-5-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230419114320.13674-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta0xTZxjOOaccTjE1hwLjE4aWOjbRQFst9QPlEkB3NsgkYctgCWMNPVAC tF0vw42Y4Rg6UEAuNVjYZIIU8MJFMAi0m2UgF4EIApNZxQyWIIOCZLINHWtt3fz3vM/7PO/t y0dg7G+cvYh0mZpWysSZXNyFcb3Xf1dAzlS4hK8r3Ambh/ox+NWZ5xi8ZC7B4WLvEwSeXfkL g7M/hEPDcpUTvPfjDRT2XChDYeOlPhR2f7+Kwr7NJRyWmaYQOD+pQ6FhZg/sMQwy4ERXNQ7P 1887Q1N5Hgo7544j8PrGeQxeXbQw4MCMNxx7fsspAlATd2Mo3cMRnLqhMztTYw9aGdTEiIZq ayrAqWt1X1Ld93JxqihvGacsxkmcKm5vQqhrwznUWtt2qm1uCY3b+lHGQSktltBKDi1LkUvS ZWmh3Jj45KjkIBFfECAIhvu5HJk4iw7lRsfGBRxOz7RegMv5TJypsVJxYpWKyws7qJRr1DRH KlepQ7m0QpKpECoCVeIslUaWFiij1SECPn9vkFX4SYb00axCUbTr6B+jejwXafAtRAgCkELw z0JMIeJCsMluBNSPn0bswRME5PdfQO3BUwR0FOVZM8wXjrpcvUNlQEBLrh63JdhkPgouVylt ZXFyDxjeJGwad/IEBlbnCxi2ACPHUfCs3YLZDG5kErh9s8DJZmCQfuCnSpWNZpEhoMI4zLCP xwMlD11tNJM8APTGcswucQWD5+YYNoyRO0BeRxVmKw/IWiZYmTjFsA8aDf5sXEXt2A08vtXu bMdeYG3ZgNtxNmisaMDt5q8RoJvWObYMB/lDJZhtCIz0B81dPDvtA7RDV1F7462gaGPOUZ8F Or97iXeCy801jvrbwNT6cQemwLkas+NWxQhYMCSeQTi6V/bRvbKP7v/ONQjWhGyjFaqsNFoV pBDI6Oz/njhFntWGvPgSu9/pRMyzK4EmBCUQEwIIjOvOun0oRMJmScSff0Er5clKTSatMiFB 1nOXYl4eKXLrn5KpkwXCYL5QJBIJg/eJBFxP1luhgylsMk2spjNoWkErX/pQgumViyrX17qz Uenab67RpCk/8SRTO7rkWn/FM2JlqOJbz9oy0bF9PuXR7//Oh368w+MdRPWk79iy4VdtoDau 1Bg5O51E+MW6BGUcm0w40BMR4q5wmxekayO1v/RXfuwXo2fSG33FTib3qvulwTXM0/qL1S2a SsvK23d2eJteOzX6uuXnO2bWyadHUsOq14ExyX/AP6HcaaEjMOTBmwPKTzcs5qO8zb3enhc/ CAu7X8D2bd2e86jl7t9+71WOJH7YWvNu3nSncEtkk8cb5XUtuv2Pn/EUvctGjhorSsjJiT9R OxPV39DVcZPZH6UtiJVsiZd6+BgGj6QW1/Kau1LHzi5eIdo9uQyVVCzYjSlV4n8BIa8zkZsE AAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrAIsWRmVeSWpSXmKPExsWy7bCSnO7pK/YpBtP2S1isP3WM2aJpwl9m i9V3+9ksXh/+xGgx7cNPZosH++0t9r6bzWpx88BOJos9iyYxWaxcfZTJYvfCj0wWR/+/ZbOY dOgao8XTq7OYLPbe0rbYs/cki8XlXXPYLOYve8pucWhyM5PFjieNjBbbfs9ntlj3+j2LxYlb 0hbn/x5ndZDwuHzF22PW/bNsHjtn3WX3OH9vI4vH5bOlHptWdbJ5bF5S77H7ZgObR2/zOzaP 9/uusnn0bVnF6LH5dLXH501yHpuevGUK4IvisklJzcksSy3St0vgynj4oKCgV6Pi67nlbA2M KxS7GDk5JARMJJY0LGfsYuTiEBLYzShx6OZmFoiEpMSyv0eYIWxhiZX/nrNDFDUzSTzcMRso wcHBJqAtcfo/B0hcRGACs8Sl+w1sIA6zwAMmifPPv7CCdAsLREt0b1jICNLAIqAqcWRGMUiY V8BKYsq+0ywgYQkBfYn++4IgYU4Ba4nl+yaD7RUCKuk59I8RolxQ4uTMJ2C3MQvISzRvnc08 gVFgFpLULCSpBYxMqxglUwuKc9Nziw0LDPNSy/WKE3OLS/PS9ZLzczcxgiNZS3MH4/ZVH/QO MTJxMB5ilOBgVhLhdbexSxHiTUmsrEotyo8vKs1JLT7EKM3BoiTOe6HrZLyQQHpiSWp2ampB ahFMlomDU6qBqS1r1vfc7a2zrXasU2KY8oTB/+CnS2p7RQQVCm1DdNbPPCGlprK4xOPXsbDH uraLX+Ubz03RmJd66sddedHsheH1hukhlnXd05WC6mNOfFSXOq/b+tmK/97y7Ss3cItJWzAq 182ZsJH1WM2MC02rEuflXI3zet11SD9r20+r/jbPc1/FzvIu6l6uOGllyJkEs8Tr91t+F19J 9hSzjHQRe1HPNr83ab2RkUL+7b+KE/o7H7hkJlwq03z1N3fatyKREu8r6vOlF7zYfunTxA0v 2Y3+Saw5YpWpofB/95dnUcfd7DjmGdTMCE7+8T6QuaL69G972VcH94kqvWDcHH5126X0UCuz RywbeGbrndWuVGIpzkg01GIuKk4EAERXN8JTAwAA X-CMS-MailID: 20230419114723epcas5p461a6d54ffc6cc5c32ee9d5ab37978135 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230419114723epcas5p461a6d54ffc6cc5c32ee9d5ab37978135 References: <20230419114320.13674-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org For direct block device opened with O_DIRECT, use copy_file_range to issue device copy offload, and fallback to generic_copy_file_range incase device copy offload capability is absent. Modify checks to allow bdevs to use copy_file_range. Suggested-by: Ming Lei Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty --- block/blk-lib.c | 23 +++++++++++++++++++++++ block/fops.c | 20 ++++++++++++++++++++ fs/read_write.c | 11 +++++++++-- include/linux/blkdev.h | 3 +++ 4 files changed, 55 insertions(+), 2 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index ba32545eb8d5..7d6ef85692a6 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -523,6 +523,29 @@ int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, } EXPORT_SYMBOL_GPL(blkdev_issue_copy); +/* Returns the length of bytes copied */ +int blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + gfp_t gfp_mask) +{ + struct request_queue *in_q = bdev_get_queue(bdev_in); + struct request_queue *out_q = bdev_get_queue(bdev_out); + int ret = 0; + + if (blkdev_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len)) + return 0; + + if (blk_queue_copy(in_q) && blk_queue_copy(out_q)) { + ret = __blkdev_copy_offload(bdev_in, pos_in, bdev_out, pos_out, + len, NULL, NULL, gfp_mask); + if (ret < 0) + return 0; + } + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_copy_offload); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/block/fops.c b/block/fops.c index d2e6be4e3d1c..042a62c81468 100644 --- a/block/fops.c +++ b/block/fops.c @@ -611,6 +611,25 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to) return ret; } +static ssize_t blkdev_copy_file_range(struct file *file_in, loff_t pos_in, + struct file *file_out, loff_t pos_out, + size_t len, unsigned int flags) +{ + struct block_device *in_bdev = I_BDEV(bdev_file_inode(file_in)); + struct block_device *out_bdev = I_BDEV(bdev_file_inode(file_out)); + int comp_len = 0; + + if ((file_in->f_iocb_flags & IOCB_DIRECT) && + (file_out->f_iocb_flags & IOCB_DIRECT)) + comp_len = blkdev_copy_offload(in_bdev, pos_in, out_bdev, + pos_out, len, GFP_KERNEL); + if (comp_len != len) + comp_len = generic_copy_file_range(file_in, pos_in + comp_len, + file_out, pos_out + comp_len, len - comp_len, flags); + + return comp_len; +} + #define BLKDEV_FALLOC_FL_SUPPORTED \ (FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE | \ FALLOC_FL_ZERO_RANGE | FALLOC_FL_NO_HIDE_STALE) @@ -694,6 +713,7 @@ const struct file_operations def_blk_fops = { .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, .fallocate = blkdev_fallocate, + .copy_file_range = blkdev_copy_file_range, }; static __init int blkdev_init(void) diff --git a/fs/read_write.c b/fs/read_write.c index a21ba3be7dbe..47e848fcfd42 100644 --- a/fs/read_write.c +++ b/fs/read_write.c @@ -20,6 +20,7 @@ #include #include #include +#include #include "internal.h" #include @@ -1447,7 +1448,11 @@ static int generic_copy_file_checks(struct file *file_in, loff_t pos_in, return -EOVERFLOW; /* Shorten the copy to EOF */ - size_in = i_size_read(inode_in); + if (S_ISBLK(inode_in->i_mode)) + size_in = bdev_nr_bytes(I_BDEV(file_in->f_mapping->host)); + else + size_in = i_size_read(inode_in); + if (pos_in >= size_in) count = 0; else @@ -1708,7 +1713,9 @@ int generic_file_rw_checks(struct file *file_in, struct file *file_out) /* Don't copy dirs, pipes, sockets... */ if (S_ISDIR(inode_in->i_mode) || S_ISDIR(inode_out->i_mode)) return -EISDIR; - if (!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) + + if ((!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) && + (!S_ISBLK(inode_in->i_mode) || !S_ISBLK(inode_out->i_mode))) return -EINVAL; if (!(file_in->f_mode & FMODE_READ) || diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index a54153610800..533ad682e0ca 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1057,6 +1057,9 @@ int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); +int blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + gfp_t gfp_mask); struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, gfp_t gfp_mask); void bio_map_kern_endio(struct bio *bio); From patchwork Wed Apr 19 11:43:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13216706 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50EA6C6FD18 for ; Wed, 19 Apr 2023 11:55:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233129AbjDSLzU (ORCPT ); Wed, 19 Apr 2023 07:55:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46222 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233130AbjDSLy4 (ORCPT ); Wed, 19 Apr 2023 07:54:56 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E01E71545C for ; Wed, 19 Apr 2023 04:54:43 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230419115442epoutp0360051777eeeea8905c257c96fe8cd476~XU-F3xmus1374113741epoutp03y for ; Wed, 19 Apr 2023 11:54:42 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230419115442epoutp0360051777eeeea8905c257c96fe8cd476~XU-F3xmus1374113741epoutp03y DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681905282; bh=1WJtwnO5PvXiBgPu8IyPQ8G/PqQ97PZnyPYo9MnZUQA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dO2oFM5dl339CH+Du1PWozcyTczt8prhJa8FfdCYgiUHCk9pk5TSJzPsqEf7g9Lzh XKW4En/tIQKbpJe2mS7WtZ0glL+4bvI6APiMCUkF3PMcbXkgTBHy3yGnTBMecvmP/7 5x5js571AzEDBXUweeVu5SKUTdu3sQGin+ZIrpF4= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230419115441epcas5p3c54c366b69eaaa2f0bf6955f4b2f2ac1~XU-FTvAWr1261212612epcas5p3I; Wed, 19 Apr 2023 11:54:41 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.174]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4Q1fNJ12lnz4x9Pp; Wed, 19 Apr 2023 11:54:40 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id CB.14.09987.086DF346; Wed, 19 Apr 2023 20:54:40 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230419114734epcas5p327483315c82893ca4b45bc3f3f871fb3~XU43G3an40477504775epcas5p39; Wed, 19 Apr 2023 11:47:34 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230419114734epsmtrp2912052a0ef1a2b7bcf1bfc1b37f10ae4~XU43F0fCP2737927379epsmtrp2W; Wed, 19 Apr 2023 11:47:34 +0000 (GMT) X-AuditID: b6c32a4b-7fbff70000002703-37-643fd6808231 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id CE.83.08609.6D4DF346; Wed, 19 Apr 2023 20:47:34 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230419114729epsmtip1fe85992e92f8efac5c40992d7651fbbd~XU4yoR5sV1717917179epsmtip1O; Wed, 19 Apr 2023 11:47:29 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , =?utf-8?q?Javi?= =?utf-8?q?er_Gonz=C3=A1lez?= , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v10 5/9] nvme: add copy offload support Date: Wed, 19 Apr 2023 17:13:10 +0530 Message-Id: <20230419114320.13674-6-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230419114320.13674-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta1BUZRjuO2c5nKWBOQs4fSASs04pOMBuLPBxE0tyjuIPCqvJnIEd9gQI e3EvEDmTXDTwwqXFiJYEZJgUqCUuERdXaxEQFgSHgGQUEVi6EOiyiinj0C4Hyn/P87zv896+ +UjctcTRk0yVqRmlTJzOJ5w4bd2+vv7Z49ESwcU1HDUO9OIot+Q5jhruFhNooXsZoLKHT3E0 fS0aGZYqHNDtnzswdKVGi6G6hh4MdV20YGj2jtUR9awtEkhrHAfIPKbDkGFyF7pi6Oeg0c5v CFT1rdkRGUvzMNQ+lwNQ22oVjvQLDzjoxuRWNPy8z2GPBz36ayytuzdE0B26u4708FQThx4d 0tDN9acJuqX2BN11O5ugC/OWCPrB1TGCLmqtB3SL6Thtbfamm+cWsTiXw2mRKYxYwih9GFmS XJIqS47ix8Yn7E0IDhEI/YVhKJTvIxNLmSh+zME4/32p6bYz8H0yxOkamxQnVqn4gbsjlXKN mvFJkavUUXxGIUlXiBQBKrFUpZElB8gYdbhQIHgj2JaYmJZiGR4EiqmMT1bNkw7Z4GTiGcAl ISWCd671ADt2pboAXJvdeQY42fAygEaLCWeJFcALI9eJTUdvxSxgA50A/l2YS7DkFAYbzt+w EZIkqF3QtEbadXfqcxxazKc5doJTWhzWXjq73tCNCoU1eZXrmEO9BkeMSw527EyFw9GpVo69 EKQCYfE9nl3mUhHw0tVSnE3hwf6v5zh2jFOvwrwfK9ZHhVQXF+b/VoWz3hhY3fc+O7Ub/Kuv 1ZHFntC6ZNjYJhPWnb9MsN6TAOomdIANRMNTA8XrdXDKFzZ2BrLyNvjlgB5j+7rAwtU5jNWd YXvlJt4Ov2us3qjvAcef5GxgGuqfLWLssYoANM00ESXAR/fCProX9tH937oa4PXAg1GopMmM KlgRJGMy/3vlJLm0Gax/Db/YdjAz/TDACDASGAEkcb678+Db4RJXZ4k461NGKU9QatIZlREE 2+79Be65JUlu+1sydYJQFCYQhYSEiMKCQoT8V5x3RPUnuVLJYjWTxjAKRrnpw0iuZzb2pxO5 uJ9X5WAwTIg+PhCUaIl5NDU2Xmb2xYpmagU9fqm38PJufkvbStrv41ulC/lrN/U/Vc9W5nt9 6JRQnKHRkT9MGsnk3Pnyyv1joQdNn41o3avI7W8+mpU063PyMlxd/sl6aZ/3OLVyxP/+yrle 997BsmM8rT9eMB3s3ffMI7LosOWIqakFeH/fEFwQO3/i2A4RtZdnuTmQVfmOZHG5rtyQzIxd OPd6wZY2bmbtTutC16GJrzo+Cn2cxAvTvuV1+bFXfGmE/oNDqYsq6+r8tnfdMlLi9EdhSe/9 J7KI8qNc862n8ZzQCWNg+R/HD3S+Z6F2B7nVnL2O/1Il3TMkyXmZz1GliIV+uFIl/hfoWMsN owQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02Sf0yMcRzHfZ/nuafn2uLbVe6b/MhNy64p2ZmvCtk0z7AyY4yNO/UoVM49 lfwYx2ETcnRSd4TOj7ozcipHd34cUaqduc44rmzKj6VImvlRuJrx3/vzfr8+788/H4YUeakx zPrsHE6VrciU0P5U7T3J+ClPW+ekTfU6KHzl0QMS79EOkNjsPULjrnufAS7+9I3Er27PwfYe gwA/v3ODwLbyYwSuNNcTuO5sL4Ffv+zzw/W/uml8zPEU4E63nsB2TxS22Rsp7Lp5ksanL3T6 YUeRhsDWjt0A1/44TeLLXR8p3OAJw86Bh4LEUNbVupDVt7fQ7A291491tl2lWFdLLmsxHaDZ a+d2sXXP1TR7WNNDsx9vuWm2sNoE2GtN29k+y3jW0tFNLB650j8hjctcn8epYmbL/TN6nc1A 2ZaX/6PTI1CDvfICIGQQlKEHhtegAPgzImgFqEhbDIaDUHRh4D45rINQ5eBbv2FIQyDz+8I/ AcPQMAo1/WJ8fjDUkuhJu5r2DSQ8Q6Kjpibatx0EZ6ByTdlQKwUj0GNHj8CnA2AccrVVU74i BGPQkfZAny2E8ejiraKhw6I/yCHHIBjGA1FjaccQTsJIdKVM5LNJOAFpagykFgTq/6P0/yj9 f9QZQJpAKKfks9Kz+FjltGxuSzSvyOJzs9OjUzdlWcDQB0ilVmAzfYp2AIIBDoAYUhIc0JwU lyYKSFNs3capNq1R5WZyvAOEMZREHPC4oHGNCKYrcriNHKfkVH9TghGOURO6qLDKhN5VluSz HeGqBYZIaTJ/WZlU/aUq+edxV37DQfH37rGzrIH4WZv8LjvKvT3hakpiJBydMvDhTkLllgax vWzemxCv05TqeTcYudnWX9JcVtovqzErb0JF8eCHitCq+KVyYbnxZYYxZv+SPrfK0CKN7RLN elHjKa2TxUUso4vtvC7FqmyUrd5Q8uL+xsPe/JkVkxfviPi62rj8Yd6rfCj+rDu1R5yomLQT Flw3uM9tdS5tVe8zPto3I8gcPDEmuKpe961zuk27Xx4eH9JfWzN51LoV6WCZsDc15PyicZYT lwSl8wWJ4dM3HNAJemTGEa2Fc/s0OfakL1TFWrOE4jMUsVJSxSt+AxNCiVRwAwAA X-CMS-MailID: 20230419114734epcas5p327483315c82893ca4b45bc3f3f871fb3 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230419114734epcas5p327483315c82893ca4b45bc3f3f871fb3 References: <20230419114320.13674-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org For device supporting native copy, nvme driver receives read and write request with BLK_COPY op flags. For read request the nvme driver populates the payload with source information. For write request the driver converts it to nvme copy command using the source information in the payload and submits to the device. current design only supports single source range. This design is courtesy Mikulas Patocka's token based copy trace event support for nvme_copy_cmd. Set the device copy limits to queue limits. Signed-off-by: Kanchan Joshi Signed-off-by: Nitesh Shetty Signed-off-by: Javier González Signed-off-by: Anuj Gupta --- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 103 +++++++++++++++++++++++++++++++++- drivers/nvme/host/fc.c | 5 ++ drivers/nvme/host/nvme.h | 7 +++ drivers/nvme/host/pci.c | 27 ++++++++- drivers/nvme/host/rdma.c | 7 +++ drivers/nvme/host/tcp.c | 16 ++++++ drivers/nvme/host/trace.c | 19 +++++++ include/linux/nvme.h | 43 +++++++++++++- 9 files changed, 220 insertions(+), 8 deletions(-) diff --git a/drivers/nvme/host/constants.c b/drivers/nvme/host/constants.c index bc523ca02254..01be882b726f 100644 --- a/drivers/nvme/host/constants.c +++ b/drivers/nvme/host/constants.c @@ -19,6 +19,7 @@ static const char * const nvme_ops[] = { [nvme_cmd_resv_report] = "Reservation Report", [nvme_cmd_resv_acquire] = "Reservation Acquire", [nvme_cmd_resv_release] = "Reservation Release", + [nvme_cmd_copy] = "Copy Offload", [nvme_cmd_zone_mgmt_send] = "Zone Management Send", [nvme_cmd_zone_mgmt_recv] = "Zone Management Receive", [nvme_cmd_zone_append] = "Zone Management Append", diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index ccb6eb1282f8..aef7b59dbd61 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -754,6 +754,77 @@ static inline void nvme_setup_flush(struct nvme_ns *ns, cmnd->common.nsid = cpu_to_le32(ns->head->ns_id); } +static inline void nvme_setup_copy_read(struct nvme_ns *ns, struct request *req) +{ + struct bio *bio = req->bio; + struct nvme_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + + token->subsys = "nvme"; + token->ns = ns; + token->src_sector = bio->bi_iter.bi_sector; + token->sectors = bio->bi_iter.bi_size >> 9; +} + +static inline blk_status_t nvme_setup_copy_write(struct nvme_ns *ns, + struct request *req, struct nvme_command *cmnd) +{ + struct nvme_copy_range *range = NULL; + struct bio *bio = req->bio; + struct nvme_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + sector_t src_sector, dst_sector, n_sectors; + u64 src_lba, dst_lba, n_lba; + unsigned short nr_range = 1; + u16 control = 0; + + if (unlikely(memcmp(token->subsys, "nvme", 4))) + return BLK_STS_NOTSUPP; + if (unlikely(token->ns != ns)) + return BLK_STS_NOTSUPP; + + src_sector = token->src_sector; + dst_sector = bio->bi_iter.bi_sector; + n_sectors = token->sectors; + if (WARN_ON(n_sectors != bio->bi_iter.bi_size >> 9)) + return BLK_STS_NOTSUPP; + + src_lba = nvme_sect_to_lba(ns, src_sector); + dst_lba = nvme_sect_to_lba(ns, dst_sector); + n_lba = nvme_sect_to_lba(ns, n_sectors); + + if (WARN_ON(!n_lba)) + return BLK_STS_NOTSUPP; + + if (req->cmd_flags & REQ_FUA) + control |= NVME_RW_FUA; + + if (req->cmd_flags & REQ_FAILFAST_DEV) + control |= NVME_RW_LR; + + memset(cmnd, 0, sizeof(*cmnd)); + cmnd->copy.opcode = nvme_cmd_copy; + cmnd->copy.nsid = cpu_to_le32(ns->head->ns_id); + cmnd->copy.sdlba = cpu_to_le64(dst_lba); + + range = kmalloc_array(nr_range, sizeof(*range), + GFP_ATOMIC | __GFP_NOWARN); + if (!range) + return BLK_STS_RESOURCE; + + range[0].slba = cpu_to_le64(src_lba); + range[0].nlb = cpu_to_le16(n_lba - 1); + + cmnd->copy.nr_range = 0; + + req->special_vec.bv_page = virt_to_page(range); + req->special_vec.bv_offset = offset_in_page(range); + req->special_vec.bv_len = sizeof(*range) * nr_range; + req->rq_flags |= RQF_SPECIAL_PAYLOAD; + + cmnd->copy.control = cpu_to_le16(control); + + return BLK_STS_OK; +} + static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req, struct nvme_command *cmnd) { @@ -988,10 +1059,16 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req) ret = nvme_setup_discard(ns, req, cmd); break; case REQ_OP_READ: - ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_read); + if (unlikely(req->cmd_flags & REQ_COPY)) + nvme_setup_copy_read(ns, req); + else + ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_read); break; case REQ_OP_WRITE: - ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_write); + if (unlikely(req->cmd_flags & REQ_COPY)) + ret = nvme_setup_copy_write(ns, req, cmd); + else + ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_write); break; case REQ_OP_ZONE_APPEND: ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_zone_append); @@ -1698,6 +1775,26 @@ static void nvme_config_discard(struct gendisk *disk, struct nvme_ns *ns) blk_queue_max_write_zeroes_sectors(queue, UINT_MAX); } +static void nvme_config_copy(struct gendisk *disk, struct nvme_ns *ns, + struct nvme_id_ns *id) +{ + struct nvme_ctrl *ctrl = ns->ctrl; + struct request_queue *q = disk->queue; + + if (!(ctrl->oncs & NVME_CTRL_ONCS_COPY)) { + blk_queue_max_copy_sectors_hw(q, 0); + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + return; + } + + /* setting copy limits */ + if (blk_queue_flag_test_and_set(QUEUE_FLAG_COPY, q)) + return; + + blk_queue_max_copy_sectors_hw(q, + nvme_lba_to_sect(ns, le16_to_cpu(id->mssrl))); +} + static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b) { return uuid_equal(&a->uuid, &b->uuid) && @@ -1897,6 +1994,7 @@ static void nvme_update_disk_info(struct gendisk *disk, set_capacity_and_notify(disk, capacity); nvme_config_discard(disk, ns); + nvme_config_copy(disk, ns, id); blk_queue_max_write_zeroes_sectors(disk->queue, ns->ctrl->max_zeroes_sectors); } @@ -5343,6 +5441,7 @@ static inline void _nvme_check_size(void) BUILD_BUG_ON(sizeof(struct nvme_download_firmware) != 64); BUILD_BUG_ON(sizeof(struct nvme_format_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_dsm_cmd) != 64); + BUILD_BUG_ON(sizeof(struct nvme_copy_command) != 64); BUILD_BUG_ON(sizeof(struct nvme_write_zeroes_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_abort_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_get_log_page_command) != 64); diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 2ed75923507d..db2e22b4ca7f 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -2807,6 +2807,11 @@ nvme_fc_queue_rq(struct blk_mq_hw_ctx *hctx, if (ret) return ret; + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_end_request(rq, BLK_STS_OK); + return BLK_STS_OK; + } /* * nvme core doesn't quite treat the rq opaquely. Commands such * as WRITE ZEROES will return a non-zero rq payload_bytes yet diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index bf46f122e9e1..66af37170bff 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -500,6 +500,13 @@ struct nvme_ns { }; +struct nvme_copy_token { + char *subsys; + struct nvme_ns *ns; + sector_t src_sector; + sector_t sectors; +}; + /* NVMe ns supports metadata actions by the controller (generate/strip) */ static inline bool nvme_ns_has_pi(struct nvme_ns *ns) { diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 7f25c0fe3a0b..d5d094fa2fd1 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -495,16 +495,19 @@ static inline void nvme_sq_copy_cmd(struct nvme_queue *nvmeq, nvmeq->sq_tail = 0; } -static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +static inline void nvme_commit_sq_db(struct nvme_queue *nvmeq) { - struct nvme_queue *nvmeq = hctx->driver_data; - spin_lock(&nvmeq->sq_lock); if (nvmeq->sq_tail != nvmeq->last_sq_tail) nvme_write_sq_db(nvmeq, true); spin_unlock(&nvmeq->sq_lock); } +static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +{ + nvme_commit_sq_db(hctx->driver_data); +} + static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req, int nseg) { @@ -848,6 +851,12 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, struct request *req) if (ret) return ret; + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_start_request(req); + return BLK_STS_OK; + } + if (blk_rq_nr_phys_segments(req)) { ret = nvme_map_data(dev, req, &iod->cmd); if (ret) @@ -894,6 +903,18 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, ret = nvme_prep_rq(dev, req); if (unlikely(ret)) return ret; + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_set_request_complete(req); + blk_mq_end_request(req, BLK_STS_OK); + /* Commit the sq if copy read was the last req in the list, + * as copy read deoesn't update sq db + */ + if (bd->last) + nvme_commit_sq_db(nvmeq); + return ret; + } + spin_lock(&nvmeq->sq_lock); nvme_sq_copy_cmd(nvmeq, &iod->cmd); nvme_write_sq_db(nvmeq, bd->last); diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 0eb79696fb73..be1d20ac8bb0 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -2038,6 +2038,13 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, nvme_start_request(rq); + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_end_request(rq, BLK_STS_OK); + ret = BLK_STS_OK; + goto unmap_qe; + } + if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) && queue->pi_support && (c->common.opcode == nvme_cmd_write || diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index bf0230442d57..5ba1bb35c557 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -2373,6 +2373,11 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns, if (ret) return ret; + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + return BLK_STS_OK; + } + req->state = NVME_TCP_SEND_CMD_PDU; req->status = cpu_to_le16(NVME_SC_SUCCESS); req->offset = 0; @@ -2441,6 +2446,17 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx, nvme_start_request(rq); + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_set_request_complete(rq); + blk_mq_end_request(rq, BLK_STS_OK); + /* if copy read is the last req queue tcp reqs */ + if (bd->last && nvme_tcp_queue_more(queue)) + queue_work_on(queue->io_cpu, nvme_tcp_wq, + &queue->io_work); + return ret; + } + nvme_tcp_queue_request(req, true, bd->last); return BLK_STS_OK; diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c index 1c36fcedea20..da4a7494e5a7 100644 --- a/drivers/nvme/host/trace.c +++ b/drivers/nvme/host/trace.c @@ -150,6 +150,23 @@ static const char *nvme_trace_read_write(struct trace_seq *p, u8 *cdw10) return ret; } +static const char *nvme_trace_copy(struct trace_seq *p, u8 *cdw10) +{ + const char *ret = trace_seq_buffer_ptr(p); + u64 slba = get_unaligned_le64(cdw10); + u8 nr_range = get_unaligned_le16(cdw10 + 8); + u16 control = get_unaligned_le16(cdw10 + 10); + u32 dsmgmt = get_unaligned_le32(cdw10 + 12); + u32 reftag = get_unaligned_le32(cdw10 + 16); + + trace_seq_printf(p, + "slba=%llu, nr_range=%u, ctrl=0x%x, dsmgmt=%u, reftag=%u", + slba, nr_range, control, dsmgmt, reftag); + trace_seq_putc(p, 0); + + return ret; +} + static const char *nvme_trace_dsm(struct trace_seq *p, u8 *cdw10) { const char *ret = trace_seq_buffer_ptr(p); @@ -243,6 +260,8 @@ const char *nvme_trace_parse_nvm_cmd(struct trace_seq *p, return nvme_trace_zone_mgmt_send(p, cdw10); case nvme_cmd_zone_mgmt_recv: return nvme_trace_zone_mgmt_recv(p, cdw10); + case nvme_cmd_copy: + return nvme_trace_copy(p, cdw10); default: return nvme_trace_common(p, cdw10); } diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 779507ac750b..6582b26e532c 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -337,7 +337,7 @@ struct nvme_id_ctrl { __u8 nvscc; __u8 nwpc; __le16 acwu; - __u8 rsvd534[2]; + __le16 ocfs; __le32 sgls; __le32 mnan; __u8 rsvd544[224]; @@ -365,6 +365,7 @@ enum { NVME_CTRL_ONCS_WRITE_ZEROES = 1 << 3, NVME_CTRL_ONCS_RESERVATIONS = 1 << 5, NVME_CTRL_ONCS_TIMESTAMP = 1 << 6, + NVME_CTRL_ONCS_COPY = 1 << 8, NVME_CTRL_VWC_PRESENT = 1 << 0, NVME_CTRL_OACS_SEC_SUPP = 1 << 0, NVME_CTRL_OACS_NS_MNGT_SUPP = 1 << 3, @@ -414,7 +415,10 @@ struct nvme_id_ns { __le16 npdg; __le16 npda; __le16 nows; - __u8 rsvd74[18]; + __le16 mssrl; + __le32 mcl; + __u8 msrc; + __u8 rsvd91[11]; __le32 anagrpid; __u8 rsvd96[3]; __u8 nsattr; @@ -796,6 +800,7 @@ enum nvme_opcode { nvme_cmd_resv_report = 0x0e, nvme_cmd_resv_acquire = 0x11, nvme_cmd_resv_release = 0x15, + nvme_cmd_copy = 0x19, nvme_cmd_zone_mgmt_send = 0x79, nvme_cmd_zone_mgmt_recv = 0x7a, nvme_cmd_zone_append = 0x7d, @@ -819,7 +824,8 @@ enum nvme_opcode { nvme_opcode_name(nvme_cmd_resv_release), \ nvme_opcode_name(nvme_cmd_zone_mgmt_send), \ nvme_opcode_name(nvme_cmd_zone_mgmt_recv), \ - nvme_opcode_name(nvme_cmd_zone_append)) + nvme_opcode_name(nvme_cmd_zone_append), \ + nvme_opcode_name(nvme_cmd_copy)) @@ -996,6 +1002,36 @@ struct nvme_dsm_range { __le64 slba; }; +struct nvme_copy_command { + __u8 opcode; + __u8 flags; + __u16 command_id; + __le32 nsid; + __u64 rsvd2; + __le64 metadata; + union nvme_data_ptr dptr; + __le64 sdlba; + __u8 nr_range; + __u8 rsvd12; + __le16 control; + __le16 rsvd13; + __le16 dspec; + __le32 ilbrt; + __le16 lbat; + __le16 lbatm; +}; + +struct nvme_copy_range { + __le64 rsvd0; + __le64 slba; + __le16 nlb; + __le16 rsvd18; + __le32 rsvd20; + __le32 eilbrt; + __le16 elbat; + __le16 elbatm; +}; + struct nvme_write_zeroes_cmd { __u8 opcode; __u8 flags; @@ -1757,6 +1793,7 @@ struct nvme_command { struct nvme_download_firmware dlfw; struct nvme_format_cmd format; struct nvme_dsm_cmd dsm; + struct nvme_copy_command copy; struct nvme_write_zeroes_cmd write_zeroes; struct nvme_zone_mgmt_send_cmd zms; struct nvme_zone_mgmt_recv_cmd zmr; From patchwork Wed Apr 19 11:43:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13216707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 216AFC6FD18 for ; Wed, 19 Apr 2023 11:55:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233161AbjDSLza (ORCPT ); Wed, 19 Apr 2023 07:55:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232693AbjDSLy5 (ORCPT ); Wed, 19 Apr 2023 07:54:57 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C725E14F63 for ; Wed, 19 Apr 2023 04:54:47 -0700 (PDT) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230419115446epoutp0104249e8ffe26243ad669e6b9f80868ec~XU-JU4pEU1494314943epoutp01f for ; Wed, 19 Apr 2023 11:54:46 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230419115446epoutp0104249e8ffe26243ad669e6b9f80868ec~XU-JU4pEU1494314943epoutp01f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681905286; bh=V0h50UZL6Pzbltx28/YeEpk8ZPO0bBrEuep2yMzOsFA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UUweP2jT4OQoaa+RyHHEwRL/MJWcI2z41IbIHIvRzYG2lt/BF6bxvs45rdk/0NpFN 0D1oC/CSuLkNRTcYMsEBqhv5F+lNzu4TtX/xXbyQfYYvDB/8rKGUEvIyAhTfocuNRg IFBoIPsW5VBaotuDjjuNPhAxQdZQx0+Na0mb4cUo= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230419115445epcas5p23004130fd2cf0bfe8717f938b8114447~XU-Ii0bRw1844418444epcas5p2d; Wed, 19 Apr 2023 11:54:45 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.177]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4Q1fNM4lj4z4x9Pr; Wed, 19 Apr 2023 11:54:43 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 82.8D.09540.386DF346; Wed, 19 Apr 2023 20:54:43 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230419114743epcas5p155559f1777d242e9d68c43cb61eb5777~XU4-aDjn30344203442epcas5p1_; Wed, 19 Apr 2023 11:47:43 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230419114743epsmtrp1a134ebcb049f457752518b814bf27c26~XU4-Yz49h1843518435epsmtrp1V; Wed, 19 Apr 2023 11:47:43 +0000 (GMT) X-AuditID: b6c32a4a-4afff70000002544-7a-643fd683b348 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id D1.93.08609.ED4DF346; Wed, 19 Apr 2023 20:47:43 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230419114739epsmtip1627264bbec8b39e9107854556df05662~XU48URkOI2496324963epsmtip1E; Wed, 19 Apr 2023 11:47:39 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v10 6/9] nvmet: add copy command support for bdev and file ns Date: Wed, 19 Apr 2023 17:13:11 +0530 Message-Id: <20230419114320.13674-7-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230419114320.13674-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01TbUxbZRT23tveli5dLgW3d0wYqS4LEKDdaPeCFEzE7UbMJJq4TFnKXXtH kdI2/YANp6PiUCDANlBZUUCcIuAoMqjQDjQlsFFgX3yFZRNYQLN142PgUD6cLXfo/j3nOed5 zznPm8PFBIWcAG6axkjrNZRaiPNYtq6QXeG5I/FKkcsphFZXDwY/Pr2GwYY7JTh0dz1C4Bdz f2Nw4pd42DFTwYZjv7aj8FLNWRTWNXSj0PHNPAq7nzzE4VnnCAKnhy0o7LgVBi919LLgoP0r HFZ9P82BztJcFLZNmRFoW6nCYKN7lgWv3NoOr61dZr8CyMGhRNIyPoCT7ZY7HPLabz+xyMEB E9lcn4+TF8+fJB1jOThZlDuDk7OdwzhZ3FKPkBf7PiAXmoPI5qmHaNLmd9NjVTSlpPXBtEah VaZpUmXCxLflr8olUpE4XBwN9wqDNVQGLRMmvJEUvi9N7XFAGJxJqU0eKokyGISRcbF6rclI B6u0BqNMSOuUal2ULsJAZRhMmtQIDW2MEYtEuyWewpR0VdPo72zdavwx19KnnBzEJilAfLiA iALupat4AcLjCggHAhqrpzlM8AgB9gsWNhMsIGC0fBHfkFTfPcf2YgFhR4Ct4SRTdAoFLmux J8Hl4kQY6HvC9fL+RB4G5qfzWd4AI26iYLVlFvOq/Yg3gXttiOXFLGInGMivW+/AJ2LAnw2j HO9DgIgEJeO+XtqHeBnUdpZiTIkv6D03tS7FiB0gt7UC874PiFofcHu0H2MmTQCdTR1sBvuB +5dbOAwOAPdK8p7iLFBX9gPOiD9BgGXUgjCJeHDKVYJ5h8CIEGC1RzJ0IPjc1YgyjTeDopUp lOH5oK1yA78IfrRWP3VrGxhZMuPMLiRo79ExZhUjoGLuPHoaCbY8s4/lmX0s/3euRrB6ZBut M2Sk0gaJbreGzvrvlxXajGZk/SpCX29DJifmIpwIykWcCOBiQn9+/2sxSgFfSR3PpvVaud6k pg1OROLx+wwW8LxC6zkrjVEujooWRUml0qjoPVKxcCt/l6xXISBSKSOdTtM6Wr+hQ7k+ATlo Ya/z8byqrpSX4xrf7yj/duW52OQU/+Sx2SH3YHK2/9jkg6tl1ukXMiUiATdBNDJ3Iab+ndC8 /P3OmYM/Lw7JIm53s3jXzbM7Uk64T9xc5Rdc2dIF5ETgJlvN42FJ6+R2bdfhDFZYYCO1dWYx y7LFHnnoBnmMV1FuLhawviw7QC2Yj3/mLI5Lte0cMSkFJTxkz1plUFxQbcVf9srDw++F//Fh SMm965l9q3cn5AH6t6iXEojl5aOUqj887uiNrMhD0uUse+K+fyqBNbDq4MIm3x5bv8hcmKeo OfDAFZR0xNIFfEjD160OOrtpsMnvo6C97xceWS2immSO+2L1hOI7IcugosShmN5A/QteINsc ngQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02RWUwTYRSF889Mp0MNOi2oP+ASx2AMIi5o/MWlarSO4oIxUQNxaWQsFVqa Vtx4EKkJAdmUSGJBrEYhlIBalLWgFEEQK8aypBSoC+CCKGIJcSmYQkx8O/d+95zzcClcVEf4 UnLlKU6tlMYwpIAoq2fmLXW0iSOXDxdj6N7zRhwlZrpwVNSTQaLB+hGAsod/4ujNYzGq+ZrD Q7YnlRgy3b6KocKiBgxV3/qOoYaJIRJdNXcA1N+uw1BN1xJkqmkmkLUql0Q38/v5yJylxVBF 30WAyn7fxFHJ4DcCNXX5oVbXM94myFrbQlmdw0KylboePtva+4BgrZY41mhIJtnSOxfYalsC yaZpv5Lst9p2kk1/aABsaUs8+8M4jzX2DWFh08MF6yO5GPlpTr1s4zFB1P3OAZ7qj/js87Ek fgIoW50CPChIr4L6d9d5KUBAiegKAC83/CKmgA/Mdz3Fp7QXLBz/wHdrEa3FoMu0LQVQFEkv gS0TlNvrTWfi8LUjgXQPOP0Gg60fnDy3wYveDe2mwclQgvaHluRC0q096RA4WtTJdwdBehnM cAjdaw96HSyozcKnukJgqnkcTJ0LYfP1vskYnJ4PtY9y8ExA6/5Duv+QHmAG4MOpNAqZQrNC tVLJnQnSSBWaOKUs6HiswggmvxwQUAFMhuEgM8AoYAaQwhlvzxfbQiJFnpHSc+c5dexRdVwM pzEDP4pgZnu+Smk+KqJl0lNcNMepOPU/ilEevglYKrd4Z2OJ05btr0hb+XKGZOehXYmx2kfE GmES1hC6/EZ02kFy7PfEaLqDI4KtuQMgr2WtNVk/In+lrBplkgR7a0d/tS70aiqfyY86Gy+5 UtXUnZpncy4ynAgMlIU75+zIicgLu58vL9hRfjKZiZW/LaltPMLsqVubcUe/py3M/r345cef LwpKGMmGa0KbOOLKpWk9Pu38+UBcOD4yY4CN6Q6tD4zYP7JAni5B/kGbjR37otSqiHfvJWlD WUgfv37W59sbVVa/Un97Bt1fXZw91ms/LOt1Fnxh7METdb7YlrubOpTRTeGlQp1FlfjR5ZNL fuJbnjZmzd3uvW7rAYbQRElXBOBqjfQv+uYkj1QDAAA= X-CMS-MailID: 20230419114743epcas5p155559f1777d242e9d68c43cb61eb5777 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230419114743epcas5p155559f1777d242e9d68c43cb61eb5777 References: <20230419114320.13674-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Add support for handling target command on target. For bdev-ns we call into blkdev_issue_copy, which the block layer completes by a offloaded copy request to backend bdev or by emulating the request. For file-ns we call vfs_copy_file_range to service our request. Currently target always shows copy capability by setting NVME_CTRL_ONCS_COPY in controller ONCS. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- drivers/nvme/target/admin-cmd.c | 9 +++-- drivers/nvme/target/io-cmd-bdev.c | 58 +++++++++++++++++++++++++++++++ drivers/nvme/target/io-cmd-file.c | 52 +++++++++++++++++++++++++++ drivers/nvme/target/loop.c | 6 ++++ drivers/nvme/target/nvmet.h | 1 + 5 files changed, 124 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c index 39cb570f833d..8a09f99a2185 100644 --- a/drivers/nvme/target/admin-cmd.c +++ b/drivers/nvme/target/admin-cmd.c @@ -433,8 +433,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) id->nn = cpu_to_le32(NVMET_MAX_NAMESPACES); id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES); id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM | - NVME_CTRL_ONCS_WRITE_ZEROES); - + NVME_CTRL_ONCS_WRITE_ZEROES | NVME_CTRL_ONCS_COPY); /* XXX: don't report vwc if the underlying device is write through */ id->vwc = NVME_CTRL_VWC_PRESENT; @@ -536,6 +535,12 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req) if (req->ns->bdev) nvmet_bdev_set_limits(req->ns->bdev, id); + else { + id->msrc = (u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16(BIO_MAX_VECS << + (PAGE_SHIFT - SECTOR_SHIFT)); + id->mcl = cpu_to_le32(le16_to_cpu(id->mssrl)); + } /* * We just provide a single LBA format that matches what the diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c index c2d6cea0236b..0af273097aa4 100644 --- a/drivers/nvme/target/io-cmd-bdev.c +++ b/drivers/nvme/target/io-cmd-bdev.c @@ -46,6 +46,19 @@ void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id) id->npda = id->npdg; /* NOWS = Namespace Optimal Write Size */ id->nows = to0based(bdev_io_opt(bdev) / bdev_logical_block_size(bdev)); + + /*Copy limits*/ + if (bdev_max_copy_sectors(bdev)) { + id->msrc = id->msrc; + id->mssrl = cpu_to_le16((bdev_max_copy_sectors(bdev) << + SECTOR_SHIFT) / bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32(id->mssrl); + } else { + id->msrc = (u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16((BIO_MAX_VECS << PAGE_SHIFT) / + bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32(id->mssrl); + } } void nvmet_bdev_ns_disable(struct nvmet_ns *ns) @@ -184,6 +197,19 @@ static void nvmet_bio_done(struct bio *bio) nvmet_req_bio_put(req, bio); } +static void nvmet_bdev_copy_end_io(void *private, int comp_len) +{ + struct nvmet_req *req = (struct nvmet_req *)private; + + if (comp_len == req->copy_len) { + req->cqe->result.u32 = cpu_to_le32(1); + nvmet_req_complete(req, errno_to_nvme_status(req, 0)); + } else { + req->cqe->result.u32 = cpu_to_le32(0); + nvmet_req_complete(req, blk_to_nvme_status(req, BLK_STS_IOERR)); + } +} + #ifdef CONFIG_BLK_DEV_INTEGRITY static int nvmet_bdev_alloc_bip(struct nvmet_req *req, struct bio *bio, struct sg_mapping_iter *miter) @@ -450,6 +476,34 @@ static void nvmet_bdev_execute_write_zeroes(struct nvmet_req *req) } } +/* At present we handle only one range entry */ +static void nvmet_bdev_execute_copy(struct nvmet_req *req) +{ + struct nvme_copy_range range; + struct nvme_command *cmnd = req->cmd; + int ret; + + + ret = nvmet_copy_from_sgl(req, 0, &range, sizeof(range)); + if (ret) + goto out; + + ret = blkdev_issue_copy(req->ns->bdev, + le64_to_cpu(cmnd->copy.sdlba) << req->ns->blksize_shift, + req->ns->bdev, + le64_to_cpu(range.slba) << req->ns->blksize_shift, + (le16_to_cpu(range.nlb) + 1) << req->ns->blksize_shift, + nvmet_bdev_copy_end_io, (void *)req, GFP_KERNEL); + if (ret) { + req->cqe->result.u32 = cpu_to_le32(0); + nvmet_req_complete(req, blk_to_nvme_status(req, BLK_STS_IOERR)); + } + + return; +out: + nvmet_req_complete(req, errno_to_nvme_status(req, ret)); +} + u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) { switch (req->cmd->common.opcode) { @@ -468,6 +522,10 @@ u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_bdev_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_bdev_execute_copy; + return 0; + default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c index 2d068439b129..69f198ecec77 100644 --- a/drivers/nvme/target/io-cmd-file.c +++ b/drivers/nvme/target/io-cmd-file.c @@ -322,6 +322,49 @@ static void nvmet_file_dsm_work(struct work_struct *w) } } +static void nvmet_file_copy_work(struct work_struct *w) +{ + struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); + int nr_range; + loff_t pos; + struct nvme_command *cmnd = req->cmd; + int ret = 0, len = 0, src, id; + + nr_range = cmnd->copy.nr_range + 1; + pos = le64_to_cpu(req->cmd->copy.sdlba) << req->ns->blksize_shift; + if (unlikely(pos + req->transfer_len > req->ns->size)) { + nvmet_req_complete(req, errno_to_nvme_status(req, -ENOSPC)); + return; + } + + for (id = 0 ; id < nr_range; id++) { + struct nvme_copy_range range; + + ret = nvmet_copy_from_sgl(req, id * sizeof(range), &range, + sizeof(range)); + if (ret) + goto out; + + len = (le16_to_cpu(range.nlb) + 1) << (req->ns->blksize_shift); + src = (le64_to_cpu(range.slba) << (req->ns->blksize_shift)); + ret = vfs_copy_file_range(req->ns->file, src, req->ns->file, + pos, len, 0); +out: + if (ret != len) { + pos += ret; + req->cqe->result.u32 = cpu_to_le32(id); + nvmet_req_complete(req, ret < 0 ? + errno_to_nvme_status(req, ret) : + errno_to_nvme_status(req, -EIO)); + return; + + } else + pos += len; + } + + nvmet_req_complete(req, 0); + +} static void nvmet_file_execute_dsm(struct nvmet_req *req) { if (!nvmet_check_data_len_lte(req, nvmet_dsm_len(req))) @@ -330,6 +373,12 @@ static void nvmet_file_execute_dsm(struct nvmet_req *req) queue_work(nvmet_wq, &req->f.work); } +static void nvmet_file_execute_copy(struct nvmet_req *req) +{ + INIT_WORK(&req->f.work, nvmet_file_copy_work); + queue_work(nvmet_wq, &req->f.work); +} + static void nvmet_file_write_zeroes_work(struct work_struct *w) { struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); @@ -376,6 +425,9 @@ u16 nvmet_file_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_file_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_file_execute_copy; + return 0; default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c index f2d24b2d992f..d18ed8067a15 100644 --- a/drivers/nvme/target/loop.c +++ b/drivers/nvme/target/loop.c @@ -146,6 +146,12 @@ static blk_status_t nvme_loop_queue_rq(struct blk_mq_hw_ctx *hctx, return ret; nvme_start_request(req); + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_set_request_complete(req); + blk_mq_end_request(req, BLK_STS_OK); + return BLK_STS_OK; + } iod->cmd.common.flags |= NVME_CMD_SGL_METABUF; iod->req.port = queue->ctrl->port; if (!nvmet_req_init(&iod->req, &queue->nvme_cq, diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h index dc60a22646f7..1615dc9194ba 100644 --- a/drivers/nvme/target/nvmet.h +++ b/drivers/nvme/target/nvmet.h @@ -393,6 +393,7 @@ struct nvmet_req { struct device *p2p_client; u16 error_loc; u64 error_slba; + size_t copy_len; }; #define NVMET_MAX_MPOOL_BVEC 16 From patchwork Wed Apr 19 11:43:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13216708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE6C2C77B73 for ; Wed, 19 Apr 2023 11:55:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233182AbjDSLzc (ORCPT ); Wed, 19 Apr 2023 07:55:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232821AbjDSLy6 (ORCPT ); Wed, 19 Apr 2023 07:54:58 -0400 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 575BD1447C for ; Wed, 19 Apr 2023 04:54:52 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20230419115450epoutp04328d6ca9b4495d26fda499ff9933bcab~XU-Nc5qOy1729517295epoutp04v for ; Wed, 19 Apr 2023 11:54:50 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20230419115450epoutp04328d6ca9b4495d26fda499ff9933bcab~XU-Nc5qOy1729517295epoutp04v DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681905290; bh=1P2BJZZ/eKH4o2N5bmfZBmLKhHrfzdv6MvJHTRtfiUg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nAU6wTNjtajIT/PXkVH2o1l5ki14ETpvxojrYz7qm38OG+sxVsphMdWw1MqNz4+if L/frOi8/7RUhJZZU5sPE9OfAn7hgA7eTtX7JIB42gN0Xoh9zFnUHsHNBBDlJ8bRGBq y8HRpG0uPUSEJS6wZVk/32bJwXXc+ljjJt0xJYfA= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p4.samsung.com (KnoxPortal) with ESMTP id 20230419115449epcas5p4b6fef3a1490cdf7179b603d68e3a331e~XU-M4v5Wx0259102591epcas5p4q; Wed, 19 Apr 2023 11:54:49 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.176]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4Q1fNS1RwPz4x9Pw; Wed, 19 Apr 2023 11:54:48 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 21.9D.09540.886DF346; Wed, 19 Apr 2023 20:54:48 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230419114751epcas5p19249dff6e6e2c37795c80f973fd7eee3~XU5HWS7WI1730317303epcas5p1C; Wed, 19 Apr 2023 11:47:51 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230419114751epsmtrp1e418d528391252100c289c23958e45e2~XU5HU-fbr1843518435epsmtrp1Z; Wed, 19 Apr 2023 11:47:51 +0000 (GMT) X-AuditID: b6c32a4a-4afff70000002544-95-643fd688b523 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 77.93.08609.7E4DF346; Wed, 19 Apr 2023 20:47:51 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230419114748epsmtip19e3a900286e835c811f8e90d86182423~XU5ET-Z4x2496324963epsmtip1F; Wed, 19 Apr 2023 11:47:48 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v10 7/9] dm: Add support for copy offload Date: Wed, 19 Apr 2023 17:13:12 +0530 Message-Id: <20230419114320.13674-8-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230419114320.13674-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02TeVATVxzH+3bDZqHGWRHGZ2pbGkYtUEKCITwuYabgrFoZevzRsX/gDlmO AkkmR7m0TUSpoICcrUENSkcEFUaucopFkUuEIUoHKEgZcFSMiDClqJQSAq3/fX7H93e9eSRu f4rLJ2PkGlYlZ+IEhB2n7pbLTvcTg4Ey0ePrBKrsvoOjo6eXcHRlNJtA07deAlT4YhFH462B qOV5kQ0autmAoeaLuRgqu9KOoaYLsxhqXzYTKLdtEKCpBwYMtQy7oeaWLg4yNZ4lkPHSFBe1 5aViqH5SD1DdayOOKqZnOKhz+D3Ut9RhEwRp0/39tOFhL0E3GEa5dN/YdQ5t6tXSVeXpBF39 yw9005COoDNTnxP0zI0HBJ1VUw7o6p4Ueq7qA7pq0oyFbTwY6x/NMjJW5cTKIxSyGHlUgGD/ l+GfhntJRWJ3sQ/yFjjJmXg2QBD8WZj7npi4lQsInL5j4rQrrjBGrRZ47PZXKbQa1ilaodYE CFilLE4pUQrVTLxaK48SylmNr1gk8vRaSTwUGz1ZUgKUNfzEiceduA40OmYAWxJSEniq/w4n A9iR9lQTgOdzytaMlwDW6qfXjDkAnxaP2KxL8otMhDXQCGCf7hlhCdhTxzGYNSHLACRJUG6w Z5m05DhQaTicnUpfrYRTAxh8UzODWwSbKR84d21gtSqH2g7H7w+tMo/yhaUlr3BLIUh5wOyH myxuW8oPlt7Iw60pm2DXmUmOhXHqQ5haW4Rb6kOqxBaaF65i1kmDobG0FbfyZvi0o4ZrZT58 kp22xgmwLP8yYRUfA9DwuwFYA4HweHf26hA45QIrGz2s7vdhQXcFZm28EWa+nlzrxYP159fZ GV6tLCasvBUOLujXmIY9f2Vg1stlAdhU2M05DZwMby1keGshw/+tiwFeDraySnV8FKv2UnrK 2YT/njlCEV8FVr+F67568Of4C2EbwEjQBiCJCxx4d0N8ZfY8GZOUzKoU4SptHKtuA14rB8/B +Y4RipV/JdeEiyU+IolUKpX47JKKBVt4OwO6IuypKEbDxrKsklWt6zDSlq/DPE8W6oJ+3fBz inna8/KjeZQZqQ+u/7x9b/zI3/cOG/v8Q0JTooeoNO5NRh+7QRKEA5KhyTfN7t7mAZ1v6xHj SOS2tJRDJ0w970qnL4REmKuEs5ec60IXOz+e65h1EO7+YmZJdDCL+0eu//jXJ9+p7j98+5ue A3NuttWu9Hx/KG/gWWJHcmT695pCfUHyaC9Wm9g/fG7q0UI+Xzh/jQ/ctBNnikSlvq29LpRz YKaRPHskM2dB3Ji6/OP2eYef/DySbjumNW175ZddYfeJuJkdKx08MP/RV//sSeof6s7bV3Ds yW+mgC2DA0xLTHKDZq8qXHrOc8e9sF3e3x69m9C3OJYbJOCooxmxK65SM/8CI3yEVZ8EAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02Rf0yMcRzH932ep6fn4uxx/fC9apkzP3ZWZJpv+ZERHmPJH0YZ7uaeFeqc u87vqWSjU8rVqKuUSnXXMJVc3JFLiOzoynYXSi6ZSj+uLWmFu9j899r79f58Pn98KJxnInyp g9JEVi4VxwtID6KuURAQ2NsWLlmWNTof3Xn5DEfnsiZxVPUhk0R9jSMAXR0ax1HX43Bk/J7v hqwN9RgylKgxpK1qwtDDG8MYavo1QCK16R1APe0aDBltS5DB2Ewgy4MCEhWV97gjU3YqhvT2 FIDqJopwdLtvkEAvbH7IPPncbR1kLG1bGU3na5Kp13xwZ8wf7xKM5bWSqdalkUxNWRLz0JpM Mhmp30lm8FE7yVyu1QGm5tVpxlEdwFTbB7CoWTEeqyVs/MFjrHzpWpFHnL20FMhqfU90f32B J4MH3irAoSC9AubkW0gV8KB4tB7Asif38WnBh+WTT/+yJ9RO9bpPl1IxaFRdBCpAUSS9BL76 RTlzLzoLh62dya5NON2FQXPvqJtz2pMOhY5brS4m6AWwq83qYi4dBitKf+LORZBeCjM7Zztj Dr0KVjzKdh3m/amkm6bAdH02bM6zE07G6bkw9V4+ngVozX9K858qBpgO8FmZIiE2QREsWy5l jwcpxAkKpTQ26MCRhGrgerNQqAcG3VCQCWAUMAFI4QIvbsvGMAmPKxGfPMXKj+yXK+NZhQn4 UYRgDveNqnk/j44VJ7KHWVbGyv9ZjOL4JmPlV/KCpyICaiPCutMtaYW7ZlRmYCLOuu4c9cJF UeqNXfXbDGMF45LbSSPplZ9ztEHvSDyiBno3fLQP7NYj2Bi5cy/G507tSnGIvIo3ZPhfQwEV uZttcSVfrmsL40Wx/mWd4Rda2yr4jw3ejsCWe4sLA/fYrMdvPn2/KeSyT4ZxWPSytoM/sSN8 3oRx7JsmNL09LXrm2TXD/XXtMR1q8/bNsvuO8TkWn9ZPE9FeX2MKFnL0N/fdamw545MTEuk2 FtmwXizN6x+2zli5vq5HvKAD+JaURWf2s99WLtv29pBnWqSsv8+2yFEojFJajmZfz5WiuPMh Qu2Wk5d+NFm+PFcKCEWcOFiIyxXi3zWsV8RVAwAA X-CMS-MailID: 20230419114751epcas5p19249dff6e6e2c37795c80f973fd7eee3 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230419114751epcas5p19249dff6e6e2c37795c80f973fd7eee3 References: <20230419114320.13674-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Before enabling copy for dm target, check if underlying devices and dm target support copy. Avoid split happening inside dm target. Fail early if the request needs split, currently splitting copy request is not supported. Signed-off-by: Nitesh Shetty --- drivers/md/dm-table.c | 41 +++++++++++++++++++++++++++++++++++ drivers/md/dm.c | 7 ++++++ include/linux/device-mapper.h | 5 +++++ 3 files changed, 53 insertions(+) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 1398f1d6e83e..b3269271e761 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1867,6 +1867,39 @@ static bool dm_table_supports_nowait(struct dm_table *t) return true; } +static int device_not_copy_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + struct request_queue *q = bdev_get_queue(dev->bdev); + + return !blk_queue_copy(q); +} + +static bool dm_table_supports_copy(struct dm_table *t) +{ + struct dm_target *ti; + unsigned int i; + + for (i = 0; i < t->num_targets; i++) { + ti = dm_table_get_target(t, i); + + if (!ti->copy_offload_supported) + return false; + + /* + * target provides copy support (as implied by setting + * 'copy_offload_supported') + * and it relies on _all_ data devices having copy support. + */ + if (!ti->type->iterate_devices || + ti->type->iterate_devices(ti, + device_not_copy_capable, NULL)) + return false; + } + + return true; +} + static int device_not_discard_capable(struct dm_target *ti, struct dm_dev *dev, sector_t start, sector_t len, void *data) { @@ -1949,6 +1982,14 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, q->limits.discard_misaligned = 0; } + if (!dm_table_supports_copy(t)) { + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + q->limits.max_copy_sectors = 0; + q->limits.max_copy_sectors_hw = 0; + } else { + blk_queue_flag_set(QUEUE_FLAG_COPY, q); + } + if (!dm_table_supports_secure_erase(t)) q->limits.max_secure_erase_sectors = 0; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 3b694ba3a106..ab9069090a7d 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1720,6 +1720,13 @@ static blk_status_t __split_and_process_bio(struct clone_info *ci) if (unlikely(ci->is_abnormal_io)) return __process_abnormal_io(ci, ti); + if ((unlikely(op_is_copy(ci->bio->bi_opf)) && + max_io_len(ti, ci->sector) < ci->sector_count)) { + DMERR("Error, IO size(%u) > max target size(%llu)\n", + ci->sector_count, max_io_len(ti, ci->sector)); + return BLK_STS_IOERR; + } + /* * Only support bio polling for normal IO, and the target io is * exactly inside the dm_io instance (verified in dm_poll_dm_io) diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index a52d2b9a6846..04016bd76e73 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -398,6 +398,11 @@ struct dm_target { * bio_set_dev(). NOTE: ideally a target should _not_ need this. */ bool needs_bio_set_dev:1; + + /* + * copy offload is supported + */ + bool copy_offload_supported:1; }; void *dm_per_bio_data(struct bio *bio, size_t data_size); From patchwork Wed Apr 19 11:43:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13216709 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD9FCC77B75 for ; Wed, 19 Apr 2023 11:55:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233192AbjDSLzm (ORCPT ); Wed, 19 Apr 2023 07:55:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233145AbjDSLy7 (ORCPT ); Wed, 19 Apr 2023 07:54:59 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 103C315474 for ; Wed, 19 Apr 2023 04:54:55 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230419115454epoutp0170d121a5ae65e76ba7f0f4036d7bacf0~XU-Q-MAhv1494514945epoutp01s for ; Wed, 19 Apr 2023 11:54:54 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230419115454epoutp0170d121a5ae65e76ba7f0f4036d7bacf0~XU-Q-MAhv1494514945epoutp01s DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681905294; bh=cp2ikyfpBdIhGRYkH19yhVPh/sP+Xdj/kjqikt163DM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EY6qvk7TzRqRxwm7hJ015z3ImKC0JmnhvBHdL44aKGa8rS4A5QByhu2/HTJvl3ES8 i2tcTEZFAgmYjvT/+itmWPq3gVPGfnlzF4e2uyHUSFsfxjUYTheMIrE0GiXzjMCc71 HSg5qlWXlLV3Jjva96JNaHjNMJc59UFFJiK3Jtvk= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230419115453epcas5p38f000f783a08296ce1442ed383ed5af3~XU-QVDK4C2577325773epcas5p3M; Wed, 19 Apr 2023 11:54:53 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.176]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4Q1fNW5y7Qz4x9Pr; Wed, 19 Apr 2023 11:54:51 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id E4.9D.09540.B86DF346; Wed, 19 Apr 2023 20:54:51 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230419114801epcas5p2eb7e9c375817d827d5175468de34f0cb~XU5QHChIv1524715247epcas5p2l; Wed, 19 Apr 2023 11:48:01 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230419114801epsmtrp1476d5be0b44a2a0755f0370122e2d9eb~XU5QFxr0I1843518435epsmtrp1j; Wed, 19 Apr 2023 11:48:01 +0000 (GMT) X-AuditID: b6c32a4a-70dfa70000002544-a1-643fd68b68b7 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 4A.93.08609.0F4DF346; Wed, 19 Apr 2023 20:48:00 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230419114757epsmtip146c11a6792a756842f64d0b81903c6bf~XU5Mf0aLU2050920509epsmtip1S; Wed, 19 Apr 2023 11:47:57 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v10 8/9] dm: Enable copy offload for dm-linear target Date: Wed, 19 Apr 2023 17:13:13 +0530 Message-Id: <20230419114320.13674-9-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230419114320.13674-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Te0xTZxTfd297W0iKl8LmR/ci3YNhR6EM6gfyMoq7QxbZiPxhNN1NewMM aJs+QHCLRR4BfPDSMQoTVIKzqASojIdlrAwRHDORARGDMFfYAxWBCSGusJbi5n+/8/vO7/zO OV8OF+cXcwTcVKWO0SjpdCHhzmrv8/cLOD4WrQjKHxKj5qEbODpWZsdR02Qpgeb6FgH66skq jqa/j0aWxzVsdLe3E0PXz1dg6FJTP4a6zy1gqH/9EYEqrGMAzYwaMWSZEKHrlkEWGumqJVBd 4wwHWSvzMNRhywWo/Vkdjq7OzbPQzYlX0W37ADsGUiO/7KWMU8ME1Wmc5FC377ewqJFhPdVq KiaotoajVPddA0GdzHtMUPM9owR1ymwCVNutI9RS6xtUq+0RluBxIC0ihaEVjMaXUcpVilRl cqRwb6JslyxUGiQJkISh7UJfJZ3BRAp3xycE7ElNd2xA6JtJp+sdVAKt1QoDoyI0Kr2O8U1R aXWRQkatSFeHqMVaOkOrVyaLlYwuXBIUFBzqSPwsLaXKVMVW57IPz9p/IAzga1YJ4HIhGQJ7 2hJKgDuXT3YDaLl8heMKFgG8s9ZIlAA3R7AEYFev3ImdgsrRuk2+C8C2p1yXoACDXfb2jaoE KYK31jd4b7IQhwszxSxngJN3MPiPeR53qr3IWDh4fo3lxCzyHbhwrXcD88hwWGZrIVztBcLS KU8n7UbugBd7KnFXiiccrLZtpOPkmzDvWg3urA/JOjdYZT5BuDrdDccNw5vYC/41YOa4sAD+ WVq4ibPgpdPfEi5xPoDGcSNwPUTDgqFS3NkETvrD5q5AF/06PDN0FXMZe8CTz2yYi+fBjrPP 8VvwcnP9pq8PHFvJ3ZyFgg1nRK5lnQJwYW6VXQZ8jS/MY3xhHuP/zvUANwEfRq3NSGa0oepg JZP13x/LVRmtYOMmtsV1gF+nn4itAOMCK4BcXOjN+yk2XMHnKejsHEajkmn06YzWCkId+y7H BS/LVY6jUupkkpCwoBCpVBoS9oFUItzK84sclPPJZFrHpDGMmtE812FcN4EB+47WZSUtfVyQ xEmSH2+hLk738xRMzJZvjm6J5qZ9aZ5iPaw4JIt5lz29JrFHLt970K23fLE8UYQO/ahM3LF2 MHCrT8z99wqjgstPPBTON/dNdWtmpYGrT6kPDTm5WtuAELpNFyX18ffXAHoyy/z5cL7J+8LZ gwNRr436Uce48tPEhI+lc2dS277IcXH9in02YPGwx7nkcjr205Xfx4bcFWnc2oa/398z2UzL DDli4P0gu5oXX50iCEaviOKkff5FF/jR8S+JboZmNmVmCxLj8MyP1oPLl/9YXXs73NxBY2D7 Pc+fE0zY/gOyTEPjziteEUc4nE+sv+3Lrp1jbiyIhCxtCi3Zhmu09L/Q6E0JnAQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02Ra0hTYRzGe885Ox4Hs9MSfVObtehDSpZi8GqmVqiHkqwIArvYagcnzTk2 LbMkbRBkmUMjdWam1cQl2TTFyyxb3pa38JJleaOZlTg198XErZwEfXt4fs/z/D/8KZxvJDyo BFkyq5CJpEKSS9S9FQp2zg+GiXdbi0JQ1bt2HN1Qr+Do2WgOiWbe/gLo/vwSjiZeh6FmSxEH fWppwJChLBdDFc/aMNRUuoChNvssiXKNHwCaGtJgqHnEFxmaTQQaaHxAohLtlBMy5qkwVG/O BKhuuQRHz2fmCNQ54on6Vjo44ZAZGDzMaMZ7SKZBM+rE9I3pCWagJ4Wp1t0imZon15mmTxkk k62ykMzcqyGSuftSB5iarqvMYrWAqTbPYkddYrkhYlaacIlV7Ao9x5Xk6/I58kxO6reVN2QG KCCygDMF6UCYN1RCZgEuxafrAfz+sRxbA5ugdqUVX9MbYYVt2mktpMLgzGjlX0BRJO0Lu+zU qu9Kq3HYP57hWMLpCQz2TVs5q+2NdAQ0ldkc5wh6O1yobXFoHh0M1WY9uToE6V0wZ3zDqu1M 74Xlr/Ich/l/I3eMNrAW3wBNhWZHFae9oaq2CFcDWvMf0vyHHgFMBzaxcmVifKLSXx4gYy/7 KUWJyhRZvN+FpMRq4Hizj089MOjm/YwAo4ARQAoXuvK6I4LFfJ5YdCWNVSTFKVKkrNIIPClC 6M57n2WK49PxomT2IsvKWcU/ilHOHhlY59abr5fm08/GhCD7xMlrj3sq0iUhQT4zB7PFEcqX 3sVF0vsCVlXu+8Z2Wuu0bfyA3SPotibetP78noLfWw5MVgmx4bljP7WD52vaR+CZL43Lbmxq IPQ+U+hle1ozdbU4/9SP9IAk16Z9tMRWuWT4udx3OTl0gccXp81N+Ftj9IV6uTRHgLoNOxYK Jq3am3pL/+eH97idA7cY21JvvnOBb7TF3a33+LRK3TGyOXQxHIwJetbx/LfebvULjj0S5akv OdFSK4mdtNmHItUpluj9L1KjSiWHjqmqIs05dd2HYABuGlQjN1nl8W+/JGUue6z9ix6jX4V5 XieGXRrVQkIpEfn74Aql6A8UcQ2tVQMAAA== X-CMS-MailID: 20230419114801epcas5p2eb7e9c375817d827d5175468de34f0cb X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230419114801epcas5p2eb7e9c375817d827d5175468de34f0cb References: <20230419114320.13674-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Setting copy_offload_supported flag to enable offload. Signed-off-by: Nitesh Shetty --- drivers/md/dm-linear.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c index f4448d520ee9..1d1ee30bbefb 100644 --- a/drivers/md/dm-linear.c +++ b/drivers/md/dm-linear.c @@ -62,6 +62,7 @@ static int linear_ctr(struct dm_target *ti, unsigned int argc, char **argv) ti->num_discard_bios = 1; ti->num_secure_erase_bios = 1; ti->num_write_zeroes_bios = 1; + ti->copy_offload_supported = 1; ti->private = lc; return 0; From patchwork Wed Apr 19 11:43:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nitesh Shetty X-Patchwork-Id: 13216710 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CC0DC77B73 for ; Wed, 19 Apr 2023 11:55:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233084AbjDSLz4 (ORCPT ); Wed, 19 Apr 2023 07:55:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45842 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233099AbjDSLzO (ORCPT ); Wed, 19 Apr 2023 07:55:14 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 045B115616 for ; Wed, 19 Apr 2023 04:54:58 -0700 (PDT) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230419115457epoutp03e85d49f856ca24367ff685262787177f~XU-T2jhDO1371913719epoutp03E for ; Wed, 19 Apr 2023 11:54:57 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230419115457epoutp03e85d49f856ca24367ff685262787177f~XU-T2jhDO1371913719epoutp03E DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681905297; bh=/MVZrDYgNIh/mjvCvKoJUXnOyDfXEO11lYxG5YriEZA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GK2s24DISr8KuidwXO1T2DqTNYOmvHPFeThTGJS0iUeAkwqwzMBaWoFyvf1i+Sfuq uS0lUdNYFIBe1NNb6uwmcS0VYOZpgMmk+CSdkRiR3HIS01nknMzhB2JnpNeF4jdRSE FIOOPvPae8BpPrTQByCmQNoS+wVBp0nZfdDu5PdQ= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230419115456epcas5p2fe3ba66a84ae572af0d21e62bc772135~XU-TbEuka1844418444epcas5p26; Wed, 19 Apr 2023 11:54:56 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.177]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4Q1fNb2m7sz4x9Py; Wed, 19 Apr 2023 11:54:55 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id C6.34.09987.F86DF346; Wed, 19 Apr 2023 20:54:55 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230419114810epcas5p3b10b7eddf9dae9ddc41940f09b483813~XU5Yporf_0874908749epcas5p3G; Wed, 19 Apr 2023 11:48:10 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230419114810epsmtrp142f95b564da6872f61544594de8071cf~XU5YofKW61896418964epsmtrp1C; Wed, 19 Apr 2023 11:48:10 +0000 (GMT) X-AuditID: b6c32a4b-a67fd70000002703-72-643fd68fccb9 Received: from epsmtip1.samsung.com ( [182.195.34.30]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 4E.97.08279.AF4DF346; Wed, 19 Apr 2023 20:48:10 +0900 (KST) Received: from green245.sa.corp.samsungelectronics.net (unknown [107.99.41.245]) by epsmtip1.samsung.com (KnoxPortal) with ESMTPA id 20230419114806epsmtip15324f8047867f14d72c9bdf9f1f60c86~XU5VStBYa2332323323epsmtip1n; Wed, 19 Apr 2023 11:48:06 +0000 (GMT) From: Nitesh Shetty To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Damien Le Moal , Vincent Fu , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v10 9/9] null_blk: add support for copy offload Date: Wed, 19 Apr 2023 17:13:14 +0530 Message-Id: <20230419114320.13674-10-nj.shetty@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230419114320.13674-1-nj.shetty@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Te0xbZRTPd297W3CFy8P4DZnDmmU85FFWygeOIQ7wEtSQLCZmQbChN4UA bW3pcBId7zAYYzDArMzR6RwUykNgjJWHGwR5CwkDBeQhDze3DFj3ILMCtlzQ/ff7/c453znn 9+VwcfvLHCdugiyFVsrESXzCmtXa43bYs2gyWOJj7LBBDYM/4yjzwiaOameLCPSwxwhQ+foL HJlGRnG08FMw6lytYKOp27cw1PFdCYZ0tb0Yar/6GEO9248IVNI9CdDKhAZDndMeqKNzgIXG DZcJVHl9hYO6L2ZhqG05A6BWUyWO6h+usVD/9OtoqSAPoNHNPva7TtT43UhKMz9CULc0sxxq dO5HFjU+oqaaas4SVPO1M1T7VDpBFWatmhNy5tnUWtcEQZ1vqQFU81Aa9aTpDapp+REWZXsy 8Wg8LZbQShdaFieXJMikQfzIE7HHY/1EPgJPQQDy57vIxMl0ED/0gyjP8IQksyF8l1PiJLVZ ihKrVHzvY0eVcnUK7RIvV6UE8WmFJEkhVHipxMkqtUzqJaNTAgU+Pr5+5sTPEuPv11WxFd8G f5GdN4qngxVhPrDiQlIIx/4Z5+QDa6492Q7glDaXzRAjgBV5dwBDngO4/EMmsVeSUfcbwQQ6 AZzRGHGG5GBwVL/FygdcLkF6wKFtrkV3JHNx+HjlLMtCcFKPw4V1E7A85UAGwxddZSwLZpGH oK7iHMeCeeQ7sHp6BbM8BElvWDRvZ5GtzHJV10WcSbGDA5eWd0px8iDMulGxMwQkm63gXPaf bGbUUPi3qYXFYAf4oK+Fw2An+FdR7i5OhbrSaoIpzgZQ86sGMIFgmDNYhFuGwEk32GDwZuQD sGywHmMa28BC0zLG6DzYdmUPvwX1Ddpdu/bDyY2MXUzBkmc5uwafB1B3bQZcAC6alxbSvLSQ 5v/WWoDXgP20QpUspVV+iiMyOvW/f46TJzeBnTNxj2wDiwvrXt0A44JuALk435E3HBYosedJ xKe/pJXyWKU6iVZ1Az+z4cW406txcvOdyVJiBcIAH6FIJBIGHBEJ+K/xDgcNxNmTUnEKnUjT Clq5V4dxrZzSMfu3i4lGf35Irr7g3r6bDc6+gVfebz02vRBeVfv75xNd4eWNlbdfic5zfR5z AeW2xGgXP+nvaSrNsbmrhalrb+rqn6TBG6cChALOmW3pgW+eHgoYc/7jl5D0j+fC+13j7qgf 0Lw5laeo3HV4yXEpsjE+ctNoJ6sN/cgdi0itc19Uv6d/aviKN+w/FJXe67BqWLW6et9F4FHY 9SGh8/16ONr/2RQdRri2xxSknjgXlpNh083S62yr08Y2JBEzUyHVtupE6b1Y0F+6NXvSmdto 6FMWa/dJRxsK2J/a6Oc1Gb1sg9H6Zual69+bTi+7FcUG5G95lx3sj4vmqjd0DhFt8mQ+SxUv FrjjSpX4XzyXjwevBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02Ra0hTcRjG+59zdnYcLc+m0V8lq0FSRqZk9ceGMyo7dKOwL2ZSSw/mfWxa WVRbWtk0My3DWWgqLq+lZrq8VFMrK5l5KyWdkqNE5iwpMVFzjsBvz/v8nvd9PrwULuwgnKnw mDhWHiONEpE84nmzyHXz325JqGfqKImevH+Do6vpszgqHbhNorHmXwBlTUzjaKbdgKOhlxLU OJ7DQX2vdBhqyM/AUHFpK4bqH/3EUOu8mUQZ+l6ATD0aDDX2b0INjW0E6nrxgES5RSYu0mcm YqhuRAXQ85lcHFWMWQj0rt8FfUtJBsgw+5bj58x0dR9gNMZ2ktFpBriMYbCSYLra45mqkpsk U114hanvU5LMrcTxhcA1I4exNPWQTNqzEsBUf7jITFa5MlUjZuyI/XGeOJSNCj/Lyrf4nuKd +VGu5cgeSs4nJRtwJTB5q4EdBWlvqCr/QqoBjxLS9QA2t01xbcAJFs224DbtAIvnvnNtoUQM ZtaqFwaKIulN8MM8ZfUd6XQcdhqVpHUBpxtwWJy3y6odaAmcbrpHWDVBr4fFOamLBXx6J3zc b8KsdyC9Bd42Cqy23YKtbcpc7BXSPjBVPwdscQFsyx4hbOfXwMSaHDwd0JolSLME5QGsBDix MkV0WLTCS+YVw57zUEijFfExYR4hsdFVYPHp7hvrQG3JhIceYBTQA0jhIkf+PrFvqJAfKk24 wMpjT8rjo1iFHrhQhGgVv0PddlJIh0nj2EiWlbHy/xSj7JyVWNqpG/LA1kP+E8I66d1cex/0 60bOdp9PQ5Eha0V8nRepTcjMsjMOtFfMsC2/z1eeHr+8p2Zq9bJgUeClr70rXCTm6d4f+UEf C7PPqTIEY/al3Y6elQJVSM+qmnXu/uSwutBNpjV4kBnkn9jlh7UJrkYsJWD/DoeAhGCZZ++h j4LG0b3cstTfZjytLMjk6J2XJubyXr9O4URYfLFa3VFjXOw+3bA2r3tr/aibr2inf+SJyXdh k9UbjsbrAgqcWhuOPJWYx7XK8Cmx3qewYHfSSnFdCieOf+zzEGO5479rSnXB0nF9c74f96Bx /Tx0v9+5tnrw8l7hNj+niCD9ntVcEaE4I/Vyx+UK6T+V/wzJYwMAAA== X-CMS-MailID: 20230419114810epcas5p3b10b7eddf9dae9ddc41940f09b483813 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230419114810epcas5p3b10b7eddf9dae9ddc41940f09b483813 References: <20230419114320.13674-1-nj.shetty@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Implementaion is based on existing read and write infrastructure. copy_max_bytes: A new configfs and module parameter is introduced, which can be used to set hardware/driver supported maximum copy limit. Suggested-by: Damien Le Moal Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu --- drivers/block/null_blk/main.c | 108 ++++++++++++++++++++++++++++-- drivers/block/null_blk/null_blk.h | 8 +++ 2 files changed, 111 insertions(+), 5 deletions(-) diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index 5d54834d8690..9028aae1efbd 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -157,6 +157,10 @@ static int g_max_sectors; module_param_named(max_sectors, g_max_sectors, int, 0444); MODULE_PARM_DESC(max_sectors, "Maximum size of a command (in 512B sectors)"); +static unsigned long g_copy_max_bytes = COPY_MAX_BYTES; +module_param_named(copy_max_bytes, g_copy_max_bytes, ulong, 0444); +MODULE_PARM_DESC(copy_max_bytes, "Maximum size of a copy command (in bytes)"); + static unsigned int nr_devices = 1; module_param(nr_devices, uint, 0444); MODULE_PARM_DESC(nr_devices, "Number of devices to register"); @@ -409,6 +413,7 @@ NULLB_DEVICE_ATTR(home_node, uint, NULL); NULLB_DEVICE_ATTR(queue_mode, uint, NULL); NULLB_DEVICE_ATTR(blocksize, uint, NULL); NULLB_DEVICE_ATTR(max_sectors, uint, NULL); +NULLB_DEVICE_ATTR(copy_max_bytes, uint, NULL); NULLB_DEVICE_ATTR(irqmode, uint, NULL); NULLB_DEVICE_ATTR(hw_queue_depth, uint, NULL); NULLB_DEVICE_ATTR(index, uint, NULL); @@ -550,6 +555,7 @@ static struct configfs_attribute *nullb_device_attrs[] = { &nullb_device_attr_queue_mode, &nullb_device_attr_blocksize, &nullb_device_attr_max_sectors, + &nullb_device_attr_copy_max_bytes, &nullb_device_attr_irqmode, &nullb_device_attr_hw_queue_depth, &nullb_device_attr_index, @@ -656,7 +662,8 @@ static ssize_t memb_group_features_show(struct config_item *item, char *page) "poll_queues,power,queue_mode,shared_tag_bitmap,size," "submit_queues,use_per_node_hctx,virt_boundary,zoned," "zone_capacity,zone_max_active,zone_max_open," - "zone_nr_conv,zone_offline,zone_readonly,zone_size\n"); + "zone_nr_conv,zone_offline,zone_readonly,zone_size," + "copy_max_bytes\n"); } CONFIGFS_ATTR_RO(memb_group_, features); @@ -722,6 +729,7 @@ static struct nullb_device *null_alloc_dev(void) dev->queue_mode = g_queue_mode; dev->blocksize = g_bs; dev->max_sectors = g_max_sectors; + dev->copy_max_bytes = g_copy_max_bytes; dev->irqmode = g_irqmode; dev->hw_queue_depth = g_hw_queue_depth; dev->blocking = g_blocking; @@ -1271,6 +1279,78 @@ static int null_transfer(struct nullb *nullb, struct page *page, return err; } +static inline void nullb_setup_copy_read(struct nullb *nullb, struct bio *bio) +{ + struct nullb_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + + token->subsys = "nullb"; + token->sector_in = bio->bi_iter.bi_sector; + token->nullb = nullb; + token->sectors = bio->bi_iter.bi_size >> SECTOR_SHIFT; +} + +static inline int nullb_setup_copy_write(struct nullb *nullb, + struct bio *bio, bool is_fua) +{ + struct nullb_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + sector_t sector_in, sector_out; + void *in, *out; + size_t rem, temp; + unsigned long offset_in, offset_out; + struct nullb_page *t_page_in, *t_page_out; + int ret = -EIO; + + if (unlikely(memcmp(token->subsys, "nullb", 5))) + return -EINVAL; + if (unlikely(token->nullb != nullb)) + return -EINVAL; + if (WARN_ON(token->sectors != bio->bi_iter.bi_size >> SECTOR_SHIFT)) + return -EINVAL; + + sector_in = token->sector_in; + sector_out = bio->bi_iter.bi_sector; + rem = token->sectors << SECTOR_SHIFT; + + spin_lock_irq(&nullb->lock); + while (rem > 0) { + temp = min_t(size_t, nullb->dev->blocksize, rem); + offset_in = (sector_in & SECTOR_MASK) << SECTOR_SHIFT; + offset_out = (sector_out & SECTOR_MASK) << SECTOR_SHIFT; + + if (null_cache_active(nullb) && !is_fua) + null_make_cache_space(nullb, PAGE_SIZE); + + t_page_in = null_lookup_page(nullb, sector_in, false, + !null_cache_active(nullb)); + if (!t_page_in) + goto err; + t_page_out = null_insert_page(nullb, sector_out, + !null_cache_active(nullb) || is_fua); + if (!t_page_out) + goto err; + + in = kmap_local_page(t_page_in->page); + out = kmap_local_page(t_page_out->page); + + memcpy(out + offset_out, in + offset_in, temp); + kunmap_local(out); + kunmap_local(in); + __set_bit(sector_out & SECTOR_MASK, t_page_out->bitmap); + + if (is_fua) + null_free_sector(nullb, sector_out, true); + + rem -= temp; + sector_in += temp >> SECTOR_SHIFT; + sector_out += temp >> SECTOR_SHIFT; + } + + ret = 0; +err: + spin_unlock_irq(&nullb->lock); + return ret; +} + static int null_handle_rq(struct nullb_cmd *cmd) { struct request *rq = cmd->rq; @@ -1280,13 +1360,20 @@ static int null_handle_rq(struct nullb_cmd *cmd) sector_t sector = blk_rq_pos(rq); struct req_iterator iter; struct bio_vec bvec; + bool fua = rq->cmd_flags & REQ_FUA; + + if (rq->cmd_flags & REQ_COPY) { + if (op_is_write(req_op(rq))) + return nullb_setup_copy_write(nullb, rq->bio, fua); + nullb_setup_copy_read(nullb, rq->bio); + return 0; + } spin_lock_irq(&nullb->lock); rq_for_each_segment(bvec, rq, iter) { len = bvec.bv_len; err = null_transfer(nullb, bvec.bv_page, len, bvec.bv_offset, - op_is_write(req_op(rq)), sector, - rq->cmd_flags & REQ_FUA); + op_is_write(req_op(rq)), sector, fua); if (err) { spin_unlock_irq(&nullb->lock); return err; @@ -1307,13 +1394,20 @@ static int null_handle_bio(struct nullb_cmd *cmd) sector_t sector = bio->bi_iter.bi_sector; struct bio_vec bvec; struct bvec_iter iter; + bool fua = bio->bi_opf & REQ_FUA; + + if (bio->bi_opf & REQ_COPY) { + if (op_is_write(bio_op(bio))) + return nullb_setup_copy_write(nullb, bio, fua); + nullb_setup_copy_read(nullb, bio); + return 0; + } spin_lock_irq(&nullb->lock); bio_for_each_segment(bvec, bio, iter) { len = bvec.bv_len; err = null_transfer(nullb, bvec.bv_page, len, bvec.bv_offset, - op_is_write(bio_op(bio)), sector, - bio->bi_opf & REQ_FUA); + op_is_write(bio_op(bio)), sector, fua); if (err) { spin_unlock_irq(&nullb->lock); return err; @@ -2157,6 +2251,10 @@ static int null_add_dev(struct nullb_device *dev) dev->max_sectors = queue_max_hw_sectors(nullb->q); dev->max_sectors = min(dev->max_sectors, BLK_DEF_MAX_SECTORS); blk_queue_max_hw_sectors(nullb->q, dev->max_sectors); + blk_queue_max_copy_sectors_hw(nullb->q, + dev->copy_max_bytes >> SECTOR_SHIFT); + if (dev->copy_max_bytes) + blk_queue_flag_set(QUEUE_FLAG_COPY, nullb->disk->queue); if (dev->virt_boundary) blk_queue_virt_boundary(nullb->q, PAGE_SIZE - 1); diff --git a/drivers/block/null_blk/null_blk.h b/drivers/block/null_blk/null_blk.h index 929f659dd255..3dda593b0747 100644 --- a/drivers/block/null_blk/null_blk.h +++ b/drivers/block/null_blk/null_blk.h @@ -67,6 +67,13 @@ enum { NULL_Q_MQ = 2, }; +struct nullb_copy_token { + char *subsys; + struct nullb *nullb; + sector_t sector_in; + sector_t sectors; +}; + struct nullb_device { struct nullb *nullb; struct config_group group; @@ -107,6 +114,7 @@ struct nullb_device { unsigned int queue_mode; /* block interface */ unsigned int blocksize; /* block size */ unsigned int max_sectors; /* Max sectors per command */ + unsigned long copy_max_bytes; /* Max copy offload length in bytes */ unsigned int irqmode; /* IRQ completion handler */ unsigned int hw_queue_depth; /* queue depth */ unsigned int index; /* index of the disk, only valid with a disk */