From patchwork Tue Apr 11 08:10:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13207168 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF886C76196 for ; Tue, 11 Apr 2023 08:19:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230235AbjDKITL (ORCPT ); Tue, 11 Apr 2023 04:19:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230032AbjDKITB (ORCPT ); Tue, 11 Apr 2023 04:19:01 -0400 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C58A7E7B for ; Tue, 11 Apr 2023 01:18:59 -0700 (PDT) Received: from epcas5p1.samsung.com (unknown [182.195.41.39]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20230411081858epoutp043e4638be51001c544cebf21a5d300ba3~U04chXRV80030600306epoutp04v for ; Tue, 11 Apr 2023 08:18:58 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20230411081858epoutp043e4638be51001c544cebf21a5d300ba3~U04chXRV80030600306epoutp04v DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681201138; bh=azNz458ro0wDubj3yA6JaBDeCE9nCzpzPvVQaV4RgaA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qecUkqtlNj5NHCQ4B/eCYhY6DQ/bDrHCQE8e0jvBtB8UTeDb5MyEyfAEpIIFcotf8 K8IpB5aTe0vMr1h0MZwpI6osWlaA1wSI7bllc3Ln9iYo4UkiC9y34KNJxl1KljNpEi qu5olvbK89WEggX74YqMZQRdU2UUCw04W66HgeAI= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230411081856epcas5p1d40b52ec6c71413049c66a8f4569a722~U04axPziP1310613106epcas5p1Q; Tue, 11 Apr 2023 08:18:56 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.180]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4Pwdz24X9Hz4x9Q2; Tue, 11 Apr 2023 08:18:54 +0000 (GMT) Received: from epcas5p3.samsung.com ( [182.195.41.41]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id 60.AC.09987.EE715346; Tue, 11 Apr 2023 17:18:54 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230411081241epcas5p15b9c54d8f86db6dda2901d15f4c834db~U0y_GV3zo1543815438epcas5p1_; Tue, 11 Apr 2023 08:12:41 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230411081241epsmtrp2985da42dbd36d667b60fc0c204c599e6~U0y_FXbsb2545725457epsmtrp2X; Tue, 11 Apr 2023 08:12:41 +0000 (GMT) X-AuditID: b6c32a4b-7fbff70000002703-b6-643517eee8ee Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id B3.0F.08279.97615346; Tue, 11 Apr 2023 17:12:41 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230411081237epsmtip242be83d7d3286979f87d447cbe28d778~U0y6Rm8JJ2386223862epsmtip2J; Tue, 11 Apr 2023 08:12:37 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v9 1/9] block: Introduce queue limits for copy-offload support Date: Tue, 11 Apr 2023 13:40:28 +0530 Message-Id: <20230411081041.5328-2-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230411081041.5328-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Tf0wTZxjHd3fttRhLTnDhBbbJLjqDjh8dbXmZVMkw7jaMspHFREOway+A lGvttcJGHAWCCBuCTFGKUuemGwVBCxIEC1uZMDAUJrIIWYEsrTMqoDCNUn6spbD53/d53s83 z483Dx/zK+EF8dMZLa1hZEoSX8Np6QoNDZsKECsinx97Gzb2dWMwv3wBg3X2Mhw+6ppBYOWT lxic6NwBLVPVXDjy8w0U3rxYgcLaulsobP/uKQpvLU3isML6BwKdwwYUWka3wpuWXg4cajuH Q+NlJw9avy1AYasjD4EtLiMGGx5Nc+Bvo8FwYKGHGweoobsJlGG8H6duGOw8amDsGoca6tdR ZlMxTjX9kEu1j+hxqrRgCqemO4Zx6kSzCaGabudQs+a3KLNjEk303Z8Rm0bLFLQmhGbkKkU6 kyolE5JS4lPEkkhhmDAGRpMhjCyTlpI7dyeG7UpXujdAhhyRKXXuVKKMZcmI7bEalU5Lh6Sp WK2UpNUKpVqkDmdlmayOSQ1naO37wsjI98Ru8GBG2sjcdvWVxOy5xgZcj9jiSxAfPiBEoKK9 hVuCrOH7Ee0IOP2wH/MGMwi4PVjG8VB+xHMETD7eu+p4XPgjzwtZ3I7h+7g3KEDBP81mzEPh xGbw69+FiOdhPXEMA0+dxRxPgBF3UDDfPO2m+Hx/IhG0joV7DBxiE1hwzCEeLSAgaHUucj0I ICJA2fg6T9qHiAHz54/jXmQd6K1yLHeHERtAwfXq5bYBYfQBlSYD6m11J1i09XC92h887Gnm eXUQmJ2y4F6dCl4MOVd4NSjo7kC8egco7CtbbhMjQkFjW4Q3/SY43deAeuv6glKXY8UqAK01 q5oERbXnVjQAFpt+RVOgq+b6yq6/QcBMmxMpR0IMr8xjeGUew/+lLyCYCQmk1WxmKs2K1VEM nfXfJ8tVmWZk+Si2JLQif008CbciKB+xIoCPkesFz5aiFH4CheyLL2mNKkWjU9KsFRG7930S C3pdrnJfFaNNEYpiIkUSiUQUEyURkgGCzdJeuR+RKtPSGTStpjWrPpTvE6RHM/vXVmSzX48N WkHTKQ3/YkaxXF9RmrvVaecKAs/MJu/7oLDwqKl3sOi+s9EVGu/7WsCEv9x+dYMpaZN4REQG Vm1cVL5TKRDqDuyunwfA3HH0UEBsln9ncd694vkw4Uf5cRfs9cK4wfQXJ+qzlQ+SyjWYdDTZ X+RyfWI1mXz50ZcObZxY5A1cJd498+lZXytx2Mjtu/TTZ7kME7ynvur8x3u6q/FgG7O2JsHn Zd4vB48Yc2r3aodOTRRdadrfEOA4rr/nsidTl7cdHhmNDn5wwEh8+P1Y5ecd27KcX01159vu JFxbOCnd9waH7TgbUafpNP25YMsez3kmv5s/uCT5XUdy2DSZcAumYWX/AuUB5D6dBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrIIsWRmVeSWpSXmKPExsWy7bCSvG6lmGmKwa0LRhbrTx1jtmia8JfZ YvXdfjaL14c/MVpM+/CT2eLBfnuLve9ms1rcPLCTyWLPoklMFitXH2Wy2L3wI5PF0f9v2Swm HbrGaPH06iwmi723tC327D3JYnF51xw2i/nLnrJbHJrczGSx40kjo8W23/OZLda9fs9iceKW tMX5v8dZHSQ8Ll/x9ph1/yybx85Zd9k9zt/byOJx+Wypx6ZVnWwem5fUe+y+2cDm0dv8js3j /b6rbB59W1Yxemw+Xe3xeZOcx6Ynb5kC+KK4bFJSczLLUov07RK4Mm7+sitYG1Dxa/06tgbG c85djJwcEgImEm9al7N3MXJxCAnsZpToe9LCBJGQkDj1chkjhC0ssfLfc6iiRiaJTd/vgxWx CahLHHneygiSEBGYwCxx6X4DG4jDLPCASeL88y+sIFXCAn4Sk9ecAxvFIqAq8ffJLzCbV8BC YsfTf0A1HEAr9CX67wuChDkFLCX+zO1gAwkLAZUs2cMCUS0ocXLmEzCbWUBeonnrbOYJjAKz kKRmIUktYGRaxSiZWlCcm55bbFhgmJdarlecmFtcmpeul5yfu4kRHMtamjsYt6/6oHeIkYmD 8RCjBAezkgjv1//GKUK8KYmVValF+fFFpTmpxYcYpTlYlMR5L3SdjBcSSE8sSc1OTS1ILYLJ MnFwSjUwaf54E1Cx4Pa2ZwtvxQhYLJ72UixSNvjNvTJ3OYXi74rrfe8l6c3tuhoR+uRo9tYj PwL+HK1b3MRd4mei1SMmv2sSj2GXwfpVf4t0Nu7s9v6R8/NRreiyxvuTvkT3qi+rssna4+S4 W4SLxYyjdJf7N2Zv9481SzcHmr95IPfsSW4l19H9jb1Lo5fXqrfGflcUnP14Z8whp7aAXU2x BbzXW/rrvCqeB3ZoF59PvHRS29v0+lLpnauXnJ97eNNjUenzMV47r37u9+P1rD3Ecq3a44bE hIDjXnJ751Ruc95b9qo2MIJf2ifbx7363jOV0M6lqx/KVOtEn5Nc0WL0S+lveEP8xqmfDy5i lbgTLfleiaU4I9FQi7moOBEAk5Qmm1QDAAA= X-CMS-MailID: 20230411081241epcas5p15b9c54d8f86db6dda2901d15f4c834db X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230411081241epcas5p15b9c54d8f86db6dda2901d15f4c834db References: <20230411081041.5328-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Nitesh Shetty Add device limits as sysfs entries, - copy_offload (RW) - copy_max_bytes (RW) - copy_max_bytes_hw (RO) Above limits help to split the copy payload in block layer. copy_offload: used for setting copy offload(1) or emulation(0). copy_max_bytes: maximum total length of copy in single payload. copy_max_bytes_hw: Reflects the device supported maximum limit. Reviewed-by: Hannes Reinecke Signed-off-by: Nitesh Shetty Signed-off-by: Kanchan Joshi Signed-off-by: Anuj Gupta --- Documentation/ABI/stable/sysfs-block | 33 ++++++++++++++ block/blk-settings.c | 24 +++++++++++ block/blk-sysfs.c | 64 ++++++++++++++++++++++++++++ include/linux/blkdev.h | 12 ++++++ include/uapi/linux/fs.h | 3 ++ 5 files changed, 136 insertions(+) diff --git a/Documentation/ABI/stable/sysfs-block b/Documentation/ABI/stable/sysfs-block index c57e5b7cb532..e4d31132f77c 100644 --- a/Documentation/ABI/stable/sysfs-block +++ b/Documentation/ABI/stable/sysfs-block @@ -155,6 +155,39 @@ Description: last zone of the device which may be smaller. +What: /sys/block//queue/copy_offload +Date: April 2023 +Contact: linux-block@vger.kernel.org +Description: + [RW] When read, this file shows whether offloading copy to a + device is enabled (1) or disabled (0). Writing '0' to this + file will disable offloading copies for this device. + Writing any '1' value will enable this feature. If the device + does not support offloading, then writing 1, will result in + error. + + +What: /sys/block//queue/copy_max_bytes +Date: April 2023 +Contact: linux-block@vger.kernel.org +Description: + [RW] This is the maximum number of bytes, that the block layer + will allow for copy request. This will be smaller or equal to + the maximum size allowed by the hardware, indicated by + 'copy_max_bytes_hw'. Attempt to set value higher than + 'copy_max_bytes_hw' will truncate this to 'copy_max_bytes_hw'. + + +What: /sys/block//queue/copy_max_bytes_hw +Date: April 2023 +Contact: linux-block@vger.kernel.org +Description: + [RO] This is the maximum number of bytes, that the hardware + will allow in a single data copy request. + A value of 0 means that the device does not support + copy offload. + + What: /sys/block//queue/crypto/ Date: February 2022 Contact: linux-block@vger.kernel.org diff --git a/block/blk-settings.c b/block/blk-settings.c index 896b4654ab00..23aff2d4dcba 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -59,6 +59,8 @@ void blk_set_default_limits(struct queue_limits *lim) lim->zoned = BLK_ZONED_NONE; lim->zone_write_granularity = 0; lim->dma_alignment = 511; + lim->max_copy_sectors_hw = 0; + lim->max_copy_sectors = 0; } /** @@ -82,6 +84,8 @@ void blk_set_stacking_limits(struct queue_limits *lim) lim->max_dev_sectors = UINT_MAX; lim->max_write_zeroes_sectors = UINT_MAX; lim->max_zone_append_sectors = UINT_MAX; + lim->max_copy_sectors_hw = ULONG_MAX; + lim->max_copy_sectors = ULONG_MAX; } EXPORT_SYMBOL(blk_set_stacking_limits); @@ -183,6 +187,22 @@ void blk_queue_max_discard_sectors(struct request_queue *q, } EXPORT_SYMBOL(blk_queue_max_discard_sectors); +/** + * blk_queue_max_copy_sectors_hw - set max sectors for a single copy payload + * @q: the request queue for the device + * @max_copy_sectors: maximum number of sectors to copy + **/ +void blk_queue_max_copy_sectors_hw(struct request_queue *q, + unsigned int max_copy_sectors) +{ + if (max_copy_sectors > (COPY_MAX_BYTES >> SECTOR_SHIFT)) + max_copy_sectors = COPY_MAX_BYTES >> SECTOR_SHIFT; + + q->limits.max_copy_sectors_hw = max_copy_sectors; + q->limits.max_copy_sectors = max_copy_sectors; +} +EXPORT_SYMBOL_GPL(blk_queue_max_copy_sectors_hw); + /** * blk_queue_max_secure_erase_sectors - set max sectors for a secure erase * @q: the request queue for the device @@ -578,6 +598,10 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, t->max_segment_size = min_not_zero(t->max_segment_size, b->max_segment_size); + t->max_copy_sectors = min(t->max_copy_sectors, b->max_copy_sectors); + t->max_copy_sectors_hw = min(t->max_copy_sectors_hw, + b->max_copy_sectors_hw); + t->misaligned |= b->misaligned; alignment = queue_limit_alignment_offset(b, start); diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index 1a743b4f2958..dccb162cf318 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -213,6 +213,63 @@ static ssize_t queue_discard_zeroes_data_show(struct request_queue *q, char *pag return queue_var_show(0, page); } +static ssize_t queue_copy_offload_show(struct request_queue *q, char *page) +{ + return queue_var_show(blk_queue_copy(q), page); +} + +static ssize_t queue_copy_offload_store(struct request_queue *q, + const char *page, size_t count) +{ + s64 copy_offload; + ssize_t ret = queue_var_store64(©_offload, page); + + if (ret < 0) + return ret; + + if (copy_offload && !q->limits.max_copy_sectors_hw) + return -EINVAL; + + if (copy_offload) + blk_queue_flag_set(QUEUE_FLAG_COPY, q); + else + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + + return count; +} + +static ssize_t queue_copy_max_hw_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors_hw << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_show(struct request_queue *q, char *page) +{ + return sprintf(page, "%llu\n", (unsigned long long) + q->limits.max_copy_sectors << SECTOR_SHIFT); +} + +static ssize_t queue_copy_max_store(struct request_queue *q, + const char *page, size_t count) +{ + s64 max_copy; + ssize_t ret = queue_var_store64(&max_copy, page); + + if (ret < 0) + return ret; + + if (max_copy & (queue_logical_block_size(q) - 1)) + return -EINVAL; + + max_copy >>= SECTOR_SHIFT; + if (max_copy > q->limits.max_copy_sectors_hw) + max_copy = q->limits.max_copy_sectors_hw; + + q->limits.max_copy_sectors = max_copy; + return count; +} + static ssize_t queue_write_same_max_show(struct request_queue *q, char *page) { return queue_var_show(0, page); @@ -591,6 +648,10 @@ QUEUE_RO_ENTRY(queue_nr_zones, "nr_zones"); QUEUE_RO_ENTRY(queue_max_open_zones, "max_open_zones"); QUEUE_RO_ENTRY(queue_max_active_zones, "max_active_zones"); +QUEUE_RW_ENTRY(queue_copy_offload, "copy_offload"); +QUEUE_RO_ENTRY(queue_copy_max_hw, "copy_max_bytes_hw"); +QUEUE_RW_ENTRY(queue_copy_max, "copy_max_bytes"); + QUEUE_RW_ENTRY(queue_nomerges, "nomerges"); QUEUE_RW_ENTRY(queue_rq_affinity, "rq_affinity"); QUEUE_RW_ENTRY(queue_poll, "io_poll"); @@ -638,6 +699,9 @@ static struct attribute *queue_attrs[] = { &queue_discard_max_entry.attr, &queue_discard_max_hw_entry.attr, &queue_discard_zeroes_data_entry.attr, + &queue_copy_offload_entry.attr, + &queue_copy_max_hw_entry.attr, + &queue_copy_max_entry.attr, &queue_write_same_max_entry.attr, &queue_write_zeroes_max_entry.attr, &queue_zone_append_max_entry.attr, diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index e3242e67a8e3..200338f2ec2e 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -298,6 +298,9 @@ struct queue_limits { unsigned int discard_alignment; unsigned int zone_write_granularity; + unsigned long max_copy_sectors_hw; + unsigned long max_copy_sectors; + unsigned short max_segments; unsigned short max_integrity_segments; unsigned short max_discard_segments; @@ -564,6 +567,7 @@ struct request_queue { #define QUEUE_FLAG_NOWAIT 29 /* device supports NOWAIT */ #define QUEUE_FLAG_SQ_SCHED 30 /* single queue style io dispatch */ #define QUEUE_FLAG_SKIP_TAGSET_QUIESCE 31 /* quiesce_tagset skip the queue*/ +#define QUEUE_FLAG_COPY 32 /* supports copy offload */ #define QUEUE_FLAG_MQ_DEFAULT ((1UL << QUEUE_FLAG_IO_STAT) | \ (1UL << QUEUE_FLAG_SAME_COMP) | \ @@ -584,6 +588,7 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q); test_bit(QUEUE_FLAG_STABLE_WRITES, &(q)->queue_flags) #define blk_queue_io_stat(q) test_bit(QUEUE_FLAG_IO_STAT, &(q)->queue_flags) #define blk_queue_add_random(q) test_bit(QUEUE_FLAG_ADD_RANDOM, &(q)->queue_flags) +#define blk_queue_copy(q) test_bit(QUEUE_FLAG_COPY, &(q)->queue_flags) #define blk_queue_zone_resetall(q) \ test_bit(QUEUE_FLAG_ZONE_RESETALL, &(q)->queue_flags) #define blk_queue_dax(q) test_bit(QUEUE_FLAG_DAX, &(q)->queue_flags) @@ -902,6 +907,8 @@ extern void blk_queue_chunk_sectors(struct request_queue *, unsigned int); extern void blk_queue_max_segments(struct request_queue *, unsigned short); extern void blk_queue_max_discard_segments(struct request_queue *, unsigned short); +extern void blk_queue_max_copy_sectors_hw(struct request_queue *q, + unsigned int max_copy_sectors); void blk_queue_max_secure_erase_sectors(struct request_queue *q, unsigned int max_sectors); extern void blk_queue_max_segment_size(struct request_queue *, unsigned int); @@ -1221,6 +1228,11 @@ static inline unsigned int bdev_discard_granularity(struct block_device *bdev) return bdev_get_queue(bdev)->limits.discard_granularity; } +static inline unsigned int bdev_max_copy_sectors(struct block_device *bdev) +{ + return bdev_get_queue(bdev)->limits.max_copy_sectors; +} + static inline unsigned int bdev_max_secure_erase_sectors(struct block_device *bdev) { diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index b7b56871029c..8879567791fa 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -64,6 +64,9 @@ struct fstrim_range { __u64 minlen; }; +/* maximum total copy length */ +#define COPY_MAX_BYTES (1 << 27) + /* extent-same (dedupe) ioctls; these MUST match the btrfs ioctl definitions */ #define FILE_DEDUPE_RANGE_SAME 0 #define FILE_DEDUPE_RANGE_DIFFERS 1 From patchwork Tue Apr 11 08:10:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13207184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E592AC76196 for ; Tue, 11 Apr 2023 08:19:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230237AbjDKITg (ORCPT ); Tue, 11 Apr 2023 04:19:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57406 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230032AbjDKIT0 (ORCPT ); Tue, 11 Apr 2023 04:19:26 -0400 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 719DA40D1 for ; Tue, 11 Apr 2023 01:19:11 -0700 (PDT) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20230411081908epoutp02042c77fe6bfbcdd9a9a81d27a7a919d2~U04l0Xy-R0346203462epoutp02X for ; Tue, 11 Apr 2023 08:19:08 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20230411081908epoutp02042c77fe6bfbcdd9a9a81d27a7a919d2~U04l0Xy-R0346203462epoutp02X DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681201148; bh=qxRFqjFGypnIZQAYX8SzKxiOsIEG20pyvhgntQcUwTY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OTdHw/dvhkOCK4XVSwxDqqSQyyx76mQqyAIuvL6LqL2iHGthTYGu601Eqe4Yo2kBX 28/afqTFDwC1ub9m8VWtR1vc8pmltLUvTaTYKvIN2iASsqDFJ7zSyTfgZwhMDcDPaQ ZLsLNkkz68KBsLI1O1hXZkYGnMMRBxjDX4RRb+mM= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230411081907epcas5p14cc7243acedb09df278a7e44c4ca1aa0~U04lG_xqE0857608576epcas5p1z; Tue, 11 Apr 2023 08:19:07 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.177]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4PwdzF5q6Bz4x9QF; Tue, 11 Apr 2023 08:19:05 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 4D.1F.09540.9F715346; Tue, 11 Apr 2023 17:19:05 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230411081252epcas5p3ce3b26d13bbc302a0119c09c34a5eb49~U0zIWmE9l3265932659epcas5p3F; Tue, 11 Apr 2023 08:12:52 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230411081252epsmtrp1c460376282546ec7c44c142812180726~U0zISiMuA1831018310epsmtrp1W; Tue, 11 Apr 2023 08:12:52 +0000 (GMT) X-AuditID: b6c32a4a-70dfa70000002544-fd-643517f93d46 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id F1.ED.08609.48615346; Tue, 11 Apr 2023 17:12:52 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230411081247epsmtip2caf0c01f1eee6b3179f9aef98b1f99ed~U0zDhWjdK2397423974epsmtip2I; Tue, 11 Apr 2023 08:12:47 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v9 2/9] block: Add copy offload support infrastructure Date: Tue, 11 Apr 2023 13:40:29 +0530 Message-Id: <20230411081041.5328-3-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230411081041.5328-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTdxTH+d3b3hY33KWw+BuLiE0YkQ1osbAfCw8XndzBggzmXBYddvQG GKVt+pgCS+hsYEhg5SEqLVB1i1OIIoU5BMqW8gYBtUORBMSlnQ8e5ZEtEmGupWXzv88553t+ 5/HLYeOcMpYfO0uipOUSoZhLbGFc794VFLK6LULEu/T4HdQ01IejE+XrOGqc0hJotnsZoNOL qzia+TUOmRb0THT/txsY6rxQiaHLjb0Y6ji/hKHeF/MEqjTfBcg2rsOQafJt1GkaZCBLey2B DBdtLGSu0mCozfotQNefG3B0ddbOQAOTb6Kx9X7mHkhZfk+kdA9GCOqGbopFjU03MyjLiIoy NpwkqJYfC6iO+2qCKtMsEJS9a5ygvm9tAFTLcD61YvSnjNZ5LHnr59nRmbRQRMsDaEm6VJQl yYjhJqam7U2LiOTxQ/hR6F1ugESYQ8dw932UHLI/S+zYADfga6FY5XAlCxUKblhstFyqUtIB mVKFMoZLy0RimUAWqhDmKFSSjFAJrXyPz+OFRziER7MzuyduMWRnDh23T68z1eBsfAnwZENS AIvHG7ESsIXNITsArCnSsVzGMoD1Yz+4jRUAF6rWsc2Umz2PGK5AO4DlhTa3ocHgH2MGplNF kEGw51EhcAZ8ySIcLtlObqhw8g4G11rtuFPlQ+6Hy7+sACczyECo11VvZHuRCP7ZaHRo2I56 YVD7wNvp9iSj4FpdMeGSeMPBGivDyTi5A2p+1uPO9yFp8IT3+kdYrl73QcvTfjf7wKf9rW72 g0+0RW7OgM8sNvdsMqjp6wIujoOFQ9qNHnByF2xqD3O5t8PqoauYq+5WWPbc6k71gm31m8yF 312udTOEplG1mynY1WXFXdsqBbC/pwMvBwG6l+bRvTSP7v/S5wDeAN6gZYqcDFoRIQuX0Mf+ ++d0aY4RbNxFcEIbeDizGGoGGBuYAWTjXF+vv17sFnG8RMLcPFouTZOrxLTCDCIc+67A/V5P lzoOS6JM4wuieILIyEhB1O5IPnebV1DMYDqHzBAq6WyaltHyzTyM7emnxpijAwc/PCO7oh7+ ZioJsnwtmZdq8p6N/XQ3hDUnX11I5Hi8qmoKn7UmFQXegxKNIZ5IqHllu872z+PUr5r9P9Bn 6wPnTvXemTgwdbi5Y/b4nrcaE/v0KaXTFXWVg6aHU0dyW47Ec7QsjeF90jotV8deq+7Jry0/ UZHAOpbyJK3+fLNnrz3pdrw5VYtRt+f+vuX/sb1EHGr4tKKgICWYmBw+N1jZFnQxmaReqzMN 7/CZYC4tjZYO1MKG6cNnjd6Cm815ycX8a2ttqZ1HDx3Y6ZF04Us2yLdn5+zt/mTnsGox2R7L js71OjjfVcFZWvSYjLtSPvOZpb2XN6P+IrzudHgVl6HIFPKDcblC+C8ZS2NNoAQAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA02Ra0hTYRjHe99zdjxbLY5T6jWrxUgqq2ll8RZl9aE6diOwCxRkq53U1Cmb riyklRY1UKdR2px4QYxmFE1bWtNs3lJbrja7rHRR2rqqaRJRak4J+vY8/9+P//PhoQmRlZxF xyqSOaVCFi+hBKS5QSJemjFjpTy0Ji8U32prJvBZ3QiBK7qyKfylYRDgKwO/CPz2wXpc21fA w6/qayC2lOZCfL2iCeL7Jd8hbhr7RuFc63OAezv1ENe6FmNLbSuJHfcMFC4q7/XB1kvpEFf3 nAHY/LuIwDe/9JP4kSsQd4y08DYg1uHcxurdNoqt0Xf5sB3dt0nWYUthTcaLFFtZdpq9/0pD sZnpfRTbX9dJsVlVRsBWtp9ih0xzWVPPN7hr+n7BWjkXH6vmlCHhhwQxDS/tZFLevhP93SM8 DcjfogV8GjFh6HGjh9QCAS1iqgGymw3UJECo7VM5mJz90PVRj8+kdAai6rp8wgsoZgFq9JwD XuDP6Aj0zK2hvAvBvIWow/OD57X8mM1o8O7QRBXJBKEC/eWJXMhg9KHCNN5Ej58IQdluX2/M Z1ajP4UXKG8sGlfKLOSk7Ytar/ZMzAQjRul3CggdYPT/If1/qBhAIwjgklQJ0QmqZUnLFdxx qUqWoEpRREuPJCaYwMSbg4OrgcU4ILUCSAMrQDQh8RcOj62Qi4RyWepJTpkYpUyJ51RWEEiT kplCu7Y1SsREy5K5OI5L4pT/KKT5szTQr1Peu9vZ0vXTkqYUwtdvujNPl0R9Hqsqc69ZooPN VUrbok2rsr4uDcvzlT89mhFQWncwLmBOfYEnom3HrT04+GHFtPbGYn+c/86krrM9tRNqywEo zs1xXLJMvWq48TrHEmRz8neubdl2zulI3NRoDqUGIgsV81qa1OdD9FjQqc3QDuydNjzqIy5P C3gfEffDGBphv/aEENuk29NGUldEslnLxW61UGF4oRnM+Z6qqf8V/7UvZL4rs/RYeOQo7Dq8 gJ/6kecaPTDDJdmYvT23eSio6BFXme+84jAML1wcs67dfCGcF/biokyT2DT7vO7N/EWBaVsL k6fsqZLG9ktIVYxsWTChVMn+AinWwe5VAwAA X-CMS-MailID: 20230411081252epcas5p3ce3b26d13bbc302a0119c09c34a5eb49 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230411081252epcas5p3ce3b26d13bbc302a0119c09c34a5eb49 References: <20230411081041.5328-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Nitesh Shetty Introduce blkdev_issue_copy which takes similar arguments as copy_file_range and performs copy offload between two bdevs. Introduce REQ_COPY copy offload operation flag. Create a read-write bio pair with a token as payload and submitted to the device in order. Read request populates token with source specific information which is then passed with write request. This design is courtesy Mikulas Patocka's token based copy Larger copy will be divided, based on max_copy_sectors limit. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- block/blk-lib.c | 230 ++++++++++++++++++++++++++++++++++++++ block/blk.h | 2 + include/linux/blk_types.h | 25 +++++ include/linux/blkdev.h | 3 + 4 files changed, 260 insertions(+) diff --git a/block/blk-lib.c b/block/blk-lib.c index e59c3069e835..b5914a357763 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -115,6 +115,236 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector, } EXPORT_SYMBOL(blkdev_issue_discard); +/* + * For synchronous copy offload/emulation, wait and process all in-flight BIOs. + * This must only be called once all bios have been issued so that the refcount + * can only decrease. This just waits for all bios to make it through + * blkdev_copy_write_endio. + */ +static int blkdev_copy_wait_completion(struct cio *cio) +{ + int ret; + + if (cio->endio) + return 0; + + if (atomic_read(&cio->refcount)) { + __set_current_state(TASK_UNINTERRUPTIBLE); + blk_io_schedule(); + } + + ret = cio->comp_len; + kfree(cio); + + return ret; +} + +static void blkdev_copy_offload_write_endio(struct bio *bio) +{ + struct copy_ctx *ctx = bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (bio->bi_status) { + clen = (bio->bi_iter.bi_sector << SECTOR_SHIFT) - cio->pos_out; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + } + __free_page(bio->bi_io_vec[0].bv_page); + bio_put(bio); + + kfree(ctx); + if (!atomic_dec_and_test(&cio->refcount)) + return; + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); +} + +static void blkdev_copy_offload_read_endio(struct bio *read_bio) +{ + struct copy_ctx *ctx = read_bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (read_bio->bi_status) { + clen = (read_bio->bi_iter.bi_sector << SECTOR_SHIFT) + - cio->pos_in; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + __free_page(read_bio->bi_io_vec[0].bv_page); + bio_put(ctx->write_bio); + bio_put(read_bio); + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } + return; + } + + schedule_work(&ctx->dispatch_work); + bio_put(read_bio); +} + +static void blkdev_copy_dispatch_work(struct work_struct *work) +{ + struct copy_ctx *ctx = container_of(work, struct copy_ctx, + dispatch_work); + + submit_bio(ctx->write_bio); +} + +/* + * __blkdev_copy_offload - Use device's native copy offload feature. + * we perform copy operation by sending 2 bio. + * 1. First we send a read bio with REQ_COPY flag along with a token and source + * and length. Once read bio reaches driver layer, device driver adds all the + * source info to token and does a fake completion. + * 2. Once read operation completes, we issue write with REQ_COPY flag with same + * token. In driver layer, token info is used to form a copy offload command. + * + * returns the length of bytes copied + */ +static int __blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct cio *cio; + struct copy_ctx *ctx; + struct bio *read_bio, *write_bio; + struct page *token; + sector_t copy_len; + sector_t rem, max_copy_len; + + cio = kzalloc(sizeof(struct cio), GFP_KERNEL); + if (!cio) + return 0; + atomic_set(&cio->refcount, 0); + cio->waiter = current; + cio->endio = endio; + cio->private = private; + + max_copy_len = min(bdev_max_copy_sectors(bdev_in), + bdev_max_copy_sectors(bdev_out)) << SECTOR_SHIFT; + + cio->pos_in = pos_in; + cio->pos_out = pos_out; + /* If there is a error, comp_len will be set to least successfully + * completed copied length */ + cio->comp_len = len; + for (rem = len; rem > 0; rem -= copy_len) { + copy_len = min(rem, max_copy_len); + + token = alloc_page(gfp_mask); + if (unlikely(!token)) + goto err_token; + + ctx = kzalloc(sizeof(struct copy_ctx), gfp_mask); + if (!ctx) + goto err_ctx; + read_bio = bio_alloc(bdev_in, 1, REQ_OP_READ | REQ_COPY + | REQ_SYNC | REQ_NOMERGE, gfp_mask); + if (!read_bio) + goto err_read_bio; + write_bio = bio_alloc(bdev_out, 1, REQ_OP_WRITE + | REQ_COPY | REQ_SYNC | REQ_NOMERGE, gfp_mask); + if (!write_bio) + goto err_write_bio; + + ctx->cio = cio; + ctx->write_bio = write_bio; + INIT_WORK(&ctx->dispatch_work, blkdev_copy_dispatch_work); + + __bio_add_page(read_bio, token, PAGE_SIZE, 0); + read_bio->bi_iter.bi_size = copy_len; + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_end_io = blkdev_copy_offload_read_endio; + read_bio->bi_private = ctx; + + __bio_add_page(write_bio, token, PAGE_SIZE, 0); + write_bio->bi_iter.bi_size = copy_len; + write_bio->bi_end_io = blkdev_copy_offload_write_endio; + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_private = ctx; + + atomic_inc(&cio->refcount); + submit_bio(read_bio); + pos_in += copy_len; + pos_out += copy_len; + } + + /* Wait for completion of all IO's*/ + return blkdev_copy_wait_completion(cio); + +err_write_bio: + bio_put(read_bio); +err_read_bio: + kfree(ctx); +err_ctx: + __free_page(token); +err_token: + cio->comp_len = min_t(sector_t, cio->comp_len, (len - rem)); + return blkdev_copy_wait_completion(cio); +} + +static inline int blkdev_copy_sanity_check(struct block_device *bdev_in, + loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, + size_t len) +{ + unsigned int align = max(bdev_logical_block_size(bdev_out), + bdev_logical_block_size(bdev_in)) - 1; + + if (bdev_read_only(bdev_out)) + return -EPERM; + + if ((pos_in & align) || (pos_out & align) || (len & align) || !len || + len >= COPY_MAX_BYTES) + return -EINVAL; + + return 0; +} + +/* + * @bdev_in: source block device + * @pos_in: source offset + * @bdev_out: destination block device + * @pos_out: destination offset + * @len: length in bytes to be copied + * @endio: endio function to be called on completion of copy operation, + * for synchronous operation this should be NULL + * @private: endio function will be called with this private data, should be + * NULL, if operation is synchronous in nature + * @gfp_mask: memory allocation flags (for bio_alloc) + * + * Returns the length of bytes copied + * + * Description: + * Copy source offset from source block device to destination block + * device. Max total length of copy is limited to MAX_COPY_TOTAL_LENGTH + */ +int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct request_queue *q_in = bdev_get_queue(bdev_in); + struct request_queue *q_out = bdev_get_queue(bdev_out); + int ret = 0; + + if (blkdev_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len)) + return 0; + + if (blk_queue_copy(q_in) && blk_queue_copy(q_out)) + ret = __blkdev_copy_offload(bdev_in, pos_in, bdev_out, pos_out, + len, endio, private, gfp_mask); + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_issue_copy); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/block/blk.h b/block/blk.h index d65d96994a94..684b8fa121db 100644 --- a/block/blk.h +++ b/block/blk.h @@ -311,6 +311,8 @@ static inline bool bio_may_exceed_limits(struct bio *bio, break; } + if (unlikely(op_is_copy(bio->bi_opf))) + return false; /* * All drivers must accept single-segments bios that are <= PAGE_SIZE. * This is a quick and dirty check that relies on the fact that diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 80670a641cc2..da07ce399881 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -416,6 +416,7 @@ enum req_flag_bits { */ /* for REQ_OP_WRITE_ZEROES: */ __REQ_NOUNMAP, /* do not free blocks when zeroing */ + __REQ_COPY, /* copy request */ __REQ_NR_BITS, /* stops here */ }; @@ -440,6 +441,7 @@ enum req_flag_bits { #define REQ_POLLED (__force blk_opf_t)(1ULL << __REQ_POLLED) #define REQ_ALLOC_CACHE (__force blk_opf_t)(1ULL << __REQ_ALLOC_CACHE) #define REQ_SWAP (__force blk_opf_t)(1ULL << __REQ_SWAP) +#define REQ_COPY ((__force blk_opf_t)(1ULL << __REQ_COPY)) #define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV) #define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE) @@ -470,6 +472,11 @@ static inline bool op_is_write(blk_opf_t op) return !!(op & (__force blk_opf_t)1); } +static inline bool op_is_copy(blk_opf_t op) +{ + return op & REQ_COPY; +} + /* * Check if the bio or request is one that needs special treatment in the * flush state machine. @@ -529,4 +536,22 @@ struct blk_rq_stat { u64 batch; }; +typedef void (cio_iodone_t)(void *private, int comp_len); + +struct cio { + struct task_struct *waiter; /* waiting task (NULL if none) */ + atomic_t refcount; + loff_t pos_in; + loff_t pos_out; + size_t comp_len; + cio_iodone_t *endio; /* applicable for async operation */ + void *private; /* applicable for async operation */ +}; + +struct copy_ctx { + struct cio *cio; + struct work_struct dispatch_work; + struct bio *write_bio; +}; + #endif /* __LINUX_BLK_TYPES_H */ diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 200338f2ec2e..1bb43697d43d 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1054,6 +1054,9 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop); int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp); +int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Tue Apr 11 08:10:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13207185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BA2FC7619A for ; Tue, 11 Apr 2023 08:19:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230193AbjDKITu (ORCPT ); Tue, 11 Apr 2023 04:19:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230310AbjDKIT3 (ORCPT ); Tue, 11 Apr 2023 04:19:29 -0400 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9CE740CC for ; Tue, 11 Apr 2023 01:19:14 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20230411081913epoutp02b0648cd697c9a0c180afd39b23c0ad7b~U04qX9SBV0479104791epoutp027 for ; Tue, 11 Apr 2023 08:19:13 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20230411081913epoutp02b0648cd697c9a0c180afd39b23c0ad7b~U04qX9SBV0479104791epoutp027 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681201153; bh=fYGUDfCyG7FlU2TedDj9Ugdogzie90RjzpLc0LEeThE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dACYGeoh4xTZMKGd9ycv7DVd8FB89J71jO1RYflDD3SqxLu2BlRyg9gHSARA1H+Hg AsqChg42MQTJoT8LuRUCFsxQY4se24Zz25h9gSEKKrIBcQ3tFL0tCEtvd1bQr3NR1B 1cCni94QlY+KIBYpeX+cHZYXtHlN4DII4mcaKhhs= Received: from epsnrtp2.localdomain (unknown [182.195.42.163]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230411081912epcas5p2fa20847fcb0874f420450a4623b3e033~U04ps1gvw0242802428epcas5p2Q; Tue, 11 Apr 2023 08:19:12 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.179]) by epsnrtp2.localdomain (Postfix) with ESMTP id 4PwdzL2BYGz4x9Pw; Tue, 11 Apr 2023 08:19:10 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id C5.5B.09961.EF715346; Tue, 11 Apr 2023 17:19:10 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230411081303epcas5p2a9bd49cc4cf49257fd7119fcb0739fa2~U0zRzJPW01765917659epcas5p2f; Tue, 11 Apr 2023 08:13:03 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230411081303epsmtrp2b562a2251d012cc891bc0ca02ee50126~U0zRx-S742545625456epsmtrp2t; Tue, 11 Apr 2023 08:13:03 +0000 (GMT) X-AuditID: b6c32a49-2c1ff700000026e9-1d-643517fe3a68 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id E6.ED.08609.E8615346; Tue, 11 Apr 2023 17:13:03 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230411081258epsmtip2c30f0e17964872050b47f9cd2bc99526~U0zNqQkkt2386223862epsmtip2L; Tue, 11 Apr 2023 08:12:58 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Vincent Fu , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v9 3/9] block: add emulation for copy Date: Tue, 11 Apr 2023 13:40:30 +0530 Message-Id: <20230411081041.5328-4-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230411081041.5328-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xbdRTH97u33Ba0y6V08gOj4lVWYQFaoOXHtrKZIV7DTDBMpzCDlV5b QmmbtiiKCUgD47GNMQLBwtjLOXlsTF7hDSuvAUGCrCZggD14TAjvoS4MsaWg++9zXr/vOeeX w8F5eWx3TpzawOjUMhVFOLHqO70EPv+4iuVCY5ETqurvwVHa+U0cVYznEmi+cxWgwuWnOLrf fgS1LhY7oNGORgy1XL2AobKKbgw1X1nBUPfWAoEumH8DaNpiwlDr2AHU0trHQiNNJQS69OM0 G5nzjRhqmPoOoPqNSzi6Nb/EQnfHXkaPcjIBGtrsdTjqRo/cC6dNk4ME3WgaZ9NDEz+z6JHB RLq6PIuga35IoZtHUwn6rHGRoJfaLAR9rrYc0DUDyfRa9at09dQCFrE3Kv6wkpHJGZ0Ho47V yOPUCikVHhlzLEYsEYp8RMEoiPJQyxIYKRV6PMInLE5lXQPl8aVMlWh1Rcj0esov5LBOk2hg PJQavUFKMVq5Shuo9dXLEvSJaoWvmjEcFAmF/mJr4mfxyvaxAqAdlSZNProBUsFD/2zgyIFk INyau+dgYx7ZDOBsRmQ2cLLyKoCduZcd7MYagLnXMsBuxYOKOZY90ARgWVYt224YMbg+UbSd RZAC2DWbDmwBPpmBw5XprO0SnPwLg5OzNduKLqQYblU+3mYW6Qnzy8vYNuaSCJZO5WDZgGPV 84O5k842tyMZDJ9dzCTsKc6w7/splo1x8jVorCvGbe9DssIRtueMsuy9hsKBqkHczi5wrreW bWd3uLbYSthZAf8emcbsrIXGnradOY/A9P5c3NYDTnrBqiY/u/sVWNB/C7Pr7oVnN6Z2Srmw oXSXKXi6rGSHIWz9JXWHaZgxnAbs2zoD4M0FM/s88DA9N4/puXlM/0tfBng5cGO0+gQFoxdr RWrmq/++OVaTUA22b8P7vQYwfn/Z1wwwDjADyMEpPnd9K0DO48plX3/D6DQxukQVozcDsXXf ebj7vliN9bjUhhhRYLAwUCKRBAYHSESUK1cg7YvlkQqZgYlnGC2j263DOI7uqZjUhedl4ZdO VPJVgpE3xnOOOw/L4zOvJfMnrqSvByxyT9VLyl+QpIaEhK5Fz7zz+53i5Koby/3CJSbBc4aX xd2zr/eA8PqA+5NPTB/efqmudLaQ+mn5D77u7Wy9fH8U/lb4+30t87N+aaenu4beDNMqzgjy HzPv7h2s2z/sUh3U80HXM+UXBS1/NlwdsWTNR+ft+ZVq/Dzayc272e/EQ0tl9ME7JRvHwuhN WVFJmDLlY52r2rnQcorylfE+4jMvvs5q8dfeFbSxJReTTp5Yyc+vi3JMOnS7Y5Wo8lbOZHTf HIr59lP2+pMHQebxoIbKiI6T1+snIw95aNoLxa6eR5+mBBnrKJZeKRN54zq97F95XxPCpAQA AA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrOIsWRmVeSWpSXmKPExsWy7bCSvG6/mGmKQedjRov1p44xWzRN+Mts sfpuP5vF68OfGC2mffjJbPFgv73F3nezWS1uHtjJZLFn0SQmi5WrjzJZ7F74kcni6P+3bBaT Dl1jtHh6dRaTxd5b2hZ79p5ksbi8aw6bxfxlT9ktDk1uZrLY8aSR0WLb7/nMFutev2exOHFL 2uJxdwejxfm/x1kdJD0uX/H2mHX/LJvHzll32T3O39vI4nH5bKnHplWdbB6bl9R77L7ZwObR 2/yOzeP9vqtsHn1bVjF6bD5d7fF5k5zHpidvmQL4orhsUlJzMstSi/TtErgy9t+aylhw07bi /uPljA2Mj4y6GDk5JARMJB6ufsXSxcjFISSwg1FizrFWJoiEhMSpl8sYIWxhiZX/nrNDFDUy SXTuWwdWxCagLnHkeSsjSEJEYAKzxKX7DWwgDrNAO7PEm2UfWECqhAVMJf6vecEKYrMIqEpM XrWSHcTmFbCQmPekG2gSB9AKfYn++4IgYU4BS4k/czvYQMJCQCVL9rBAVAtKnJz5BMxmFpCX aN46m3kCo8AsJKlZSFILGJlWMUqmFhTnpucWGxYY5aWW6xUn5haX5qXrJefnbmIEx7WW1g7G Pas+6B1iZOJgPMQowcGsJML79b9xihBvSmJlVWpRfnxRaU5q8SFGaQ4WJXHeC10n44UE0hNL UrNTUwtSi2CyTBycUg1MSewezD9tbl0X0kl0vbh9n2ZhyLwfbd7BEqunHj+7MKrn6Icd5/Ys 7Q+2e/7mQRl3dkjsygKNf9z/f2wtd3BZk5Yvo5L2+8xKa4bSsKnStYxH7HIPpLXueCm3Rnfa vr+uGy47JfO1HNrzpnj9/UviuXN6bhh3zL+Tp/NxpaGaYnPNttznTOUbk4tv1oVs5jcNUvw6 4cDrTE2r6qCjgQGbXI59v7PWXlZk7k4RvymRppt+a7Xbn/gY6u7yf/vNvg8hZXN/5X/eudOd r4Czp6SRYe7TaZNE5RWlJCNZn+tLCAt5ZCjuCzTjYD39cUGWt7ukBDt7c8RnWWslP/lvpk2s 6/5cCLxi57nMecGpkFwlluKMREMt5qLiRABTy4M5WgMAAA== X-CMS-MailID: 20230411081303epcas5p2a9bd49cc4cf49257fd7119fcb0739fa2 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230411081303epcas5p2a9bd49cc4cf49257fd7119fcb0739fa2 References: <20230411081041.5328-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Nitesh Shetty For the devices which does not support copy, copy emulation is added. It is required for in-kernel users like fabrics, where file descriptor is not available and hence they can't use copy_file_range. Copy-emulation is implemented by reading from source into memory and writing to the corresponding destination asynchronously. Also emulation is used, if copy offload fails or partially completes. Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu Signed-off-by: Anuj Gupta --- block/blk-lib.c | 167 +++++++++++++++++++++++++++++++++++++++++ block/blk-map.c | 4 +- include/linux/blkdev.h | 3 + 3 files changed, 172 insertions(+), 2 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index b5914a357763..2b6e0f5b1f31 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -291,6 +291,169 @@ static int __blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, return blkdev_copy_wait_completion(cio); } +static void *blkdev_copy_alloc_buf(sector_t req_size, sector_t *alloc_size, + gfp_t gfp_mask) +{ + int min_size = PAGE_SIZE; + void *buf; + + while (req_size >= min_size) { + buf = kvmalloc(req_size, gfp_mask); + if (buf) { + *alloc_size = req_size; + return buf; + } + /* retry half the requested size */ + req_size >>= 1; + } + + return NULL; +} + +static void blkdev_copy_emulate_write_endio(struct bio *bio) +{ + struct copy_ctx *ctx = bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (bio->bi_status) { + clen = (bio->bi_iter.bi_sector << SECTOR_SHIFT) - cio->pos_out; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + } + kvfree(page_address(bio->bi_io_vec[0].bv_page)); + bio_map_kern_endio(bio); + kfree(ctx); + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } +} + +static void blkdev_copy_emulate_read_endio(struct bio *read_bio) +{ + struct copy_ctx *ctx = read_bio->bi_private; + struct cio *cio = ctx->cio; + sector_t clen; + + if (read_bio->bi_status) { + clen = (read_bio->bi_iter.bi_sector << SECTOR_SHIFT) - + cio->pos_in; + cio->comp_len = min_t(sector_t, clen, cio->comp_len); + __free_page(read_bio->bi_io_vec[0].bv_page); + bio_map_kern_endio(read_bio); + kfree(ctx); + + if (atomic_dec_and_test(&cio->refcount)) { + if (cio->endio) { + cio->endio(cio->private, cio->comp_len); + kfree(cio); + } else + blk_wake_io_task(cio->waiter); + } + } + schedule_work(&ctx->dispatch_work); + kfree(read_bio); +} + +/* + * If native copy offload feature is absent, this function tries to emulate, + * by copying data from source to a temporary buffer and from buffer to + * destination device. + * returns the length of bytes copied + */ +static int __blkdev_copy_emulate(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct request_queue *in = bdev_get_queue(bdev_in); + struct request_queue *out = bdev_get_queue(bdev_out); + struct bio *read_bio, *write_bio; + void *buf = NULL; + struct copy_ctx *ctx; + struct cio *cio; + sector_t buf_len, req_len, rem = 0; + sector_t max_src_hw_len = min_t(unsigned int, + queue_max_hw_sectors(in), + queue_max_segments(in) << (PAGE_SHIFT - SECTOR_SHIFT)) + << SECTOR_SHIFT; + sector_t max_dst_hw_len = min_t(unsigned int, + queue_max_hw_sectors(out), + queue_max_segments(out) << (PAGE_SHIFT - SECTOR_SHIFT)) + << SECTOR_SHIFT; + sector_t max_hw_len = min_t(unsigned int, + max_src_hw_len, max_dst_hw_len); + + cio = kzalloc(sizeof(struct cio), GFP_KERNEL); + if (!cio) + return 0; + atomic_set(&cio->refcount, 0); + cio->pos_in = pos_in; + cio->pos_out = pos_out; + cio->waiter = current; + cio->endio = endio; + cio->private = private; + + for (rem = len; rem > 0; rem -= buf_len) { + req_len = min_t(int, max_hw_len, rem); + + buf = blkdev_copy_alloc_buf(req_len, &buf_len, gfp_mask); + if (!buf) + goto err_alloc_buf; + + ctx = kzalloc(sizeof(struct copy_ctx), gfp_mask); + if (!ctx) + goto err_ctx; + + read_bio = bio_map_kern(in, buf, buf_len, gfp_mask); + if (IS_ERR(read_bio)) + goto err_read_bio; + + write_bio = bio_map_kern(out, buf, buf_len, gfp_mask); + if (IS_ERR(write_bio)) + goto err_write_bio; + + ctx->cio = cio; + ctx->write_bio = write_bio; + INIT_WORK(&ctx->dispatch_work, blkdev_copy_dispatch_work); + + read_bio->bi_iter.bi_sector = pos_in >> SECTOR_SHIFT; + read_bio->bi_iter.bi_size = buf_len; + read_bio->bi_opf = REQ_OP_READ | REQ_SYNC; + bio_set_dev(read_bio, bdev_in); + read_bio->bi_end_io = blkdev_copy_emulate_read_endio; + read_bio->bi_private = ctx; + + write_bio->bi_iter.bi_size = buf_len; + write_bio->bi_opf = REQ_OP_WRITE | REQ_SYNC; + bio_set_dev(write_bio, bdev_out); + write_bio->bi_end_io = blkdev_copy_emulate_write_endio; + write_bio->bi_iter.bi_sector = pos_out >> SECTOR_SHIFT; + write_bio->bi_private = ctx; + + atomic_inc(&cio->refcount); + submit_bio(read_bio); + + pos_in += buf_len; + pos_out += buf_len; + } + + /* Wait for completion of all IO's*/ + return blkdev_copy_wait_completion(cio); + +err_write_bio: + bio_put(read_bio); +err_read_bio: + kfree(ctx); +err_ctx: + kvfree(buf); +err_alloc_buf: + cio->comp_len -= min_t(sector_t, cio->comp_len, len - rem); + return blkdev_copy_wait_completion(cio); +} + static inline int blkdev_copy_sanity_check(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len) @@ -341,6 +504,10 @@ int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, ret = __blkdev_copy_offload(bdev_in, pos_in, bdev_out, pos_out, len, endio, private, gfp_mask); + if (ret != len) + ret = __blkdev_copy_emulate(bdev_in, pos_in + ret, bdev_out, + pos_out + ret, len - ret, endio, private, gfp_mask); + return ret; } EXPORT_SYMBOL_GPL(blkdev_issue_copy); diff --git a/block/blk-map.c b/block/blk-map.c index 3551c3ff17cf..e75bae459cfa 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -363,7 +363,7 @@ static void bio_invalidate_vmalloc_pages(struct bio *bio) #endif } -static void bio_map_kern_endio(struct bio *bio) +void bio_map_kern_endio(struct bio *bio) { bio_invalidate_vmalloc_pages(bio); bio_uninit(bio); @@ -380,7 +380,7 @@ static void bio_map_kern_endio(struct bio *bio) * Map the kernel address into a bio suitable for io to a block * device. Returns an error pointer in case of error. */ -static struct bio *bio_map_kern(struct request_queue *q, void *data, +struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, gfp_t gfp_mask) { unsigned long kaddr = (unsigned long)data; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 1bb43697d43d..a54153610800 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1057,6 +1057,9 @@ int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); +struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, + gfp_t gfp_mask); +void bio_map_kern_endio(struct bio *bio); #define BLKDEV_ZERO_NOUNMAP (1 << 0) /* do not free blocks */ #define BLKDEV_ZERO_NOFALLBACK (1 << 1) /* don't write explicit zeroes */ From patchwork Tue Apr 11 08:10:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13207187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC5D8C76196 for ; Tue, 11 Apr 2023 08:20:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230351AbjDKIUS (ORCPT ); Tue, 11 Apr 2023 04:20:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230348AbjDKITq (ORCPT ); Tue, 11 Apr 2023 04:19:46 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8DAA8422A for ; Tue, 11 Apr 2023 01:19:29 -0700 (PDT) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230411081928epoutp03073c7c7beadbe238871b4cff7d0fa861~U044TZNbr1232212322epoutp03n for ; Tue, 11 Apr 2023 08:19:28 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230411081928epoutp03073c7c7beadbe238871b4cff7d0fa861~U044TZNbr1232212322epoutp03n DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681201168; bh=2CFOC+DxaW6OTaR0o8hk4qN6SiHkCpPQrQBM4maV6/8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YQUz+pIlUAaCKy0AfXgjOp5vH9UrObGAckYbjYkPFEZ1ykdH8qdgujYms5ffNZXOI qg8vrqReMWDisBseU8uAykikWEGzezZ024h2qIW5A2hNKEWfGDu+6rfVbsBSzW5nWY rtxrhyeTTYPcPcG4HDB6vjr5PK/2Bt69gK9NGvF8= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230411081927epcas5p1713b44eac41715d509f27fc66575b7be~U043rdlaf0857608576epcas5p1S; Tue, 11 Apr 2023 08:19:27 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.183]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4PwdzX62G3z4x9QJ; Tue, 11 Apr 2023 08:19:20 +0000 (GMT) Received: from epcas5p4.samsung.com ( [182.195.41.42]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 24.6B.09961.80815346; Tue, 11 Apr 2023 17:19:20 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230411081313epcas5p1ce798f50564cefde203c730980b7d557~U0zb5qb6z0300903009epcas5p15; Tue, 11 Apr 2023 08:13:13 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230411081313epsmtrp13c39e473fafb0aee76348d6b1013b46a~U0zb2-1Jp1886118861epsmtrp1a; Tue, 11 Apr 2023 08:13:13 +0000 (GMT) X-AuditID: b6c32a49-2c1ff700000026e9-46-64351808a27a Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id B0.1F.08279.99615346; Tue, 11 Apr 2023 17:13:13 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230411081308epsmtip29cdda8c4b5592a8bf683a23dae46fb91~U0zXIqfFz2397423974epsmtip2S; Tue, 11 Apr 2023 08:13:08 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v9 4/9] fs, block: copy_file_range for def_blk_ops for direct block device Date: Tue, 11 Apr 2023 13:40:31 +0530 Message-Id: <20230411081041.5328-5-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230411081041.5328-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Tf1CTdRzH7/s842Fw4T0C5rdZiLs8EOXHgOEXk0hDfYyyReeV5h0+xx42 ZL9uG2JJ55Ik9YAhFXdtxDDMFC6xMX4IDGooP2Y0dJChArMbdbnjl/YDRaCNjfK/1+f9eX/u 8+N7XzYeXOzPYefI1IxSRku4RCCrqWtDVDQb8oVxro4AVG/txtHxsnkc1Y1oCeTqegBQxfQj HDk6U5F5Uu+Hhr+/gqH2r8oxdLHuGobazs5g6NriBIHKLT8DND6kw5D59kbUbu5jIXtrJYEM 58f9keXTQgy1OD8CqGnOgKNLrikW6r29Btnme/xegZR9MJ3SjfUT1BXdiD9lG/2ORdn78yhj 7SmCajh3jGob1hBUSeEkQU11DBFUqakWUA3Xj1IPjWGU0TmBCVbsz90qZmghowxnZFlyYY5M lMJNfzvz1Ux+UhwvmpeMNnPDZbSUSeGmvS6I3pkjcV+AG36YluS5JQGtUnFjX96qlOepmXCx XKVO4TIKoUSRqIhR0VJVnkwUI2PUW3hxcfF8t/Fgrrh5oJGlMEcemW66ztKAe+tOgwA2JBPh yW/+YZ0Ggexgsg1AU9MFzBs8ALCxVO/vDf4G0PbI4b9cYrwxgHsTZgBL+yYwTyKYLMRg58Ie DxNkBLz6+wngMYWSRTicGT+11AQnb2LwiWkK97hCyANweLKO8DCLXA8d395a0oNIBMe6PB62 u10s1I6t9MgBZDJ88uVJwmtZCfu+cLI8jJNrYWGjfmkiSNYEwM6OO75R06Dx8aifl0Pg/R6T T+fAP7RFPhbBWfs45mUFLOzuAF5OhSes2qUZcHIDrG+N9covwM+tlzBv3xWwZM7pKw2CLVXL zIWfXKz0MYTmnzQ+pqCj8q7vvsUAnq224mUgXPfUPrqn9tH937oa4LXgOUahkooYFV/BkzH5 /z1zllxqBEvfImp3CxhxTMdYAMYGFgDZODc06K/FBGFwkJB+/wNGKc9U5kkYlQXw3fc+g3NW Zcnd/0qmzuQlJsclJiUlJSYnJPG4q4MiUvqygkkRrWZyGUbBKJfrMHYAR4MRmyrC7nXkdxU0 RN3NzrSaY4Ra5y8i28Bq/f6121+aM2QM1QfGM8KD2c3KdTSnsq77xcCPhX39ijc32WbWHNn2 WFZbtTujuBe+eznk16ODeMjzN8MqLux4LX1wb4vcaM6QzUi1HJWgyH5s431RjUS/paxeYhd0 5g+lDYw9bK3KyE4Xv1G/w9ZzKGFeLdp2CDdk74kB3akHjEWDmq/PzTrYkTWCH3ZG7Ns1sj2+ JMXFIeL7z7v2ts+i4xZnb2TdO6yFD/naGwu7bq0qslXvC3zWdNXVcjmnuYC3/hmp3pb+lmnz j38WRCzayqtR7FDob/MnNe+FfGZtvzNqEA/k9BimD3NZKjHNi8KVKvpfXh6PFZ8EAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02Rf0yMcRzHfZ/n6ekpTk9Xy5em5tSIXD/IviYhP/YIYzbLCnXcs6t1d7W7 ToTuqmXJUsuWdbGU3LqIdalz1ZEnItWKyI/JZaL87C6OoaSr2fz3+rxf7+39x4fC+beJeVSi PJVVyEVSAelKNLQKfJeVeIWJg01jQnTtQRuOsgrHcXS5v4BEH1tHASq2/sTRwK21yPyl1Ak9 bzFhqLmiCEP6y3cx1FRuw9Ddic8kKuL6AHr7RIsh84ulqNncTqDexnMkKtO9dUbcmWwM3RjM BKjhdxmOrn4cIdD9F96oe/ye0zrI9D7eymgtXSRj0vY7M92vagmmt0vFGKpPkkxdpZppeq4h mfzsLyQzcvMJyZy+Xg2Yuo6jzFeDD2MY/IztnB3jGi5mpYmHWEVQRLxrgrGnnkgxLz5sbegg NOD1gjzgQkF6BTQ87MEdzKebAOzpjJ3OIXzwXgem2QPq/ww55wHXyU4mBu2Gr4RDkPQieGco BziEJ12Iw0cWDek4cHoAg91D35wcLQ86BlYMXSQdTND+cKDm6dQcj0bQ0joyydTkRBAssLg7 Yhd6FRw7n0s6Yv5kpbKZmG67w/aSwSnGaV+YXV+KFwJa+5/S/qcuAKwazGVTlDKJTBmSEiJn 04RKkUypkkuEB5NlBjD15SUBN4Cx2irkAEYBDkAKF3jy7BPLxXyeWHQknVUkxylUUlbJAW+K EMzh9eS1x/FpiSiVTWLZFFbxz2KUyzwNVhPWmGMdu7lv2MbNYH2yohK3bbpe07Zs7/YATYws /mRRFld2bmzYu8QYMGC8doqcMypfE9vQ7rV6YmNSllNz9K/wE5Gm0R9ViVigWu3XFxN8wHip fs/p+Ee5z/QbhtuqvLaEznoXJayJCOesobWa2HQ3CdhhD9b6JekWtKiucNmekYzJQtqWZ5pm f6+dqbIFNI6/rJDkzkjz0m2Q6swRb0Lz2eRLgatyPtmP709LJ4LOrm/N2JWm3CK2rZRGdx6z R2121xMFt0M6jhs/VPXBfvVrl2TZ/F1uJeqrNnUkxSveL4jy317RRbhnuA2OdGWUa/csjLX5 HKtq0u8u2H1FQCgTRCFLcIVS9Be63jIhVAMAAA== X-CMS-MailID: 20230411081313epcas5p1ce798f50564cefde203c730980b7d557 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230411081313epcas5p1ce798f50564cefde203c730980b7d557 References: <20230411081041.5328-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Nitesh Shetty For direct block device opened with O_DIRECT, use copy_file_range to issue device copy offload, and fallback to generic_copy_file_range incase device copy offload capability is absent. Modify checks to allow bdevs to use copy_file_range. Suggested-by: Ming Lei Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty --- block/blk-lib.c | 19 +++++++++++++++++++ block/fops.c | 20 ++++++++++++++++++++ fs/read_write.c | 11 +++++++++-- include/linux/blkdev.h | 3 +++ 4 files changed, 51 insertions(+), 2 deletions(-) diff --git a/block/blk-lib.c b/block/blk-lib.c index 2b6e0f5b1f31..b6e193603789 100644 --- a/block/blk-lib.c +++ b/block/blk-lib.c @@ -512,6 +512,25 @@ int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, } EXPORT_SYMBOL_GPL(blkdev_issue_copy); +int blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t endio, void *private, gfp_t gfp_mask) +{ + struct request_queue *in_q = bdev_get_queue(bdev_in); + struct request_queue *out_q = bdev_get_queue(bdev_out); + int ret = 0; + + if (blkdev_copy_sanity_check(bdev_in, pos_in, bdev_out, pos_out, len)) + return 0; + + if (blk_queue_copy(in_q) && blk_queue_copy(out_q)) + ret = __blkdev_copy_offload(bdev_in, pos_in, bdev_out, pos_out, + len, endio, private, gfp_mask); + + return ret; +} +EXPORT_SYMBOL_GPL(blkdev_copy_offload); + static int __blkdev_issue_write_zeroes(struct block_device *bdev, sector_t sector, sector_t nr_sects, gfp_t gfp_mask, struct bio **biop, unsigned flags) diff --git a/block/fops.c b/block/fops.c index d2e6be4e3d1c..3b7c05831d5c 100644 --- a/block/fops.c +++ b/block/fops.c @@ -611,6 +611,25 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, struct iov_iter *to) return ret; } +static ssize_t blkdev_copy_file_range(struct file *file_in, loff_t pos_in, + struct file *file_out, loff_t pos_out, + size_t len, unsigned int flags) +{ + struct block_device *in_bdev = I_BDEV(bdev_file_inode(file_in)); + struct block_device *out_bdev = I_BDEV(bdev_file_inode(file_out)); + int comp_len = 0; + + if ((file_in->f_iocb_flags & IOCB_DIRECT) && + (file_out->f_iocb_flags & IOCB_DIRECT)) + comp_len = blkdev_copy_offload(in_bdev, pos_in, out_bdev, + pos_out, len, NULL, NULL, GFP_KERNEL); + if (comp_len != len) + comp_len = generic_copy_file_range(file_in, pos_in + comp_len, + file_out, pos_out + comp_len, len - comp_len, flags); + + return comp_len; +} + #define BLKDEV_FALLOC_FL_SUPPORTED \ (FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE | \ FALLOC_FL_ZERO_RANGE | FALLOC_FL_NO_HIDE_STALE) @@ -694,6 +713,7 @@ const struct file_operations def_blk_fops = { .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, .fallocate = blkdev_fallocate, + .copy_file_range = blkdev_copy_file_range, }; static __init int blkdev_init(void) diff --git a/fs/read_write.c b/fs/read_write.c index a21ba3be7dbe..47e848fcfd42 100644 --- a/fs/read_write.c +++ b/fs/read_write.c @@ -20,6 +20,7 @@ #include #include #include +#include #include "internal.h" #include @@ -1447,7 +1448,11 @@ static int generic_copy_file_checks(struct file *file_in, loff_t pos_in, return -EOVERFLOW; /* Shorten the copy to EOF */ - size_in = i_size_read(inode_in); + if (S_ISBLK(inode_in->i_mode)) + size_in = bdev_nr_bytes(I_BDEV(file_in->f_mapping->host)); + else + size_in = i_size_read(inode_in); + if (pos_in >= size_in) count = 0; else @@ -1708,7 +1713,9 @@ int generic_file_rw_checks(struct file *file_in, struct file *file_out) /* Don't copy dirs, pipes, sockets... */ if (S_ISDIR(inode_in->i_mode) || S_ISDIR(inode_out->i_mode)) return -EISDIR; - if (!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) + + if ((!S_ISREG(inode_in->i_mode) || !S_ISREG(inode_out->i_mode)) && + (!S_ISBLK(inode_in->i_mode) || !S_ISBLK(inode_out->i_mode))) return -EINVAL; if (!(file_in->f_mode & FMODE_READ) || diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index a54153610800..468d5f3378e2 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1057,6 +1057,9 @@ int blkdev_issue_secure_erase(struct block_device *bdev, sector_t sector, int blkdev_issue_copy(struct block_device *bdev_in, loff_t pos_in, struct block_device *bdev_out, loff_t pos_out, size_t len, cio_iodone_t end_io, void *private, gfp_t gfp_mask); +int blkdev_copy_offload(struct block_device *bdev_in, loff_t pos_in, + struct block_device *bdev_out, loff_t pos_out, size_t len, + cio_iodone_t end_io, void *private, gfp_t gfp_mask); struct bio *bio_map_kern(struct request_queue *q, void *data, unsigned int len, gfp_t gfp_mask); void bio_map_kern_endio(struct bio *bio); From patchwork Tue Apr 11 08:10:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13207186 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AE34C76196 for ; Tue, 11 Apr 2023 08:20:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230349AbjDKIUP (ORCPT ); Tue, 11 Apr 2023 04:20:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58424 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230339AbjDKITp (ORCPT ); Tue, 11 Apr 2023 04:19:45 -0400 Received: from mailout3.samsung.com (mailout3.samsung.com [203.254.224.33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC2263C11 for ; Tue, 11 Apr 2023 01:19:28 -0700 (PDT) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout3.samsung.com (KnoxPortal) with ESMTP id 20230411081927epoutp0380b436c811afa060fc425867dacb909e~U043odr9b1232212322epoutp03m for ; Tue, 11 Apr 2023 08:19:27 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout3.samsung.com 20230411081927epoutp0380b436c811afa060fc425867dacb909e~U043odr9b1232212322epoutp03m DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681201167; bh=vB3u7+eUdlx2QSFI1UvvfPgkWkEEXblxw0rGBh0MicI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PtZWg5YwJ+7PZDmpoeZUH1Sxrthn8f3+YOJKYrZztrCK73Pp3WYZBetmF05c5w17l nc0eD2xM0eiGBBusSgXnoqAx9qst/YsjDOAtVzvcjW6n3SMSBBewrFg1QgCgiwh6yi A61rxGBIKh9O8Kxog91EhMjPXEGYJmnR7D1ENKwA= Received: from epsnrtp3.localdomain (unknown [182.195.42.164]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230411081926epcas5p348239af51fa255c7f70502c4d871ffee~U042-cFFT2309423094epcas5p35; Tue, 11 Apr 2023 08:19:26 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.178]) by epsnrtp3.localdomain (Postfix) with ESMTP id 4Pwdzc4GLtz4x9Q7; Tue, 11 Apr 2023 08:19:24 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id E2.CC.09987.C0815346; Tue, 11 Apr 2023 17:19:24 +0900 (KST) Received: from epsmtrp2.samsung.com (unknown [182.195.40.14]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230411081323epcas5p3372cfe0159cfd9da0948d607aa548405~U0zkr0JcO2478124781epcas5p3q; Tue, 11 Apr 2023 08:13:23 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp2.samsung.com (KnoxPortal) with ESMTP id 20230411081323epsmtrp2c44dfad17473186ac55fc49d3f0da82b~U0zkqj3wd2545725457epsmtrp23; Tue, 11 Apr 2023 08:13:23 +0000 (GMT) X-AuditID: b6c32a4b-a67fd70000002703-20-6435180c6dda Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 94.1F.08279.3A615346; Tue, 11 Apr 2023 17:13:23 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230411081319epsmtip29a1c8e1a5c6587a749693064e745c356~U0zhGRB2A1199111991epsmtip2R; Tue, 11 Apr 2023 08:13:19 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , =?utf-8?q?Javi?= =?utf-8?q?er_Gonz=C3=A1lez?= , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v9 5/9] nvme: add copy offload support Date: Tue, 11 Apr 2023 13:40:32 +0530 Message-Id: <20230411081041.5328-6-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230411081041.5328-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta0xTZxjOd045FDLY4aafbGNdlw3QUFop3YfI2OLtRMhCYky2xVkbegJI aZueMpz7MbYiAgqt5RLX4hTmUEAEgQmI5VIQuciUoYw7DGEYQOQS0QUYoxzc/Pc8z/e8eZ/3 /fJycddz9p7cGKWW1ihlCj7hyLnZ5Ovt9wYMlAvXGnmotL0FRz8YVnFUPKQn0HTTAkA5c3/j aLQ+FFlmzXaor6EGQ7fzjRgqLL6Dodq8eQw9Hly0R3fWnhLIaO0BaOKRCUOW/h3otqWNg7pv 5RLoYsGEPbJm6jBUPf49QDeXL+Lo+vQzDmrtfwvdX71r98k2qvthGGUa6SSoGtOQPXV/+AaH 6u6Mp8qLUgmq4vJ3VG1fIkGl62YJ6lndI4LKqCwCVEXHt9RiuRdVPv4Ui3D+MnZ3NC2T0xoe rYxUyWOUUSH8sEPSPdJAiVDkJwpCH/F5SlkcHcLfGx7htz9Gsb4GPu9rmSJ+XYqQMQzf/+Pd GlW8luZFqxhtCJ9WyxVqsVrAyOKYeGWUQElrd4mEwp2B68ZjsdHPr1Xj6uqEE+a8LpAI7h1L A1wuJMXw4R8H0oAj15WsBTCpYQRjyQKAw6tjBEuWALw6kGyfBhw2Kuo7ngD2wQKgpSAVZ4kO g+biMdzmIkhv2Dx5asPlTibjcH4ilWMjOGnE4eUrZ4DN5UZK4ExZJWHDHPIDWDZVyLFhJxLB Gd08YBP6Q/2Ii012IIPgyoUUgrW4wLYfxzfsOPku1P1q3kgBySoHmP37KIfNuheulLwgWOwG p+5Wbs7gCRdnLZt6FHzZPYGxWA11LXWAxaHwVLset2XASV9Yesufld+B2e3XMbavM0xfHt8s dYLVP73CfHi6MHcTQ2j5LXETUzCpbtqO3dZZAM/9WWVvADzTa/OYXpvH9H/rSwAvAttoNRMX RTOB6gAlnfDfN0eq4srBxm1sD6sGY6NzAivAuMAKIBfnuzs9XwuQuzrJZd+cpDUqqSZeQTNW ELi+73O4p0ekav24lFqpSBwkFEskEnFQgETE3+rkHdIW6UpGybR0LE2rac2rOozr4JmI2Xkc 7DGc9fr56Iy15MMtqoaErJy4AWpHaU7Wpfkrht4HD5ybJKIvapprn7xcazZ91Vqu542sFBkj u9BxH3Lw5HGnmjJ0oXu5ct+BwTNzUcmfSmdG5+tbG9Oc8yIs+pbTFm1s//m+pab0LdckAzVT HVOZB9/f6RPe0du4Xy/xriyrOkQLdg15aZnD/aY5o++LfGbfvYUjnfmWnqvue5KotwvCw/rU hs5lj0VD53nHkPeCMZVYkJLbpZvzm1z6/PBnN9b+8fQ6Mlv0S+Fjx9CsPnmo0bx10C1YvTZz lHQRnPDhZCiE2rTgdn22R69iuMSamJnx10KvOWWyouvNOiWulvI5TLRMtB3XMLJ/AS6VHhik BAAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrLIsWRmVeSWpSXmKPExsWy7bCSvO5iMdMUg92XBSzWnzrGbNE04S+z xeq7/WwWrw9/YrSY9uEns8WD/fYWe9/NZrW4eWAnk8WeRZOYLFauPspksXvhRyaLx3c+s1sc /f+WzWLSoWuMFk+vzmKy2HtL22LP3pMsFpd3zWGzmL/sKbvFocnNTBY7njQyWmz7PZ/ZYt3r 9ywWJ25JW5z/e5zVQdLj8hVvj1n3z7J57Jx1l93j/L2NLB6Xz5Z6bFrVyeaxeUm9x+6bDWwe vc3v2Dze77vK5tG3ZRWjx+bT1R6fN8l5bHrylimAL4rLJiU1J7MstUjfLoEr4+uaHcwFO8or Zi+8yNjAeCahi5GTQ0LARGL/6ReMXYxcHEICuxkldp/9wwKRkJA49XIZI4QtLLHy33N2iKJG Jok7E2YwgyTYBNQljjxvBesWEZjALHHpfgMbiMMssIBZYuKq02wgVcICZhJvNmwBs1kEVCU2 vFoJtoJXwELiTfNHoG4OoBX6Ev33BUHCnAKWEn/mdrCBhIWASpbsgaoWlDg58wkLSJgZaO/6 eUIgYWYBeYnmrbOZJzAKzkJSNQuhahaSqgWMzKsYJVMLinPTc4sNCwzzUsv1ihNzi0vz0vWS 83M3MYJTgJbmDsbtqz7oHWJk4mA8xCjBwawkwvv1v3GKEG9KYmVValF+fFFpTmrxIUZpDhYl cd4LXSfjhQTSE0tSs1NTC1KLYLJMHJxSDUziD4xzj/WXTfROC9XL0L7xIHFB1sKDzRUe1059 KFR905P6iSU5sWFpCMeFD1cuzA8JUjuxMGQr5+qdkdUfJEuF3r1wafd/36hYXpKe2S1qezX4 R+IPsxOrOJ/N2RJ392lliZtGQrz3443H/6swaVg/NDDWFPyRGDPVLaTzrrfggylzDa+Y2lw6 dDZoCZ+DkcSGfxW7V/Dx2J06JWSglmwYoZV2JTi3Jfjt9aBLVoV2xvVn9O2XyXjHfGfsZazx O7/r553Pr9xjntxv3ekbk+GT0ua7esZinZ+L5l/hbMzQfPD5fFOq7WKvYxvmTTomvMZqvk+u ZfWzyo/B4k/ni22+8ChYuv3vXoEqrfeyD5VYijMSDbWYi4oTAYLRc19wAwAA X-CMS-MailID: 20230411081323epcas5p3372cfe0159cfd9da0948d607aa548405 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230411081323epcas5p3372cfe0159cfd9da0948d607aa548405 References: <20230411081041.5328-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Nitesh Shetty For device supporting native copy, nvme driver receives read and write request with BLK_COPY op flags. For read request the nvme driver populates the payload with source information. For write request the driver converts it to nvme copy command using the source information in the payload and submits to the device. current design only supports single source range. This design is courtesy Mikulas Patocka's token based copy trace event support for nvme_copy_cmd. Set the device copy limits to queue limits. Signed-off-by: Kanchan Joshi Signed-off-by: Nitesh Shetty Signed-off-by: Javier González Signed-off-by: Anuj Gupta --- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 106 +++++++++++++++++++++++++++++++++- drivers/nvme/host/fc.c | 5 ++ drivers/nvme/host/nvme.h | 7 +++ drivers/nvme/host/pci.c | 27 ++++++++- drivers/nvme/host/rdma.c | 7 +++ drivers/nvme/host/tcp.c | 16 +++++ drivers/nvme/host/trace.c | 19 ++++++ include/linux/nvme.h | 43 +++++++++++++- 9 files changed, 223 insertions(+), 8 deletions(-) diff --git a/drivers/nvme/host/constants.c b/drivers/nvme/host/constants.c index bc523ca02254..01be882b726f 100644 --- a/drivers/nvme/host/constants.c +++ b/drivers/nvme/host/constants.c @@ -19,6 +19,7 @@ static const char * const nvme_ops[] = { [nvme_cmd_resv_report] = "Reservation Report", [nvme_cmd_resv_acquire] = "Reservation Acquire", [nvme_cmd_resv_release] = "Reservation Release", + [nvme_cmd_copy] = "Copy Offload", [nvme_cmd_zone_mgmt_send] = "Zone Management Send", [nvme_cmd_zone_mgmt_recv] = "Zone Management Receive", [nvme_cmd_zone_append] = "Zone Management Append", diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 3ffdc80ebb6c..745c512c324b 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -754,6 +754,80 @@ static inline void nvme_setup_flush(struct nvme_ns *ns, cmnd->common.nsid = cpu_to_le32(ns->head->ns_id); } +static inline blk_status_t nvme_setup_copy_read(struct nvme_ns *ns, + struct request *req) +{ + struct bio *bio = req->bio; + struct nvme_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + + memcpy(token->subsys, "nvme", 4); + token->ns = ns; + token->src_sector = bio->bi_iter.bi_sector; + token->sectors = bio->bi_iter.bi_size >> 9; + + return BLK_STS_OK; +} + +static inline blk_status_t nvme_setup_copy_write(struct nvme_ns *ns, + struct request *req, struct nvme_command *cmnd) +{ + struct nvme_copy_range *range = NULL; + struct bio *bio = req->bio; + struct nvme_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + sector_t src_sector, dst_sector, n_sectors; + u64 src_lba, dst_lba, n_lba; + unsigned short nr_range = 1; + u16 control = 0; + + if (unlikely(memcmp(token->subsys, "nvme", 4))) + return BLK_STS_NOTSUPP; + if (unlikely(token->ns != ns)) + return BLK_STS_NOTSUPP; + + src_sector = token->src_sector; + dst_sector = bio->bi_iter.bi_sector; + n_sectors = token->sectors; + if (WARN_ON(n_sectors != bio->bi_iter.bi_size >> 9)) + return BLK_STS_NOTSUPP; + + src_lba = nvme_sect_to_lba(ns, src_sector); + dst_lba = nvme_sect_to_lba(ns, dst_sector); + n_lba = nvme_sect_to_lba(ns, n_sectors); + + if (WARN_ON(!n_lba)) + return BLK_STS_NOTSUPP; + + if (req->cmd_flags & REQ_FUA) + control |= NVME_RW_FUA; + + if (req->cmd_flags & REQ_FAILFAST_DEV) + control |= NVME_RW_LR; + + memset(cmnd, 0, sizeof(*cmnd)); + cmnd->copy.opcode = nvme_cmd_copy; + cmnd->copy.nsid = cpu_to_le32(ns->head->ns_id); + cmnd->copy.sdlba = cpu_to_le64(dst_lba); + + range = kmalloc_array(nr_range, sizeof(*range), + GFP_ATOMIC | __GFP_NOWARN); + if (!range) + return BLK_STS_RESOURCE; + + range[0].slba = cpu_to_le64(src_lba); + range[0].nlb = cpu_to_le16(n_lba - 1); + + cmnd->copy.nr_range = 0; + + req->special_vec.bv_page = virt_to_page(range); + req->special_vec.bv_offset = offset_in_page(range); + req->special_vec.bv_len = sizeof(*range) * nr_range; + req->rq_flags |= RQF_SPECIAL_PAYLOAD; + + cmnd->copy.control = cpu_to_le16(control); + + return BLK_STS_OK; +} + static blk_status_t nvme_setup_discard(struct nvme_ns *ns, struct request *req, struct nvme_command *cmnd) { @@ -988,10 +1062,16 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req) ret = nvme_setup_discard(ns, req, cmd); break; case REQ_OP_READ: - ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_read); + if (unlikely(req->cmd_flags & REQ_COPY)) + ret = nvme_setup_copy_read(ns, req); + else + ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_read); break; case REQ_OP_WRITE: - ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_write); + if (unlikely(req->cmd_flags & REQ_COPY)) + ret = nvme_setup_copy_write(ns, req, cmd); + else + ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_write); break; case REQ_OP_ZONE_APPEND: ret = nvme_setup_rw(ns, req, cmd, nvme_cmd_zone_append); @@ -1698,6 +1778,26 @@ static void nvme_config_discard(struct gendisk *disk, struct nvme_ns *ns) blk_queue_max_write_zeroes_sectors(queue, UINT_MAX); } +static void nvme_config_copy(struct gendisk *disk, struct nvme_ns *ns, + struct nvme_id_ns *id) +{ + struct nvme_ctrl *ctrl = ns->ctrl; + struct request_queue *q = disk->queue; + + if (!(ctrl->oncs & NVME_CTRL_ONCS_COPY)) { + blk_queue_max_copy_sectors_hw(q, 0); + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + return; + } + + /* setting copy limits */ + if (blk_queue_flag_test_and_set(QUEUE_FLAG_COPY, q)) + return; + + blk_queue_max_copy_sectors_hw(q, + nvme_lba_to_sect(ns, le16_to_cpu(id->mssrl))); +} + static bool nvme_ns_ids_equal(struct nvme_ns_ids *a, struct nvme_ns_ids *b) { return uuid_equal(&a->uuid, &b->uuid) && @@ -1897,6 +1997,7 @@ static void nvme_update_disk_info(struct gendisk *disk, set_capacity_and_notify(disk, capacity); nvme_config_discard(disk, ns); + nvme_config_copy(disk, ns, id); blk_queue_max_write_zeroes_sectors(disk->queue, ns->ctrl->max_zeroes_sectors); } @@ -5346,6 +5447,7 @@ static inline void _nvme_check_size(void) BUILD_BUG_ON(sizeof(struct nvme_download_firmware) != 64); BUILD_BUG_ON(sizeof(struct nvme_format_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_dsm_cmd) != 64); + BUILD_BUG_ON(sizeof(struct nvme_copy_command) != 64); BUILD_BUG_ON(sizeof(struct nvme_write_zeroes_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_abort_cmd) != 64); BUILD_BUG_ON(sizeof(struct nvme_get_log_page_command) != 64); diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index 2ed75923507d..db2e22b4ca7f 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -2807,6 +2807,11 @@ nvme_fc_queue_rq(struct blk_mq_hw_ctx *hctx, if (ret) return ret; + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_end_request(rq, BLK_STS_OK); + return BLK_STS_OK; + } /* * nvme core doesn't quite treat the rq opaquely. Commands such * as WRITE ZEROES will return a non-zero rq payload_bytes yet diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index bf46f122e9e1..257f91ee1f2d 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -500,6 +500,13 @@ struct nvme_ns { }; +struct nvme_copy_token { + char subsys[4]; + struct nvme_ns *ns; + u64 src_sector; + u64 sectors; +}; + /* NVMe ns supports metadata actions by the controller (generate/strip) */ static inline bool nvme_ns_has_pi(struct nvme_ns *ns) { diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 282d808400c5..9fdf56256cdb 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -496,16 +496,19 @@ static inline void nvme_sq_copy_cmd(struct nvme_queue *nvmeq, nvmeq->sq_tail = 0; } -static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +static inline void nvme_commit_sq_db(struct nvme_queue *nvmeq) { - struct nvme_queue *nvmeq = hctx->driver_data; - spin_lock(&nvmeq->sq_lock); if (nvmeq->sq_tail != nvmeq->last_sq_tail) nvme_write_sq_db(nvmeq, true); spin_unlock(&nvmeq->sq_lock); } +static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) +{ + nvme_commit_sq_db(hctx->driver_data); +} + static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req, int nseg) { @@ -849,6 +852,12 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, struct request *req) if (ret) return ret; + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_start_request(req); + return BLK_STS_OK; + } + if (blk_rq_nr_phys_segments(req)) { ret = nvme_map_data(dev, req, &iod->cmd); if (ret) @@ -895,6 +904,18 @@ static blk_status_t nvme_queue_rq(struct blk_mq_hw_ctx *hctx, ret = nvme_prep_rq(dev, req); if (unlikely(ret)) return ret; + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_set_request_complete(req); + blk_mq_end_request(req, BLK_STS_OK); + /* Commit the sq if copy read was the last req in the list, + * as copy read deoesn't update sq db + */ + if (bd->last) + nvme_commit_sq_db(nvmeq); + return ret; + } + spin_lock(&nvmeq->sq_lock); nvme_sq_copy_cmd(nvmeq, &iod->cmd); nvme_write_sq_db(nvmeq, bd->last); diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index bbad26b82b56..a8bf2a87f42a 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -2043,6 +2043,13 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, nvme_start_request(rq); + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_end_request(rq, BLK_STS_OK); + ret = BLK_STS_OK; + goto unmap_qe; + } + if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) && queue->pi_support && (c->common.opcode == nvme_cmd_write || diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 49c9e7bc9116..aaeb761c56a1 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -2370,6 +2370,11 @@ static blk_status_t nvme_tcp_setup_cmd_pdu(struct nvme_ns *ns, if (ret) return ret; + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + return BLK_STS_OK; + } + req->state = NVME_TCP_SEND_CMD_PDU; req->status = cpu_to_le16(NVME_SC_SUCCESS); req->offset = 0; @@ -2438,6 +2443,17 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx, nvme_start_request(rq); + if (unlikely((rq->cmd_flags & REQ_COPY) && + (req_op(rq) == REQ_OP_READ))) { + blk_mq_set_request_complete(rq); + blk_mq_end_request(rq, BLK_STS_OK); + /* if copy read is the last req queue tcp reqs */ + if (bd->last && nvme_tcp_queue_more(queue)) + queue_work_on(queue->io_cpu, nvme_tcp_wq, + &queue->io_work); + return ret; + } + nvme_tcp_queue_request(req, true, bd->last); return BLK_STS_OK; diff --git a/drivers/nvme/host/trace.c b/drivers/nvme/host/trace.c index 1c36fcedea20..da4a7494e5a7 100644 --- a/drivers/nvme/host/trace.c +++ b/drivers/nvme/host/trace.c @@ -150,6 +150,23 @@ static const char *nvme_trace_read_write(struct trace_seq *p, u8 *cdw10) return ret; } +static const char *nvme_trace_copy(struct trace_seq *p, u8 *cdw10) +{ + const char *ret = trace_seq_buffer_ptr(p); + u64 slba = get_unaligned_le64(cdw10); + u8 nr_range = get_unaligned_le16(cdw10 + 8); + u16 control = get_unaligned_le16(cdw10 + 10); + u32 dsmgmt = get_unaligned_le32(cdw10 + 12); + u32 reftag = get_unaligned_le32(cdw10 + 16); + + trace_seq_printf(p, + "slba=%llu, nr_range=%u, ctrl=0x%x, dsmgmt=%u, reftag=%u", + slba, nr_range, control, dsmgmt, reftag); + trace_seq_putc(p, 0); + + return ret; +} + static const char *nvme_trace_dsm(struct trace_seq *p, u8 *cdw10) { const char *ret = trace_seq_buffer_ptr(p); @@ -243,6 +260,8 @@ const char *nvme_trace_parse_nvm_cmd(struct trace_seq *p, return nvme_trace_zone_mgmt_send(p, cdw10); case nvme_cmd_zone_mgmt_recv: return nvme_trace_zone_mgmt_recv(p, cdw10); + case nvme_cmd_copy: + return nvme_trace_copy(p, cdw10); default: return nvme_trace_common(p, cdw10); } diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 779507ac750b..6582b26e532c 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -337,7 +337,7 @@ struct nvme_id_ctrl { __u8 nvscc; __u8 nwpc; __le16 acwu; - __u8 rsvd534[2]; + __le16 ocfs; __le32 sgls; __le32 mnan; __u8 rsvd544[224]; @@ -365,6 +365,7 @@ enum { NVME_CTRL_ONCS_WRITE_ZEROES = 1 << 3, NVME_CTRL_ONCS_RESERVATIONS = 1 << 5, NVME_CTRL_ONCS_TIMESTAMP = 1 << 6, + NVME_CTRL_ONCS_COPY = 1 << 8, NVME_CTRL_VWC_PRESENT = 1 << 0, NVME_CTRL_OACS_SEC_SUPP = 1 << 0, NVME_CTRL_OACS_NS_MNGT_SUPP = 1 << 3, @@ -414,7 +415,10 @@ struct nvme_id_ns { __le16 npdg; __le16 npda; __le16 nows; - __u8 rsvd74[18]; + __le16 mssrl; + __le32 mcl; + __u8 msrc; + __u8 rsvd91[11]; __le32 anagrpid; __u8 rsvd96[3]; __u8 nsattr; @@ -796,6 +800,7 @@ enum nvme_opcode { nvme_cmd_resv_report = 0x0e, nvme_cmd_resv_acquire = 0x11, nvme_cmd_resv_release = 0x15, + nvme_cmd_copy = 0x19, nvme_cmd_zone_mgmt_send = 0x79, nvme_cmd_zone_mgmt_recv = 0x7a, nvme_cmd_zone_append = 0x7d, @@ -819,7 +824,8 @@ enum nvme_opcode { nvme_opcode_name(nvme_cmd_resv_release), \ nvme_opcode_name(nvme_cmd_zone_mgmt_send), \ nvme_opcode_name(nvme_cmd_zone_mgmt_recv), \ - nvme_opcode_name(nvme_cmd_zone_append)) + nvme_opcode_name(nvme_cmd_zone_append), \ + nvme_opcode_name(nvme_cmd_copy)) @@ -996,6 +1002,36 @@ struct nvme_dsm_range { __le64 slba; }; +struct nvme_copy_command { + __u8 opcode; + __u8 flags; + __u16 command_id; + __le32 nsid; + __u64 rsvd2; + __le64 metadata; + union nvme_data_ptr dptr; + __le64 sdlba; + __u8 nr_range; + __u8 rsvd12; + __le16 control; + __le16 rsvd13; + __le16 dspec; + __le32 ilbrt; + __le16 lbat; + __le16 lbatm; +}; + +struct nvme_copy_range { + __le64 rsvd0; + __le64 slba; + __le16 nlb; + __le16 rsvd18; + __le32 rsvd20; + __le32 eilbrt; + __le16 elbat; + __le16 elbatm; +}; + struct nvme_write_zeroes_cmd { __u8 opcode; __u8 flags; @@ -1757,6 +1793,7 @@ struct nvme_command { struct nvme_download_firmware dlfw; struct nvme_format_cmd format; struct nvme_dsm_cmd dsm; + struct nvme_copy_command copy; struct nvme_write_zeroes_cmd write_zeroes; struct nvme_zone_mgmt_send_cmd zms; struct nvme_zone_mgmt_recv_cmd zmr; From patchwork Tue Apr 11 08:10:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13207188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DEDEC7619A for ; Tue, 11 Apr 2023 08:20:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230251AbjDKIUj (ORCPT ); Tue, 11 Apr 2023 04:20:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230246AbjDKITv (ORCPT ); Tue, 11 Apr 2023 04:19:51 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2DCD40C5 for ; Tue, 11 Apr 2023 01:19:33 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230411081932epoutp01253dfa2c79f03b02620cc52feb3428e8~U048B-zbY0720007200epoutp018 for ; Tue, 11 Apr 2023 08:19:32 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230411081932epoutp01253dfa2c79f03b02620cc52feb3428e8~U048B-zbY0720007200epoutp018 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681201172; bh=k1bVpVyxbnP2n/Pv5/agKnJF+GMbukvS2GpzzjdrKh4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lYtgpgo5yzCpCS1Za/rwoIXYBDVLTHZG8ctbLssl6ASRZPS2fg+WjDzjo7wsP1K6i HOSOWr5XWUpea/4Y0SQKF2Sw4E+uubaBJGm6uqlVYT3PetlKvYOXFgRvaKQaZEJ3fn PEHIyIZpGp+2K0M7t3P14oQszuamEAa0YjuI3fF0= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230411081930epcas5p170a47f66c746b9701742d0cc9b3d1f10~U047APlX-0386303863epcas5p1H; Tue, 11 Apr 2023 08:19:30 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.180]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4Pwdzj188Bz4x9Ps; Tue, 11 Apr 2023 08:19:29 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 4F.6B.09961.01815346; Tue, 11 Apr 2023 17:19:28 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230411081332epcas5p257c090a0d1ea6abf98416ca687f6c1e1~U0ztP0i8E1989319893epcas5p2A; Tue, 11 Apr 2023 08:13:32 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230411081332epsmtrp13f71e3a58be2eae3a33c354b1ea119e8~U0ztO7KNa1918219182epsmtrp1N; Tue, 11 Apr 2023 08:13:32 +0000 (GMT) X-AuditID: b6c32a49-2c1ff700000026e9-62-64351810ea7f Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 11.FD.08609.CA615346; Tue, 11 Apr 2023 17:13:32 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230411081329epsmtip281d01e8da5afbcf0f02a3ea104ab9c3c~U0zp45Bn81199111991epsmtip2T; Tue, 11 Apr 2023 08:13:28 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v9 6/9] nvmet: add copy command support for bdev and file ns Date: Tue, 11 Apr 2023 13:40:33 +0530 Message-Id: <20230411081041.5328-7-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230411081041.5328-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02TfVDTdRzH+/5+48fG3byfYPCNTlhLWkA8DBh94XjwQrlfpweUUV6njB37 OTjGtvaQUNw1ITAwwbDsHBTKFR5MISbZxsPsQORJoJp0ogxMwQcQpmBXhEhjPyj/e30+3/fn 8XsfNu55xN2XnaPQ0mqFRM4nPFgXugMFISQUScP7KmJR88BlHBUdW8GR0V5JoNnuBYBOPFzC 0c2LiahzvtoNjf1swVBHXRWGGow9GGo//QhDPatzBKrq+h2g6VEDhjqvB6OOzn4WsrXVEKi2 ftoddR0vxpB56hBAF5ZrcdQ062ChvusvopGVXrftkLJd3UUZJocIymKwu1MjEy0syjako0yN ZQR1/rtPqPYxPUEdLZ4nKId1lKAqWhsBdX7wY2rR5EeZpuawtE3v58Zl0xIprebRiiylNEch i+fv2iNOEouiw4Uhwhj0Op+nkOTR8fwdu9NCknPkzg3weR9K5DqnK02i0fDDEuLUSp2W5mUr Ndp4Pq2SylVRqlCNJE+jU8hCFbQ2VhgeHiFyCjNzs0/Xd7irjNvzu2946cGEqBxw2JCMgnfu 6oly4MH2JNsBnF/8CTDGAoCWs/0sxlgEcGD8Gr4RYh01uDEPbQC2Nc+tG8UYvNX5PbamIkgB vHS3xJVrC1mKw0fTZa5cOPkbBp+0Oly5vMjd8JLlIlhjFhkAK2xFrDXmkgjaW1ucGrazXhis nNy85uaQMfDJN58RjGQz7D855ZLjpD8s/rEaX8sPyVoObL8zzGJ63QF/WBghGPaCM72t7gz7 wvuVpessg3/bpjGGVbD4shUwnAhLBipdPeBkIGxuC2PcW+FXA00YU3cTPLo8tR7KheZvN5gP DzfUrDOEncN6jBmFglc6uMyyPgfw9o1h/BjgGZ4Zx/DMOIb/K58CeCN4gVZp8mS0RqQSKuiD //1yljLPBFxXEfSmGdhvPgztAhgbdAHIxvlbuH+uRko9uVJJwUe0WilW6+S0pguInOv+Avd9 PkvpPCuFViyMigmPio6OjoqJjBbyfbiC+P4sT1Im0dK5NK2i1RtxGJvjq8cSdm5NfXDvQKN5 tVAiPJB5yO1pkGfy/LlVWa81veTL56Rmb6+AWz7skomIJTH76crtM96c2l/8CiOU5emn0l+O 9ZmJ6Bsqzbde8W/SQ+v+OkOkKuj41+qD93OvBjY7xh3+RWX73zq8czYsYfzMu/UB1ANzNWeZ J3gM/LrlQfcElG5sVMx5r8eYspR5Ai/I4I56F7w9kyvdti+f9UeG+x7Vpyb5iKNmG5nc3BKc +pLjXFxK+AcRfb5JwTn814ZT/tk7OCsm7MJylWmQk2S3vPHqr5NFZQl746o4J98xHtn3WGGL zHolXWBewRMNhrMef1mEqRmlk8YGczcoKGy6FgLq+CxNtkQYhKs1kn8BDrSeF54EAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02RfyyUcRzH932e5557WNceR903Ku2WVcqPkvn2Q6SVZ0st6Q9rK566Zzzj ZHf0Q6krs+qWI7aa0y9mzClxSuSUTgklzaH8uChkJodiJqGO2vrv/Xm/Xvt8/vhQuPgF4Ujx MXGcIoaNlpK2RFmN1Nnt/lJvmefllGXoYUMtji6lzeCo0JxKoqGa7wDdGJ3CUc9zP1RlyRKg 9uoKDBly0jFUUPgKQ5XZYxh6NTdMonRjG0D9rVoMVXWsR4aqegKZnt4i0d28fiEyZiRhqLzv IkBl03dxVDQ0QqC6DifUNPNa4A8ZU8teRtvdSDIVWrOQafpUQjCmxnhGr7tKMqW5F5jKdhXJ pCRZSGbkWSvJaB7pAFP65izzQ7+S0fcNYwcWH7bdLuOi+ZOcwmNHuG1kdp5BGFvof7qm014F PnmrgQ0F6c3wWatWoAa2lJguB/BKbZlwAUDYMJgHFrI9LJgdEC5IFzGY3ZYzL5H0GvhyIBlY gQOdhsPmbhVpHXC6B4NNA+MCq2VPB8GXFc/nVxG0C9SYLhHWLKIRND8qwdWA+nPCA6Z221lr G3oL/HX7CmmtxX+UXMNf2w7WZ/bNZ5x2hkmPs/A0QGv/Q9r/0D2A6cAyLlYpj5ArN8ZuiuFO uStZuTI+JsL9+Am5Hsx/2dW1HBh0o+5GgFHACCCFSx1EE3NeMrFIxp5J4BQnwhTx0ZzSCJwo QioRvVfXh4npCDaOi+K4WE7xj2KUjaMKK4qK1idf9Vu1b2+QIAccWV7dUY/koU98A0BmicuN sQEOfeXOX9/ZCRZFhGQUOzmvmPGzkRx+4BNWGx68QRNQpcZEmYKptwk+bflZrCmQTTcHJU26 7BnPKOF98zfMPs6/8Hmo+lzK7p6ZNpMhhOuTNjcbNO2qQ+zN/AbeEufrJUm8nZy4dnrXOnaC gEddhwO7ln/W534zN852TXs2rQ5eophmVsnqKvV5RXssIbzEThr6sS6Yv/aBf+ff/UWTYbZX v9gZlVXB77+OZ6aWToUfnDtaKu93K7rjmSpKKByTWAZ9J8u2um0WjB/b1vwz4IdHcW9vZ+Ao v1/RdV7XIiWUkexGV1yhZH8DcS0kglQDAAA= X-CMS-MailID: 20230411081332epcas5p257c090a0d1ea6abf98416ca687f6c1e1 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230411081332epcas5p257c090a0d1ea6abf98416ca687f6c1e1 References: <20230411081041.5328-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Nitesh Shetty Add support for handling target command on target. For bdev-ns we call into blkdev_issue_copy, which the block layer completes by a offloaded copy request to backend bdev or by emulating the request. For file-ns we call vfs_copy_file_range to service our request. Currently target always shows copy capability by setting NVME_CTRL_ONCS_COPY in controller ONCS. Signed-off-by: Nitesh Shetty Signed-off-by: Anuj Gupta --- drivers/nvme/target/admin-cmd.c | 9 +++-- drivers/nvme/target/io-cmd-bdev.c | 58 +++++++++++++++++++++++++++++++ drivers/nvme/target/io-cmd-file.c | 52 +++++++++++++++++++++++++++ drivers/nvme/target/loop.c | 6 ++++ drivers/nvme/target/nvmet.h | 1 + 5 files changed, 124 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c index 80099df37314..978786ec6a9e 100644 --- a/drivers/nvme/target/admin-cmd.c +++ b/drivers/nvme/target/admin-cmd.c @@ -433,8 +433,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) id->nn = cpu_to_le32(NVMET_MAX_NAMESPACES); id->mnan = cpu_to_le32(NVMET_MAX_NAMESPACES); id->oncs = cpu_to_le16(NVME_CTRL_ONCS_DSM | - NVME_CTRL_ONCS_WRITE_ZEROES); - + NVME_CTRL_ONCS_WRITE_ZEROES | NVME_CTRL_ONCS_COPY); /* XXX: don't report vwc if the underlying device is write through */ id->vwc = NVME_CTRL_VWC_PRESENT; @@ -536,6 +535,12 @@ static void nvmet_execute_identify_ns(struct nvmet_req *req) if (req->ns->bdev) nvmet_bdev_set_limits(req->ns->bdev, id); + else { + id->msrc = (u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16(BIO_MAX_VECS << + (PAGE_SHIFT - SECTOR_SHIFT)); + id->mcl = cpu_to_le32(le16_to_cpu(id->mssrl)); + } /* * We just provide a single LBA format that matches what the diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c index c2d6cea0236b..0af273097aa4 100644 --- a/drivers/nvme/target/io-cmd-bdev.c +++ b/drivers/nvme/target/io-cmd-bdev.c @@ -46,6 +46,19 @@ void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id) id->npda = id->npdg; /* NOWS = Namespace Optimal Write Size */ id->nows = to0based(bdev_io_opt(bdev) / bdev_logical_block_size(bdev)); + + /*Copy limits*/ + if (bdev_max_copy_sectors(bdev)) { + id->msrc = id->msrc; + id->mssrl = cpu_to_le16((bdev_max_copy_sectors(bdev) << + SECTOR_SHIFT) / bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32(id->mssrl); + } else { + id->msrc = (u8)to0based(BIO_MAX_VECS - 1); + id->mssrl = cpu_to_le16((BIO_MAX_VECS << PAGE_SHIFT) / + bdev_logical_block_size(bdev)); + id->mcl = cpu_to_le32(id->mssrl); + } } void nvmet_bdev_ns_disable(struct nvmet_ns *ns) @@ -184,6 +197,19 @@ static void nvmet_bio_done(struct bio *bio) nvmet_req_bio_put(req, bio); } +static void nvmet_bdev_copy_end_io(void *private, int comp_len) +{ + struct nvmet_req *req = (struct nvmet_req *)private; + + if (comp_len == req->copy_len) { + req->cqe->result.u32 = cpu_to_le32(1); + nvmet_req_complete(req, errno_to_nvme_status(req, 0)); + } else { + req->cqe->result.u32 = cpu_to_le32(0); + nvmet_req_complete(req, blk_to_nvme_status(req, BLK_STS_IOERR)); + } +} + #ifdef CONFIG_BLK_DEV_INTEGRITY static int nvmet_bdev_alloc_bip(struct nvmet_req *req, struct bio *bio, struct sg_mapping_iter *miter) @@ -450,6 +476,34 @@ static void nvmet_bdev_execute_write_zeroes(struct nvmet_req *req) } } +/* At present we handle only one range entry */ +static void nvmet_bdev_execute_copy(struct nvmet_req *req) +{ + struct nvme_copy_range range; + struct nvme_command *cmnd = req->cmd; + int ret; + + + ret = nvmet_copy_from_sgl(req, 0, &range, sizeof(range)); + if (ret) + goto out; + + ret = blkdev_issue_copy(req->ns->bdev, + le64_to_cpu(cmnd->copy.sdlba) << req->ns->blksize_shift, + req->ns->bdev, + le64_to_cpu(range.slba) << req->ns->blksize_shift, + (le16_to_cpu(range.nlb) + 1) << req->ns->blksize_shift, + nvmet_bdev_copy_end_io, (void *)req, GFP_KERNEL); + if (ret) { + req->cqe->result.u32 = cpu_to_le32(0); + nvmet_req_complete(req, blk_to_nvme_status(req, BLK_STS_IOERR)); + } + + return; +out: + nvmet_req_complete(req, errno_to_nvme_status(req, ret)); +} + u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) { switch (req->cmd->common.opcode) { @@ -468,6 +522,10 @@ u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_bdev_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_bdev_execute_copy; + return 0; + default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c index 2d068439b129..69f198ecec77 100644 --- a/drivers/nvme/target/io-cmd-file.c +++ b/drivers/nvme/target/io-cmd-file.c @@ -322,6 +322,49 @@ static void nvmet_file_dsm_work(struct work_struct *w) } } +static void nvmet_file_copy_work(struct work_struct *w) +{ + struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); + int nr_range; + loff_t pos; + struct nvme_command *cmnd = req->cmd; + int ret = 0, len = 0, src, id; + + nr_range = cmnd->copy.nr_range + 1; + pos = le64_to_cpu(req->cmd->copy.sdlba) << req->ns->blksize_shift; + if (unlikely(pos + req->transfer_len > req->ns->size)) { + nvmet_req_complete(req, errno_to_nvme_status(req, -ENOSPC)); + return; + } + + for (id = 0 ; id < nr_range; id++) { + struct nvme_copy_range range; + + ret = nvmet_copy_from_sgl(req, id * sizeof(range), &range, + sizeof(range)); + if (ret) + goto out; + + len = (le16_to_cpu(range.nlb) + 1) << (req->ns->blksize_shift); + src = (le64_to_cpu(range.slba) << (req->ns->blksize_shift)); + ret = vfs_copy_file_range(req->ns->file, src, req->ns->file, + pos, len, 0); +out: + if (ret != len) { + pos += ret; + req->cqe->result.u32 = cpu_to_le32(id); + nvmet_req_complete(req, ret < 0 ? + errno_to_nvme_status(req, ret) : + errno_to_nvme_status(req, -EIO)); + return; + + } else + pos += len; + } + + nvmet_req_complete(req, 0); + +} static void nvmet_file_execute_dsm(struct nvmet_req *req) { if (!nvmet_check_data_len_lte(req, nvmet_dsm_len(req))) @@ -330,6 +373,12 @@ static void nvmet_file_execute_dsm(struct nvmet_req *req) queue_work(nvmet_wq, &req->f.work); } +static void nvmet_file_execute_copy(struct nvmet_req *req) +{ + INIT_WORK(&req->f.work, nvmet_file_copy_work); + queue_work(nvmet_wq, &req->f.work); +} + static void nvmet_file_write_zeroes_work(struct work_struct *w) { struct nvmet_req *req = container_of(w, struct nvmet_req, f.work); @@ -376,6 +425,9 @@ u16 nvmet_file_parse_io_cmd(struct nvmet_req *req) case nvme_cmd_write_zeroes: req->execute = nvmet_file_execute_write_zeroes; return 0; + case nvme_cmd_copy: + req->execute = nvmet_file_execute_copy; + return 0; default: return nvmet_report_invalid_opcode(req); } diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c index f2d24b2d992f..d18ed8067a15 100644 --- a/drivers/nvme/target/loop.c +++ b/drivers/nvme/target/loop.c @@ -146,6 +146,12 @@ static blk_status_t nvme_loop_queue_rq(struct blk_mq_hw_ctx *hctx, return ret; nvme_start_request(req); + if (unlikely((req->cmd_flags & REQ_COPY) && + (req_op(req) == REQ_OP_READ))) { + blk_mq_set_request_complete(req); + blk_mq_end_request(req, BLK_STS_OK); + return BLK_STS_OK; + } iod->cmd.common.flags |= NVME_CMD_SGL_METABUF; iod->req.port = queue->ctrl->port; if (!nvmet_req_init(&iod->req, &queue->nvme_cq, diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h index 89bedfcd974c..69ed4c8469e5 100644 --- a/drivers/nvme/target/nvmet.h +++ b/drivers/nvme/target/nvmet.h @@ -393,6 +393,7 @@ struct nvmet_req { struct device *p2p_client; u16 error_loc; u64 error_slba; + size_t copy_len; }; #define NVMET_MAX_MPOOL_BVEC 16 From patchwork Tue Apr 11 08:10:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13207189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52AF3C7619A for ; Tue, 11 Apr 2023 08:21:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230333AbjDKIU6 (ORCPT ); Tue, 11 Apr 2023 04:20:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230344AbjDKIUP (ORCPT ); Tue, 11 Apr 2023 04:20:15 -0400 Received: from mailout1.samsung.com (mailout1.samsung.com [203.254.224.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2A4044A8 for ; Tue, 11 Apr 2023 01:19:46 -0700 (PDT) Received: from epcas5p3.samsung.com (unknown [182.195.41.41]) by mailout1.samsung.com (KnoxPortal) with ESMTP id 20230411081945epoutp01fa7c2251cff76ff386a80f8b5fddbe2d~U05IbIzJ_0717807178epoutp01u for ; Tue, 11 Apr 2023 08:19:45 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.samsung.com 20230411081945epoutp01fa7c2251cff76ff386a80f8b5fddbe2d~U05IbIzJ_0717807178epoutp01u DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681201185; bh=3oNvHZOTqNqlN+q+wZYXheAb4+r+epzc3PSTrslmBBc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EHtWEoyXYfSlNY7aTlWuec23AQwEYVn79UB/NiKGbGZuD23gCJajKbNf1S6kvZ6o/ soXRaSpn9Acs65Tum3RP3vWmybtPbxv/tlMZh0jzkLhyt6zGTCLXpU3dPlFZmgHEjc NgUB53Eq4L4qj8ypy+8gkMsv23dZKqAa4Kbe1k+o= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p3.samsung.com (KnoxPortal) with ESMTP id 20230411081944epcas5p32c059cbc4322d2f40a830470c4961fae~U05Hws4EY0205802058epcas5p3V; Tue, 11 Apr 2023 08:19:44 +0000 (GMT) Received: from epsmges5p3new.samsung.com (unknown [182.195.38.182]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4Pwdzy6Jcqz4x9QK; Tue, 11 Apr 2023 08:19:42 +0000 (GMT) Received: from epcas5p1.samsung.com ( [182.195.41.39]) by epsmges5p3new.samsung.com (Symantec Messaging Gateway) with SMTP id A1.EC.09987.E1815346; Tue, 11 Apr 2023 17:19:42 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p2.samsung.com (KnoxPortal) with ESMTPA id 20230411081342epcas5p22a4c587babd6a373cfa709d9609f65f4~U0z2KF_911873318733epcas5p2C; Tue, 11 Apr 2023 08:13:42 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230411081342epsmtrp165a97aca7b9f744fe5e1ac9e658333dc~U0z2HYLTt1918219182epsmtrp1T; Tue, 11 Apr 2023 08:13:42 +0000 (GMT) X-AuditID: b6c32a4b-7fbff70000002703-5d-6435181e2d29 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id 4A.FD.08609.5B615346; Tue, 11 Apr 2023 17:13:42 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230411081338epsmtip254352bf2c68432e6e66ec343cd492f17~U0zyW-oCI2386223862epsmtip2T; Tue, 11 Apr 2023 08:13:37 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v9 7/9] dm: Add support for copy offload Date: Tue, 11 Apr 2023 13:40:34 +0530 Message-Id: <20230411081041.5328-8-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230411081041.5328-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0zTVxTHc3+/8mshKSkg2wUcdl0IAQRaWsoFAcl0+kvYli5mD0kca+gv lEcf6WPKyBAhPETkUdBAYbzHAmw4ChIEioaXFoPMoU1kMhnSoGOAIkQIICu0bP73Oed+z/2e c24uA3ctpHsykuQaSiUXp3IIJ1r3kJ9voDcMlXAXSv3RtbFRHGWVbOOobbqYQAtDKwBdfbGB o5mbR5FxqcoBPbp1A0P9DToMtbSNYKiv/iWGRnYWCaQbNANkeajHkHEqAPUbTTQ02VtNoNpm Cx0NlmVjqGfuAkDdm7U4al9YpqE7U15oYvu2QwwkJx/Ekvon4wR5Qz9NJyf+7KCRk+Na0tB6 kSA7m86TfY8yCfJy9hJBLg88JMiirlZAdt5NJ18ZvEnD3CImco5LiZRSYgmlYlPyBIUkSZ4Y xYk9FX8sPlTI5QXywlEYhy0Xy6gozvGPRYEnklKtG+CwvxWnaq0pkVit5gRHR6oUWg3FlirU migOpZSkKgXKILVYptbKE4PklCaCx+WGhFqF36RI51vqgPKp57l7Dfm0TDDrXgAcGZAlgI39 r0EBcGK4svoA7FlsxG3BCoD3LR10W/AKQHPJG3y/pKW8gLAd9AL4uvmxXZWNwbH1sj0VwfKF w/M5excfYOXi8KXlIm03wFm/Y3Cra3lP5cZC8A9DDX2XaSwfqNus32OmNZ+1U2H1YFj9gmHx E5fdtCMrHG79kE/YJC7QVDlH22WcdQhmX6+yt1frCCfKP7fxcdi+cMXBxm7w79tddBt7wufF uXZOhOuTFszGSpg9OgBsfBTmjBXjuy3gLD94rTfYln4PXhlrx2y2zvDy5py9lAl7avaZA/Na qu0MofFepp1JOD72o32/hQAO360DJYCtf2sc/Vvj6P+3rgN4K/CglGpZIqUOVfLl1Nn/njlB ITOAvW/hH9sDZmdeBA0CjAEGAWTgnAPMtR2+xJUpEad9R6kU8SptKqUeBKHWdZfinu4JCuu/ kmvieYJwrkAoFArC+UIe512mb5QpwZWVKNZQKRSlpFT7dRjD0TMTy7iZPntm2Xzy/cPsIw61 t76mzqokHZEzRc3HHie7Vw7FO05GbZ8Y4eeMrDYItCerYngtSvPzhW0vXd5qepylMV/oKnBL ykxLkM0MdKw+2OHxL60tf2ZuE/nFBjr7pFXi2s7DF6i1S3e2RoimkI+CTn0ljT6vypuqPXOw YqN7RVfE/cL7YENAtCmLbzkkW6molpm6yqZLRXlvapoCMlSn/zLyTsvjNvuXPtkq73lWXubh nszI4jwN1xjDvIaPuCRvlGb88v2Q9JzH5gemyJ/DIr4Uad8pkcy3job8thFjqE+bzv3QSee3 XvTp+n3zs5+md9JH1/+5Wni9jNnZ+SsZ4VPD59DUUjHPH1epxf8C1XKlyJ8EAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrEIsWRmVeSWpSXmKPExsWy7bCSvO42MdMUg+mfmCzWnzrGbNE04S+z xeq7/WwWrw9/YrSY9uEns8WD/fYWe9/NZrW4eWAnk8WeRZOYLFauPspksXvhRyaLo//fsllM OnSN0eLp1VlMFntvaVvs2XuSxeLyrjlsFvOXPWW3ODS5mclix5NGRottv+czW6x7/Z7F4sQt aYvzf4+zOkh4XL7i7THr/lk2j52z7rJ7nL+3kcXj8tlSj02rOtk8Ni+p99h9s4HNo7f5HZvH +31X2Tz6tqxi9Nh8utrj8yY5j01P3jIF8EVx2aSk5mSWpRbp2yVwZTxfuYCx4LFUxblFHSwN jI9Euxg5OSQETCRWTuli62Lk4hAS2MEocetVAwtEQkLi1MtljBC2sMTKf8/ZIYoamSTmzWxn BkmwCahLHHneygiSEBGYwCxx6X4D2ChmgQdMEueff2EFqRIWsJC4vWkeO4jNIqAqMen3QjCb Fyje9H8GUAMH0Ap9if77giBhTgFLiT9zO8DCQkAlS/awQFQLSpyc+QTMZhaQl2jeOpt5AqPA LCSpWUhSCxiZVjFKphYU56bnFhsWGOWllusVJ+YWl+al6yXn525iBEezltYOxj2rPugdYmTi YDzEKMHBrCTC+/W/cYoQb0piZVVqUX58UWlOavEhRmkOFiVx3gtdJ+OFBNITS1KzU1MLUotg skwcnFINTPmCcb2L42xae/jWPFtmeUK74bLT6knaaQFir3TUC3ZoPuqY53mBf09PchmDau6z 7NJtv/w64s/OesG7b0sxi627huG1T5qS3ou4r/4LLRX6O79Ys+qk1ayU47P33pfwjpq5QvvW Ib3Hlyy/TfM9kbFcWKgqke+jbKDc1GSneSxnTBY0xb1X6n05UYHD2t81j9v80++0E0k/zn2L /nghz7r4u9DPBfJOrcuPh2//9YFBWKA0sCdoxfSvuzYtyfFjPbdJw9xie1mvg1X3p/nHVfdF h1ufCVcsesB18Hm3nWBE1dfvHxfXsHClhBZlx95eEm2+evWKr6lv1S1XTQhm+R9YeWz1/jNV Do07KiRYlViKMxINtZiLihMBKOh6SlUDAAA= X-CMS-MailID: 20230411081342epcas5p22a4c587babd6a373cfa709d9609f65f4 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230411081342epcas5p22a4c587babd6a373cfa709d9609f65f4 References: <20230411081041.5328-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Nitesh Shetty Before enabling copy for dm target, check if underlying devices and dm target support copy. Avoid split happening inside dm target. Fail early if the request needs split, currently splitting copy request is not supported. Signed-off-by: Nitesh Shetty --- drivers/md/dm-table.c | 41 +++++++++++++++++++++++++++++++++++ drivers/md/dm.c | 7 ++++++ include/linux/device-mapper.h | 5 +++++ 3 files changed, 53 insertions(+) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 1398f1d6e83e..b3269271e761 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1867,6 +1867,39 @@ static bool dm_table_supports_nowait(struct dm_table *t) return true; } +static int device_not_copy_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + struct request_queue *q = bdev_get_queue(dev->bdev); + + return !blk_queue_copy(q); +} + +static bool dm_table_supports_copy(struct dm_table *t) +{ + struct dm_target *ti; + unsigned int i; + + for (i = 0; i < t->num_targets; i++) { + ti = dm_table_get_target(t, i); + + if (!ti->copy_offload_supported) + return false; + + /* + * target provides copy support (as implied by setting + * 'copy_offload_supported') + * and it relies on _all_ data devices having copy support. + */ + if (!ti->type->iterate_devices || + ti->type->iterate_devices(ti, + device_not_copy_capable, NULL)) + return false; + } + + return true; +} + static int device_not_discard_capable(struct dm_target *ti, struct dm_dev *dev, sector_t start, sector_t len, void *data) { @@ -1949,6 +1982,14 @@ int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, q->limits.discard_misaligned = 0; } + if (!dm_table_supports_copy(t)) { + blk_queue_flag_clear(QUEUE_FLAG_COPY, q); + q->limits.max_copy_sectors = 0; + q->limits.max_copy_sectors_hw = 0; + } else { + blk_queue_flag_set(QUEUE_FLAG_COPY, q); + } + if (!dm_table_supports_secure_erase(t)) q->limits.max_secure_erase_sectors = 0; diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 20c6b72a0245..1492bea4d605 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1709,6 +1709,13 @@ static blk_status_t __split_and_process_bio(struct clone_info *ci) if (unlikely(ci->is_abnormal_io)) return __process_abnormal_io(ci, ti); + if ((unlikely(op_is_copy(ci->bio->bi_opf)) && + max_io_len(ti, ci->sector) < ci->sector_count)) { + DMERR("Error, IO size(%u) > max target size(%llu)\n", + ci->sector_count, max_io_len(ti, ci->sector)); + return BLK_STS_IOERR; + } + /* * Only support bio polling for normal IO, and the target io is * exactly inside the dm_io instance (verified in dm_poll_dm_io) diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index 8aa6b3ea91fa..70142fff417c 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -386,6 +386,11 @@ struct dm_target { * bio_set_dev(). NOTE: ideally a target should _not_ need this. */ bool needs_bio_set_dev:1; + + /* + * copy offload is supported + */ + bool copy_offload_supported:1; }; void *dm_per_bio_data(struct bio *bio, size_t data_size); From patchwork Tue Apr 11 08:10:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13207190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C1F3C76196 for ; Tue, 11 Apr 2023 08:21:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230403AbjDKIVN (ORCPT ); Tue, 11 Apr 2023 04:21:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230378AbjDKIUj (ORCPT ); Tue, 11 Apr 2023 04:20:39 -0400 Received: from mailout2.samsung.com (mailout2.samsung.com [203.254.224.25]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8BBFF40E5 for ; Tue, 11 Apr 2023 01:19:50 -0700 (PDT) Received: from epcas5p4.samsung.com (unknown [182.195.41.42]) by mailout2.samsung.com (KnoxPortal) with ESMTP id 20230411081949epoutp021b12bcc91300e583e0617f50816531f8~U05L0NBSo0589505895epoutp02R for ; Tue, 11 Apr 2023 08:19:49 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout2.samsung.com 20230411081949epoutp021b12bcc91300e583e0617f50816531f8~U05L0NBSo0589505895epoutp02R DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681201189; bh=LSWdD3MQyN53kHS/yCDz0oi55FM+zMca65KKpSIzhUQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=A+gRVZy8wxEWhLMjG7PTNg3FIqPbv0U9tdhLuaMIGsy0OfazrWjkNIHl7jkkw6FUY GFPLYyLm9l+O7zTdDS1byHi8gcm4gPBgPXUWxFnQZiUvWfXbxk6EYmFB0MFdAbWD9C W+LVgX/MdsLaFU/M44vdj8CvWEsTUzRhAp0byDgs= Received: from epsnrtp4.localdomain (unknown [182.195.42.165]) by epcas5p2.samsung.com (KnoxPortal) with ESMTP id 20230411081948epcas5p259aff7c3e4caa110a36836bab23b8110~U05LUqq1Z0093700937epcas5p2X; Tue, 11 Apr 2023 08:19:48 +0000 (GMT) Received: from epsmges5p2new.samsung.com (unknown [182.195.38.174]) by epsnrtp4.localdomain (Postfix) with ESMTP id 4Pwf0218Y5z4x9QF; Tue, 11 Apr 2023 08:19:46 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p2new.samsung.com (Symantec Messaging Gateway) with SMTP id 67.4F.09540.22815346; Tue, 11 Apr 2023 17:19:46 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p3.samsung.com (KnoxPortal) with ESMTPA id 20230411081351epcas5p3c2a85087cce368f6c2e0ffdbf18f29b2~U0z_dpTVe2461924619epcas5p3v; Tue, 11 Apr 2023 08:13:51 +0000 (GMT) Received: from epsmgms1p2.samsung.com (unknown [182.195.42.42]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230411081351epsmtrp14c71460f7f29de5d9fca595fba7313c5~U0z_cD27P1918219182epsmtrp1f; Tue, 11 Apr 2023 08:13:51 +0000 (GMT) X-AuditID: b6c32a4a-4afff70000002544-86-64351822eb6f Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p2.samsung.com (Symantec Messaging Gateway) with SMTP id A0.0E.08609.EB615346; Tue, 11 Apr 2023 17:13:50 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230411081347epsmtip224a26827cc4aeef86f955c4275720480~U0z7VJIRW2247222472epsmtip2J; Tue, 11 Apr 2023 08:13:47 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v9 8/9] dm: Enable copy offload for dm-linear target Date: Tue, 11 Apr 2023 13:40:35 +0530 Message-Id: <20230411081041.5328-9-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230411081041.5328-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA01Ta0xTZxjeOac9FJOaQ5HsswusO5Nx2YBWWvaxiDhh25mYjcSMZMsSOGlP CqG0tZcJOhHoAK1yGQaiVUS2AQIGJgWCQBFbi4JRNxkymSJmJaTIRXRkAwXW0rL573mf73ne 25eXg/GO+/A5GUodo1HSChLfxOq0hYVGkEAiE5b3s2Hr0AAGC8pXMNj8sAyHT2zPEFj1dAmD E1fioWXuDBve77+Mwt4fKlDY2GxHYU/tAgrta7M4rLDeQ+DkiAmFlrF3Ya9lkAWHu8/isKZ+ 0gdaTxpQ2OXIR2DnixoMtjyZZ8EbY2/AOyvX2bsANfxbEmV6dAunLpse+lB3xi+xqOFbeqqt 6RhOmX86QvXcz8OpEsMcTs33jeBUaXsTQplvHqKetwVRbY5ZNHnzV5k70hlaxmgEjFKqkmUo 5XFk0r7UhFRJjFAUIYqF75MCJZ3FxJGJe5MjPs5QuDZACr6hFXoXlUxrtWTUzh0alV7HCNJV Wl0cyahlCrVYHamls7R6pTxSyeg+EAmF2yUuYVpmuuW0BVX/zM4ebPgdz0P6WUbElwMIMVgc mECMyCYOj+hBQKf9Ku4JniHAeLuf7VbxiOcImGnhbDgej5awPKJuBPxV+yvqCQwoMC84ELcK J0LAtanC9bxbiCIMLEweW7dgxF0UvGyfx9wqfyIRnCvtcz1wOCwiGCyN5bppLgHBgHXKx00D IgqUPfJz075ELHhZfRT3SPzA4GnH+gwY8SYwdJzB3OkB8aMvqDZPoJ5WE8Hx72a82B9MX2/3 8WA+cJYVebEc/DM86dWogWGgD/HgeFA4VIa5e8CIMNDaHeWhA0HlUAvqqbsZlLxweK1c0HVu A5OguPGsFwNguZ3nxRToLTzqXdYJBBQ7Z9ByRGB6ZR7TK/OY/i99HsGakK2MWpslZ7QS9XYl c+C/X5aqstqQ9asI39OFPJ54GmlFUA5iRQAHI7dwF9eiZTyujM45yGhUqRq9gtFaEYlr3d9j /ACpynVWSl2qSBwrFMfExIhjo2NE5OvckLhBKY+Q0zomk2HUjGbDh3J8+XmocO7CbloSNjvl 5zxouQDtxqaUd0avlugVDQ9Gt3WkBSaV8Bucb4vQsYjSX2jV4V3SafP5FFFZ4I0VefCIriCU qH6LU4kf+XY8qOqkYz9bWuGU/uFUJ1QGhWjj2eG59HJxrm5uKrE2WtZqBdJJ6ejsobp8mz1/ 1NgUXFB4wD/FMHzlRFXO8h5q+WLIXsFA8r7VbsZeVL/zvbtLmpxVMatW0VjBj1zK67bVhVzk 9szV3fs0bf9HsubPtg0tKDhjhyXT1akqv69j47fSAae6Vj9/8Mlato1KSVisCb1mSzYbvyxs GN/92qlLAdn1X3yIUQoz8Tevvlx+07955M+OfpJkadNpUTim0dL/AjrFP6meBAAA X-Brightmail-Tracker: H4sIAAAAAAAAA02Ra0xSYRjHe885HI822xEVX+/JdDkqS7v4mq360IezVcvWyuXalOKkTSEC 7bqVZi7NLIarKWZXVxOdGqZBgikWimZWZqVl2NJMTaOLlhmRaG19+z//32/P8+GhcG4j4UPt laSyMokwhU+6ELVN/MDF9bwVoqWW7ihU2WrC0QmFDUdlvedINNL0BaAL1kkc9d1biwxjRRzU 3aDDkP6aEkOlZQ8wVHf1M4Ye2EdJpDQ+B2igS4UhQ89CpDeYCdR59yKJLt8YcELG/EwMafsz AKqduoyjipFPBGrp8UUdtmbOOsh0PtvAqCztJKNT9ToxHW9uEUxnexqjUeeQTHXJcaauO51k 8jLHSOZTfRfJnL2tBkx121HmqyaA0fSPYjHz4lxWi9iUvQdY2ZI1CS5JhkIDJq3iHDLffEmm gwbiNHCmIL0cvn2RN51dKC6tBfDRzXwwCyBsHbrxN7vD0t+DTrNSBgYbs16TDkDSC+D9wSzg AB60AodPLemkY8DpPgx2DH7jOCx3ej28dLZ++gZFEXQInOw55qhdaQRNRsdWavrCEnjO4uao neko+Ks4m3TU3GmlRE/M2m7QXNg/k3E6EGbWFOEKQKv+Q6r/0BWAqYE3K5WLE8XycGmEhD0Y JheK5WmSxLDd+8QaMPNlgUAL9GprmBFgFDACSOF8D9dx+zIR11UkPHyEle2Ll6WlsHIj8KUI vpfr49PmeC6dKExlk1lWysr+UYxy9knHuN5YsdAzIfNI+2f7z7aVHisq9QWx3/uI7DR33vgO +aKoC0Mm0+bhBv+J6qD3T/g8WwFWsGuP23yOOJYKHbYNfLm+1i/k1caIBJ/muYuCeb6Lkzc9 bVMuU47MrwxNprWC1KKPkruwMVfzQ69rjcm9Wi6Az7wzdkuL74zRy8MqckYmz+/3iq88YEiO brL5lE4UFgfH7ekNjAxomvP15MqCDy3bcu2KbRvnlueEr5l8e+xhxpBOqVd75g17Tg3HfHj0 whrsro3OX6U61aXoeee/dTzAmrW9MIg3YLrU3F0u3FWyX+fRZiurD4qsCzdD/zOsTbTQvrN5 S43VK8Kv6qXEwifkScJwAS6TC/8ACa8Kg1QDAAA= X-CMS-MailID: 20230411081351epcas5p3c2a85087cce368f6c2e0ffdbf18f29b2 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230411081351epcas5p3c2a85087cce368f6c2e0ffdbf18f29b2 References: <20230411081041.5328-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Nitesh Shetty Setting copy_offload_supported flag to enable offload. Signed-off-by: Nitesh Shetty --- drivers/md/dm-linear.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/md/dm-linear.c b/drivers/md/dm-linear.c index f4448d520ee9..1d1ee30bbefb 100644 --- a/drivers/md/dm-linear.c +++ b/drivers/md/dm-linear.c @@ -62,6 +62,7 @@ static int linear_ctr(struct dm_target *ti, unsigned int argc, char **argv) ti->num_discard_bios = 1; ti->num_secure_erase_bios = 1; ti->num_write_zeroes_bios = 1; + ti->copy_offload_supported = 1; ti->private = lc; return 0; From patchwork Tue Apr 11 08:10:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Gupta X-Patchwork-Id: 13207191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6F05C7619A for ; Tue, 11 Apr 2023 08:21:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230412AbjDKIVS (ORCPT ); Tue, 11 Apr 2023 04:21:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230307AbjDKIUn (ORCPT ); Tue, 11 Apr 2023 04:20:43 -0400 Received: from mailout4.samsung.com (mailout4.samsung.com [203.254.224.34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3F35946A8 for ; Tue, 11 Apr 2023 01:19:54 -0700 (PDT) Received: from epcas5p2.samsung.com (unknown [182.195.41.40]) by mailout4.samsung.com (KnoxPortal) with ESMTP id 20230411081952epoutp04ea69e8a5ae47b0ca180b4c4f9a4303d8~U05PSI1T13273132731epoutp04d for ; Tue, 11 Apr 2023 08:19:52 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout4.samsung.com 20230411081952epoutp04ea69e8a5ae47b0ca180b4c4f9a4303d8~U05PSI1T13273132731epoutp04d DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1681201192; bh=XHv/zSeinRbm2d/hOX8s3Jm3bpY0hPt0hIkX6sypYh0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OT0Jhtn3DPZqb+OjidVMNtQQzWBBdxUQkzkbgL5NGPSretU2M6SwJTOmUyYXcbR/n UEABlbpucZwtpFoHBWa4xXxSJCf3TwTy+F6MlvJPHCc5Xlgg0667+Y3fcLGxTm/rLi aJl0Q2rUStx05ntvsRReQRkV4UUXlLcAo+D8D8yA= Received: from epsnrtp1.localdomain (unknown [182.195.42.162]) by epcas5p1.samsung.com (KnoxPortal) with ESMTP id 20230411081952epcas5p13d0164cc4c6fc74079317057dbb8f959~U05OscMoq0228502285epcas5p1J; Tue, 11 Apr 2023 08:19:52 +0000 (GMT) Received: from epsmges5p1new.samsung.com (unknown [182.195.38.177]) by epsnrtp1.localdomain (Postfix) with ESMTP id 4Pwf0647fBz4x9Pr; Tue, 11 Apr 2023 08:19:50 +0000 (GMT) Received: from epcas5p2.samsung.com ( [182.195.41.40]) by epsmges5p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 43.9B.09961.62815346; Tue, 11 Apr 2023 17:19:50 +0900 (KST) Received: from epsmtrp1.samsung.com (unknown [182.195.40.13]) by epcas5p1.samsung.com (KnoxPortal) with ESMTPA id 20230411081400epcas5p151186138b36daf361520b08618300502~U00HSKMnJ1534615346epcas5p1-; Tue, 11 Apr 2023 08:14:00 +0000 (GMT) Received: from epsmgms1p1new.samsung.com (unknown [182.195.42.41]) by epsmtrp1.samsung.com (KnoxPortal) with ESMTP id 20230411081400epsmtrp146444ac54d2b6b40e011a9ecd7a73d6a~U00HP9d9s1886118861epsmtrp1Y; Tue, 11 Apr 2023 08:14:00 +0000 (GMT) X-AuditID: b6c32a49-2c1ff700000026e9-ab-643518266927 Received: from epsmtip2.samsung.com ( [182.195.34.31]) by epsmgms1p1new.samsung.com (Symantec Messaging Gateway) with SMTP id 85.2F.08279.8C615346; Tue, 11 Apr 2023 17:14:00 +0900 (KST) Received: from green5.sa.corp.samsungelectronics.net (unknown [107.110.206.5]) by epsmtip2.samsung.com (KnoxPortal) with ESMTPA id 20230411081356epsmtip2a79113ab5665be1b002625fbcee5446c~U00D5G5402416524165epsmtip2Y; Tue, 11 Apr 2023 08:13:56 +0000 (GMT) From: Anuj Gupta To: Jens Axboe , Alasdair Kergon , Mike Snitzer , dm-devel@redhat.com, Keith Busch , Christoph Hellwig , Sagi Grimberg , James Smart , Chaitanya Kulkarni , Alexander Viro , Christian Brauner Cc: bvanassche@acm.org, hare@suse.de, ming.lei@redhat.com, dlemoal@kernel.org, anuj20.g@samsung.com, joshi.k@samsung.com, nitheshshetty@gmail.com, gost.dev@samsung.com, Nitesh Shetty , Damien Le Moal , Vincent Fu , linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v9 9/9] null_blk: add support for copy offload Date: Tue, 11 Apr 2023 13:40:36 +0530 Message-Id: <20230411081041.5328-10-anuj20.g@samsung.com> X-Mailer: git-send-email 2.35.1.500.gb896f729e2 In-Reply-To: <20230411081041.5328-1-anuj20.g@samsung.com> MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA02Te0xTVxzHc+7tC5bOa9F4YJt2nQOLAdrR4oHwSmbmXViUTLYlbK7etTeF UNqmLehGFh4dojwFAxuVyUM35bHiChIEixsMBRwroaDIgpBYHpOnLKtuSFxLYfO/z/d3vt/z O+d3cjg4r4Ltx0lWG2idmlIJWN6Mth7hviB/KFWInJd2o+aBWzjKObuOo8aJEhaa71kFqGLl bxytDdpwNHUzBlmXzjPR/Z+uY+hGXRmG6ht7MdRZ+xhDvc8XWais+y5A06MmDFnH96Mb1n4G sndUsVD199Ns1H3OiKF2RzZAbWvVODLPLzNQ3/gr6GHBaYBs67eZsX6kfSSONE0Ossjrpgk2 aXvwI4O0D6aRloYzLLLlUibZeT+LRRYZl1yG3Ekmudw1yiKLWxsA2XIng/zTspu0OBax+G2J KZFJNKWgdXxaLdcoktXKKEHcUdnbMmmYSBwkDkcHBHw1lUpHCQ6+Fx/0TrLKNRABP51SpblK 8ZReLwiJjtRp0gw0P0mjN0QJaK1CpZVog/VUqj5NrQxW04YIsUj0ltRlPJ6S9NTsq22OOGlb WmJmgTxxPuBwICGBU1XKfODN4RGdAPY09WEesQrg5a+fgXzg5RJOAIeW+VuB8WyZx2MFsMl5 jekRRgyO2Wdxd4BFBMBfZnOBe2EHcQqHj6fPMNwCJ5pwOLWytrGtDxEN7zbksN3MIN6ED62V G2kuEQ7Nl2fZnnYhsGRyu7vs5So/+/Y0y2PZDvsrHQw348QeaLx2HnfvD4lmL2gdKHAJtksc hKZItwUSPvDR7Va2h/3gHyWnNlkJn9qnMQ9rofFWF/BwDMwdKMHdJ8AJIWzuCPGUX4PlA2bM 0/VlWLTm2IxyYfuFLRbAvPqqTYbQ+lvWJpNw7tf2zVkVAnix8wH7LOCbXriN6YXbmP5vXQPw BuBLa/WpSlov1YrV9In/XliuSbWAjQ8S+G47mJhaCe4GGAd0A8jBBTu4fz0PVfC4CurzL2id RqZLU9H6biB1TbsU99sp17h+mNogE0vCRZKwsDBJeGiYWLCLGxDVL+cRSspAp9C0ltZt5TCO l18WFkAJwTxciD0x71zv0AXRPHlN3ZGdKHNE6ftN3OG5J63lDREc8+sJ8uCamSHK8Zl4m2kw 9upYsdOxq/CCP1/0e3PtVczBmX/jJfzLOxE/rHqVNh5O/0Sc+F15QiSltTGiPpjrevWY//CH 9OLxEXq4LEHR32tpqT1Ut/yx5GRFYPhXe73juQf22Hmj769Lbl7M6Esqym4XF+d+FF0yZFmw 5lV1Fe4LbmsaHPlUnxEaU70qHSuFlZlC5yH+vSUwc8555cpk42j1wnrlUVmZ2rzWNjxXaOy3 /zwpnKnfm2jrqDt7jDmrKkjZ/0/XE++YUYnPvUfm9OqiI53C9CBeTlwsS8DQJ1HiQFynp/4F EBl72KkEAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA02Re0hTYRjG+845Ox5X1tmM+qZhMYtISxsWfN3sip0ospsRktjQk1lujR1N LcjZuuhWuSys1sIVq3CG0rznpjaztLKVZpRd7KKJ2exqWmbmSYL+e/j9Hh5eeClcfJ/woeKV iaxaKU+QkkKitFY6eVb9hLmxs3vNElR45xaODhgGcZT/IotE3bVfAMr59ANHA40uHL2qXowc PecE6GlNBYbsF7MxlJdfh6HKC58xVDfkJlG28zFAHS1GDDlaA5Hd0UCg5usmEuVe7vBAzpNa DJW3pwNUOpCLo4LujwSqb/VFb/UZALkGbwuW+DDNj1YzxrZGkqkwvvBgXC+vEUxzYxJjs2aS TJEljal8qiGZY9qe4cKhNgHzsaqFZI4XWwFTdHcf89Xmx9ja3di6cZHChbFsQvweVh0cuk24 o79Aoiqcn+Lq6RFowBGZDlAUpOfA1vRoHRBSYroSwKGSPg8d8BzmEN7pugxGsjfM+93pMVJK x2BfRy/GC5KeDm92HgK8GE8bcNjUpiF5gdN2HOaZl/LZmw6Fj60H/q4S9DT41nEW57MXPQ8W XOFX+SuCYVabiMeew/jX+QySx2IaQYudGGmLYMPZdmJkfTLUlpzDDYA2/qeM/ykzwKxAwqo4 RZyCk6lkSjY5iJMruCRlXFDMboUN/H15wIxyUGb9FOQEGAWcAFK4dLxX71BIrNgrVp66l1Xv jlYnJbCcE/hShHSi1wNdQ7SYjpMnsrtYVsWq/1mM8vTRYEXFdaHPw0HOfq06O9Mt6k9JqF8V 1QjDfbve9cHaUeVdFdy0kgWv1dPx1aqFz/z199yhhmxFxnrFWl1u2kHL7Qfbk7+79cr+ljGq gWLzoinajBXCDWlhY6MMd8MOc7VR1nhN2JVEOqc+5sw7SVbA0Z/Bz/JF/mW+ET9TR4tuCru/ 9JqqTHLqjUX0QR939US65VtVvN9KSbXAvHHFVf/AglOZL8c1Zaru+W2qSQWXroV0w0d2WdOy pSURhcmFi0/uDOlkAiO8XyWelqSsnDol+knH2BuTMNMH2xpqv2BQeTiylXwYc0m/td/QEhkw Mxy9Vym3OEzLSddmvClmn6iyQkpwO+SyAFzNyf8ANvLyWGEDAAA= X-CMS-MailID: 20230411081400epcas5p151186138b36daf361520b08618300502 X-Msg-Generator: CA X-Sendblock-Type: REQ_APPROVE CMS-TYPE: 105P DLP-Filter: Pass X-CFilter-Loop: Reflected X-CMS-RootMailID: 20230411081400epcas5p151186138b36daf361520b08618300502 References: <20230411081041.5328-1-anuj20.g@samsung.com> Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Nitesh Shetty Implementaion is based on existing read and write infrastructure. copy_max_bytes: A new configfs and module parameter is introduced, which can be used to set hardware/driver supported maximum copy limit. Suggested-by: Damien Le Moal Signed-off-by: Anuj Gupta Signed-off-by: Nitesh Shetty Signed-off-by: Vincent Fu --- drivers/block/null_blk/main.c | 101 ++++++++++++++++++++++++++++++ drivers/block/null_blk/null_blk.h | 8 +++ 2 files changed, 109 insertions(+) diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index bc2c58724df3..e273e18ace74 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -157,6 +157,10 @@ static int g_max_sectors; module_param_named(max_sectors, g_max_sectors, int, 0444); MODULE_PARM_DESC(max_sectors, "Maximum size of a command (in 512B sectors)"); +static int g_copy_max_bytes = COPY_MAX_BYTES; +module_param_named(copy_max_bytes, g_copy_max_bytes, int, 0444); +MODULE_PARM_DESC(copy_max_bytes, "Maximum size of a copy command (in bytes)"); + static unsigned int nr_devices = 1; module_param(nr_devices, uint, 0444); MODULE_PARM_DESC(nr_devices, "Number of devices to register"); @@ -409,6 +413,7 @@ NULLB_DEVICE_ATTR(home_node, uint, NULL); NULLB_DEVICE_ATTR(queue_mode, uint, NULL); NULLB_DEVICE_ATTR(blocksize, uint, NULL); NULLB_DEVICE_ATTR(max_sectors, uint, NULL); +NULLB_DEVICE_ATTR(copy_max_bytes, uint, NULL); NULLB_DEVICE_ATTR(irqmode, uint, NULL); NULLB_DEVICE_ATTR(hw_queue_depth, uint, NULL); NULLB_DEVICE_ATTR(index, uint, NULL); @@ -550,6 +555,7 @@ static struct configfs_attribute *nullb_device_attrs[] = { &nullb_device_attr_queue_mode, &nullb_device_attr_blocksize, &nullb_device_attr_max_sectors, + &nullb_device_attr_copy_max_bytes, &nullb_device_attr_irqmode, &nullb_device_attr_hw_queue_depth, &nullb_device_attr_index, @@ -631,6 +637,7 @@ static ssize_t memb_group_features_show(struct config_item *item, char *page) "badblocks,blocking,blocksize,cache_size," "completion_nsec,discard,home_node,hw_queue_depth," "irqmode,max_sectors,mbps,memory_backed,no_sched," + "copy_max_bytes," "poll_queues,power,queue_mode,shared_tag_bitmap,size," "submit_queues,use_per_node_hctx,virt_boundary,zoned," "zone_capacity,zone_max_active,zone_max_open," @@ -693,6 +700,7 @@ static struct nullb_device *null_alloc_dev(void) dev->queue_mode = g_queue_mode; dev->blocksize = g_bs; dev->max_sectors = g_max_sectors; + dev->copy_max_bytes = g_copy_max_bytes; dev->irqmode = g_irqmode; dev->hw_queue_depth = g_hw_queue_depth; dev->blocking = g_blocking; @@ -1242,6 +1250,81 @@ static int null_transfer(struct nullb *nullb, struct page *page, return err; } +static inline int nullb_setup_copy_read(struct nullb *nullb, + struct bio *bio) +{ + struct nullb_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + + memcpy(token->subsys, "nullb", 5); + token->sector_in = bio->bi_iter.bi_sector; + token->nullb = nullb; + token->sectors = bio->bi_iter.bi_size >> SECTOR_SHIFT; + + return 0; +} + +static inline int nullb_setup_copy_write(struct nullb *nullb, + struct bio *bio, bool is_fua) +{ + struct nullb_copy_token *token = bvec_kmap_local(&bio->bi_io_vec[0]); + sector_t sector_in, sector_out; + void *in, *out; + size_t rem, temp; + unsigned long offset_in, offset_out; + struct nullb_page *t_page_in, *t_page_out; + int ret = -EIO; + + if (unlikely(memcmp(token->subsys, "nullb", 5))) + return -EINVAL; + if (unlikely(token->nullb != nullb)) + return -EINVAL; + if (WARN_ON(token->sectors != bio->bi_iter.bi_size >> SECTOR_SHIFT)) + return -EINVAL; + + sector_in = token->sector_in; + sector_out = bio->bi_iter.bi_sector; + rem = token->sectors << SECTOR_SHIFT; + + spin_lock_irq(&nullb->lock); + while (rem > 0) { + temp = min_t(size_t, nullb->dev->blocksize, rem); + offset_in = (sector_in & SECTOR_MASK) << SECTOR_SHIFT; + offset_out = (sector_out & SECTOR_MASK) << SECTOR_SHIFT; + + if (null_cache_active(nullb) && !is_fua) + null_make_cache_space(nullb, PAGE_SIZE); + + t_page_in = null_lookup_page(nullb, sector_in, false, + !null_cache_active(nullb)); + if (!t_page_in) + goto err; + t_page_out = null_insert_page(nullb, sector_out, + !null_cache_active(nullb) || is_fua); + if (!t_page_out) + goto err; + + in = kmap_local_page(t_page_in->page); + out = kmap_local_page(t_page_out->page); + + memcpy(out + offset_out, in + offset_in, temp); + kunmap_local(out); + kunmap_local(in); + __set_bit(sector_out & SECTOR_MASK, t_page_out->bitmap); + + if (is_fua) + null_free_sector(nullb, sector_out, true); + + rem -= temp; + sector_in += temp >> SECTOR_SHIFT; + sector_out += temp >> SECTOR_SHIFT; + } + + ret = 0; +err: + spin_unlock_irq(&nullb->lock); + return ret; +} + static int null_handle_rq(struct nullb_cmd *cmd) { struct request *rq = cmd->rq; @@ -1252,6 +1335,13 @@ static int null_handle_rq(struct nullb_cmd *cmd) struct req_iterator iter; struct bio_vec bvec; + if (rq->cmd_flags & REQ_COPY) { + if (op_is_write(req_op(rq))) + return nullb_setup_copy_write(nullb, rq->bio, + rq->cmd_flags & REQ_FUA); + return nullb_setup_copy_read(nullb, rq->bio); + } + spin_lock_irq(&nullb->lock); rq_for_each_segment(bvec, rq, iter) { len = bvec.bv_len; @@ -1279,6 +1369,13 @@ static int null_handle_bio(struct nullb_cmd *cmd) struct bio_vec bvec; struct bvec_iter iter; + if (bio->bi_opf & REQ_COPY) { + if (op_is_write(bio_op(bio))) + return nullb_setup_copy_write(nullb, bio, + bio->bi_opf & REQ_FUA); + return nullb_setup_copy_read(nullb, bio); + } + spin_lock_irq(&nullb->lock); bio_for_each_segment(bvec, bio, iter) { len = bvec.bv_len; @@ -2109,6 +2206,10 @@ static int null_add_dev(struct nullb_device *dev) dev->max_sectors = queue_max_hw_sectors(nullb->q); dev->max_sectors = min(dev->max_sectors, BLK_DEF_MAX_SECTORS); blk_queue_max_hw_sectors(nullb->q, dev->max_sectors); + blk_queue_max_copy_sectors_hw(nullb->q, + dev->copy_max_bytes >> SECTOR_SHIFT); + if (dev->copy_max_bytes) + blk_queue_flag_set(QUEUE_FLAG_COPY, nullb->disk->queue); if (dev->virt_boundary) blk_queue_virt_boundary(nullb->q, PAGE_SIZE - 1); diff --git a/drivers/block/null_blk/null_blk.h b/drivers/block/null_blk/null_blk.h index eb5972c50be8..c67c098d92fa 100644 --- a/drivers/block/null_blk/null_blk.h +++ b/drivers/block/null_blk/null_blk.h @@ -67,6 +67,13 @@ enum { NULL_Q_MQ = 2, }; +struct nullb_copy_token { + char subsys[5]; + struct nullb *nullb; + u64 sector_in; + u64 sectors; +}; + struct nullb_device { struct nullb *nullb; struct config_item item; @@ -102,6 +109,7 @@ struct nullb_device { unsigned int queue_mode; /* block interface */ unsigned int blocksize; /* block size */ unsigned int max_sectors; /* Max sectors per command */ + unsigned long copy_max_bytes; /* Max copy offload length in bytes */ unsigned int irqmode; /* IRQ completion handler */ unsigned int hw_queue_depth; /* queue depth */ unsigned int index; /* index of the disk, only valid with a disk */