From patchwork Wed Oct 18 15:18:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13427225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5508BCDB484 for ; Wed, 18 Oct 2023 15:24:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231235AbjJRPYo (ORCPT ); Wed, 18 Oct 2023 11:24:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231344AbjJRPYn (ORCPT ); Wed, 18 Oct 2023 11:24:43 -0400 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA63FFA for ; Wed, 18 Oct 2023 08:24:41 -0700 (PDT) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 39I84NIO015181 for ; Wed, 18 Oct 2023 08:24:41 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=ZuYobI9FB5FgUYWyl3A8aqmEEA9/aSHKCp9svw5nXYo=; b=S6tLdoKL40BlrJRzHUyP9JJ/8sFcIKhJZBqyBAcmrDx2wACJ5ShPufZQlpwVq+/uaw/J LQOIKfrCqGaPdH/X4hzfdgw9epSzhqfYFcbchDgSzcLNmPjA7bG7XS2r/tXmeU53MQsz WipJA6kTYd5oXer3bc3k1SjAO6E0Il9aCqV73phYlX+B5ucbApSwK/xv3LxbYf3fhd26 HpKI076LrotZWQgpN1glqpvmAzufZgSqsRZIAtAQ40xdHkxOPI8ZCIUDWltyhg4w+GAO wxTNxg9ex6QBK8OJ854p3cIvJmSWw2o1l+wNW+B3ML0DSkUH+W95GSChWj2x3d14tcSN Gw== Received: from maileast.thefacebook.com ([163.114.130.16]) by m0089730.ppops.net (PPS) with ESMTPS id 3ttbhatctk-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 18 Oct 2023 08:24:40 -0700 Received: from twshared5242.08.ash8.facebook.com (2620:10d:c0a8:1b::30) by mail.thefacebook.com (2620:10d:c0a8:82::b) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Wed, 18 Oct 2023 08:24:32 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 6DD66205F43B2; Wed, 18 Oct 2023 08:19:39 -0700 (PDT) From: Keith Busch To: , , CC: , , , , Keith Busch Subject: [PATCH 1/4] block: bio-integrity: add support for user buffers Date: Wed, 18 Oct 2023 08:18:40 -0700 Message-ID: <20231018151843.3542335-2-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231018151843.3542335-1-kbusch@meta.com> References: <20231018151843.3542335-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: OUHj-t2NJlbAtFM1A_kemgfi7cah_wbl X-Proofpoint-ORIG-GUID: OUHj-t2NJlbAtFM1A_kemgfi7cah_wbl X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-18_13,2023-10-18_01,2023-05-22_02 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch User space passthrough commands that utilize metadata currently need to bounce the "integrity" buffer through the kernel. This adds unnecessary overhead and memory pressure. Add support for mapping user space directly so that we can avoid this costly copy. This is similiar to how the bio payload utilizes user addresses with bio_map_user_iov(). Signed-off-by: Keith Busch --- block/bio-integrity.c | 67 +++++++++++++++++++++++++++++++++++++++++++ include/linux/bio.h | 8 ++++++ 2 files changed, 75 insertions(+) diff --git a/block/bio-integrity.c b/block/bio-integrity.c index ec8ac8cf6e1b9..08f70b837a29b 100644 --- a/block/bio-integrity.c +++ b/block/bio-integrity.c @@ -91,6 +91,19 @@ struct bio_integrity_payload *bio_integrity_alloc(struct bio *bio, } EXPORT_SYMBOL(bio_integrity_alloc); +static void bio_integrity_unmap_user(struct bio_integrity_payload *bip) +{ + bool dirty = bio_data_dir(bip->bip_bio) == READ; + struct bvec_iter iter; + struct bio_vec bv; + + bip_for_each_vec(bv, bip, iter) { + if (dirty && !PageCompound(bv.bv_page)) + set_page_dirty_lock(bv.bv_page); + unpin_user_page(bv.bv_page); + } +} + /** * bio_integrity_free - Free bio integrity payload * @bio: bio containing bip to be freed @@ -105,6 +118,8 @@ void bio_integrity_free(struct bio *bio) if (bip->bip_flags & BIP_BLOCK_INTEGRITY) kfree(bvec_virt(bip->bip_vec)); + else if (bip->bip_flags & BIP_INTEGRITY_USER) + bio_integrity_unmap_user(bip);; __bio_integrity_free(bs, bip); bio->bi_integrity = NULL; @@ -160,6 +175,58 @@ int bio_integrity_add_page(struct bio *bio, struct page *page, } EXPORT_SYMBOL(bio_integrity_add_page); +int bio_integrity_map_user(struct bio *bio, void __user *ubuf, unsigned int len, + u32 seed, u32 maxvecs) +{ + struct request_queue *q = bdev_get_queue(bio->bi_bdev); + unsigned long align = q->dma_pad_mask | queue_dma_alignment(q); + struct page *stack_pages[UIO_FASTIOV]; + size_t offset = offset_in_page(ubuf); + unsigned long ptr = (uintptr_t)ubuf; + struct page **pages = stack_pages; + struct bio_integrity_payload *bip; + int npages, ret, i; + + if (bio_integrity(bio) || ptr & align || maxvecs > UIO_FASTIOV) + return -EINVAL; + + bip = bio_integrity_alloc(bio, GFP_KERNEL, maxvecs); + if (IS_ERR(bip)) + return PTR_ERR(bip); + + ret = pin_user_pages_fast(ptr, UIO_FASTIOV, FOLL_WRITE, pages); + if (unlikely(ret < 0)) + goto free_bip; + + npages = ret; + for (i = 0; i < npages; i++) { + u32 bytes = min_t(u32, len, PAGE_SIZE - offset); + ret = bio_integrity_add_page(bio, pages[i], bytes, offset); + if (ret != bytes) { + ret = -EINVAL; + goto release_pages; + } + len -= ret; + offset = 0; + } + + if (len) { + ret = -EINVAL; + goto release_pages; + } + + bip->bip_iter.bi_sector = seed; + bip->bip_flags |= BIP_INTEGRITY_USER; + return 0; + +release_pages: + unpin_user_pages(pages, npages); +free_bip: + bio_integrity_free(bio); + return ret; +} +EXPORT_SYMBOL(bio_integrity_map_user); + /** * bio_integrity_process - Process integrity metadata for a bio * @bio: bio to generate/verify integrity metadata for diff --git a/include/linux/bio.h b/include/linux/bio.h index 41d417ee13499..144cc280b6ad3 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -324,6 +324,7 @@ enum bip_flags { BIP_CTRL_NOCHECK = 1 << 2, /* disable HBA integrity checking */ BIP_DISK_NOCHECK = 1 << 3, /* disable disk integrity checking */ BIP_IP_CHECKSUM = 1 << 4, /* IP checksum */ + BIP_INTEGRITY_USER = 1 << 5, /* Integrity payload is user address */ }; /* @@ -720,6 +721,7 @@ static inline bool bioset_initialized(struct bio_set *bs) extern struct bio_integrity_payload *bio_integrity_alloc(struct bio *, gfp_t, unsigned int); extern int bio_integrity_add_page(struct bio *, struct page *, unsigned int, unsigned int); +extern int bio_integrity_map_user(struct bio *, void __user *, unsigned int, u32, u32); extern bool bio_integrity_prep(struct bio *); extern void bio_integrity_advance(struct bio *, unsigned int); extern void bio_integrity_trim(struct bio *); @@ -789,6 +791,12 @@ static inline int bio_integrity_add_page(struct bio *bio, struct page *page, return 0; } +static inline int bio_integrity_map_user(struct bio *bio, void __user *ubuf, + unsigned int len, u32 seed, u32 maxvecs) +{ + return -EINVAL +} + #endif /* CONFIG_BLK_DEV_INTEGRITY */ /* From patchwork Wed Oct 18 15:18:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13427237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1D13CDB483 for ; Wed, 18 Oct 2023 15:24:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231542AbjJRPYw (ORCPT ); Wed, 18 Oct 2023 11:24:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49636 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231563AbjJRPYv (ORCPT ); Wed, 18 Oct 2023 11:24:51 -0400 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 79F5210F for ; Wed, 18 Oct 2023 08:24:48 -0700 (PDT) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 39I84NIe015181 for ; Wed, 18 Oct 2023 08:24:47 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=ic7A44rt4K/WcfaYrUmFA/FKaGJZgx0TCYOoC0ab5ks=; b=d/A6rj8Vsy4sJ0MT7KemWmd+/ewipJ59CpGgPa7Q0RWyQa4hlolzZkuwf5GzVNL6Ffsc FU7olrSfv7YIybZzH8crK1qqqKjt1+z4+89q4Ymyzj4RVkVMxURYD5Mu42dr7ukRQZwi YuRCC5RYMDZ+Koe2+TcmlrglktVqvIuVfJKpzfh74k30NUf67Znk82URS0WSStKZZfOS OPSUIUWTZLxRPk5eEtvd2haOXbZFLOfj41pityo04Fn2mc8Pv8ue4lU+LnUnsfU+H745 nkQczAs645ohup1yNGXRGu84n247b3y8SpEIF872a/b4yK9ZaXz5dlk3wfrHAKw4BI2r kg== Received: from maileast.thefacebook.com ([163.114.130.16]) by m0089730.ppops.net (PPS) with ESMTPS id 3ttbhatctk-19 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 18 Oct 2023 08:24:47 -0700 Received: from twshared34392.14.frc2.facebook.com (2620:10d:c0a8:1c::11) by mail.thefacebook.com (2620:10d:c0a8:82::b) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Wed, 18 Oct 2023 08:24:32 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 57D5C205F4408; Wed, 18 Oct 2023 08:24:21 -0700 (PDT) From: Keith Busch To: , , CC: , , , , Keith Busch Subject: [PATCH 2/4] nvme: use bio_integrity_map_user Date: Wed, 18 Oct 2023 08:18:41 -0700 Message-ID: <20231018151843.3542335-3-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231018151843.3542335-1-kbusch@meta.com> References: <20231018151843.3542335-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: KWW97MCWd4zimF_hQUsZuqtHbQOGg3xh X-Proofpoint-ORIG-GUID: KWW97MCWd4zimF_hQUsZuqtHbQOGg3xh X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-18_13,2023-10-18_01,2023-05-22_02 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch Map user metadata buffers directly instead of maintaining a complicated copy buffer. Now that the bio tracks the metadata through its bip, nvme doesn't need special metadata handling, callbacks, or additional fields in the pdu. This greatly simplifies passthrough handling and avoids a "might_fault" copy_to_user in the completion path. This also creates pdu space to track the original request separately from its bio, further simplifying polling without relying on special iouring fields. The downside is that nvme requires the metadata buffer be physically contiguous, so user space will need to utilize huge pages if the buffer needs to span multiple pages. In practice, metadata payload sizes are a small fraction of the main payload, so this shouldn't be a problem. Signed-off-by: Keith Busch --- drivers/nvme/host/ioctl.c | 174 ++++++-------------------------------- 1 file changed, 27 insertions(+), 147 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index d8ff796fd5f21..a4889536ca4c6 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -96,52 +96,17 @@ static void __user *nvme_to_user_ptr(uintptr_t ptrval) return (void __user *)ptrval; } -static void *nvme_add_user_metadata(struct request *req, void __user *ubuf, +static int nvme_add_user_metadata(struct request *req, void __user *ubuf, unsigned len, u32 seed) { - struct bio_integrity_payload *bip; - int ret = -ENOMEM; - void *buf; - struct bio *bio = req->bio; - - buf = kmalloc(len, GFP_KERNEL); - if (!buf) - goto out; - - ret = -EFAULT; - if ((req_op(req) == REQ_OP_DRV_OUT) && copy_from_user(buf, ubuf, len)) - goto out_free_meta; - - bip = bio_integrity_alloc(bio, GFP_KERNEL, 1); - if (IS_ERR(bip)) { - ret = PTR_ERR(bip); - goto out_free_meta; - } + int ret; - bip->bip_iter.bi_sector = seed; - ret = bio_integrity_add_page(bio, virt_to_page(buf), len, - offset_in_page(buf)); - if (ret != len) { - ret = -ENOMEM; - goto out_free_meta; - } + ret = bio_integrity_map_user(req->bio, ubuf, len, seed, 1); + if (ret) + return ret; req->cmd_flags |= REQ_INTEGRITY; - return buf; -out_free_meta: - kfree(buf); -out: - return ERR_PTR(ret); -} - -static int nvme_finish_user_metadata(struct request *req, void __user *ubuf, - void *meta, unsigned len, int ret) -{ - if (!ret && req_op(req) == REQ_OP_DRV_IN && - copy_to_user(ubuf, meta, len)) - ret = -EFAULT; - kfree(meta); - return ret; + return 0; } static struct request *nvme_alloc_user_request(struct request_queue *q, @@ -160,14 +125,12 @@ static struct request *nvme_alloc_user_request(struct request_queue *q, static int nvme_map_user_request(struct request *req, u64 ubuffer, unsigned bufflen, void __user *meta_buffer, unsigned meta_len, - u32 meta_seed, void **metap, struct io_uring_cmd *ioucmd, - unsigned int flags) + u32 meta_seed, struct io_uring_cmd *ioucmd, unsigned int flags) { struct request_queue *q = req->q; struct nvme_ns *ns = q->queuedata; struct block_device *bdev = ns ? ns->disk->part0 : NULL; struct bio *bio = NULL; - void *meta = NULL; int ret; if (ioucmd && (ioucmd->flags & IORING_URING_CMD_FIXED)) { @@ -194,13 +157,10 @@ static int nvme_map_user_request(struct request *req, u64 ubuffer, bio_set_dev(bio, bdev); if (bdev && meta_buffer && meta_len) { - meta = nvme_add_user_metadata(req, meta_buffer, meta_len, + ret = nvme_add_user_metadata(req, meta_buffer, meta_len, meta_seed); - if (IS_ERR(meta)) { - ret = PTR_ERR(meta); + if (ret) goto out_unmap; - } - *metap = meta; } return ret; @@ -221,7 +181,6 @@ static int nvme_submit_user_cmd(struct request_queue *q, struct nvme_ns *ns = q->queuedata; struct nvme_ctrl *ctrl; struct request *req; - void *meta = NULL; struct bio *bio; u32 effects; int ret; @@ -233,7 +192,7 @@ static int nvme_submit_user_cmd(struct request_queue *q, req->timeout = timeout; if (ubuffer && bufflen) { ret = nvme_map_user_request(req, ubuffer, bufflen, meta_buffer, - meta_len, meta_seed, &meta, NULL, flags); + meta_len, meta_seed, NULL, flags); if (ret) return ret; } @@ -245,9 +204,6 @@ static int nvme_submit_user_cmd(struct request_queue *q, ret = nvme_execute_rq(req, false); if (result) *result = le64_to_cpu(nvme_req(req)->result.u64); - if (meta) - ret = nvme_finish_user_metadata(req, meta_buffer, meta, - meta_len, ret); if (bio) blk_rq_unmap_user(bio); blk_mq_free_request(req); @@ -442,19 +398,10 @@ struct nvme_uring_data { * Expect build errors if this grows larger than that. */ struct nvme_uring_cmd_pdu { - union { - struct bio *bio; - struct request *req; - }; - u32 meta_len; - u32 nvme_status; - union { - struct { - void *meta; /* kernel-resident buffer */ - void __user *meta_buffer; - }; - u64 result; - } u; + struct request *req; + struct bio *bio; + u64 result; + int status; }; static inline struct nvme_uring_cmd_pdu *nvme_uring_cmd_pdu( @@ -463,31 +410,6 @@ static inline struct nvme_uring_cmd_pdu *nvme_uring_cmd_pdu( return (struct nvme_uring_cmd_pdu *)&ioucmd->pdu; } -static void nvme_uring_task_meta_cb(struct io_uring_cmd *ioucmd, - unsigned issue_flags) -{ - struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); - struct request *req = pdu->req; - int status; - u64 result; - - if (nvme_req(req)->flags & NVME_REQ_CANCELLED) - status = -EINTR; - else - status = nvme_req(req)->status; - - result = le64_to_cpu(nvme_req(req)->result.u64); - - if (pdu->meta_len) - status = nvme_finish_user_metadata(req, pdu->u.meta_buffer, - pdu->u.meta, pdu->meta_len, status); - if (req->bio) - blk_rq_unmap_user(req->bio); - blk_mq_free_request(req); - - io_uring_cmd_done(ioucmd, status, result, issue_flags); -} - static void nvme_uring_task_cb(struct io_uring_cmd *ioucmd, unsigned issue_flags) { @@ -495,8 +417,7 @@ static void nvme_uring_task_cb(struct io_uring_cmd *ioucmd, if (pdu->bio) blk_rq_unmap_user(pdu->bio); - - io_uring_cmd_done(ioucmd, pdu->nvme_status, pdu->u.result, issue_flags); + io_uring_cmd_done(ioucmd, pdu->status, pdu->result, issue_flags); } static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, @@ -505,50 +426,24 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, struct io_uring_cmd *ioucmd = req->end_io_data; struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); - req->bio = pdu->bio; if (nvme_req(req)->flags & NVME_REQ_CANCELLED) - pdu->nvme_status = -EINTR; + pdu->status = -EINTR; else - pdu->nvme_status = nvme_req(req)->status; - pdu->u.result = le64_to_cpu(nvme_req(req)->result.u64); + pdu->status = nvme_req(req)->status; + pdu->result = le64_to_cpu(nvme_req(req)->result.u64); /* * For iopoll, complete it directly. * Otherwise, move the completion to task work. */ - if (blk_rq_is_poll(req)) { - WRITE_ONCE(ioucmd->cookie, NULL); + if (blk_rq_is_poll(req)) nvme_uring_task_cb(ioucmd, IO_URING_F_UNLOCKED); - } else { + else io_uring_cmd_do_in_task_lazy(ioucmd, nvme_uring_task_cb); - } return RQ_END_IO_FREE; } -static enum rq_end_io_ret nvme_uring_cmd_end_io_meta(struct request *req, - blk_status_t err) -{ - struct io_uring_cmd *ioucmd = req->end_io_data; - struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); - - req->bio = pdu->bio; - pdu->req = req; - - /* - * For iopoll, complete it directly. - * Otherwise, move the completion to task work. - */ - if (blk_rq_is_poll(req)) { - WRITE_ONCE(ioucmd->cookie, NULL); - nvme_uring_task_meta_cb(ioucmd, IO_URING_F_UNLOCKED); - } else { - io_uring_cmd_do_in_task_lazy(ioucmd, nvme_uring_task_meta_cb); - } - - return RQ_END_IO_NONE; -} - static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, struct io_uring_cmd *ioucmd, unsigned int issue_flags, bool vec) { @@ -560,7 +455,6 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, struct request *req; blk_opf_t rq_flags = REQ_ALLOC_CACHE; blk_mq_req_flags_t blk_flags = 0; - void *meta = NULL; int ret; c.common.opcode = READ_ONCE(cmd->opcode); @@ -608,27 +502,17 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, if (d.addr && d.data_len) { ret = nvme_map_user_request(req, d.addr, d.data_len, nvme_to_user_ptr(d.metadata), - d.metadata_len, 0, &meta, ioucmd, vec); + d.metadata_len, 0, ioucmd, vec); if (ret) return ret; } - if (blk_rq_is_poll(req)) { - ioucmd->flags |= IORING_URING_CMD_POLLED; - WRITE_ONCE(ioucmd->cookie, req); - } /* to free bio on completion, as req->bio will be null at that time */ pdu->bio = req->bio; - pdu->meta_len = d.metadata_len; + pdu->req = req; req->end_io_data = ioucmd; - if (pdu->meta_len) { - pdu->u.meta = meta; - pdu->u.meta_buffer = nvme_to_user_ptr(d.metadata); - req->end_io = nvme_uring_cmd_end_io_meta; - } else { - req->end_io = nvme_uring_cmd_end_io; - } + req->end_io = nvme_uring_cmd_end_io; blk_execute_rq_nowait(req, false); return -EIOCBQUEUED; } @@ -779,16 +663,12 @@ int nvme_ns_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, struct io_comp_batch *iob, unsigned int poll_flags) { - struct request *req; - int ret = 0; - - if (!(ioucmd->flags & IORING_URING_CMD_POLLED)) - return 0; + struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); + struct request *req = pdu->req; - req = READ_ONCE(ioucmd->cookie); if (req && blk_rq_is_poll(req)) - ret = blk_rq_poll(req, iob, poll_flags); - return ret; + return blk_rq_poll(req, iob, poll_flags); + return 0; } #ifdef CONFIG_NVME_MULTIPATH static int nvme_ns_head_ctrl_ioctl(struct nvme_ns *ns, unsigned int cmd, From patchwork Wed Oct 18 15:18:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13427224 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70F13CDB47E for ; Wed, 18 Oct 2023 15:24:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231210AbjJRPYi (ORCPT ); Wed, 18 Oct 2023 11:24:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230391AbjJRPYh (ORCPT ); Wed, 18 Oct 2023 11:24:37 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72F85109 for ; Wed, 18 Oct 2023 08:24:36 -0700 (PDT) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 39I4Xcjw026840 for ; Wed, 18 Oct 2023 08:24:35 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=Bq/y6Ay8FGWy4bqRyOewKN5HseTbnUMhXdfqoo3Xi2w=; b=MCFJOt0EEzV+lPQ8Ulbj9Z8LbMbOLV52ixk6rERRXZw6QgtGrmtRjgjTu9T3lJHKLVDW sYIiAvdj9/y1Wb1vycn2WMgocggMowB+NS9Fhf2VG+UP0vRDIyAlx0CRqXhpaMGvN9ip JmwQ6fXuG9n33nswWv1F5sh4dMfQPnqK3dQk9U1AEl8doLRLfkUne3hhExRtFYtrUsCa xw24CNympXpuAiPm2UuI+lX2aD/7E6x7btQG3Cj127cVYMljBBRV5vaxdhGwCQkAcYwv QtU/dY5LUZl8I5yYehp9xJBVa76Vl0CYUkmcfMUQboeECnwtV3ji+1hEhodWjxuNv2z1 mQ== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3ts86xq5bv-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 18 Oct 2023 08:24:35 -0700 Received: from twshared5242.08.ash8.facebook.com (2620:10d:c0a8:1c::1b) by mail.thefacebook.com (2620:10d:c0a8:83::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Wed, 18 Oct 2023 08:24:32 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 7E7C2205F440D; Wed, 18 Oct 2023 08:24:27 -0700 (PDT) From: Keith Busch To: , , CC: , , , , Keith Busch Subject: [PATCH 3/4] iouring: remove IORING_URING_CMD_POLLED Date: Wed, 18 Oct 2023 08:18:42 -0700 Message-ID: <20231018151843.3542335-4-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231018151843.3542335-1-kbusch@meta.com> References: <20231018151843.3542335-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: 6_qHmDyudYTbzCSHK4B6IN66LJ4JNT2b X-Proofpoint-ORIG-GUID: 6_qHmDyudYTbzCSHK4B6IN66LJ4JNT2b X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-18_13,2023-10-18_01,2023-05-22_02 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch No more users of this flag. Signed-off-by: Keith Busch Reviewed-by: Christoph Hellwig --- include/uapi/linux/io_uring.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 8e61f8b7c2ced..10e724370b612 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -249,10 +249,8 @@ enum io_uring_op { * sqe->uring_cmd_flags * IORING_URING_CMD_FIXED use registered buffer; pass this flag * along with setting sqe->buf_index. - * IORING_URING_CMD_POLLED driver use only */ #define IORING_URING_CMD_FIXED (1U << 0) -#define IORING_URING_CMD_POLLED (1U << 31) /* From patchwork Wed Oct 18 15:18:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13427226 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03BD5C46CA1 for ; Wed, 18 Oct 2023 15:24:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231344AbjJRPYq (ORCPT ); Wed, 18 Oct 2023 11:24:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231472AbjJRPYq (ORCPT ); Wed, 18 Oct 2023 11:24:46 -0400 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A69D198 for ; Wed, 18 Oct 2023 08:24:44 -0700 (PDT) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 39I84NIU015181 for ; Wed, 18 Oct 2023 08:24:44 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=a+CBAEnmZsjKkElSI8KyeZYdLbO9F0M1lGnJEP99RWs=; b=N8CshzJ9Eq9BIS21Znz+dzoSBoCZigVCBULa8imL7wj00d7+P2QvPQcEJfhBbfsrsCrE KFdAQE11y/uTPNNfMYhJk6wciRVGJkPeLZVYBeUXUMLqIRNZ0bz3yjExy+MfL77IPt8P azeGX1r9d/H3wtR4xaxG4w3EQOgzGxsuvwmsQtPCKwgFlMG6+sRS7EAjxS6cAOmWw822 eTP3oAjZcDvyVmL+urGvSDXmNOZtVkxECdaNgtqXVbR8O/+nWgbKpcONcpbema0SvdkP a7mQQJPcAY/pchvwuEuuD0SS3ZA1XEQ3hB8tR6WzpvsljCxLQBU+qsO880MCvypb7P7d eA== Received: from maileast.thefacebook.com ([163.114.130.16]) by m0089730.ppops.net (PPS) with ESMTPS id 3ttbhatctk-11 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Wed, 18 Oct 2023 08:24:43 -0700 Received: from twshared34392.14.frc2.facebook.com (2620:10d:c0a8:1b::2d) by mail.thefacebook.com (2620:10d:c0a8:82::b) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Wed, 18 Oct 2023 08:24:32 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 90DE9205F440F; Wed, 18 Oct 2023 08:24:27 -0700 (PDT) From: Keith Busch To: , , CC: , , , , Keith Busch Subject: [PATCH 4/4] io_uring: remove uring_cmd cookie Date: Wed, 18 Oct 2023 08:18:43 -0700 Message-ID: <20231018151843.3542335-5-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231018151843.3542335-1-kbusch@meta.com> References: <20231018151843.3542335-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: c22ermQ3dVE-gZ4QVLYD-QqRgADtxshn X-Proofpoint-ORIG-GUID: c22ermQ3dVE-gZ4QVLYD-QqRgADtxshn X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-18_13,2023-10-18_01,2023-05-22_02 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch No more users of this field. Signed-off-by: Keith Busch --- include/linux/io_uring.h | 8 ++------ io_uring/uring_cmd.c | 1 - 2 files changed, 2 insertions(+), 7 deletions(-) diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index 106cdc55ff3bd..30d3db4bc61a7 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -25,12 +25,8 @@ enum io_uring_cmd_flags { struct io_uring_cmd { struct file *file; const struct io_uring_sqe *sqe; - union { - /* callback to defer completions to task context */ - void (*task_work_cb)(struct io_uring_cmd *cmd, unsigned); - /* used for polled completion */ - void *cookie; - }; + /* callback to defer completions to task context */ + void (*task_work_cb)(struct io_uring_cmd *cmd, unsigned); u32 cmd_op; u32 flags; u8 pdu[32]; /* available inline for free use */ diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 537795fddc87d..56f3ef8206057 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -133,7 +133,6 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags) return -EOPNOTSUPP; issue_flags |= IO_URING_F_IOPOLL; req->iopoll_completed = 0; - WRITE_ONCE(ioucmd->cookie, NULL); } ret = file->f_op->uring_cmd(ioucmd, issue_flags);