From patchwork Fri Apr 7 19:16:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13205255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92154C77B73 for ; Fri, 7 Apr 2023 19:17:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230171AbjDGTRK (ORCPT ); Fri, 7 Apr 2023 15:17:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230113AbjDGTRJ (ORCPT ); Fri, 7 Apr 2023 15:17:09 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7602BDFB for ; Fri, 7 Apr 2023 12:17:02 -0700 (PDT) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 337GhF0b030268 for ; Fri, 7 Apr 2023 12:17:02 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=+UFhzrSYqKN+JTPh69ZvXohlHMlA6GbEecv7ZNmfRBc=; b=Kx5ChT/ZAC2veEgWkzuJZCBtuUkWcS/9HIW46R16xXGeFjHqh1i5MaP6yeuz7v0l7Ari jD7Mpi++ZQ9Bh9YNmTZEEVfR67NGTf1YYcfoRKDAOf1YtPj3IIzk0tWRfBWHQNDj6k1j erJTKwL/Q6S8WxYJnbMzBhjwg/FBYKlWWJbprDB7PNpGYrlwtEXcEEq/cj3NuLme2Tmh Dzr0CR77f++g+xcgB/RLqNuT9eTwqNFe2iCxSI/gT5MSyiov/9b/+T73gcpjZsR/Eizg n+PWf47AvXUdB0aHjc6k+KyasGZZSkSshAQfBqy48W0QmaHhh+DGftUfJui37Gl4DN1p 2w== Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3psyun9816-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 07 Apr 2023 12:17:02 -0700 Received: from twshared25760.37.frc1.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Fri, 7 Apr 2023 12:17:01 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 6CCB9157B5F7F; Fri, 7 Apr 2023 12:16:47 -0700 (PDT) From: Keith Busch To: , , , , CC: , , Keith Busch Subject: [PATCHv2 1/5] block: add request polling helper Date: Fri, 7 Apr 2023 12:16:32 -0700 Message-ID: <20230407191636.2631046-2-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230407191636.2631046-1-kbusch@meta.com> References: <20230407191636.2631046-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: zOzSx7Tx8ajkXykGmaBWygwyFRJndO5G X-Proofpoint-ORIG-GUID: zOzSx7Tx8ajkXykGmaBWygwyFRJndO5G X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-07_12,2023-04-06_03,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch This will be used by drivers that allocate polling requests. Signed-off-by: Keith Busch --- block/blk-mq.c | 18 ++++++++++++++++++ include/linux/blk-mq.h | 2 ++ 2 files changed, 20 insertions(+) diff --git a/block/blk-mq.c b/block/blk-mq.c index 1b304f66f4e8e..67707a488b7e5 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -4737,6 +4737,24 @@ int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, struct io_comp_batch * return 0; } +int blk_rq_poll(struct request *rq, struct io_comp_batch *iob, + unsigned int poll_flags) +{ + struct request_queue *q = rq->q; + int ret; + + if (!blk_rq_is_poll(rq)) + return 0; + if (!percpu_ref_tryget(&q->q_usage_counter)) + return 0; + + ret = blk_mq_poll(q, blk_rq_to_qc(rq), iob, poll_flags); + blk_queue_exit(q); + + return ret; +} +EXPORT_SYMBOL_GPL(blk_rq_poll); + unsigned int blk_mq_rq_cpu(struct request *rq) { return rq->mq_ctx->cpu; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 06caacd77ed66..579818fa1f91d 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -722,6 +722,8 @@ int blk_mq_alloc_sq_tag_set(struct blk_mq_tag_set *set, void blk_mq_free_tag_set(struct blk_mq_tag_set *set); void blk_mq_free_request(struct request *rq); +int blk_rq_poll(struct request *rq, struct io_comp_batch *iob, + unsigned int poll_flags); bool blk_mq_queue_inflight(struct request_queue *q); From patchwork Fri Apr 7 19:16:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13205256 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1C91C77B6E for ; Fri, 7 Apr 2023 19:17:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230316AbjDGTRM (ORCPT ); Fri, 7 Apr 2023 15:17:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230269AbjDGTRL (ORCPT ); Fri, 7 Apr 2023 15:17:11 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52082C147 for ; Fri, 7 Apr 2023 12:17:03 -0700 (PDT) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 337GhEIq030225 for ; Fri, 7 Apr 2023 12:17:03 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=NRUfXoo+zRRKwmGsi/Srk58rVRQNCgmkeDuGygd3+t8=; b=kDNSLrSU+tNp1M3FXO1PreRyCln4G5QB6Sj6SqR3C+Cj4Nsbq3yYRXcx8aLaiSBBh2UL 7w8u196u3XX/d3ux9vqO/imUR9MVsXLPBzo8c0MGZH5VyxuX8TiabdH5yciAwDnzlxzY IlySIVX4TUaQEa01bhlScb6XLYxs58RiCF5a99SXsciGC4IdwMtDLwQvEj3QAR6mWM7C qu7dhG7xxthhNb9QTwtAY77KjbS1zA9XwaywdJsOa+cjZVff61PQFlRO74ei3C7ZbSRn YYPbANQj92DoRY4osLkD+vKOsmie/pDOLXlpfMOjlDgXdnVHHmkF6Gf2clGsJG/xGJ52 ug== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3psyun9818-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 07 Apr 2023 12:17:02 -0700 Received: from twshared15216.17.frc2.facebook.com (2620:10d:c085:108::8) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Fri, 7 Apr 2023 12:17:02 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id B210B157B5F82; Fri, 7 Apr 2023 12:16:47 -0700 (PDT) From: Keith Busch To: , , , , CC: , , Keith Busch Subject: [PATCHv2 2/5] nvme: simplify passthrough bio cleanup Date: Fri, 7 Apr 2023 12:16:33 -0700 Message-ID: <20230407191636.2631046-3-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230407191636.2631046-1-kbusch@meta.com> References: <20230407191636.2631046-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: 32QrxKMTfhKs969xKEDaJfyxMFihXo3G X-Proofpoint-ORIG-GUID: 32QrxKMTfhKs969xKEDaJfyxMFihXo3G X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-07_12,2023-04-06_03,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch Set the bio's bi_end_io to handle the cleanup so that uring_cmd doesn't need this complex pdu->{bio,req} switchero and restore. Signed-off-by: Keith Busch --- drivers/nvme/host/ioctl.c | 26 +++++++++----------------- 1 file changed, 9 insertions(+), 17 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index d24ea2e051564..278c57ee0db91 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -159,6 +159,11 @@ static struct request *nvme_alloc_user_request(struct request_queue *q, return req; } +static void nvme_uring_bio_end_io(struct bio *bio) +{ + blk_rq_unmap_user(bio); +} + static int nvme_map_user_request(struct request *req, u64 ubuffer, unsigned bufflen, void __user *meta_buffer, unsigned meta_len, u32 meta_seed, void **metap, struct io_uring_cmd *ioucmd, @@ -204,6 +209,7 @@ static int nvme_map_user_request(struct request *req, u64 ubuffer, *metap = meta; } + bio->bi_end_io = nvme_uring_bio_end_io; return ret; out_unmap: @@ -249,8 +255,6 @@ static int nvme_submit_user_cmd(struct request_queue *q, if (meta) ret = nvme_finish_user_metadata(req, meta_buffer, meta, meta_len, ret); - if (bio) - blk_rq_unmap_user(bio); blk_mq_free_request(req); if (effects) @@ -443,10 +447,7 @@ struct nvme_uring_data { * Expect build errors if this grows larger than that. */ struct nvme_uring_cmd_pdu { - union { - struct bio *bio; - struct request *req; - }; + struct request *req; u32 meta_len; u32 nvme_status; union { @@ -482,8 +483,6 @@ static void nvme_uring_task_meta_cb(struct io_uring_cmd *ioucmd, if (pdu->meta_len) status = nvme_finish_user_metadata(req, pdu->u.meta_buffer, pdu->u.meta, pdu->meta_len, status); - if (req->bio) - blk_rq_unmap_user(req->bio); blk_mq_free_request(req); io_uring_cmd_done(ioucmd, status, result, issue_flags); @@ -494,9 +493,6 @@ static void nvme_uring_task_cb(struct io_uring_cmd *ioucmd, { struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); - if (pdu->bio) - blk_rq_unmap_user(pdu->bio); - io_uring_cmd_done(ioucmd, pdu->nvme_status, pdu->u.result, issue_flags); } @@ -507,7 +503,6 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); void *cookie = READ_ONCE(ioucmd->cookie); - req->bio = pdu->bio; if (nvme_req(req)->flags & NVME_REQ_CANCELLED) pdu->nvme_status = -EINTR; else @@ -533,9 +528,6 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io_meta(struct request *req, struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); void *cookie = READ_ONCE(ioucmd->cookie); - req->bio = pdu->bio; - pdu->req = req; - /* * For iopoll, complete it directly. * Otherwise, move the completion to task work. @@ -624,8 +616,8 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, req->bio->bi_opf |= REQ_POLLED; } } - /* to free bio on completion, as req->bio will be null at that time */ - pdu->bio = req->bio; + + pdu->req = req; pdu->meta_len = d.metadata_len; req->end_io_data = ioucmd; if (pdu->meta_len) { From patchwork Fri Apr 7 19:16:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13205257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F11EEC6FD1D for ; Fri, 7 Apr 2023 19:17:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230248AbjDGTRa (ORCPT ); Fri, 7 Apr 2023 15:17:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229932AbjDGTR1 (ORCPT ); Fri, 7 Apr 2023 15:17:27 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D815BB98 for ; Fri, 7 Apr 2023 12:17:17 -0700 (PDT) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 337GhV1o030727 for ; Fri, 7 Apr 2023 12:17:17 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=jnhpZENk/xcLV2g88wDxXsdXF0VZ73uIc8pUlWQlfgk=; b=FipN7gqaWccdnx0Ru8b6j0swgGQA6exOU3PegXSroFE5AIzPrh3BiteE1OneqUGGZs3R C0/LdfVHOJ6RAUUayQDWreJCdxULESHP4zAqdLPlj5WTfD2Kaj9NvAcj3i/lQ6pvoZIV o1IIQWpP3KpOpG57gF5F2E7pEScY0pDFYZfgowrDraG9zdMaVDU4jG76SI2XSOlZCDtl sxKXZjeKXFvq7TxcFmwW/DGgh4OlgHvnoMweMuUeSxBnbnJWF76X+k96O/Im/+Vxuf/H 9/i0hLT+sp5C5EOY//GYLtn2yu0YnPLG6CvOiAFVc4uVnPq0ArTTZ+njm60ZitcdrVjS jw== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3pt7e6np55-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 07 Apr 2023 12:17:17 -0700 Received: from twshared0333.05.ash8.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Fri, 7 Apr 2023 12:17:00 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 2FEA8157B5F85; Fri, 7 Apr 2023 12:16:47 -0700 (PDT) From: Keith Busch To: , , , , CC: , , Keith Busch Subject: [PATCHv2 3/5] nvme: unify nvme request end_io Date: Fri, 7 Apr 2023 12:16:34 -0700 Message-ID: <20230407191636.2631046-4-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230407191636.2631046-1-kbusch@meta.com> References: <20230407191636.2631046-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: 2W9GCQD4fmAAVO3leW4C6NUV8Ajc8FfE X-Proofpoint-GUID: 2W9GCQD4fmAAVO3leW4C6NUV8Ajc8FfE X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-07_12,2023-04-06_03,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch We can finish the metadata copy inline with the completion. After that, there's really nothing else different between the meta and non-meta data end_io callbacks, so unify them. Signed-off-by: Keith Busch --- drivers/nvme/host/ioctl.c | 57 +++++++-------------------------------- 1 file changed, 9 insertions(+), 48 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 278c57ee0db91..a1e0a14dadedc 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -465,29 +465,6 @@ static inline struct nvme_uring_cmd_pdu *nvme_uring_cmd_pdu( return (struct nvme_uring_cmd_pdu *)&ioucmd->pdu; } -static void nvme_uring_task_meta_cb(struct io_uring_cmd *ioucmd, - unsigned issue_flags) -{ - struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); - struct request *req = pdu->req; - int status; - u64 result; - - if (nvme_req(req)->flags & NVME_REQ_CANCELLED) - status = -EINTR; - else - status = nvme_req(req)->status; - - result = le64_to_cpu(nvme_req(req)->result.u64); - - if (pdu->meta_len) - status = nvme_finish_user_metadata(req, pdu->u.meta_buffer, - pdu->u.meta, pdu->meta_len, status); - blk_mq_free_request(req); - - io_uring_cmd_done(ioucmd, status, result, issue_flags); -} - static void nvme_uring_task_cb(struct io_uring_cmd *ioucmd, unsigned issue_flags) { @@ -502,11 +479,16 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, struct io_uring_cmd *ioucmd = req->end_io_data; struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); void *cookie = READ_ONCE(ioucmd->cookie); + int status = nvme_req(req)->status; if (nvme_req(req)->flags & NVME_REQ_CANCELLED) - pdu->nvme_status = -EINTR; - else - pdu->nvme_status = nvme_req(req)->status; + status = -EINTR; + + if (pdu->meta_len) + status = nvme_finish_user_metadata(req, pdu->u.meta_buffer, + pdu->u.meta, pdu->meta_len, status); + + pdu->nvme_status = status; pdu->u.result = le64_to_cpu(nvme_req(req)->result.u64); /* @@ -521,25 +503,6 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, return RQ_END_IO_FREE; } -static enum rq_end_io_ret nvme_uring_cmd_end_io_meta(struct request *req, - blk_status_t err) -{ - struct io_uring_cmd *ioucmd = req->end_io_data; - struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); - void *cookie = READ_ONCE(ioucmd->cookie); - - /* - * For iopoll, complete it directly. - * Otherwise, move the completion to task work. - */ - if (cookie != NULL && blk_rq_is_poll(req)) - nvme_uring_task_meta_cb(ioucmd, IO_URING_F_UNLOCKED); - else - io_uring_cmd_complete_in_task(ioucmd, nvme_uring_task_meta_cb); - - return RQ_END_IO_NONE; -} - static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, struct io_uring_cmd *ioucmd, unsigned int issue_flags, bool vec) { @@ -620,12 +583,10 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, pdu->req = req; pdu->meta_len = d.metadata_len; req->end_io_data = ioucmd; + req->end_io = nvme_uring_cmd_end_io; if (pdu->meta_len) { pdu->u.meta = meta; pdu->u.meta_buffer = nvme_to_user_ptr(d.metadata); - req->end_io = nvme_uring_cmd_end_io_meta; - } else { - req->end_io = nvme_uring_cmd_end_io; } blk_execute_rq_nowait(req, false); return -EIOCBQUEUED; From patchwork Fri Apr 7 19:16:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13205254 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38538C76196 for ; Fri, 7 Apr 2023 19:17:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230241AbjDGTRG (ORCPT ); Fri, 7 Apr 2023 15:17:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230171AbjDGTRF (ORCPT ); Fri, 7 Apr 2023 15:17:05 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A16AAD14 for ; Fri, 7 Apr 2023 12:16:59 -0700 (PDT) Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 337GhAIQ028111 for ; Fri, 7 Apr 2023 12:16:58 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=O8LsWxCTGAleK5IzhkYFJhCU1D7RgCZW4eSmA+et/fE=; b=UL1+B5lo+dBdK/DUMTOLMIbXazsjDeetXqtNHTWk1yIUOnmK+KVcm3IOGBHI2aKHWtyC WMX9e2NjR9RxwuC2GqHCfWzDCFMW54Y2KZQ2HSlGnpu8hcn0ngzL3/5+GZ32zUE5MrHk rgqWkYwMLaNxiqB39d63+8p1SwL9CyiDIofJJNrydLNFiVQ+3XjeTNYDGyC6gmfex+FM SNQSE2st+tEKbNHslr8F0F2HQ7yWapprd11IQWSpvX34fs2R4hHT3aStOGTCoNSt2ozP 04dFs+fLn1E5ARis4mRqZKBE+7NLyWzkeKHIOuaYSFlBJjtoO5F5GmWjUcoNv3rSe7XL 6Q== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3pt6y25w71-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 07 Apr 2023 12:16:58 -0700 Received: from twshared21760.39.frc1.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:11d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Fri, 7 Apr 2023 12:16:56 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 9DC37157B5F88; Fri, 7 Apr 2023 12:16:48 -0700 (PDT) From: Keith Busch To: , , , , CC: , , Keith Busch Subject: [PATCHv2 4/5] nvme: use blk-mq polling for uring commands Date: Fri, 7 Apr 2023 12:16:35 -0700 Message-ID: <20230407191636.2631046-5-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230407191636.2631046-1-kbusch@meta.com> References: <20230407191636.2631046-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: 1oG6oLpb1T4LFYIRU4l622NkK1_B_RtJ X-Proofpoint-GUID: 1oG6oLpb1T4LFYIRU4l622NkK1_B_RtJ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-07_12,2023-04-06_03,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch Drivers can now poll requests directly, and the nvme uring command unconditionally saves the request now, so use that. We just need to clear it on completion to prevent any races among multiple threads polling the same queues during a tagset teardown. The first advantage is that unshared and multipath namespaces can use the same polling callback, and multipath is guaranteed to get the same queue as the command was submitted on. Previously multipath polling might check a different path and poll the wrong info. The other advantage is that we don't need a bio payload in order to poll, allowing commands like 'flush' and 'write zeroes' to be submitted on the same high priority queue as read and write commands. This can also allow for a future optimization for the driver since we no longer need to create special hidden block devices to back nvme-generic char dev's with unsupported command sets. Signed-off-by: Keith Busch --- drivers/nvme/host/ioctl.c | 58 +++++------------------------------ drivers/nvme/host/multipath.c | 2 +- drivers/nvme/host/nvme.h | 2 -- 3 files changed, 9 insertions(+), 53 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index a1e0a14dadedc..3604166752e4b 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -478,9 +478,9 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, { struct io_uring_cmd *ioucmd = req->end_io_data; struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); - void *cookie = READ_ONCE(ioucmd->cookie); int status = nvme_req(req)->status; + WRITE_ONCE(pdu->req, NULL); if (nvme_req(req)->flags & NVME_REQ_CANCELLED) status = -EINTR; @@ -495,7 +495,7 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, * For iopoll, complete it directly. * Otherwise, move the completion to task work. */ - if (cookie != NULL && blk_rq_is_poll(req)) + if (blk_rq_is_poll(req)) nvme_uring_task_cb(ioucmd, IO_URING_F_UNLOCKED); else io_uring_cmd_complete_in_task(ioucmd, nvme_uring_task_cb); @@ -554,7 +554,6 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, if (issue_flags & IO_URING_F_IOPOLL) rq_flags |= REQ_POLLED; -retry: req = nvme_alloc_user_request(q, &c, rq_flags, blk_flags); if (IS_ERR(req)) return PTR_ERR(req); @@ -568,19 +567,7 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, return ret; } - if (issue_flags & IO_URING_F_IOPOLL && rq_flags & REQ_POLLED) { - if (unlikely(!req->bio)) { - /* we can't poll this, so alloc regular req instead */ - blk_mq_free_request(req); - rq_flags &= ~REQ_POLLED; - goto retry; - } else { - WRITE_ONCE(ioucmd->cookie, req->bio); - req->bio->bi_opf |= REQ_POLLED; - } - } - - pdu->req = req; + WRITE_ONCE(pdu->req, req); pdu->meta_len = d.metadata_len; req->end_io_data = ioucmd; req->end_io = nvme_uring_cmd_end_io; @@ -735,18 +722,14 @@ int nvme_ns_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, struct io_comp_batch *iob, unsigned int poll_flags) { - struct bio *bio; + struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); + struct request *req; int ret = 0; - struct nvme_ns *ns; - struct request_queue *q; rcu_read_lock(); - bio = READ_ONCE(ioucmd->cookie); - ns = container_of(file_inode(ioucmd->file)->i_cdev, - struct nvme_ns, cdev); - q = ns->queue; - if (test_bit(QUEUE_FLAG_POLL, &q->queue_flags) && bio && bio->bi_bdev) - ret = bio_poll(bio, iob, poll_flags); + req = READ_ONCE(pdu->req); + if (req && blk_rq_is_poll(req)) + ret = blk_rq_poll(req, iob, poll_flags); rcu_read_unlock(); return ret; } @@ -838,31 +821,6 @@ int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd, srcu_read_unlock(&head->srcu, srcu_idx); return ret; } - -int nvme_ns_head_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, - struct io_comp_batch *iob, - unsigned int poll_flags) -{ - struct cdev *cdev = file_inode(ioucmd->file)->i_cdev; - struct nvme_ns_head *head = container_of(cdev, struct nvme_ns_head, cdev); - int srcu_idx = srcu_read_lock(&head->srcu); - struct nvme_ns *ns = nvme_find_path(head); - struct bio *bio; - int ret = 0; - struct request_queue *q; - - if (ns) { - rcu_read_lock(); - bio = READ_ONCE(ioucmd->cookie); - q = ns->queue; - if (test_bit(QUEUE_FLAG_POLL, &q->queue_flags) && bio - && bio->bi_bdev) - ret = bio_poll(bio, iob, poll_flags); - rcu_read_unlock(); - } - srcu_read_unlock(&head->srcu, srcu_idx); - return ret; -} #endif /* CONFIG_NVME_MULTIPATH */ int nvme_dev_uring_cmd(struct io_uring_cmd *ioucmd, unsigned int issue_flags) diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c index 9171452e2f6d4..f17be1c72f4de 100644 --- a/drivers/nvme/host/multipath.c +++ b/drivers/nvme/host/multipath.c @@ -470,7 +470,7 @@ static const struct file_operations nvme_ns_head_chr_fops = { .unlocked_ioctl = nvme_ns_head_chr_ioctl, .compat_ioctl = compat_ptr_ioctl, .uring_cmd = nvme_ns_head_chr_uring_cmd, - .uring_cmd_iopoll = nvme_ns_head_chr_uring_cmd_iopoll, + .uring_cmd_iopoll = nvme_ns_chr_uring_cmd_iopoll, }; static int nvme_add_ns_head_cdev(struct nvme_ns_head *head) diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index bf46f122e9e1e..ca4ea89333660 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -847,8 +847,6 @@ long nvme_dev_ioctl(struct file *file, unsigned int cmd, unsigned long arg); int nvme_ns_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, struct io_comp_batch *iob, unsigned int poll_flags); -int nvme_ns_head_chr_uring_cmd_iopoll(struct io_uring_cmd *ioucmd, - struct io_comp_batch *iob, unsigned int poll_flags); int nvme_ns_chr_uring_cmd(struct io_uring_cmd *ioucmd, unsigned int issue_flags); int nvme_ns_head_chr_uring_cmd(struct io_uring_cmd *ioucmd, From patchwork Fri Apr 7 19:16:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13205258 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1954AC76196 for ; Fri, 7 Apr 2023 19:18:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229586AbjDGTSG (ORCPT ); Fri, 7 Apr 2023 15:18:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229932AbjDGTSF (ORCPT ); Fri, 7 Apr 2023 15:18:05 -0400 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7B88976F for ; Fri, 7 Apr 2023 12:18:04 -0700 (PDT) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.19/8.17.1.19) with ESMTP id 337GiLuv028093 for ; Fri, 7 Apr 2023 12:18:04 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=OUahnU2vQF7v2u8mdfu+tv4js0n8ogHNDJ3QjG1WFzs=; b=M09D4pjBtkENGGsAcvpvv5MsN+ic1NkADHSeasf5TJsmgU64TGEq10veYT5dnaIOWddh sxTrhwYhr3icf3iGxRecM6j/xfIy+htS/zIIGurWplAvqMtobpyAQ0EHTohX7kdXnmUn VtJl0IAi+Bnd7cjKuGEyURoAFibYz3oG3bBdGCG5a7uAZpzHFsyYUHq/QRCKWXdaMad4 s9r3gvg+cKTXGE3pGIi2mOWfwt/S+IFPqDtkoKaLGF2M35acRdskWJ9lE0XZPbR0UV/L ZmvbbgQ3FaLj8VikP+IY6C+cxhCBYpbGaY6xHAPrG8fGjwedxPaN84wQH2GspF3Dk+C2 pA== Received: from mail.thefacebook.com ([163.114.132.120]) by m0089730.ppops.net (PPS) with ESMTPS id 3pt9yhn2jf-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 07 Apr 2023 12:18:03 -0700 Received: from twshared15216.17.frc2.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Fri, 7 Apr 2023 12:17:02 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id CC8D1157B5F8C; Fri, 7 Apr 2023 12:16:48 -0700 (PDT) From: Keith Busch To: , , , , CC: , , Keith Busch Subject: [PATCHv2 5/5] io_uring: remove uring_cmd cookie Date: Fri, 7 Apr 2023 12:16:36 -0700 Message-ID: <20230407191636.2631046-6-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230407191636.2631046-1-kbusch@meta.com> References: <20230407191636.2631046-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: mo8FCSJuEdet6c5oFjJbTrN7_LPghfSc X-Proofpoint-ORIG-GUID: mo8FCSJuEdet6c5oFjJbTrN7_LPghfSc X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-07_12,2023-04-06_03,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch No users of this field anymore, so remove it. Signed-off-by: Keith Busch --- include/linux/io_uring.h | 8 ++------ io_uring/uring_cmd.c | 1 - 2 files changed, 2 insertions(+), 7 deletions(-) diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index 35b9328ca3352..235307bf6a072 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -25,12 +25,8 @@ enum io_uring_cmd_flags { struct io_uring_cmd { struct file *file; const void *cmd; - union { - /* callback to defer completions to task context */ - void (*task_work_cb)(struct io_uring_cmd *cmd, unsigned); - /* used for polled completion */ - void *cookie; - }; + /* callback to defer completions to task context */ + void (*task_work_cb)(struct io_uring_cmd *cmd, unsigned); u32 cmd_op; u32 flags; u8 pdu[32]; /* available inline for free use */ diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index f7a96bc76ea13..94586f691984c 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -127,7 +127,6 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags) return -EOPNOTSUPP; issue_flags |= IO_URING_F_IOPOLL; req->iopoll_completed = 0; - WRITE_ONCE(ioucmd->cookie, NULL); } ret = file->f_op->uring_cmd(ioucmd, issue_flags);