From patchwork Fri Apr 7 19:16:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Keith Busch X-Patchwork-Id: 13205257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F11EEC6FD1D for ; Fri, 7 Apr 2023 19:17:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230248AbjDGTRa (ORCPT ); Fri, 7 Apr 2023 15:17:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229932AbjDGTR1 (ORCPT ); Fri, 7 Apr 2023 15:17:27 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D815BB98 for ; Fri, 7 Apr 2023 12:17:17 -0700 (PDT) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 337GhV1o030727 for ; Fri, 7 Apr 2023 12:17:17 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=s2048-2021-q4; bh=jnhpZENk/xcLV2g88wDxXsdXF0VZ73uIc8pUlWQlfgk=; b=FipN7gqaWccdnx0Ru8b6j0swgGQA6exOU3PegXSroFE5AIzPrh3BiteE1OneqUGGZs3R C0/LdfVHOJ6RAUUayQDWreJCdxULESHP4zAqdLPlj5WTfD2Kaj9NvAcj3i/lQ6pvoZIV o1IIQWpP3KpOpG57gF5F2E7pEScY0pDFYZfgowrDraG9zdMaVDU4jG76SI2XSOlZCDtl sxKXZjeKXFvq7TxcFmwW/DGgh4OlgHvnoMweMuUeSxBnbnJWF76X+k96O/Im/+Vxuf/H 9/i0hLT+sp5C5EOY//GYLtn2yu0YnPLG6CvOiAFVc4uVnPq0ArTTZ+njm60ZitcdrVjS jw== Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3pt7e6np55-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 07 Apr 2023 12:17:17 -0700 Received: from twshared0333.05.ash8.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Fri, 7 Apr 2023 12:17:00 -0700 Received: by devbig007.nao1.facebook.com (Postfix, from userid 544533) id 2FEA8157B5F85; Fri, 7 Apr 2023 12:16:47 -0700 (PDT) From: Keith Busch To: , , , , CC: , , Keith Busch Subject: [PATCHv2 3/5] nvme: unify nvme request end_io Date: Fri, 7 Apr 2023 12:16:34 -0700 Message-ID: <20230407191636.2631046-4-kbusch@meta.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230407191636.2631046-1-kbusch@meta.com> References: <20230407191636.2631046-1-kbusch@meta.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: 2W9GCQD4fmAAVO3leW4C6NUV8Ajc8FfE X-Proofpoint-GUID: 2W9GCQD4fmAAVO3leW4C6NUV8Ajc8FfE X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-07_12,2023-04-06_03,2023-02-09_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Keith Busch We can finish the metadata copy inline with the completion. After that, there's really nothing else different between the meta and non-meta data end_io callbacks, so unify them. Signed-off-by: Keith Busch --- drivers/nvme/host/ioctl.c | 57 +++++++-------------------------------- 1 file changed, 9 insertions(+), 48 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 278c57ee0db91..a1e0a14dadedc 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -465,29 +465,6 @@ static inline struct nvme_uring_cmd_pdu *nvme_uring_cmd_pdu( return (struct nvme_uring_cmd_pdu *)&ioucmd->pdu; } -static void nvme_uring_task_meta_cb(struct io_uring_cmd *ioucmd, - unsigned issue_flags) -{ - struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); - struct request *req = pdu->req; - int status; - u64 result; - - if (nvme_req(req)->flags & NVME_REQ_CANCELLED) - status = -EINTR; - else - status = nvme_req(req)->status; - - result = le64_to_cpu(nvme_req(req)->result.u64); - - if (pdu->meta_len) - status = nvme_finish_user_metadata(req, pdu->u.meta_buffer, - pdu->u.meta, pdu->meta_len, status); - blk_mq_free_request(req); - - io_uring_cmd_done(ioucmd, status, result, issue_flags); -} - static void nvme_uring_task_cb(struct io_uring_cmd *ioucmd, unsigned issue_flags) { @@ -502,11 +479,16 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, struct io_uring_cmd *ioucmd = req->end_io_data; struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); void *cookie = READ_ONCE(ioucmd->cookie); + int status = nvme_req(req)->status; if (nvme_req(req)->flags & NVME_REQ_CANCELLED) - pdu->nvme_status = -EINTR; - else - pdu->nvme_status = nvme_req(req)->status; + status = -EINTR; + + if (pdu->meta_len) + status = nvme_finish_user_metadata(req, pdu->u.meta_buffer, + pdu->u.meta, pdu->meta_len, status); + + pdu->nvme_status = status; pdu->u.result = le64_to_cpu(nvme_req(req)->result.u64); /* @@ -521,25 +503,6 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, return RQ_END_IO_FREE; } -static enum rq_end_io_ret nvme_uring_cmd_end_io_meta(struct request *req, - blk_status_t err) -{ - struct io_uring_cmd *ioucmd = req->end_io_data; - struct nvme_uring_cmd_pdu *pdu = nvme_uring_cmd_pdu(ioucmd); - void *cookie = READ_ONCE(ioucmd->cookie); - - /* - * For iopoll, complete it directly. - * Otherwise, move the completion to task work. - */ - if (cookie != NULL && blk_rq_is_poll(req)) - nvme_uring_task_meta_cb(ioucmd, IO_URING_F_UNLOCKED); - else - io_uring_cmd_complete_in_task(ioucmd, nvme_uring_task_meta_cb); - - return RQ_END_IO_NONE; -} - static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, struct io_uring_cmd *ioucmd, unsigned int issue_flags, bool vec) { @@ -620,12 +583,10 @@ static int nvme_uring_cmd_io(struct nvme_ctrl *ctrl, struct nvme_ns *ns, pdu->req = req; pdu->meta_len = d.metadata_len; req->end_io_data = ioucmd; + req->end_io = nvme_uring_cmd_end_io; if (pdu->meta_len) { pdu->u.meta = meta; pdu->u.meta_buffer = nvme_to_user_ptr(d.metadata); - req->end_io = nvme_uring_cmd_end_io_meta; - } else { - req->end_io = nvme_uring_cmd_end_io; } blk_execute_rq_nowait(req, false); return -EIOCBQUEUED;