From patchwork Mon May 15 12:54:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13241357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0629DC77B75 for ; Mon, 15 May 2023 12:58:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241918AbjEOM6G (ORCPT ); Mon, 15 May 2023 08:58:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242087AbjEOM5v (ORCPT ); Mon, 15 May 2023 08:57:51 -0400 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E5451BD8; Mon, 15 May 2023 05:57:40 -0700 (PDT) Received: by mail-ej1-x630.google.com with SMTP id a640c23a62f3a-96a2b6de3cbso1066625866b.1; Mon, 15 May 2023 05:57:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684155459; x=1686747459; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MBnj2lFa+PB4bOPosEwzuAw3ZW4WfbS6upUcJq1uNTI=; b=JsNN5O2Ztvssh2QEpi9nbfl/t9Sn3wkwVm3DmD2Z12g9RFC+Hkr7AQPTrknfPKL7RI iAvvhEIyzfD/UWYOlTyVzaeVZLlnBJxLtGwqN73pfKqDlVVXZPebFEoxd1ddUpDInV3q CXFhjKNEhDRcwisQ3p8TZyZGk1QnfW1RRaxqi7kolzEqR2yvKd4uvh5YgjoE30hJTPch Dd+qUMrHnWHEZS9fpjt5gO4JUrJHms5G6tQpzUVEgmmMPrH8hE1liHC8NZ0lFL1Agu/n thH7plw9sbTROyM2NnHspOU9UKqYmGRv1uRroxP1GCtTsaMv5xsBhv+g7QbyfWdD4ux1 7bzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684155459; x=1686747459; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MBnj2lFa+PB4bOPosEwzuAw3ZW4WfbS6upUcJq1uNTI=; b=ETPxSxgg7P6WSKhnErWQVf5T+Ib1MA1ATOq5oIdgJr96vIj+KPCwsolaIhVHHXf+nC hNkXN6Wjt0WmkZP4FYhz5d4hwbHPQnZo7c+MtHEkavMU/uI8jfjoYwqUOQmgHSluNMni 0LlmC53r/EKomPG7UBSgmkrP5xDE29CuUugyeRLW1W2a5cVZXnzMPQmeP4XkNWvXcMr0 w8Nr/I7uuiBNgrC5GvIej7/7vz+KlrzKPgfcV8qGu6dvRRAyGn081xpjzwU94bU63xez EbYgqiSMUNsmEdgsR+h5sIA448nWzPhbeemcelhTOfNhwdZJG45/qsxh2qI126DOzHfb Vx8g== X-Gm-Message-State: AC+VfDwyAv3R4fde1+bioRd9AxCu9r6rA3QDm4iaDcY63rVdPNeSN2r5 ZB0/F/mSHzhcN4rDiPNcX0Y= X-Google-Smtp-Source: ACHHUZ4Tu3v8GwHhl2ppjKrwpi6gObiB3V9WdWreNVleJpQ6F0XBMVo3hvhxBZLu7ZUQ++IdKXq2gw== X-Received: by 2002:a17:907:c26:b0:966:238a:c93 with SMTP id ga38-20020a1709070c2600b00966238a0c93mr30248914ejc.68.1684155458787; Mon, 15 May 2023 05:57:38 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:6366]) by smtp.gmail.com with ESMTPSA id m13-20020a17090672cd00b0096ace7ae086sm4003685ejl.174.2023.05.15.05.57.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 05:57:38 -0700 (PDT) From: Pavel Begunkov To: linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, io-uring@vger.kernel.org, axboe@kernel.dk Cc: kbusch@kernel.org, hch@lst.de, sagi@grimberg.me, joshi.k@samsung.com, Pavel Begunkov Subject: [PATCH for-next 1/2] io_uring/cmd: add cmd lazy tw wake helper Date: Mon, 15 May 2023 13:54:42 +0100 Message-Id: <5b9f6716006df7e817f18bd555aee2f8f9c8b0c3.1684154817.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We want to use IOU_F_TWQ_LAZY_WAKE in commands. First, introduce a new cmd tw helper accepting TWQ flags, and then add io_uring_cmd_do_in_task_laz() that will pass IOU_F_TWQ_LAZY_WAKE and imply the "lazy" semantics, i.e. it posts no more than 1 CQE and delaying execution of this tw should not prevent forward progress. Signed-off-by: Pavel Begunkov --- include/linux/io_uring.h | 18 ++++++++++++++++-- io_uring/uring_cmd.c | 16 ++++++++++++---- 2 files changed, 28 insertions(+), 6 deletions(-) diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index 7fe31b2cd02f..bb9c666bd584 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -46,13 +46,23 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd); void io_uring_cmd_done(struct io_uring_cmd *cmd, ssize_t ret, ssize_t res2, unsigned issue_flags); -void io_uring_cmd_complete_in_task(struct io_uring_cmd *ioucmd, - void (*task_work_cb)(struct io_uring_cmd *, unsigned)); struct sock *io_uring_get_socket(struct file *file); void __io_uring_cancel(bool cancel_all); void __io_uring_free(struct task_struct *tsk); void io_uring_unreg_ringfd(void); const char *io_uring_get_opcode(u8 opcode); +void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, + void (*task_work_cb)(struct io_uring_cmd *, unsigned), + unsigned flags); +/* users should follow semantics of IOU_F_TWQ_LAZY_WAKE */ +void io_uring_cmd_do_in_task_lazy(struct io_uring_cmd *ioucmd, + void (*task_work_cb)(struct io_uring_cmd *, unsigned)); + +static inline void io_uring_cmd_complete_in_task(struct io_uring_cmd *ioucmd, + void (*task_work_cb)(struct io_uring_cmd *, unsigned)) +{ + __io_uring_cmd_do_in_task(ioucmd, task_work_cb, 0); +} static inline void io_uring_files_cancel(void) { @@ -85,6 +95,10 @@ static inline void io_uring_cmd_complete_in_task(struct io_uring_cmd *ioucmd, void (*task_work_cb)(struct io_uring_cmd *, unsigned)) { } +static inline void io_uring_cmd_do_in_task_lazy(struct io_uring_cmd *ioucmd, + void (*task_work_cb)(struct io_uring_cmd *, unsigned)) +{ +} static inline struct sock *io_uring_get_socket(struct file *file) { return NULL; diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 5e32db48696d..476c7877ce58 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -20,16 +20,24 @@ static void io_uring_cmd_work(struct io_kiocb *req, struct io_tw_state *ts) ioucmd->task_work_cb(ioucmd, issue_flags); } -void io_uring_cmd_complete_in_task(struct io_uring_cmd *ioucmd, - void (*task_work_cb)(struct io_uring_cmd *, unsigned)) +void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, + void (*task_work_cb)(struct io_uring_cmd *, unsigned), + unsigned flags) { struct io_kiocb *req = cmd_to_io_kiocb(ioucmd); ioucmd->task_work_cb = task_work_cb; req->io_task_work.func = io_uring_cmd_work; - io_req_task_work_add(req); + __io_req_task_work_add(req, flags); +} +EXPORT_SYMBOL_GPL(__io_uring_cmd_do_in_task); + +void io_uring_cmd_do_in_task_lazy(struct io_uring_cmd *ioucmd, + void (*task_work_cb)(struct io_uring_cmd *, unsigned)) +{ + __io_uring_cmd_do_in_task(ioucmd, task_work_cb, IOU_F_TWQ_LAZY_WAKE); } -EXPORT_SYMBOL_GPL(io_uring_cmd_complete_in_task); +EXPORT_SYMBOL_GPL(io_uring_cmd_do_in_task_lazy); static inline void io_req_set_cqe32_extra(struct io_kiocb *req, u64 extra1, u64 extra2) From patchwork Mon May 15 12:54:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13241358 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B28FC7EE23 for ; Mon, 15 May 2023 12:58:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242001AbjEOM6I (ORCPT ); Mon, 15 May 2023 08:58:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45966 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242088AbjEOM5v (ORCPT ); Mon, 15 May 2023 08:57:51 -0400 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 11FA61BEF; Mon, 15 May 2023 05:57:41 -0700 (PDT) Received: by mail-ej1-x62a.google.com with SMTP id a640c23a62f3a-965fc25f009so2060999066b.3; Mon, 15 May 2023 05:57:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684155459; x=1686747459; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A48EUvATFnZh9D6jBlZ8QJ+HRRU2GPVKxJ6T/ClswYY=; b=B94E+rYQEFtywvHsGIgQ9mmQaaFbdMyn2sKZp5SXzSvrHpndqya34/Wg256icZ6H/d C6i4TBNb/nm9LFRrnldiL5RDBY0qQYgteR/qWCNp3ePZaI9bS+aDrBVGH86FuarHNLh/ qOdS7lhDJzCWsyL6E4cUVM5SBRoc4A4oZDsG+e/KSeQ8LNU1PasU71B7rd+TFN1kaXlc kHu+DujDp3gALaNiGffHF6f52CccQqkVN104W3Adg3ay1TBfsI2XomQ3y22dzIa9Hwji RsTMTDVKIRBWhune2y1mIWFfkoxRJqNJhrF7WQpPSYNAzLzVeSUCCkxLtGjCo11EGGUY ZVcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684155459; x=1686747459; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A48EUvATFnZh9D6jBlZ8QJ+HRRU2GPVKxJ6T/ClswYY=; b=jXmeilTyZYCSuipXun1SzzeTwNPE0rOctspL8kUGzf8BhtZR16jRc9MAQ4EdQ6m89g dYvOd91fIXLBp/+TLvIjLQtjQyeYOleGCJbrxnh11i1xhHwSA1IwO5YBfVmGLmDQmJB9 By2pgBEfm5J3M95EFYNBWSfdFWh9ezSm1p4dBrvY2qDwwAjMuraSp9FEjcJss1whmNbp 7t9elhDl8Meg87t/+NWM524NJVMR7qK47kRSgYndgmSVggVcS0JCFn0y0j2Aa4WTfYND iQTwwz9G2rCtdqR4aRArNjZXZoOYa1WMq/9hl8uipP1fBfKVYYW+OhQXnRLgiEzV1fpE jNSw== X-Gm-Message-State: AC+VfDyWDWSkhtY9VE/Fi5J0UBfQ+2qyokTIiDjSsvxqprGO0CfI04sZ HXBZUllu8NYeqVUE+PPrID0= X-Google-Smtp-Source: ACHHUZ7Ng0cIhxR4lpY4sfifZqKzwQl+W7KvOkOYDWeQBODzPtjKkf8J4RxdtSlK5nvP0gOSrg9eQw== X-Received: by 2002:a17:907:6088:b0:949:cb6a:b6f7 with SMTP id ht8-20020a170907608800b00949cb6ab6f7mr32952390ejc.56.1684155459464; Mon, 15 May 2023 05:57:39 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:6366]) by smtp.gmail.com with ESMTPSA id m13-20020a17090672cd00b0096ace7ae086sm4003685ejl.174.2023.05.15.05.57.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 May 2023 05:57:39 -0700 (PDT) From: Pavel Begunkov To: linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, io-uring@vger.kernel.org, axboe@kernel.dk Cc: kbusch@kernel.org, hch@lst.de, sagi@grimberg.me, joshi.k@samsung.com, Pavel Begunkov Subject: [PATCH for-next 2/2] nvme: optimise io_uring passthrough completion Date: Mon, 15 May 2023 13:54:43 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Use IOU_F_TWQ_LAZY_WAKE via iou_cmd_exec_in_task_lazy() for passthrough commands completion. It further delays the execution of task_work for DEFER_TASKRUN until there are enough of task_work items queued to meet the waiting criteria, which reduces the number of wake ups we issue. Signed-off-by: Pavel Begunkov --- drivers/nvme/host/ioctl.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 81c5c9e38477..52ed1094ccbb 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -521,7 +521,7 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, if (cookie != NULL && blk_rq_is_poll(req)) nvme_uring_task_cb(ioucmd, IO_URING_F_UNLOCKED); else - io_uring_cmd_complete_in_task(ioucmd, nvme_uring_task_cb); + io_uring_cmd_do_in_task_lazy(ioucmd, nvme_uring_task_cb); return RQ_END_IO_FREE; } @@ -543,7 +543,7 @@ static enum rq_end_io_ret nvme_uring_cmd_end_io_meta(struct request *req, if (cookie != NULL && blk_rq_is_poll(req)) nvme_uring_task_meta_cb(ioucmd, IO_URING_F_UNLOCKED); else - io_uring_cmd_complete_in_task(ioucmd, nvme_uring_task_meta_cb); + io_uring_cmd_do_in_task_lazy(ioucmd, nvme_uring_task_meta_cb); return RQ_END_IO_NONE; }