From patchwork Mon Sep 11 20:40:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13380248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6164DCA0EC7 for ; Mon, 11 Sep 2023 22:12:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236209AbjIKWKO (ORCPT ); Mon, 11 Sep 2023 18:10:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244544AbjIKUkc (ORCPT ); Mon, 11 Sep 2023 16:40:32 -0400 Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1580127 for ; Mon, 11 Sep 2023 13:40:27 -0700 (PDT) Received: by mail-pf1-x42e.google.com with SMTP id d2e1a72fcca58-68cb5278e3fso1012292b3a.1 for ; Mon, 11 Sep 2023 13:40:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1694464827; x=1695069627; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9V9a5OK9Iy8QE1Mjd/+uWyQONY2gF9HS1th7ayJRu5A=; b=u+I3ujcGd9pHknxpojvCzK00d7zeHXEPYDdcFn75gMaRDT94xJjHV3k0qArsWnh3NG uePOq92RHFVaFgyN28e4LnzMGJkmhT33kK4Gw68iX9xD3/Cy+m5V1dB0C14caxVhvJhM Z9A6vm/GrBwm7DZz5pjLq2jZ7uycyXzqJvx+vu7YxLDTAvMdisHa25eGmQth9j0J92a+ rx5sv9HnZkb4pWytTVtwR81QJuPDtZYSY+iGgDLcbUU0Yp20eLkxw9cklv9dHQC/f28b O0tk/NvPujw1xAaVfXMN3UUtj44JPKAKYvt06IBQf+UT0x6xt5ZVH67L/WeTJA88a3Xz WoSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694464827; x=1695069627; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9V9a5OK9Iy8QE1Mjd/+uWyQONY2gF9HS1th7ayJRu5A=; b=jpo1KRoMmlgdJnol7jmG1vQtz7OpjArOxQx554aKuZJso04yQ4nuBlicKzTkVKRtZK FeUB94AbhVAm8kOEh8xyW971r8Bu8oqcw4E9IEgoioq5ahaZ0IOsdpL01EpLfT9T64oA x3VBx4XznPtfWsTh1jux4FHxU+kJ6jdz7/RNMeDhlZXvKF0XIkr72lT/eiIYdikT8wXl dHaZ9qvtOUDkB7qZnRuLjS9AKJverRQj3TiTaHatpJow4f+wTe+oEgEiHDp+27+6HckJ zLyWYhkaBQdoWEWJVGUlW50QjvXsLMe1aFeCzkY2G4wh3oPHgtDllkKsDthgClt6rDpx df7Q== X-Gm-Message-State: AOJu0YwepW3WRIosFqaV1paeHAYXPBa/thStEi+X/RTahDp2wDqBPrSb opakdKXrqcMtlLC9BI97QydPpnPxr8MrzYQu8tYdmQ== X-Google-Smtp-Source: AGHT+IFtildw1ndsq0i+1vRvVH+SEaumkQPQWRSZSvmYf1GvCwT/EwEWSm2WzzSOnL7pDJKmRC3WpA== X-Received: by 2002:a05:6a00:1d08:b0:68a:6cbe:35a7 with SMTP id a8-20020a056a001d0800b0068a6cbe35a7mr10934526pfx.2.1694464826821; Mon, 11 Sep 2023 13:40:26 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id ef22-20020a056a002c9600b0068fe5a5a566sm100544pfb.142.2023.09.11.13.40.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Sep 2023 13:40:25 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 1/3] io_uring/rw: split io_read() into a helper Date: Mon, 11 Sep 2023 14:40:19 -0600 Message-Id: <20230911204021.1479172-2-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230911204021.1479172-1-axboe@kernel.dk> References: <20230911204021.1479172-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add __io_read() which does the grunt of the work, leaving the completion side to the new io_read(). No functional changes in this patch. Signed-off-by: Jens Axboe --- io_uring/rw.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/io_uring/rw.c b/io_uring/rw.c index c8c822fa7980..402e8bf002d6 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -708,7 +708,7 @@ static int io_rw_init_file(struct io_kiocb *req, fmode_t mode) return 0; } -int io_read(struct io_kiocb *req, unsigned int issue_flags) +static int __io_read(struct io_kiocb *req, unsigned int issue_flags) { struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); struct io_rw_state __s, *s = &__s; @@ -853,6 +853,17 @@ int io_read(struct io_kiocb *req, unsigned int issue_flags) /* it's faster to check here then delegate to kfree */ if (iovec) kfree(iovec); + return ret; +} + +int io_read(struct io_kiocb *req, unsigned int issue_flags) +{ + int ret; + + ret = __io_read(req, issue_flags); + if (unlikely(ret < 0)) + return ret; + return kiocb_done(req, ret, issue_flags); } From patchwork Mon Sep 11 20:40:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13380244 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F707CA0ED8 for ; Mon, 11 Sep 2023 22:11:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241018AbjIKWKC (ORCPT ); Mon, 11 Sep 2023 18:10:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244545AbjIKUkd (ORCPT ); Mon, 11 Sep 2023 16:40:33 -0400 Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A148127 for ; Mon, 11 Sep 2023 13:40:29 -0700 (PDT) Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-68e2ff0b5c4so489542b3a.0 for ; Mon, 11 Sep 2023 13:40:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1694464828; x=1695069628; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WnGBrfZZuySU2APTyOcddYlt8Geg3fQwoMh64sBxPCY=; b=d3En+Isos11fb/iqh201dqZwfOYvNgN87FivMeUbWdfS7tbMDyOhRIP8Muxi20Y3KF ZWyyYiHTjCu+tZhvxBGytZQeaT8HehMAuzUkQJbmEf3W7Lj8Hp9EhSG4wA0O6Yo7kJp9 iowICh1/0/Za02E6azgbjt1heThLFGL9VYBiHD8Vc/kiKaTvPfDvPUQ2tTulGm4hS8mf i0oyG/DoeuAK2PtiXKtjJGfp8qnOym0Ww0HWChBdCpO1yU26VavVKP2szQ9hSzT2OMmI oKGP+vsoRq3M3vB+tUv2t1dK5eofnGWIfZ/6+ljKEp2gzrjHED2u3XZnhPdf/LeFgR9Y /TtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694464828; x=1695069628; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WnGBrfZZuySU2APTyOcddYlt8Geg3fQwoMh64sBxPCY=; b=EL3m87jAjwGhxxUn3ZgVYU45OQCc8vdw+KLkxHjjvt+9fsNjUhhVHngP+YuN95ACvs ADLZ7wL1qu+EWkpQyI6zs6kD06M2m9DdAKH+voHbUwWlLjTmEk+hWE16I+l9xZqVUZmV l/3FVg6Wau435n4XUAim0rHYMw/PN6ZbHjnEULN9s3IdiWOjVWP0FYp9UZTMKR4pBdNZ rz8HVHXIQlAdxQUBfOS/KW6w6KuHmfMXDByaw0ekR9LxnZ4Laq7nTrl+Fs3pgax/Pb5I dKyK8+5BD1y3BzvKgI47E+IE0iyoGBIcasruG5bJSqRKsjdtkU3WUrJ4oRqDYZo/H6R4 OcUw== X-Gm-Message-State: AOJu0YxKVcDvVYwu32MOGUB4YY6FHe0oapgmN2cewdZhSjBpXuWtgk5+ XwJGZOw/1qB9aovmA8fwcKBAaIHV32Y9CtU47dxGzQ== X-Google-Smtp-Source: AGHT+IEGGazBX2ZumWzVAVAq0lb1AGNPM7LvwRAwP44xqCplc4eGXd3OIYqfviy4LI6WNWVaNdjfkw== X-Received: by 2002:a05:6a00:4ac3:b0:68f:c8b3:3077 with SMTP id ds3-20020a056a004ac300b0068fc8b33077mr3535020pfb.1.1694464828284; Mon, 11 Sep 2023 13:40:28 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id ef22-20020a056a002c9600b0068fe5a5a566sm100544pfb.142.2023.09.11.13.40.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Sep 2023 13:40:27 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 2/3] io_uring/rw: mark readv/writev as vectored in the opcode definition Date: Mon, 11 Sep 2023 14:40:20 -0600 Message-Id: <20230911204021.1479172-3-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230911204021.1479172-1-axboe@kernel.dk> References: <20230911204021.1479172-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This is cleaner than gating on the opcode type, particularly as more read/write type opcodes may be added. Then we can use that for the data import, and for __io_read() on whether or not we need to copy state. Signed-off-by: Jens Axboe --- io_uring/opdef.c | 2 ++ io_uring/opdef.h | 2 ++ io_uring/rw.c | 10 ++++++---- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 1d26ef10fc16..bfb7c53389c0 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -65,6 +65,7 @@ const struct io_issue_def io_issue_defs[] = { .ioprio = 1, .iopoll = 1, .iopoll_queue = 1, + .vectored = 1, .prep = io_prep_rw, .issue = io_read, }, @@ -78,6 +79,7 @@ const struct io_issue_def io_issue_defs[] = { .ioprio = 1, .iopoll = 1, .iopoll_queue = 1, + .vectored = 1, .prep = io_prep_rw, .issue = io_write, }, diff --git a/io_uring/opdef.h b/io_uring/opdef.h index c22c8696e749..9e5435ec27d0 100644 --- a/io_uring/opdef.h +++ b/io_uring/opdef.h @@ -29,6 +29,8 @@ struct io_issue_def { unsigned iopoll_queue : 1; /* opcode specific path will handle ->async_data allocation if needed */ unsigned manual_alloc : 1; + /* vectored opcode, set if 1) vectored, and 2) handler needs to know */ + unsigned vectored : 1; int (*issue)(struct io_kiocb *, unsigned int); int (*prep)(struct io_kiocb *, const struct io_uring_sqe *); diff --git a/io_uring/rw.c b/io_uring/rw.c index 402e8bf002d6..c3bf38419230 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -388,8 +388,7 @@ static struct iovec *__io_import_iovec(int ddir, struct io_kiocb *req, buf = u64_to_user_ptr(rw->addr); sqe_len = rw->len; - if (opcode == IORING_OP_READ || opcode == IORING_OP_WRITE || - (req->flags & REQ_F_BUFFER_SELECT)) { + if (!io_issue_defs[opcode].vectored || req->flags & REQ_F_BUFFER_SELECT) { if (io_do_buffer_select(req)) { buf = io_buffer_select(req, &sqe_len, issue_flags); if (!buf) @@ -776,8 +775,11 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) if (ret == -EAGAIN || (req->flags & REQ_F_REISSUE)) { req->flags &= ~REQ_F_REISSUE; - /* if we can poll, just do that */ - if (req->opcode == IORING_OP_READ && file_can_poll(req->file)) + /* + * If we can poll, just do that. For a vectored read, we'll + * need to copy state first. + */ + if (file_can_poll(req->file) && !io_issue_defs[req->opcode].vectored) return -EAGAIN; /* IOPOLL retry should happen for io-wq threads */ if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL)) From patchwork Mon Sep 11 20:40:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13380268 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8ADFCA0EC8 for ; Mon, 11 Sep 2023 22:15:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1358425AbjIKWKq (ORCPT ); Mon, 11 Sep 2023 18:10:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60164 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244555AbjIKUki (ORCPT ); Mon, 11 Sep 2023 16:40:38 -0400 Received: from mail-pf1-x429.google.com (mail-pf1-x429.google.com [IPv6:2607:f8b0:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9A00127 for ; Mon, 11 Sep 2023 13:40:30 -0700 (PDT) Received: by mail-pf1-x429.google.com with SMTP id d2e1a72fcca58-68fd6dcb6c8so131299b3a.1 for ; Mon, 11 Sep 2023 13:40:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1694464830; x=1695069630; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h5d7pLYJCHN5fqfnPez7rv1tfEDXGnkJm4TIMr7pJ0w=; b=hAV5Ah9Yr9OSl7G7aVqoGqDuMouvli7bwpmR64n9+GaPhCeymbfx03qUWcj45rvb/G JDunyg6swnCT4D3BoOi6tthH7/9vDY4KCGKqEk6q3YsUadd2E16mXGQ4Ah8SjXEexTYQ AdbV47KSo273Sq8K9zg+3sGfZxYlqce1+pMWEE1veZKba7J3Vs0Z+uFXXDbCd1rkZGSh 9P9uqQ0NacSxk9hWMKPr8pEyUf3TBnqaoTUbwVa56Ljsxf1z32rmCxDVOKuP72KsvB5x IqQSXZYTto/HuIv+S7IzEnhYBA1ZMTJFTBJAHYyGdK0DRVVCfOM1Op6UdF/CIiKPUNZi H4vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694464830; x=1695069630; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h5d7pLYJCHN5fqfnPez7rv1tfEDXGnkJm4TIMr7pJ0w=; b=EIdGdE/pX2qJuZ5GnHTnHvx0FkR2MB92BNdW7KdUUlKLvX27nCuzWwGxmJE7cApRch DdEfC6obBZ8bM/pfwtuaDoDLojBd8nOTNypNdimO6ad4LTDBE3xeuMA5BI5dvOyDw1YE koSZa8bpPZbxRn0z/D49b6dOTh5f4hbq2aQui5Agvtyg0XlbPPeqtT5AXvWANR8RnIVq jTE+mTm6nEkHsb5ngMqA4ZFy4/0gtFcYA0yTwFyrVbwONJ8yZsi/ooP0xBLl9nAOZDYc 0fOqIVfXpzIMK7+PDPqnjBbfM2XWh1q3BJHEobXYRVQV+HSlRvqqwXORxqQN0pE849vQ 9JlA== X-Gm-Message-State: AOJu0YwcLUyN8l9OrvajNZLcXrDRIpPr3veu7mbT8CpX3v93gSEZ7g26 8XKHFXQom678F5VXfbWpb/vM0TJmZUcXAUFCzBJ5EQ== X-Google-Smtp-Source: AGHT+IEzf83Tr7NQFpIW/uH4lXYhiohpOjHx+S8QBAKq7EVnCR4q9Wbn2WmYugONdeKgY44gQlD5GQ== X-Received: by 2002:a05:6a00:158e:b0:68e:3eff:e93a with SMTP id u14-20020a056a00158e00b0068e3effe93amr11574230pfk.2.1694464829789; Mon, 11 Sep 2023 13:40:29 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id ef22-20020a056a002c9600b0068fe5a5a566sm100544pfb.142.2023.09.11.13.40.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Sep 2023 13:40:28 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 3/3] io_uring/rw: add support for IORING_OP_READ_MULTISHOT Date: Mon, 11 Sep 2023 14:40:21 -0600 Message-Id: <20230911204021.1479172-4-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230911204021.1479172-1-axboe@kernel.dk> References: <20230911204021.1479172-1-axboe@kernel.dk> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This behaves like IORING_OP_READ, except: 1) It only supports pollable files (eg pipes, sockets, etc). Note that for sockets, you probably want to use recv/recvmsg with multishot instead. 2) It supports multishot mode, meaning it will repeatedly trigger a read and fill a buffer when data is available. This allows similar use to recv/recvmsg but on non-sockets, where a single request will repeatedly post a CQE whenever data is read from it. 3) Because of #2, it must be used with provided buffers. This is uniformly true across any request type that supports multishot and transfers data, with the reason being that it's obviously not possible to pass in a single buffer for the data, as multiple reads may very well trigger before an application has a chance to process previous CQEs and the data passed from them. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 1 + io_uring/opdef.c | 13 +++++++ io_uring/rw.c | 66 +++++++++++++++++++++++++++++++++++ io_uring/rw.h | 2 ++ 4 files changed, 82 insertions(+) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index daa363d1a502..c35438af679a 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -246,6 +246,7 @@ enum io_uring_op { IORING_OP_FUTEX_WAIT, IORING_OP_FUTEX_WAKE, IORING_OP_FUTEX_WAITV, + IORING_OP_READ_MULTISHOT, /* this goes last, obviously */ IORING_OP_LAST, diff --git a/io_uring/opdef.c b/io_uring/opdef.c index bfb7c53389c0..03e1a6f26fa5 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -460,6 +460,16 @@ const struct io_issue_def io_issue_defs[] = { .prep = io_eopnotsupp_prep, #endif }, + [IORING_OP_READ_MULTISHOT] = { + .needs_file = 1, + .unbound_nonreg_file = 1, + .pollin = 1, + .buffer_select = 1, + .audit_skip = 1, + .ioprio = 1, + .prep = io_read_mshot_prep, + .issue = io_read_mshot, + }, }; const struct io_cold_def io_cold_defs[] = { @@ -692,6 +702,9 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_FUTEX_WAITV] = { .name = "FUTEX_WAITV", }, + [IORING_OP_READ_MULTISHOT] = { + .name = "READ_MULTISHOT", + }, }; const char *io_uring_get_opcode(u8 opcode) diff --git a/io_uring/rw.c b/io_uring/rw.c index c3bf38419230..7305792fbbbf 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -123,6 +123,22 @@ int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } +/* + * Multishot read is prepared just like a normal read/write request, only + * difference is that we set the MULTISHOT flag. + */ +int io_read_mshot_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + int ret; + + ret = io_prep_rw(req, sqe); + if (unlikely(ret)) + return ret; + + req->flags |= REQ_F_APOLL_MULTISHOT; + return 0; +} + void io_readv_writev_cleanup(struct io_kiocb *req) { struct io_async_rw *io = req->async_data; @@ -869,6 +885,56 @@ int io_read(struct io_kiocb *req, unsigned int issue_flags) return kiocb_done(req, ret, issue_flags); } +int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags) +{ + unsigned int cflags = 0; + int ret; + + /* + * Multishot MUST be used on a pollable file + */ + if (!file_can_poll(req->file)) + return -EBADFD; + + ret = __io_read(req, issue_flags); + + /* + * If we get -EAGAIN, recycle our buffer and just let normal poll + * handling arm it. + */ + if (ret == -EAGAIN) { + io_kbuf_recycle(req, issue_flags); + return -EAGAIN; + } + + /* + * Any error will terminate a multishot request + */ + if (ret <= 0) { +finish: + io_req_set_res(req, ret, cflags); + if (issue_flags & IO_URING_F_MULTISHOT) + return IOU_STOP_MULTISHOT; + return IOU_OK; + } + + /* + * Put our buffer and post a CQE. If we fail to post a CQE, then + * jump to the termination path. This request is then done. + */ + cflags = io_put_kbuf(req, issue_flags); + + if (io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, + ret, cflags | IORING_CQE_F_MORE)) { + if (issue_flags & IO_URING_F_MULTISHOT) + return IOU_ISSUE_SKIP_COMPLETE; + else + return -EAGAIN; + } + + goto finish; +} + int io_write(struct io_kiocb *req, unsigned int issue_flags) { struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); diff --git a/io_uring/rw.h b/io_uring/rw.h index 4b89f9659366..c5aed03d42a4 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -23,3 +23,5 @@ int io_writev_prep_async(struct io_kiocb *req); void io_readv_writev_cleanup(struct io_kiocb *req); void io_rw_fail(struct io_kiocb *req); void io_req_rw_complete(struct io_kiocb *req, struct io_tw_state *ts); +int io_read_mshot_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags);