From patchwork Wed Jan 22 00:05:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 11344785 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 26CDC139A for ; Wed, 22 Jan 2020 00:06:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 055872465B for ; Wed, 22 Jan 2020 00:06:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="U4MDQ+MD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729016AbgAVAGO (ORCPT ); Tue, 21 Jan 2020 19:06:14 -0500 Received: from mail-wr1-f68.google.com ([209.85.221.68]:41947 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728912AbgAVAGM (ORCPT ); Tue, 21 Jan 2020 19:06:12 -0500 Received: by mail-wr1-f68.google.com with SMTP id c9so5395697wrw.8; Tue, 21 Jan 2020 16:06:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=foKquxBAIMXjAR3tdf5BNsaZnzetsrzrXlbRi9LRBb4=; b=U4MDQ+MDFZ4+2VQMTW1Z2qxQ0J5lVSlfpS0Ugvi7MKull33Mv3+LI6Pu0vEhpl1j55 pXL3y8RB7AumVU9f5hDAnyiu1VI7uDcNkjJW06UZmOKWY7kWy7e4K0xQd4PIcde0I3dV u90jbXknx9egI5Qt5QqYbfSZBbMixLEYPnO1L8t5tFyi6d4k2yyQ1YS2jWVGF4hCvdsP A4yGRE/5YTTPOUkNAnMruf06GBg7XlAKPwwV2fbGYtvSxn3xxkoCqKMr0J3hk+dkA0Jh P9Z+1gga6DqXVT+G2Ncxtb9H9xu0RvJQuNyE1pN9GDIKwHaNjAIGsH8EZoEZJH+ggLNv P2Eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=foKquxBAIMXjAR3tdf5BNsaZnzetsrzrXlbRi9LRBb4=; b=ZoA3CFXXPDo7e7e8KdmFs8QEiI1NuK1w6bOfzn7FLoQZjzTblhO9ocITCLLs1w9ojo AtHkUFvF+DPuaOLdfCFdwP/KzA7e0IRnmEMreY1YBW8jHONcBepoQUJ/0BK02EmfaxdM DFiLOnQWUcxVq72z44iKVzadoBdYUUFfLntMzZjfdIhMGv1+CJZxcDNlk4he7gWtBZSC ufJCDTgErvTbawrQ+R0eRSzIzWjISH/XPOosKpSV5IXdUQnPrZaHBwvM7FNdoBHCJom7 hmC70qp84Hi5dibkUBjF7mT75OAbXA7BwhCN0dx0Hk4QdB1gxL+1PBKMuHz5HdTGf4D5 SWUA== X-Gm-Message-State: APjAAAVi9mV9Kon+wT7XvFJwYH/UGz2RxPbBDWOQ7e5nsrpTbViu1T5C pkadHejG/Mnlm1u/rVakS72qUiJE X-Google-Smtp-Source: APXvYqy1+h3okdtS4UStCFTzfwEd5REidTaNSenzsWxlM58/Cew+MSrlV4Q/x3+3dagG4lzWfvV9Yw== X-Received: by 2002:a5d:44cd:: with SMTP id z13mr7840658wrr.104.1579651570339; Tue, 21 Jan 2020 16:06:10 -0800 (PST) Received: from localhost.localdomain ([109.126.145.157]) by smtp.gmail.com with ESMTPSA id o4sm54527068wrw.97.2020.01.21.16.06.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jan 2020 16:06:09 -0800 (PST) From: Pavel Begunkov To: Jens Axboe , io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Cc: Alexander Viro Subject: [PATCH 1/3] splice: make do_splice public Date: Wed, 22 Jan 2020 03:05:17 +0300 Message-Id: <525cb9fc7d729a96fd7ea57d88e8f82337973bfa.1579649589.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Make do_splice(), so other kernel parts can reuse it Signed-off-by: Pavel Begunkov --- fs/splice.c | 6 +++--- include/linux/splice.h | 3 +++ 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 3009652a41c8..6a6f30432688 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -1109,9 +1109,9 @@ static int splice_pipe_to_pipe(struct pipe_inode_info *ipipe, /* * Determine where to splice to/from. */ -static long do_splice(struct file *in, loff_t __user *off_in, - struct file *out, loff_t __user *off_out, - size_t len, unsigned int flags) +long do_splice(struct file *in, loff_t __user *off_in, + struct file *out, loff_t __user *off_out, + size_t len, unsigned int flags) { struct pipe_inode_info *ipipe; struct pipe_inode_info *opipe; diff --git a/include/linux/splice.h b/include/linux/splice.h index 74b4911ac16d..ebbbfea48aa0 100644 --- a/include/linux/splice.h +++ b/include/linux/splice.h @@ -78,6 +78,9 @@ extern ssize_t add_to_pipe(struct pipe_inode_info *, struct pipe_buffer *); extern ssize_t splice_direct_to_actor(struct file *, struct splice_desc *, splice_direct_actor *); +extern long do_splice(struct file *in, loff_t __user *off_in, + struct file *out, loff_t __user *off_out, + size_t len, unsigned int flags); /* * for dynamic pipe sizing From patchwork Wed Jan 22 00:05:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 11344789 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6F5186C1 for ; Wed, 22 Jan 2020 00:06:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4B7F12465B for ; Wed, 22 Jan 2020 00:06:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="awHCSggm" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729085AbgAVAGS (ORCPT ); Tue, 21 Jan 2020 19:06:18 -0500 Received: from mail-wr1-f67.google.com ([209.85.221.67]:38926 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726876AbgAVAGO (ORCPT ); Tue, 21 Jan 2020 19:06:14 -0500 Received: by mail-wr1-f67.google.com with SMTP id y11so5400409wrt.6; Tue, 21 Jan 2020 16:06:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2cZDDiM3BuMWFJy6YAY+ROsAm9ZEBM0CwAKXBOcjqCM=; b=awHCSggm88zDz9YWt1pnOnrUxq06tRrw4lIPbd0oNLLwcEiY5srQxTDUcgfWVnKuKZ 5Tq4IMh9Vbx++N5YEM/eOwErNVtxM+m8eUqQouWNTbDwdhfRlqTURS9x8jSZJph+8BCY 1vEABm5avZyzeGXXR9W6/RhxVgyY2yAmtZFZJCNPyPHb4er8GhbnAkrPW7ZTyoFxmAsB /HkrqkFcZ/K8DCVmj43oZmt8zwTXWSpkj8/vz3pEywgN6buyS3GaQUP18cC2tKlP3mLS DQ9Hdd6o+IYEnZz+DWwDd1OtOsC5qE0+3PAOrnWBlKuRkYaprxenrbRZIIQim5xR2HZh FN8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2cZDDiM3BuMWFJy6YAY+ROsAm9ZEBM0CwAKXBOcjqCM=; b=UGxtERDFZ42ug3H2EcYAMO21v5zgxvK79ZSgqoMqdyNDwrZAqlnFPM+0zzO3SI9aPF /LKtxSgffOlJ6iBCD5kXibTs+hxTDSGGnz1N1lNw7+G1j7aEV7HgsEc7wi11GqiOjXIl b4wDR7mYRckYGn1Tm9FACqCszP6FROEAGPtk7M0RWsXW9HNRMsuDsC/tEKjEwMf+MS2N qJJ0EYauKi98Qm2YxfYblI/3uWx8rRugY694ra1YtjDV0r1w66ifhZBnXOB0qzlPAUBF gIUEZyV3+7b/9AZ58drsz0Ja/OuW2Msd1wT7dzd7wmzjxRajXN5rDrudU5N3rViVs/cc nXAA== X-Gm-Message-State: APjAAAXUGwJmxcfr4Ylakjzu3o0dV22fC/e4oM2G7o6sQgz8xKnLyIOL RFDX5fVs0sda+SSSq16+mN+LzMSL X-Google-Smtp-Source: APXvYqybH+2v7ADk4+RbBKDPW7OBclGAoqOmcYF63BucYYn2KAQGRBNA1rdqHnWyZgkxJS8GdYSy1Q== X-Received: by 2002:adf:edc4:: with SMTP id v4mr7508248wro.336.1579651572122; Tue, 21 Jan 2020 16:06:12 -0800 (PST) Received: from localhost.localdomain ([109.126.145.157]) by smtp.gmail.com with ESMTPSA id o4sm54527068wrw.97.2020.01.21.16.06.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jan 2020 16:06:11 -0800 (PST) From: Pavel Begunkov To: Jens Axboe , io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Cc: Alexander Viro Subject: [PATCH 2/3] io_uring: add interface for getting files Date: Wed, 22 Jan 2020 03:05:18 +0300 Message-Id: <96dc9d5a58450f462a62679e67db85850370a9f6.1579649589.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Preparation without functional changes. Adds io_get_file(), that allows to grab files not only into req->file. Signed-off-by: Pavel Begunkov --- fs/io_uring.c | 66 ++++++++++++++++++++++++++++++++------------------- 1 file changed, 41 insertions(+), 25 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 8f7846cb1ebf..e9e4aee0fb99 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1161,6 +1161,15 @@ static struct io_kiocb *io_get_req(struct io_ring_ctx *ctx, return NULL; } +static inline void io_put_file(struct io_ring_ctx *ctx, struct file *file, + bool fixed) +{ + if (fixed) + percpu_ref_put(&ctx->file_data->refs); + else + fput(file); +} + static void __io_req_do_free(struct io_kiocb *req) { if (likely(!io_is_fallback_req(req))) @@ -1174,12 +1183,8 @@ static void __io_req_aux_free(struct io_kiocb *req) struct io_ring_ctx *ctx = req->ctx; kfree(req->io); - if (req->file) { - if (req->flags & REQ_F_FIXED_FILE) - percpu_ref_put(&ctx->file_data->refs); - else - fput(req->file); - } + if (req->file) + io_put_file(ctx, req->file, (req->flags & REQ_F_FIXED_FILE)); } static void __io_free_req(struct io_kiocb *req) @@ -4355,41 +4360,52 @@ static inline struct file *io_file_from_index(struct io_ring_ctx *ctx, return table->files[index & IORING_FILE_TABLE_MASK];; } -static int io_req_set_file(struct io_submit_state *state, struct io_kiocb *req, - const struct io_uring_sqe *sqe) +static int io_get_file(struct io_submit_state *state, struct io_ring_ctx *ctx, + int fd, struct file **out_file, bool fixed) { - struct io_ring_ctx *ctx = req->ctx; - unsigned flags; - int fd; - - flags = READ_ONCE(sqe->flags); - fd = READ_ONCE(sqe->fd); - - if (!io_req_needs_file(req, fd)) - return 0; + struct file *file; - if (flags & IOSQE_FIXED_FILE) { + if (fixed) { if (unlikely(!ctx->file_data || (unsigned) fd >= ctx->nr_user_files)) return -EBADF; fd = array_index_nospec(fd, ctx->nr_user_files); - req->file = io_file_from_index(ctx, fd); - if (!req->file) + file = io_file_from_index(ctx, fd); + if (!file) return -EBADF; - req->flags |= REQ_F_FIXED_FILE; percpu_ref_get(&ctx->file_data->refs); } else { - if (req->needs_fixed_file) - return -EBADF; trace_io_uring_file_get(ctx, fd); - req->file = io_file_get(state, fd); - if (unlikely(!req->file)) + file = io_file_get(state, fd); + if (unlikely(!file)) return -EBADF; } + *out_file = file; return 0; } +static int io_req_set_file(struct io_submit_state *state, struct io_kiocb *req, + const struct io_uring_sqe *sqe) +{ + struct io_ring_ctx *ctx = req->ctx; + unsigned flags; + int fd; + bool fixed; + + flags = READ_ONCE(sqe->flags); + fd = READ_ONCE(sqe->fd); + + if (!io_req_needs_file(req, fd)) + return 0; + + fixed = (flags & IOSQE_FIXED_FILE); + if (unlikely(!fixed && req->needs_fixed_file)) + return -EBADF; + + return io_get_file(state, ctx, fd, &req->file, fixed); +} + static int io_grab_files(struct io_kiocb *req) { int ret = -EBADF; From patchwork Wed Jan 22 00:05:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 11344787 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 08FF5139A for ; Wed, 22 Jan 2020 00:06:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CF1AB24673 for ; Wed, 22 Jan 2020 00:06:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iyvIsQd1" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729058AbgAVAGR (ORCPT ); Tue, 21 Jan 2020 19:06:17 -0500 Received: from mail-wr1-f68.google.com ([209.85.221.68]:34844 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729027AbgAVAGQ (ORCPT ); Tue, 21 Jan 2020 19:06:16 -0500 Received: by mail-wr1-f68.google.com with SMTP id g17so5434060wro.2; Tue, 21 Jan 2020 16:06:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=h7oPN0fHcma8hj0tfTxFypzTkTs9iot+mzcZU9jYQfA=; b=iyvIsQd17aqujrzVntjSCFkUM+uY+rJXzUF9Ifnmt+lvMk3CzQDRHgzqqiXOH/NzGH 2n5OXIAymKXe3eioxs0mmc87oVrXTUTrmojgrQ4g6WpKS2CeBsUN4TBObYV4EWflJVFe ABi/hI1m/FKYZ55gkkKxEs8R2riCvoYvt5IEJgYUUvmKMNCPkDDQGvk84J98NFDDr8lJ ibw/7+oXCpN8Xw5CA2x7vBufPrxVzZfU0Fy8+dXtfIdEOlwxGVwDtd5FM6dYN9xwZ9bK P2YhohqqkajVsDIr3/PeL46vztd5oDqXoF3qQKgtKvVpEogb9fNlQcIYxed8U3pSGYKT S7nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h7oPN0fHcma8hj0tfTxFypzTkTs9iot+mzcZU9jYQfA=; b=XJhqfvJOpimGvj3tOMPKGIxNq+UIkr8kjsFMTY0JlmyD1FBJ9+Vb6QkRlLbxSMucmy L91Do07Vt9kYkLrRyTGBiKoFA2ilyGhtD3ICEyFzCd5wlRud2wEEPOWNNedH61IULj6D RO7NWVxhMV1RaLjbIfbOhEAXvRNGukhx7PG8APmZtbsPtHf9VVGI4xFAFwzENWbe7DH7 lP1he58U0jkaYdz8qbAaa9UC7DbytbyIGlkso+tNAi5hUi6aXuUnFfMDWR/mA6h5fiI1 XgsLZN0Trc/Y/v1F5bWCmB3YJskhnyA+YK9LoquTW3GallBoXpQ1PauRGCi4dmBTEShd 2sVQ== X-Gm-Message-State: APjAAAWOhsUzT4OLrS1MyMaQqCZ1Y33oSesjxZjkT2OVvo+bQxRs09M7 fqnl8q3gr0teNf8GiLGFqvN6CCrW X-Google-Smtp-Source: APXvYqw8K8lxMVwamo2yCxmbHhShPjviREeQ2PUVqFTUN8CPzg5jWYdWXZitKgsVdsl4sTIqEcY4OA== X-Received: by 2002:adf:b605:: with SMTP id f5mr7274714wre.383.1579651573836; Tue, 21 Jan 2020 16:06:13 -0800 (PST) Received: from localhost.localdomain ([109.126.145.157]) by smtp.gmail.com with ESMTPSA id o4sm54527068wrw.97.2020.01.21.16.06.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jan 2020 16:06:13 -0800 (PST) From: Pavel Begunkov To: Jens Axboe , io-uring@vger.kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Cc: Alexander Viro Subject: [PATCH 3/3] io_uring: add splice(2) support Date: Wed, 22 Jan 2020 03:05:19 +0300 Message-Id: <8bfd9a57bf42cfc10ee7195969058d6da277deed.1579649589.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: References: MIME-Version: 1.0 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Add support for splice(2). Nothing new, just reuse do_splice(). Signed-off-by: Pavel Begunkov Reported-by: kbuild test robot Reported-by: kbuild test robot --- fs/io_uring.c | 86 +++++++++++++++++++++++++++++++++++ include/uapi/linux/io_uring.h | 16 ++++++- 2 files changed, 100 insertions(+), 2 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index e9e4aee0fb99..44ec9c63c41d 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -74,6 +74,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -373,6 +374,15 @@ struct io_rw { u64 len; }; +struct io_splice { + struct file *file_in; + struct file *file_out; + loff_t __user *off_in; + loff_t __user *off_out; + u64 len; + unsigned int flags; +}; + struct io_connect { struct file *file; struct sockaddr __user *addr; @@ -534,6 +544,7 @@ struct io_kiocb { struct io_files_update files_update; struct io_fadvise fadvise; struct io_madvise madvise; + struct io_splice splice; }; struct io_async_ctx *io; @@ -719,6 +730,11 @@ static const struct io_op_def io_op_defs[] = { .needs_file = 1, .fd_non_neg = 1, }, + [IORING_OP_SPLICE] = { + .needs_file = 1, + .hash_reg_file = 1, + .unbound_nonreg_file = 1, + } }; static void io_wq_submit_work(struct io_wq_work **workptr); @@ -730,6 +746,10 @@ static void io_queue_linked_timeout(struct io_kiocb *req); static int __io_sqe_files_update(struct io_ring_ctx *ctx, struct io_uring_files_update *ip, unsigned nr_args); +static int io_get_file(struct io_submit_state *state, + struct io_ring_ctx *ctx, + int fd, struct file **out_file, + bool fixed); static struct kmem_cache *req_cachep; @@ -2322,6 +2342,61 @@ static int io_write(struct io_kiocb *req, struct io_kiocb **nxt, return ret; } +static int io_splice_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + struct io_splice* sp = &req->splice; + + sp->file_out = NULL; + sp->off_in = u64_to_user_ptr(READ_ONCE(sqe->off)); + sp->off_out = u64_to_user_ptr(READ_ONCE(sqe->off_out)); + sp->len = READ_ONCE(sqe->splice_len); + sp->flags = READ_ONCE(sqe->splice_flags); + + if (unlikely(READ_ONCE(sqe->ioprio) || (sp->flags & ~SPLICE_F_ALL))) + return -EINVAL; + + return io_get_file(NULL, req->ctx, READ_ONCE(sqe->fd_out), + &sp->file_out, (sp->flags & IOSQE_SPLICE_FIXED_OUT)); +} + +static bool io_splice_punt(struct file *file) +{ + if (get_pipe_info(file)) + return false; + if (!io_file_supports_async(file)) + return true; + return !(file->f_mode & O_NONBLOCK); +} + +static int io_splice(struct io_kiocb *req, struct io_kiocb **nxt, + bool force_nonblock) +{ + struct io_splice* sp = &req->splice; + struct file *in = sp->file_in; + struct file *out = sp->file_out; + unsigned int flags = sp->flags; + long ret; + + if (force_nonblock) { + if (io_splice_punt(in) || io_splice_punt(out)) { + req->flags |= REQ_F_MUST_PUNT; + return -EAGAIN; + } + flags |= SPLICE_F_NONBLOCK; + } + + ret = do_splice(in, sp->off_in, out, sp->off_out, sp->len, flags); + if (force_nonblock && ret == -EAGAIN) + return -EAGAIN; + + io_put_file(req->ctx, out, (flags & IOSQE_SPLICE_FIXED_OUT)); + io_cqring_add_event(req, ret); + if (ret != sp->len) + req_set_fail_links(req); + io_put_req_find_next(req, nxt); + return 0; +} + /* * IORING_OP_NOP just posts a completion event, nothing else. */ @@ -4044,6 +4119,9 @@ static int io_req_defer_prep(struct io_kiocb *req, case IORING_OP_OPENAT2: ret = io_openat2_prep(req, sqe); break; + case IORING_OP_SPLICE: + ret = io_splice_prep(req, sqe); + break; default: printk_once(KERN_WARNING "io_uring: unhandled opcode %d\n", req->opcode); @@ -4272,6 +4350,14 @@ static int io_issue_sqe(struct io_kiocb *req, const struct io_uring_sqe *sqe, } ret = io_openat2(req, nxt, force_nonblock); break; + case IORING_OP_SPLICE: + if (sqe) { + ret = io_splice_prep(req, sqe); + if (ret < 0) + break; + } + ret = io_splice(req, nxt, force_nonblock); + break; default: ret = -EINVAL; break; diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 57d05cc5e271..f234b13e7ed3 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -23,8 +23,14 @@ struct io_uring_sqe { __u64 off; /* offset into file */ __u64 addr2; }; - __u64 addr; /* pointer to buffer or iovecs */ - __u32 len; /* buffer size or number of iovecs */ + union { + __u64 addr; /* pointer to buffer or iovecs */ + __u64 off_out; + }; + union { + __u32 len; /* buffer size or number of iovecs */ + __s32 fd_out; + }; union { __kernel_rwf_t rw_flags; __u32 fsync_flags; @@ -37,10 +43,12 @@ struct io_uring_sqe { __u32 open_flags; __u32 statx_flags; __u32 fadvise_advice; + __u32 splice_flags; }; __u64 user_data; /* data to be passed back at completion time */ union { __u16 buf_index; /* index into fixed buffers, if used */ + __u64 splice_len; __u64 __pad2[3]; }; }; @@ -67,6 +75,9 @@ enum { /* always go async */ #define IOSQE_ASYNC (1U << IOSQE_ASYNC_BIT) +/* op custom flags */ +#define IOSQE_SPLICE_FIXED_OUT (1U << 16) + /* * io_uring_setup() flags */ @@ -106,6 +117,7 @@ enum { IORING_OP_SEND, IORING_OP_RECV, IORING_OP_OPENAT2, + IORING_OP_SPLICE, /* this goes last, obviously */ IORING_OP_LAST,