From patchwork Wed Jun 22 04:15:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78164C43334 for ; Wed, 22 Jun 2022 04:16:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356787AbiFVEQF (ORCPT ); Wed, 22 Jun 2022 00:16:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356616AbiFVEP5 (ORCPT ); Wed, 22 Jun 2022 00:15:57 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D053D270 for ; Tue, 21 Jun 2022 21:15:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=Q4Xdf1aJyRzAQD48KFLYFievPt8QKiRXsjXxfb/FuAw=; b=cA/vyoooQEBOX5MxcK3uE1cy7A wmPTjzBjz4o0MmZxqYMp1sufEtZI90oV+gJD+KyvD4Ppjue1+1XzAbXyC8Pjl//knkajxVxEOnYz0 jmumMexmJlwqbkluHCYy2Soxv8nOfPPX0IgHY2lxMqDYmVkTGGuALayp82x/994VwZOwBPQwuBMJZ lAP6RPGrSGsm9mpb5rzOOftkVJjqMmBGeyxSb4A+Lc/mA7h4lyEhDF1nYZY9073FG1QpVz2VL5tIV 0GlDXCBdb+6oNS7zPT1Ix735ieGP8SZBMpprxXy6hFo6Fy/i/Ih2cr2nKbPMTQdeFHXoRcfQ3Yyzg U+Wf3ydg==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmS-0035vN-Ef; Wed, 22 Jun 2022 04:15:52 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 01/44] 9p: handling Rerror without copy_from_iter_full() Date: Wed, 22 Jun 2022 05:15:09 +0100 Message-Id: <20220622041552.737754-1-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org p9_client_zc_rpc()/p9_check_zc_errors() are playing fast and loose with copy_from_iter_full(). Reading from file is done by sending Tread request. Response consists of fixed-sized header (including the amount of data actually read) followed by the data itself. For zero-copy case we arrange the things so that the first 11 bytes of reply go into the fixed-sized buffer, with the rest going straight into the pages we want to read into. What makes the things inconvenient is that sglist describing what should go where has to be set *before* the reply arrives. As the result, if reply is an error, the things get interesting. On success we get size[4] Rread tag[2] count[4] data[count] For error layout varies depending upon the protocol variant - in original 9P and 9P2000 it's size[4] Rerror tag[2] len[2] error[len] in 9P2000.U size[4] Rerror tag[2] len[2] error[len] errno[4] in 9P2000.L size[4] Rlerror tag[2] errno[4] The last case is nice and simple - we have an 11-byte response that fits into the fixed-sized buffer we hoped to get an Rread into. In other two, though, we get a variable-length string spill into the pages we'd prepared for the data to be read. Had that been in fixed-sized buffer (which is actually 4K), we would've dealt with that the same way we handle non-zerocopy case. However, for zerocopy it doesn't end up there, so we need to copy it from those pages. The trouble is, by the time we get around to that, the references to pages in question are already dropped. As the result, p9_zc_check_errors() tries to get the data using copy_from_iter_full(). Unfortunately, the iov_iter it's trying to read from might *NOT* be capable of that. It is, after all, a data destination, not data source. In particular, if it's an ITER_PIPE one, copy_from_iter_full() will simply fail. In ->zc_request() itself we do have those pages and dealing with the problem in there would be a simple matter of memcpy_from_page() into the fixed-sized buffer. Moreover, it isn't hard to recognize the (rare) case when such copying is needed. That way we get rid of p9_zc_check_errors() entirely - p9_check_errors() can be used instead both for zero-copy and non-zero-copy cases. Signed-off-by: Al Viro Reviewed-by: Dominique Martinet Tested-by: Dominique Martinet Reviewed-by: Dominique Martinet Tested-by: Dominique Martinet --- net/9p/client.c | 86 +------------------------------------------ net/9p/trans_virtio.c | 34 +++++++++++++++++ 2 files changed, 35 insertions(+), 85 deletions(-) diff --git a/net/9p/client.c b/net/9p/client.c index 8bba0d9cf975..d403085b9ef5 100644 --- a/net/9p/client.c +++ b/net/9p/client.c @@ -550,90 +550,6 @@ static int p9_check_errors(struct p9_client *c, struct p9_req_t *req) return err; } -/** - * p9_check_zc_errors - check 9p packet for error return and process it - * @c: current client instance - * @req: request to parse and check for error conditions - * @uidata: external buffer containing error - * @in_hdrlen: Size of response protocol buffer. - * - * returns error code if one is discovered, otherwise returns 0 - * - * this will have to be more complicated if we have multiple - * error packet types - */ - -static int p9_check_zc_errors(struct p9_client *c, struct p9_req_t *req, - struct iov_iter *uidata, int in_hdrlen) -{ - int err; - int ecode; - s8 type; - char *ename = NULL; - - err = p9_parse_header(&req->rc, NULL, &type, NULL, 0); - /* dump the response from server - * This should be after parse_header which poplulate pdu_fcall. - */ - trace_9p_protocol_dump(c, &req->rc); - if (err) { - p9_debug(P9_DEBUG_ERROR, "couldn't parse header %d\n", err); - return err; - } - - if (type != P9_RERROR && type != P9_RLERROR) - return 0; - - if (!p9_is_proto_dotl(c)) { - /* Error is reported in string format */ - int len; - /* 7 = header size for RERROR; */ - int inline_len = in_hdrlen - 7; - - len = req->rc.size - req->rc.offset; - if (len > (P9_ZC_HDR_SZ - 7)) { - err = -EFAULT; - goto out_err; - } - - ename = &req->rc.sdata[req->rc.offset]; - if (len > inline_len) { - /* We have error in external buffer */ - if (!copy_from_iter_full(ename + inline_len, - len - inline_len, uidata)) { - err = -EFAULT; - goto out_err; - } - } - ename = NULL; - err = p9pdu_readf(&req->rc, c->proto_version, "s?d", - &ename, &ecode); - if (err) - goto out_err; - - if (p9_is_proto_dotu(c) && ecode < 512) - err = -ecode; - - if (!err) { - err = p9_errstr2errno(ename, strlen(ename)); - - p9_debug(P9_DEBUG_9P, "<<< RERROR (%d) %s\n", - -ecode, ename); - } - kfree(ename); - } else { - err = p9pdu_readf(&req->rc, c->proto_version, "d", &ecode); - err = -ecode; - - p9_debug(P9_DEBUG_9P, "<<< RLERROR (%d)\n", -ecode); - } - return err; - -out_err: - p9_debug(P9_DEBUG_ERROR, "couldn't parse error%d\n", err); - return err; -} - static struct p9_req_t * p9_client_rpc(struct p9_client *c, int8_t type, const char *fmt, ...); @@ -874,7 +790,7 @@ static struct p9_req_t *p9_client_zc_rpc(struct p9_client *c, int8_t type, if (err < 0) goto reterr; - err = p9_check_zc_errors(c, req, uidata, in_hdrlen); + err = p9_check_errors(c, req); trace_9p_client_res(c, type, req->rc.tag, err); if (!err) return req; diff --git a/net/9p/trans_virtio.c b/net/9p/trans_virtio.c index b24a4fb0f0a2..2a210c2f8e40 100644 --- a/net/9p/trans_virtio.c +++ b/net/9p/trans_virtio.c @@ -377,6 +377,35 @@ static int p9_get_mapped_pages(struct virtio_chan *chan, } } +static void handle_rerror(struct p9_req_t *req, int in_hdr_len, + size_t offs, struct page **pages) +{ + unsigned size, n; + void *to = req->rc.sdata + in_hdr_len; + + // Fits entirely into the static data? Nothing to do. + if (req->rc.size < in_hdr_len) + return; + + // Really long error message? Tough, truncate the reply. Might get + // rejected (we can't be arsed to adjust the size encoded in header, + // or string size for that matter), but it wouldn't be anything valid + // anyway. + if (unlikely(req->rc.size > P9_ZC_HDR_SZ)) + req->rc.size = P9_ZC_HDR_SZ; + + // data won't span more than two pages + size = req->rc.size - in_hdr_len; + n = PAGE_SIZE - offs; + if (size > n) { + memcpy_from_page(to, *pages++, offs, n); + offs = 0; + to += n; + size -= n; + } + memcpy_from_page(to, *pages, offs, size); +} + /** * p9_virtio_zc_request - issue a zero copy request * @client: client instance issuing the request @@ -503,6 +532,11 @@ p9_virtio_zc_request(struct p9_client *client, struct p9_req_t *req, kicked = 1; p9_debug(P9_DEBUG_TRANS, "virtio request kicked\n"); err = wait_event_killable(req->wq, req->status >= REQ_STATUS_RCVD); + // RERROR needs reply (== error string) in static data + if (req->status == REQ_STATUS_RCVD && + unlikely(req->rc.sdata[4] == P9_RERROR)) + handle_rerror(req, in_hdr_len, offs, in_pages); + /* * Non kernel buffers are pinned, unpin them */ From patchwork Wed Jun 22 04:15:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B20F8C433EF for ; Wed, 22 Jun 2022 04:16:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356768AbiFVEQA (ORCPT ); Wed, 22 Jun 2022 00:16:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48526 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356544AbiFVEP5 (ORCPT ); Wed, 22 Jun 2022 00:15:57 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D035E263 for ; Tue, 21 Jun 2022 21:15:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=QUDzf60Onc1SNFZ87AREnowp6ZnsYcKR6SjVT2XRbD4=; b=VVVRt94+3axDMjxuWHiTvrdp+C HTiX7gsAYQakYsOB1sWHLZ9RsITlWJ/RzLjVVT8dT7ypa+JntSQQtC/ryFD6+Te4eFS45K5CPM2ca ntMNqBomXcbmA83URzRwf4GMkGMBpK+sKIX/S2djA5q8Z4xnglw2IPTSQgUG/QYAjBrvchNmTweQ7 lazAcyELm22aQdZjsWgC51I2M6I2tM3bG35Z8vLsQE1gpFSiKXvgZjBjyoVUR/94QqxKKh8L6ieIg 9SKxajJR5jL6NuJIQ6942MzWfIhZS8Iwb8sah0iIGMrSiznoHD3HC85LEZd1CB3ne8C+gSD5IdH+q SnePNHGA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmS-0035vP-My; Wed, 22 Jun 2022 04:15:52 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 02/44] No need of likely/unlikely on calls of check_copy_size() Date: Wed, 22 Jun 2022 05:15:10 +0100 Message-Id: <20220622041552.737754-2-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org it's inline and unlikely() inside of it (including the implicit one in WARN_ON_ONCE()) suffice to convince the compiler that getting false from check_copy_size() is unlikely. Spotted-by: Jens Axboe Reviewed-by: Christoph Hellwig Reviewed-by: Christian Brauner (Microsoft) Signed-off-by: Al Viro --- arch/powerpc/include/asm/uaccess.h | 2 +- arch/s390/include/asm/uaccess.h | 4 ++-- include/linux/uaccess.h | 4 ++-- include/linux/uio.h | 15 ++++++--------- 4 files changed, 11 insertions(+), 14 deletions(-) diff --git a/arch/powerpc/include/asm/uaccess.h b/arch/powerpc/include/asm/uaccess.h index 9b82b38ff867..105f200b1e31 100644 --- a/arch/powerpc/include/asm/uaccess.h +++ b/arch/powerpc/include/asm/uaccess.h @@ -348,7 +348,7 @@ copy_mc_to_kernel(void *to, const void *from, unsigned long size) static inline unsigned long __must_check copy_mc_to_user(void __user *to, const void *from, unsigned long n) { - if (likely(check_copy_size(from, n, true))) { + if (check_copy_size(from, n, true)) { if (access_ok(to, n)) { allow_write_to_user(to, n); n = copy_mc_generic((void *)to, from, n); diff --git a/arch/s390/include/asm/uaccess.h b/arch/s390/include/asm/uaccess.h index f4511e21d646..c2c9995466e0 100644 --- a/arch/s390/include/asm/uaccess.h +++ b/arch/s390/include/asm/uaccess.h @@ -39,7 +39,7 @@ _copy_from_user_key(void *to, const void __user *from, unsigned long n, unsigned static __always_inline unsigned long __must_check copy_from_user_key(void *to, const void __user *from, unsigned long n, unsigned long key) { - if (likely(check_copy_size(to, n, false))) + if (check_copy_size(to, n, false)) n = _copy_from_user_key(to, from, n, key); return n; } @@ -50,7 +50,7 @@ _copy_to_user_key(void __user *to, const void *from, unsigned long n, unsigned l static __always_inline unsigned long __must_check copy_to_user_key(void __user *to, const void *from, unsigned long n, unsigned long key) { - if (likely(check_copy_size(from, n, true))) + if (check_copy_size(from, n, true)) n = _copy_to_user_key(to, from, n, key); return n; } diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index 5a328cf02b75..47e5d374c7eb 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -148,7 +148,7 @@ _copy_to_user(void __user *, const void *, unsigned long); static __always_inline unsigned long __must_check copy_from_user(void *to, const void __user *from, unsigned long n) { - if (likely(check_copy_size(to, n, false))) + if (check_copy_size(to, n, false)) n = _copy_from_user(to, from, n); return n; } @@ -156,7 +156,7 @@ copy_from_user(void *to, const void __user *from, unsigned long n) static __always_inline unsigned long __must_check copy_to_user(void __user *to, const void *from, unsigned long n) { - if (likely(check_copy_size(from, n, true))) + if (check_copy_size(from, n, true)) n = _copy_to_user(to, from, n); return n; } diff --git a/include/linux/uio.h b/include/linux/uio.h index 739285fe5a2f..76d305f3d4c2 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -156,19 +156,17 @@ static inline size_t copy_folio_to_iter(struct folio *folio, size_t offset, static __always_inline __must_check size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { - if (unlikely(!check_copy_size(addr, bytes, true))) - return 0; - else + if (check_copy_size(addr, bytes, true)) return _copy_to_iter(addr, bytes, i); + return 0; } static __always_inline __must_check size_t copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) { - if (unlikely(!check_copy_size(addr, bytes, false))) - return 0; - else + if (check_copy_size(addr, bytes, false)) return _copy_from_iter(addr, bytes, i); + return 0; } static __always_inline __must_check @@ -184,10 +182,9 @@ bool copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i) static __always_inline __must_check size_t copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) { - if (unlikely(!check_copy_size(addr, bytes, false))) - return 0; - else + if (check_copy_size(addr, bytes, false)) return _copy_from_iter_nocache(addr, bytes, i); + return 0; } static __always_inline __must_check From patchwork Wed Jun 22 04:15:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890076 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38A5CC43334 for ; Wed, 22 Jun 2022 04:15:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356703AbiFVEP6 (ORCPT ); Wed, 22 Jun 2022 00:15:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356486AbiFVEP4 (ORCPT ); Wed, 22 Jun 2022 00:15:56 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC604209 for ; Tue, 21 Jun 2022 21:15:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=t07ZYlu7itK5ZfJfROlY9l0+O1mByVNYEP7uyOYsI4g=; b=fA7RJO5GE12hnxIvfGGbALiDCR BW4fMZX/8lSx5pgebc1JZQk2E8Ap6iEMOxVoJ/q3CLqca0uLG9WcvQ0yRofcWgU4rr/l1TNmaVZAO wbo/5Cw0Av9uAhgGmKgFTEhW1tXKZFOGH0pB39+R+6GOyeC+ws0j7NjsL0AYI41pKvHK8qW04EK1L ndbZhAsIRYY94ug0KXGjdPSj3pikeH6z7I3mIHqlme0dp/OHP63lUpNxMO014RMpr7oJeiJjQZYCH WCn0gat4IvOZK6ni80HHYhYLP/y2Gea9WPPl+HIjtsonYh5jI7WsAhf6KGDjtZY+Kn86Tj1TLzDo6 OPw90BbQ==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmS-0035vR-TH; Wed, 22 Jun 2022 04:15:52 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 03/44] teach iomap_dio_rw() to suppress dsync Date: Wed, 22 Jun 2022 05:15:11 +0100 Message-Id: <20220622041552.737754-3-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org New flag, equivalent to removal of IOCB_DSYNC from iocb flags. This mimics what btrfs is doing (and that's what btrfs will switch to). However, I'm not at all sure that we want to suppress REQ_FUA for those - all btrfs hack really cares about is suppression of generic_write_sync(). For now let's keep the existing behaviour, but I really want to hear more detailed arguments pro or contra. [folded brain fix from willy] Suggested-by: Christoph Hellwig Reviewed-by: Christian Brauner (Microsoft) Signed-off-by: Al Viro --- fs/iomap/direct-io.c | 20 +++++++++++--------- include/linux/iomap.h | 6 ++++++ 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 370c3241618a..c10c69e2de24 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -548,17 +548,19 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, } /* for data sync or sync, we need sync completion processing */ - if (iocb->ki_flags & IOCB_DSYNC) + if (iocb->ki_flags & IOCB_DSYNC && + !(dio_flags & IOMAP_DIO_NOSYNC)) { dio->flags |= IOMAP_DIO_NEED_SYNC; - /* - * For datasync only writes, we optimistically try using FUA for - * this IO. Any non-FUA write that occurs will clear this flag, - * hence we know before completion whether a cache flush is - * necessary. - */ - if ((iocb->ki_flags & (IOCB_DSYNC | IOCB_SYNC)) == IOCB_DSYNC) - dio->flags |= IOMAP_DIO_WRITE_FUA; + /* + * For datasync only writes, we optimistically try + * using FUA for this IO. Any non-FUA write that + * occurs will clear this flag, hence we know before + * completion whether a cache flush is necessary. + */ + if (!(iocb->ki_flags & IOCB_SYNC)) + dio->flags |= IOMAP_DIO_WRITE_FUA; + } } if (dio_flags & IOMAP_DIO_OVERWRITE_ONLY) { diff --git a/include/linux/iomap.h b/include/linux/iomap.h index e552097c67e0..c8622d8f064e 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -353,6 +353,12 @@ struct iomap_dio_ops { */ #define IOMAP_DIO_PARTIAL (1 << 2) +/* + * The caller will sync the write if needed; do not sync it within + * iomap_dio_rw. Overrides IOMAP_DIO_FORCE_WAIT. + */ +#define IOMAP_DIO_NOSYNC (1 << 3) + ssize_t iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, const struct iomap_ops *ops, const struct iomap_dio_ops *dops, unsigned int dio_flags, void *private, size_t done_before); From patchwork Wed Jun 22 04:15:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EB43C433EF for ; Wed, 22 Jun 2022 04:16:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356711AbiFVEP7 (ORCPT ); Wed, 22 Jun 2022 00:15:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1354898AbiFVEP4 (ORCPT ); Wed, 22 Jun 2022 00:15:56 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D169FB71 for ; Tue, 21 Jun 2022 21:15:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=7lD6VdCM50uacW9sJatiXz/xAyinAw9nJvdvluAi4MM=; b=OUok2TU3aN4EsMFe9veVlfvE/P DNA2p8rTzhttKjXgs+yAdFQ3V7fmdJCzxDVSi3suo5Od0NGbnCPFFewyHIYU26DD3wjVOkJcAbEPt HiZAEW+XWDlP6G28nXT9U6FHR5T3DPEBxdEZLTAWddd2qEWrPIsh9ebdEhaNbX8MLojzQytIHdp7P g9s4YW8m/a/zgTyuXJDaTgdTisU17uIhqRt1s69ejOvbekuwvmlBWSYmVeBXfkOt0aUTbrrsacVFz 0aYrAoog7a0W6gmjbqePS9xBlFxQpEup8tJw/5At8hA4TFTPb0JYsluXrSxWO4iH4D71xdo1NrmUu hik2QSAg==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmT-0035vT-3W; Wed, 22 Jun 2022 04:15:53 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 04/44] btrfs: use IOMAP_DIO_NOSYNC Date: Wed, 22 Jun 2022 05:15:12 +0100 Message-Id: <20220622041552.737754-4-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org ... instead of messing with iocb flags Suggested-by: Christoph Hellwig Reviewed-by: Christoph Hellwig Reviewed-by: Christian Brauner (Microsoft) Signed-off-by: Al Viro --- fs/btrfs/file.c | 17 ----------------- fs/btrfs/inode.c | 3 ++- 2 files changed, 2 insertions(+), 18 deletions(-) diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 1fd827b99c1b..98f81e304eb1 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -1848,7 +1848,6 @@ static ssize_t check_direct_IO(struct btrfs_fs_info *fs_info, static ssize_t btrfs_direct_write(struct kiocb *iocb, struct iov_iter *from) { - const bool is_sync_write = (iocb->ki_flags & IOCB_DSYNC); struct file *file = iocb->ki_filp; struct inode *inode = file_inode(file); struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); @@ -1901,15 +1900,6 @@ static ssize_t btrfs_direct_write(struct kiocb *iocb, struct iov_iter *from) goto buffered; } - /* - * We remove IOCB_DSYNC so that we don't deadlock when iomap_dio_rw() - * calls generic_write_sync() (through iomap_dio_complete()), because - * that results in calling fsync (btrfs_sync_file()) which will try to - * lock the inode in exclusive/write mode. - */ - if (is_sync_write) - iocb->ki_flags &= ~IOCB_DSYNC; - /* * The iov_iter can be mapped to the same file range we are writing to. * If that's the case, then we will deadlock in the iomap code, because @@ -1964,13 +1954,6 @@ static ssize_t btrfs_direct_write(struct kiocb *iocb, struct iov_iter *from) btrfs_inode_unlock(inode, ilock_flags); - /* - * Add back IOCB_DSYNC. Our caller, btrfs_file_write_iter(), will do - * the fsync (call generic_write_sync()). - */ - if (is_sync_write) - iocb->ki_flags |= IOCB_DSYNC; - /* If 'err' is -ENOTBLK then it means we must fallback to buffered IO. */ if ((err < 0 && err != -ENOTBLK) || !iov_iter_count(from)) goto out; diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 81737eff92f3..fbf0aee7d66a 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -8152,7 +8152,8 @@ ssize_t btrfs_dio_rw(struct kiocb *iocb, struct iov_iter *iter, size_t done_befo struct btrfs_dio_data data; return iomap_dio_rw(iocb, iter, &btrfs_dio_iomap_ops, &btrfs_dio_ops, - IOMAP_DIO_PARTIAL, &data, done_before); + IOMAP_DIO_PARTIAL | IOMAP_DIO_NOSYNC, + &data, done_before); } static int btrfs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, From patchwork Wed Jun 22 04:15:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C92ADC433EF for ; Wed, 22 Jun 2022 04:16:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356786AbiFVEQD (ORCPT ); Wed, 22 Jun 2022 00:16:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356552AbiFVEP5 (ORCPT ); Wed, 22 Jun 2022 00:15:57 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA332B7F for ; Tue, 21 Jun 2022 21:15:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=emn3i10uOt0R9V0ItwXp76JOrUIgmJ8DtQFEUtc/55w=; b=BfXK9BVTU6LRA5epAJMuzQ7EXh UbH6BU7fhhR5zQn81wOuspOqT4b/h40mdGz7vqnXodU6z5hAkUXQ8V1XNNd1rovdd3SmuHxHfXBAY Mfb+L6lCXxsTHut5DupSF1bsoDUI7uzxWT5q5lquSlk8vKvKRacoxZrEwNoVebnvudMFuL3l94s3C M1kAmx51VyLLkWKCgfCH69tdYrdtmJnnEqQWbjeLJDYeovpzHS250LlmRD/B9G/vyKQmJYSdxFMI+ ZAw4BAtPduBLjfEXId0PbJfmdXBswM03Mfqv/AOPKhbLbW+GpPIin0oRGZLWOKbjOm3tTI1iv+lfX YTdoPjmA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmT-0035vb-7U; Wed, 22 Jun 2022 04:15:53 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 05/44] struct file: use anonymous union member for rcuhead and llist Date: Wed, 22 Jun 2022 05:15:13 +0100 Message-Id: <20220622041552.737754-5-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Once upon a time we couldn't afford anon unions; these days minimal gcc version had been raised enough to take care of that. Reviewed-by: Jan Kara Reviewed-by: Christian Brauner (Microsoft) Signed-off-by: Al Viro --- fs/file_table.c | 16 ++++++++-------- include/linux/fs.h | 6 +++--- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/fs/file_table.c b/fs/file_table.c index 5424e3a8df5f..b989e33aacda 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -45,7 +45,7 @@ static struct percpu_counter nr_files __cacheline_aligned_in_smp; static void file_free_rcu(struct rcu_head *head) { - struct file *f = container_of(head, struct file, f_u.fu_rcuhead); + struct file *f = container_of(head, struct file, f_rcuhead); put_cred(f->f_cred); kmem_cache_free(filp_cachep, f); @@ -56,7 +56,7 @@ static inline void file_free(struct file *f) security_file_free(f); if (!(f->f_mode & FMODE_NOACCOUNT)) percpu_counter_dec(&nr_files); - call_rcu(&f->f_u.fu_rcuhead, file_free_rcu); + call_rcu(&f->f_rcuhead, file_free_rcu); } /* @@ -142,7 +142,7 @@ static struct file *__alloc_file(int flags, const struct cred *cred) f->f_cred = get_cred(cred); error = security_file_alloc(f); if (unlikely(error)) { - file_free_rcu(&f->f_u.fu_rcuhead); + file_free_rcu(&f->f_rcuhead); return ERR_PTR(error); } @@ -341,13 +341,13 @@ static void delayed_fput(struct work_struct *unused) struct llist_node *node = llist_del_all(&delayed_fput_list); struct file *f, *t; - llist_for_each_entry_safe(f, t, node, f_u.fu_llist) + llist_for_each_entry_safe(f, t, node, f_llist) __fput(f); } static void ____fput(struct callback_head *work) { - __fput(container_of(work, struct file, f_u.fu_rcuhead)); + __fput(container_of(work, struct file, f_rcuhead)); } /* @@ -374,8 +374,8 @@ void fput(struct file *file) struct task_struct *task = current; if (likely(!in_interrupt() && !(task->flags & PF_KTHREAD))) { - init_task_work(&file->f_u.fu_rcuhead, ____fput); - if (!task_work_add(task, &file->f_u.fu_rcuhead, TWA_RESUME)) + init_task_work(&file->f_rcuhead, ____fput); + if (!task_work_add(task, &file->f_rcuhead, TWA_RESUME)) return; /* * After this task has run exit_task_work(), @@ -384,7 +384,7 @@ void fput(struct file *file) */ } - if (llist_add(&file->f_u.fu_llist, &delayed_fput_list)) + if (llist_add(&file->f_llist, &delayed_fput_list)) schedule_delayed_work(&delayed_fput_work, 1); } } diff --git a/include/linux/fs.h b/include/linux/fs.h index 9ad5e3520fae..6a2a4906041f 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -924,9 +924,9 @@ static inline int ra_has_index(struct file_ra_state *ra, pgoff_t index) struct file { union { - struct llist_node fu_llist; - struct rcu_head fu_rcuhead; - } f_u; + struct llist_node f_llist; + struct rcu_head f_rcuhead; + }; struct path f_path; struct inode *f_inode; /* cached value */ const struct file_operations *f_op; From patchwork Wed Jun 22 04:15:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 215D0C433EF for ; Wed, 22 Jun 2022 04:16:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356795AbiFVEQI (ORCPT ); Wed, 22 Jun 2022 00:16:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356687AbiFVEP5 (ORCPT ); Wed, 22 Jun 2022 00:15:57 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA4BC10C4 for ; Tue, 21 Jun 2022 21:15:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=vdv7Cdl9g3PWc0MQ6kvQ7n5Wg7qYgCGxU/vxDuMwimQ=; b=djIZSTqwcTeVmoa2aTIxgDePT/ Fy5v49/IXTUxbC7yudp3/5OutsW9yvGrA1PMTNL/fiiBe1XeRQ+Vd9114VOCNOptrWXwl/UqifMdc VA5DrmnWJHXBnvhLlUrpOBtAiMn7UuU4l4r2nH9J3Vrz9WjnJCo2cjZg+XD2PrSn9AfENsSNs8bkQ 5ZZIE5eSUWrsC232Ut1U1uzoFrrkDxVfCJu+SDa0hvZYkciyFMYUyb1llveNE5MeZ/7/5yFVRFfCp mU4TBr7UltqwRu9qSIAQVyguNTfG+N2eQa0pM/D8iu4/cb/LBzypO/HlD/JXadq5xKAqhs8v78WcH 77M78qLg==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmT-0035vf-CE; Wed, 22 Jun 2022 04:15:53 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 06/44] iocb: delay evaluation of IS_SYNC(...) until we want to check IOCB_DSYNC Date: Wed, 22 Jun 2022 05:15:14 +0100 Message-Id: <20220622041552.737754-6-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org New helper to be used instead of direct checks for IOCB_DSYNC: iocb_is_dsync(iocb). Checks converted, which allows to avoid the IS_SYNC(iocb->ki_filp->f_mapping->host) part (4 cache lines) from iocb_flags() - it's checked in iocb_is_dsync() instead Reviewed-by: Christian Brauner (Microsoft) Signed-off-by: Al Viro --- block/fops.c | 2 +- fs/btrfs/file.c | 2 +- fs/direct-io.c | 2 +- fs/fuse/file.c | 2 +- fs/iomap/direct-io.c | 3 +-- fs/zonefs/super.c | 2 +- include/linux/fs.h | 10 ++++++++-- 7 files changed, 14 insertions(+), 9 deletions(-) diff --git a/block/fops.c b/block/fops.c index d6b3276a6c68..6e86931ab847 100644 --- a/block/fops.c +++ b/block/fops.c @@ -37,7 +37,7 @@ static unsigned int dio_bio_write_op(struct kiocb *iocb) unsigned int op = REQ_OP_WRITE | REQ_SYNC | REQ_IDLE; /* avoid the need for a I/O completion work item */ - if (iocb->ki_flags & IOCB_DSYNC) + if (iocb_is_dsync(iocb)) op |= REQ_FUA; return op; } diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 98f81e304eb1..54358a5c9d56 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -2021,7 +2021,7 @@ ssize_t btrfs_do_write_iter(struct kiocb *iocb, struct iov_iter *from, struct file *file = iocb->ki_filp; struct btrfs_inode *inode = BTRFS_I(file_inode(file)); ssize_t num_written, num_sync; - const bool sync = iocb->ki_flags & IOCB_DSYNC; + const bool sync = iocb_is_dsync(iocb); /* * If the fs flips readonly due to some impossible error, although we diff --git a/fs/direct-io.c b/fs/direct-io.c index 840752006f60..39647eb56904 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -1210,7 +1210,7 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode, */ if (dio->is_async && iov_iter_rw(iter) == WRITE) { retval = 0; - if (iocb->ki_flags & IOCB_DSYNC) + if (iocb_is_dsync(iocb)) retval = dio_set_defer_completion(dio); else if (!dio->inode->i_sb->s_dio_done_wq) { /* diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 05caa2b9272e..00fa861aeead 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1042,7 +1042,7 @@ static unsigned int fuse_write_flags(struct kiocb *iocb) { unsigned int flags = iocb->ki_filp->f_flags; - if (iocb->ki_flags & IOCB_DSYNC) + if (iocb_is_dsync(iocb)) flags |= O_DSYNC; if (iocb->ki_flags & IOCB_SYNC) flags |= O_SYNC; diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index c10c69e2de24..31c7f1035b20 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -548,8 +548,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, } /* for data sync or sync, we need sync completion processing */ - if (iocb->ki_flags & IOCB_DSYNC && - !(dio_flags & IOMAP_DIO_NOSYNC)) { + if (iocb_is_dsync(iocb) && !(dio_flags & IOMAP_DIO_NOSYNC)) { dio->flags |= IOMAP_DIO_NEED_SYNC; /* diff --git a/fs/zonefs/super.c b/fs/zonefs/super.c index bcb21aea990a..04a98b4cd7ee 100644 --- a/fs/zonefs/super.c +++ b/fs/zonefs/super.c @@ -746,7 +746,7 @@ static ssize_t zonefs_file_dio_append(struct kiocb *iocb, struct iov_iter *from) REQ_OP_ZONE_APPEND | REQ_SYNC | REQ_IDLE, GFP_NOFS); bio->bi_iter.bi_sector = zi->i_zsector; bio->bi_ioprio = iocb->ki_ioprio; - if (iocb->ki_flags & IOCB_DSYNC) + if (iocb_is_dsync(iocb)) bio->bi_opf |= REQ_FUA; ret = bio_iov_iter_get_pages(bio, from); diff --git a/include/linux/fs.h b/include/linux/fs.h index 6a2a4906041f..380a1292f4f9 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2720,6 +2720,12 @@ extern int vfs_fsync(struct file *file, int datasync); extern int sync_file_range(struct file *file, loff_t offset, loff_t nbytes, unsigned int flags); +static inline bool iocb_is_dsync(const struct kiocb *iocb) +{ + return (iocb->ki_flags & IOCB_DSYNC) || + IS_SYNC(iocb->ki_filp->f_mapping->host); +} + /* * Sync the bytes written if this was a synchronous write. Expect ki_pos * to already be updated for the write, and will return either the amount @@ -2727,7 +2733,7 @@ extern int sync_file_range(struct file *file, loff_t offset, loff_t nbytes, */ static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count) { - if (iocb->ki_flags & IOCB_DSYNC) { + if (iocb_is_dsync(iocb)) { int ret = vfs_fsync_range(iocb->ki_filp, iocb->ki_pos - count, iocb->ki_pos - 1, (iocb->ki_flags & IOCB_SYNC) ? 0 : 1); @@ -3262,7 +3268,7 @@ static inline int iocb_flags(struct file *file) res |= IOCB_APPEND; if (file->f_flags & O_DIRECT) res |= IOCB_DIRECT; - if ((file->f_flags & O_DSYNC) || IS_SYNC(file->f_mapping->host)) + if (file->f_flags & O_DSYNC) res |= IOCB_DSYNC; if (file->f_flags & __O_SYNC) res |= IOCB_SYNC; From patchwork Wed Jun 22 04:15:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59545C433EF for ; Wed, 22 Jun 2022 04:16:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354898AbiFVEQK (ORCPT ); Wed, 22 Jun 2022 00:16:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356697AbiFVEP6 (ORCPT ); Wed, 22 Jun 2022 00:15:58 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8854060E4 for ; Tue, 21 Jun 2022 21:15:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=uRY2CcDaI7j2Qa7squjvpnb9XeTEMfUPt3NGdMa0Dmo=; b=hiUAldtQ8O9ijynsdZsiY3JhLU W0mTz2ReZeFzHKmznesp0JytEaobOsaGeS4prFW64MGq46k85T16otaTTMs9YYYJoE4+bmO6en7Vg PLqZWIPqFihhxgor5wEdYV7pTBBMUtKFV/P5Ux9Ba3HY3wsVTA3I2MssYXHPXmltQ4droozOUBim0 2/Go1MwVpvpUZOqTrwUH9Sem94p6TD9tMdgwH14mNGwnFPn7uDofgXdq3WaPJiD/0A6jilx1AoPFB qmBsfjCCU5vlwZetyasxl7Rwl5R85W3n+sjE9H9aBQflC1owPhaTUXWGytzKOPtfIwmGNz4yF+KzP dWPWoNPg==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmT-0035vj-IO; Wed, 22 Jun 2022 04:15:53 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 07/44] keep iocb_flags() result cached in struct file Date: Wed, 22 Jun 2022 05:15:15 +0100 Message-Id: <20220622041552.737754-7-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org * calculate at the time we set FMODE_OPENED (do_dentry_open() for normal opens, alloc_file() for pipe()/socket()/etc.) * update when handling F_SETFL * keep in a new field - file->f_iocb_flags; since that thing is needed only before the refcount reaches zero, we can put it into the same anon union where ->f_rcuhead and ->f_llist live - those are used only after refcount reaches zero. Reviewed-by: Christian Brauner (Microsoft) Signed-off-by: Al Viro --- drivers/nvme/target/io-cmd-file.c | 2 +- fs/aio.c | 2 +- fs/fcntl.c | 1 + fs/file_table.c | 1 + fs/io_uring.c | 2 +- fs/open.c | 1 + include/linux/fs.h | 5 ++--- 7 files changed, 8 insertions(+), 6 deletions(-) diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c index f3d58abf11e0..64b47e2a4633 100644 --- a/drivers/nvme/target/io-cmd-file.c +++ b/drivers/nvme/target/io-cmd-file.c @@ -112,7 +112,7 @@ static ssize_t nvmet_file_submit_bvec(struct nvmet_req *req, loff_t pos, iocb->ki_pos = pos; iocb->ki_filp = req->ns->file; - iocb->ki_flags = ki_flags | iocb_flags(req->ns->file); + iocb->ki_flags = ki_flags | iocb->ki_filp->f_iocb_flags; return call_iter(iocb, &iter); } diff --git a/fs/aio.c b/fs/aio.c index 3c249b938632..2bdd444d408b 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1475,7 +1475,7 @@ static int aio_prep_rw(struct kiocb *req, const struct iocb *iocb) req->ki_complete = aio_complete_rw; req->private = NULL; req->ki_pos = iocb->aio_offset; - req->ki_flags = iocb_flags(req->ki_filp); + req->ki_flags = req->ki_filp->f_iocb_flags; if (iocb->aio_flags & IOCB_FLAG_RESFD) req->ki_flags |= IOCB_EVENTFD; if (iocb->aio_flags & IOCB_FLAG_IOPRIO) { diff --git a/fs/fcntl.c b/fs/fcntl.c index 34a3faa4886d..146c9ab0cd4b 100644 --- a/fs/fcntl.c +++ b/fs/fcntl.c @@ -78,6 +78,7 @@ static int setfl(int fd, struct file * filp, unsigned long arg) } spin_lock(&filp->f_lock); filp->f_flags = (arg & SETFL_MASK) | (filp->f_flags & ~SETFL_MASK); + filp->f_iocb_flags = iocb_flags(filp); spin_unlock(&filp->f_lock); out: diff --git a/fs/file_table.c b/fs/file_table.c index b989e33aacda..905792b0521c 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -241,6 +241,7 @@ static struct file *alloc_file(const struct path *path, int flags, if ((file->f_mode & FMODE_WRITE) && likely(fop->write || fop->write_iter)) file->f_mode |= FMODE_CAN_WRITE; + file->f_iocb_flags = iocb_flags(file); file->f_mode |= FMODE_OPENED; file->f_op = fop; if ((file->f_mode & (FMODE_READ | FMODE_WRITE)) == FMODE_READ) diff --git a/fs/io_uring.c b/fs/io_uring.c index 3aab4182fd89..53424b1f019f 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -4330,7 +4330,7 @@ static int io_rw_init_file(struct io_kiocb *req, fmode_t mode) if (!io_req_ffs_set(req)) req->flags |= io_file_get_flags(file) << REQ_F_SUPPORT_NOWAIT_BIT; - kiocb->ki_flags = iocb_flags(file); + kiocb->ki_flags = file->f_iocb_flags; ret = kiocb_set_rw_flags(kiocb, req->rw.flags); if (unlikely(ret)) return ret; diff --git a/fs/open.c b/fs/open.c index 1d57fbde2feb..d80441a0bf17 100644 --- a/fs/open.c +++ b/fs/open.c @@ -862,6 +862,7 @@ static int do_dentry_open(struct file *f, f->f_mode |= FMODE_CAN_ODIRECT; f->f_flags &= ~(O_CREAT | O_EXCL | O_NOCTTY | O_TRUNC); + f->f_iocb_flags = iocb_flags(f); file_ra_state_init(&f->f_ra, f->f_mapping->host->i_mapping); diff --git a/include/linux/fs.h b/include/linux/fs.h index 380a1292f4f9..c82b9d442f56 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -926,6 +926,7 @@ struct file { union { struct llist_node f_llist; struct rcu_head f_rcuhead; + unsigned int f_iocb_flags; }; struct path f_path; struct inode *f_inode; /* cached value */ @@ -2199,13 +2200,11 @@ static inline bool HAS_UNMAPPED_ID(struct user_namespace *mnt_userns, !gid_valid(i_gid_into_mnt(mnt_userns, inode)); } -static inline int iocb_flags(struct file *file); - static inline void init_sync_kiocb(struct kiocb *kiocb, struct file *filp) { *kiocb = (struct kiocb) { .ki_filp = filp, - .ki_flags = iocb_flags(filp), + .ki_flags = filp->f_iocb_flags, .ki_ioprio = get_current_ioprio(), }; } From patchwork Wed Jun 22 04:15:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 668E8C433EF for ; Wed, 22 Jun 2022 04:16:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356872AbiFVEQO (ORCPT ); Wed, 22 Jun 2022 00:16:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356698AbiFVEP6 (ORCPT ); Wed, 22 Jun 2022 00:15:58 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8860164F2 for ; Tue, 21 Jun 2022 21:15:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=EZJVKAFRPo6DTLZ4VzXnBiLtorgf9Z60bL9Bh97bIjQ=; b=Op34Ere/Z0IOtzifpoSlBjg8iz rQdt0jqfVultYIQyLTZUPAENf6RWC6G4vxnigKEAOduaad6ph0Qh3AAS3EDwVi1SZbZodetu5kNgf vnHmvssw8LaPYE9fZpsTtdu8giRkg9tsfubbNApHOgYQhyrg6OpClALOXpAiRXLzi+jjlRmMSfebU bPxYRpH2AxPf5Z7ke+FfIcGKaSkbClycGSJTZTrJN67pJAq8+/Y32LjXVTdIlpmZYtHO5fmFUs/nH XoraGUB4kRMvHD3kvdFiMBtSYV89NaCWfCzM3lV+YaDNRTZP0DaeJaISV1Hnyc0Q+HS1PR4cyD5Xs UFwrLlXQ==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmT-0035vy-W6; Wed, 22 Jun 2022 04:15:54 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 08/44] copy_page_{to,from}_iter(): switch iovec variants to generic Date: Wed, 22 Jun 2022 05:15:16 +0100 Message-Id: <20220622041552.737754-8-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org we can do copyin/copyout under kmap_local_page(); it shouldn't overflow the kmap stack - the maximal footprint increase only by one here. Signed-off-by: Al Viro Reviewed-by: Jeff Layton Reviewed-by: Christian Brauner (Microsoft) --- lib/iov_iter.c | 191 ++----------------------------------------------- 1 file changed, 4 insertions(+), 187 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 6dd5330f7a99..4c658a25e29c 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -168,174 +168,6 @@ static int copyin(void *to, const void __user *from, size_t n) return n; } -static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t bytes, - struct iov_iter *i) -{ - size_t skip, copy, left, wanted; - const struct iovec *iov; - char __user *buf; - void *kaddr, *from; - - if (unlikely(bytes > i->count)) - bytes = i->count; - - if (unlikely(!bytes)) - return 0; - - might_fault(); - wanted = bytes; - iov = i->iov; - skip = i->iov_offset; - buf = iov->iov_base + skip; - copy = min(bytes, iov->iov_len - skip); - - if (IS_ENABLED(CONFIG_HIGHMEM) && !fault_in_writeable(buf, copy)) { - kaddr = kmap_atomic(page); - from = kaddr + offset; - - /* first chunk, usually the only one */ - left = copyout(buf, from, copy); - copy -= left; - skip += copy; - from += copy; - bytes -= copy; - - while (unlikely(!left && bytes)) { - iov++; - buf = iov->iov_base; - copy = min(bytes, iov->iov_len); - left = copyout(buf, from, copy); - copy -= left; - skip = copy; - from += copy; - bytes -= copy; - } - if (likely(!bytes)) { - kunmap_atomic(kaddr); - goto done; - } - offset = from - kaddr; - buf += copy; - kunmap_atomic(kaddr); - copy = min(bytes, iov->iov_len - skip); - } - /* Too bad - revert to non-atomic kmap */ - - kaddr = kmap(page); - from = kaddr + offset; - left = copyout(buf, from, copy); - copy -= left; - skip += copy; - from += copy; - bytes -= copy; - while (unlikely(!left && bytes)) { - iov++; - buf = iov->iov_base; - copy = min(bytes, iov->iov_len); - left = copyout(buf, from, copy); - copy -= left; - skip = copy; - from += copy; - bytes -= copy; - } - kunmap(page); - -done: - if (skip == iov->iov_len) { - iov++; - skip = 0; - } - i->count -= wanted - bytes; - i->nr_segs -= iov - i->iov; - i->iov = iov; - i->iov_offset = skip; - return wanted - bytes; -} - -static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t bytes, - struct iov_iter *i) -{ - size_t skip, copy, left, wanted; - const struct iovec *iov; - char __user *buf; - void *kaddr, *to; - - if (unlikely(bytes > i->count)) - bytes = i->count; - - if (unlikely(!bytes)) - return 0; - - might_fault(); - wanted = bytes; - iov = i->iov; - skip = i->iov_offset; - buf = iov->iov_base + skip; - copy = min(bytes, iov->iov_len - skip); - - if (IS_ENABLED(CONFIG_HIGHMEM) && !fault_in_readable(buf, copy)) { - kaddr = kmap_atomic(page); - to = kaddr + offset; - - /* first chunk, usually the only one */ - left = copyin(to, buf, copy); - copy -= left; - skip += copy; - to += copy; - bytes -= copy; - - while (unlikely(!left && bytes)) { - iov++; - buf = iov->iov_base; - copy = min(bytes, iov->iov_len); - left = copyin(to, buf, copy); - copy -= left; - skip = copy; - to += copy; - bytes -= copy; - } - if (likely(!bytes)) { - kunmap_atomic(kaddr); - goto done; - } - offset = to - kaddr; - buf += copy; - kunmap_atomic(kaddr); - copy = min(bytes, iov->iov_len - skip); - } - /* Too bad - revert to non-atomic kmap */ - - kaddr = kmap(page); - to = kaddr + offset; - left = copyin(to, buf, copy); - copy -= left; - skip += copy; - to += copy; - bytes -= copy; - while (unlikely(!left && bytes)) { - iov++; - buf = iov->iov_base; - copy = min(bytes, iov->iov_len); - left = copyin(to, buf, copy); - copy -= left; - skip = copy; - to += copy; - bytes -= copy; - } - kunmap(page); - -done: - if (skip == iov->iov_len) { - iov++; - skip = 0; - } - i->count -= wanted - bytes; - i->nr_segs -= iov - i->iov; - i->iov = iov; - i->iov_offset = skip; - return wanted - bytes; -} - #ifdef PIPE_PARANOIA static bool sanity(const struct iov_iter *i) { @@ -848,24 +680,14 @@ static inline bool page_copy_sane(struct page *page, size_t offset, size_t n) static size_t __copy_page_to_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { - if (likely(iter_is_iovec(i))) - return copy_page_to_iter_iovec(page, offset, bytes, i); - if (iov_iter_is_bvec(i) || iov_iter_is_kvec(i) || iov_iter_is_xarray(i)) { + if (unlikely(iov_iter_is_pipe(i))) { + return copy_page_to_iter_pipe(page, offset, bytes, i); + } else { void *kaddr = kmap_local_page(page); size_t wanted = _copy_to_iter(kaddr + offset, bytes, i); kunmap_local(kaddr); return wanted; } - if (iov_iter_is_pipe(i)) - return copy_page_to_iter_pipe(page, offset, bytes, i); - if (unlikely(iov_iter_is_discard(i))) { - if (unlikely(i->count < bytes)) - bytes = i->count; - i->count -= bytes; - return bytes; - } - WARN_ON(1); - return 0; } size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes, @@ -896,17 +718,12 @@ EXPORT_SYMBOL(copy_page_to_iter); size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { - if (unlikely(!page_copy_sane(page, offset, bytes))) - return 0; - if (likely(iter_is_iovec(i))) - return copy_page_from_iter_iovec(page, offset, bytes, i); - if (iov_iter_is_bvec(i) || iov_iter_is_kvec(i) || iov_iter_is_xarray(i)) { + if (page_copy_sane(page, offset, bytes)) { void *kaddr = kmap_local_page(page); size_t wanted = _copy_from_iter(kaddr + offset, bytes, i); kunmap_local(kaddr); return wanted; } - WARN_ON(1); return 0; } EXPORT_SYMBOL(copy_page_from_iter); From patchwork Wed Jun 22 04:15:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB119C43334 for ; Wed, 22 Jun 2022 04:16:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356885AbiFVEQU (ORCPT ); Wed, 22 Jun 2022 00:16:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356712AbiFVEP7 (ORCPT ); Wed, 22 Jun 2022 00:15:59 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8EA6E64F8 for ; Tue, 21 Jun 2022 21:15:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=zBIXs+Q3mDypVK92felUi8sbi0rntLVF867Dq9ukkww=; b=gPH4Cs5eElXN4eL3QpMHBeUou/ JEjs1S4ucbZP2YOdcPwhcO/IeVIzh9DCkxZr9vurKHSbAVwKfbMTQtf8CJWa4j4Xlnm1QWxS1YXLI 6d373rv/K39Xh/+4ZGYzKL++oSYDB+6JEiEI8MhXUO/FdoYLxSrA/T5yHeNM+9ZBPpyO/H65RwpTI BzzBI4YT5LWd3NNOjTgDzAojLhCNbkWIIVT/HtAnW3ipkYHLS/PUJhwM+eLO39RpmzYEL9pdsdIBH iyD/QT9fxVhuOwczlWwfBPiQBRsm/CoxE/rkM29nHum822sxhj8YelFA7kTI7TRWmBaoKTHMPv7wX Zwru55tA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmU-0035w3-3k; Wed, 22 Jun 2022 04:15:54 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 09/44] new iov_iter flavour - ITER_UBUF Date: Wed, 22 Jun 2022 05:15:17 +0100 Message-Id: <20220622041552.737754-9-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Equivalent of single-segment iovec. Initialized by iov_iter_ubuf(), checked for by iter_is_ubuf(), otherwise behaves like ITER_IOVEC ones. We are going to expose the things like ->write_iter() et.al. to those in subsequent commits. New predicate (user_backed_iter()) that is true for ITER_IOVEC and ITER_UBUF; places like direct-IO handling should use that for checking that pages we modify after getting them from iov_iter_get_pages() would need to be dirtied. DO NOT assume that replacing iter_is_iovec() with user_backed_iter() will solve all problems - there's code that uses iter_is_iovec() to decide how to poke around in iov_iter guts and for that the predicate replacement obviously won't suffice. Signed-off-by: Al Viro --- block/fops.c | 6 +-- fs/ceph/file.c | 2 +- fs/cifs/file.c | 2 +- fs/direct-io.c | 2 +- fs/fuse/dev.c | 4 +- fs/fuse/file.c | 2 +- fs/gfs2/file.c | 2 +- fs/iomap/direct-io.c | 2 +- fs/nfs/direct.c | 2 +- include/linux/uio.h | 26 ++++++++++++ lib/iov_iter.c | 94 ++++++++++++++++++++++++++++++++++---------- mm/shmem.c | 2 +- 12 files changed, 113 insertions(+), 33 deletions(-) diff --git a/block/fops.c b/block/fops.c index 6e86931ab847..3e68d69e0ee3 100644 --- a/block/fops.c +++ b/block/fops.c @@ -69,7 +69,7 @@ static ssize_t __blkdev_direct_IO_simple(struct kiocb *iocb, if (iov_iter_rw(iter) == READ) { bio_init(&bio, bdev, vecs, nr_pages, REQ_OP_READ); - if (iter_is_iovec(iter)) + if (user_backed_iter(iter)) should_dirty = true; } else { bio_init(&bio, bdev, vecs, nr_pages, dio_bio_write_op(iocb)); @@ -199,7 +199,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, } dio->size = 0; - if (is_read && iter_is_iovec(iter)) + if (is_read && user_backed_iter(iter)) dio->flags |= DIO_SHOULD_DIRTY; blk_start_plug(&plug); @@ -331,7 +331,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb, dio->size = bio->bi_iter.bi_size; if (is_read) { - if (iter_is_iovec(iter)) { + if (user_backed_iter(iter)) { dio->flags |= DIO_SHOULD_DIRTY; bio_set_pages_dirty(bio); } diff --git a/fs/ceph/file.c b/fs/ceph/file.c index 8c8226c0feac..e132adeeaf16 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -1262,7 +1262,7 @@ ceph_direct_read_write(struct kiocb *iocb, struct iov_iter *iter, size_t count = iov_iter_count(iter); loff_t pos = iocb->ki_pos; bool write = iov_iter_rw(iter) == WRITE; - bool should_dirty = !write && iter_is_iovec(iter); + bool should_dirty = !write && user_backed_iter(iter); if (write && ceph_snap(file_inode(file)) != CEPH_NOSNAP) return -EROFS; diff --git a/fs/cifs/file.c b/fs/cifs/file.c index 1618e0537d58..4b4129d9a90c 100644 --- a/fs/cifs/file.c +++ b/fs/cifs/file.c @@ -4004,7 +4004,7 @@ static ssize_t __cifs_readv( if (!is_sync_kiocb(iocb)) ctx->iocb = iocb; - if (iter_is_iovec(to)) + if (user_backed_iter(to)) ctx->should_dirty = true; if (direct) { diff --git a/fs/direct-io.c b/fs/direct-io.c index 39647eb56904..72237f49ad94 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -1245,7 +1245,7 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode, spin_lock_init(&dio->bio_lock); dio->refcount = 1; - dio->should_dirty = iter_is_iovec(iter) && iov_iter_rw(iter) == READ; + dio->should_dirty = user_backed_iter(iter) && iov_iter_rw(iter) == READ; sdio.iter = iter; sdio.final_block_in_request = end >> blkbits; diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 0e537e580dc1..8d657c2cd6f7 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -1356,7 +1356,7 @@ static ssize_t fuse_dev_read(struct kiocb *iocb, struct iov_iter *to) if (!fud) return -EPERM; - if (!iter_is_iovec(to)) + if (!user_backed_iter(to)) return -EINVAL; fuse_copy_init(&cs, 1, to); @@ -1949,7 +1949,7 @@ static ssize_t fuse_dev_write(struct kiocb *iocb, struct iov_iter *from) if (!fud) return -EPERM; - if (!iter_is_iovec(from)) + if (!user_backed_iter(from)) return -EINVAL; fuse_copy_init(&cs, 0, from); diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 00fa861aeead..c982e3afe3b4 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1465,7 +1465,7 @@ ssize_t fuse_direct_io(struct fuse_io_priv *io, struct iov_iter *iter, inode_unlock(inode); } - io->should_dirty = !write && iter_is_iovec(iter); + io->should_dirty = !write && user_backed_iter(iter); while (count) { ssize_t nres; fl_owner_t owner = current->files; diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c index 2cceb193dcd8..48e6cc74fdc1 100644 --- a/fs/gfs2/file.c +++ b/fs/gfs2/file.c @@ -780,7 +780,7 @@ static inline bool should_fault_in_pages(struct iov_iter *i, if (!count) return false; - if (!iter_is_iovec(i)) + if (!user_backed_iter(i)) return false; size = PAGE_SIZE; diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 31c7f1035b20..d5c7d019653b 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -533,7 +533,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *iter, iomi.flags |= IOMAP_NOWAIT; } - if (iter_is_iovec(iter)) + if (user_backed_iter(iter)) dio->flags |= IOMAP_DIO_DIRTY; } else { iomi.flags |= IOMAP_WRITE; diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c index 4eb2a8380a28..022e1ce63e62 100644 --- a/fs/nfs/direct.c +++ b/fs/nfs/direct.c @@ -478,7 +478,7 @@ ssize_t nfs_file_direct_read(struct kiocb *iocb, struct iov_iter *iter, if (!is_sync_kiocb(iocb)) dreq->iocb = iocb; - if (iter_is_iovec(iter)) + if (user_backed_iter(iter)) dreq->flags = NFS_ODIRECT_SHOULD_DIRTY; if (!swap) diff --git a/include/linux/uio.h b/include/linux/uio.h index 76d305f3d4c2..6ab4260c3d6c 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -26,6 +26,7 @@ enum iter_type { ITER_PIPE, ITER_XARRAY, ITER_DISCARD, + ITER_UBUF, }; struct iov_iter_state { @@ -38,6 +39,7 @@ struct iov_iter { u8 iter_type; bool nofault; bool data_source; + bool user_backed; size_t iov_offset; size_t count; union { @@ -46,6 +48,7 @@ struct iov_iter { const struct bio_vec *bvec; struct xarray *xarray; struct pipe_inode_info *pipe; + void __user *ubuf; }; union { unsigned long nr_segs; @@ -70,6 +73,11 @@ static inline void iov_iter_save_state(struct iov_iter *iter, state->nr_segs = iter->nr_segs; } +static inline bool iter_is_ubuf(const struct iov_iter *i) +{ + return iov_iter_type(i) == ITER_UBUF; +} + static inline bool iter_is_iovec(const struct iov_iter *i) { return iov_iter_type(i) == ITER_IOVEC; @@ -105,6 +113,11 @@ static inline unsigned char iov_iter_rw(const struct iov_iter *i) return i->data_source ? WRITE : READ; } +static inline bool user_backed_iter(const struct iov_iter *i) +{ + return i->user_backed; +} + /* * Total number of bytes covered by an iovec. * @@ -320,4 +333,17 @@ ssize_t __import_iovec(int type, const struct iovec __user *uvec, int import_single_range(int type, void __user *buf, size_t len, struct iovec *iov, struct iov_iter *i); +static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, + void __user *buf, size_t count) +{ + WARN_ON(direction & ~(READ | WRITE)); + *i = (struct iov_iter) { + .iter_type = ITER_UBUF, + .user_backed = true, + .data_source = direction, + .ubuf = buf, + .count = count + }; +} + #endif diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 4c658a25e29c..8275b28e886b 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -16,6 +16,16 @@ #define PIPE_PARANOIA /* for now */ +/* covers ubuf and kbuf alike */ +#define iterate_buf(i, n, base, len, off, __p, STEP) { \ + size_t __maybe_unused off = 0; \ + len = n; \ + base = __p + i->iov_offset; \ + len -= (STEP); \ + i->iov_offset += len; \ + n = len; \ +} + /* covers iovec and kvec alike */ #define iterate_iovec(i, n, base, len, off, __p, STEP) { \ size_t off = 0; \ @@ -110,7 +120,12 @@ __out: \ if (unlikely(i->count < n)) \ n = i->count; \ if (likely(n)) { \ - if (likely(iter_is_iovec(i))) { \ + if (likely(iter_is_ubuf(i))) { \ + void __user *base; \ + size_t len; \ + iterate_buf(i, n, base, len, off, \ + i->ubuf, (I)) \ + } else if (likely(iter_is_iovec(i))) { \ const struct iovec *iov = i->iov; \ void __user *base; \ size_t len; \ @@ -275,7 +290,11 @@ static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t by */ size_t fault_in_iov_iter_readable(const struct iov_iter *i, size_t size) { - if (iter_is_iovec(i)) { + if (iter_is_ubuf(i)) { + size_t n = min(size, iov_iter_count(i)); + n -= fault_in_readable(i->ubuf + i->iov_offset, n); + return size - n; + } else if (iter_is_iovec(i)) { size_t count = min(size, iov_iter_count(i)); const struct iovec *p; size_t skip; @@ -314,7 +333,11 @@ EXPORT_SYMBOL(fault_in_iov_iter_readable); */ size_t fault_in_iov_iter_writeable(const struct iov_iter *i, size_t size) { - if (iter_is_iovec(i)) { + if (iter_is_ubuf(i)) { + size_t n = min(size, iov_iter_count(i)); + n -= fault_in_safe_writeable(i->ubuf + i->iov_offset, n); + return size - n; + } else if (iter_is_iovec(i)) { size_t count = min(size, iov_iter_count(i)); const struct iovec *p; size_t skip; @@ -345,6 +368,7 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, *i = (struct iov_iter) { .iter_type = ITER_IOVEC, .nofault = false, + .user_backed = true, .data_source = direction, .iov = iov, .nr_segs = nr_segs, @@ -494,7 +518,7 @@ size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { if (unlikely(iov_iter_is_pipe(i))) return copy_pipe_to_iter(addr, bytes, i); - if (iter_is_iovec(i)) + if (user_backed_iter(i)) might_fault(); iterate_and_advance(i, bytes, base, len, off, copyout(base, addr + off, len), @@ -576,7 +600,7 @@ size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { if (unlikely(iov_iter_is_pipe(i))) return copy_mc_pipe_to_iter(addr, bytes, i); - if (iter_is_iovec(i)) + if (user_backed_iter(i)) might_fault(); __iterate_and_advance(i, bytes, base, len, off, copyout_mc(base, addr + off, len), @@ -594,7 +618,7 @@ size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) WARN_ON(1); return 0; } - if (iter_is_iovec(i)) + if (user_backed_iter(i)) might_fault(); iterate_and_advance(i, bytes, base, len, off, copyin(addr + off, base, len), @@ -882,16 +906,16 @@ void iov_iter_advance(struct iov_iter *i, size_t size) { if (unlikely(i->count < size)) size = i->count; - if (likely(iter_is_iovec(i) || iov_iter_is_kvec(i))) { + if (likely(iter_is_ubuf(i)) || unlikely(iov_iter_is_xarray(i))) { + i->iov_offset += size; + i->count -= size; + } else if (likely(iter_is_iovec(i) || iov_iter_is_kvec(i))) { /* iovec and kvec have identical layouts */ iov_iter_iovec_advance(i, size); } else if (iov_iter_is_bvec(i)) { iov_iter_bvec_advance(i, size); } else if (iov_iter_is_pipe(i)) { pipe_advance(i, size); - } else if (unlikely(iov_iter_is_xarray(i))) { - i->iov_offset += size; - i->count -= size; } else if (iov_iter_is_discard(i)) { i->count -= size; } @@ -938,7 +962,7 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll) return; } unroll -= i->iov_offset; - if (iov_iter_is_xarray(i)) { + if (iov_iter_is_xarray(i) || iter_is_ubuf(i)) { BUG(); /* We should never go beyond the start of the specified * range since we might then be straying into pages that * aren't pinned. @@ -1129,6 +1153,13 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i) unsigned long iov_iter_alignment(const struct iov_iter *i) { + if (likely(iter_is_ubuf(i))) { + size_t size = i->count; + if (size) + return ((unsigned long)i->ubuf + i->iov_offset) | size; + return 0; + } + /* iovec and kvec have identical layouts */ if (likely(iter_is_iovec(i) || iov_iter_is_kvec(i))) return iov_iter_alignment_iovec(i); @@ -1159,6 +1190,9 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i) size_t size = i->count; unsigned k; + if (iter_is_ubuf(i)) + return 0; + if (WARN_ON(!iter_is_iovec(i))) return ~0U; @@ -1287,7 +1321,19 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, return actual; } -/* must be done on non-empty ITER_IOVEC one */ +static unsigned long found_ubuf_segment(unsigned long addr, + size_t len, + size_t *size, size_t *start, + unsigned maxpages) +{ + len += (*start = addr % PAGE_SIZE); + if (len > maxpages * PAGE_SIZE) + len = maxpages * PAGE_SIZE; + *size = len; + return addr & PAGE_MASK; +} + +/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ static unsigned long first_iovec_segment(const struct iov_iter *i, size_t *size, size_t *start, size_t maxsize, unsigned maxpages) @@ -1295,6 +1341,11 @@ static unsigned long first_iovec_segment(const struct iov_iter *i, size_t skip; long k; + if (iter_is_ubuf(i)) { + unsigned long addr = (unsigned long)i->ubuf + i->iov_offset; + return found_ubuf_segment(addr, maxsize, size, start, maxpages); + } + for (k = 0, skip = i->iov_offset; k < i->nr_segs; k++, skip = 0) { unsigned long addr = (unsigned long)i->iov[k].iov_base + skip; size_t len = i->iov[k].iov_len - skip; @@ -1303,11 +1354,7 @@ static unsigned long first_iovec_segment(const struct iov_iter *i, continue; if (len > maxsize) len = maxsize; - len += (*start = addr % PAGE_SIZE); - if (len > maxpages * PAGE_SIZE) - len = maxpages * PAGE_SIZE; - *size = len; - return addr & PAGE_MASK; + return found_ubuf_segment(addr, len, size, start, maxpages); } BUG(); // if it had been empty, we wouldn't get called } @@ -1344,7 +1391,7 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, if (!maxsize) return 0; - if (likely(iter_is_iovec(i))) { + if (likely(user_backed_iter(i))) { unsigned int gup_flags = 0; unsigned long addr; @@ -1470,7 +1517,7 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, if (!maxsize) return 0; - if (likely(iter_is_iovec(i))) { + if (likely(user_backed_iter(i))) { unsigned int gup_flags = 0; unsigned long addr; @@ -1624,6 +1671,11 @@ int iov_iter_npages(const struct iov_iter *i, int maxpages) { if (unlikely(!i->count)) return 0; + if (likely(iter_is_ubuf(i))) { + unsigned offs = offset_in_page(i->ubuf + i->iov_offset); + int npages = DIV_ROUND_UP(offs + i->count, PAGE_SIZE); + return min(npages, maxpages); + } /* iovec and kvec have identical layouts */ if (likely(iter_is_iovec(i) || iov_iter_is_kvec(i))) return iov_npages(i, maxpages); @@ -1862,10 +1914,12 @@ EXPORT_SYMBOL(import_single_range); void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state) { if (WARN_ON_ONCE(!iov_iter_is_bvec(i) && !iter_is_iovec(i)) && - !iov_iter_is_kvec(i)) + !iov_iter_is_kvec(i) && !iter_is_ubuf(i)) return; i->iov_offset = state->iov_offset; i->count = state->count; + if (iter_is_ubuf(i)) + return; /* * For the *vec iters, nr_segs + iov is constant - if we increment * the vec, then we also decrement the nr_segs count. Hence we don't diff --git a/mm/shmem.c b/mm/shmem.c index a6f565308133..6b83f3971795 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2603,7 +2603,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) ret = copy_page_to_iter(page, offset, nr, to); put_page(page); - } else if (iter_is_iovec(to)) { + } else if (!user_backed_iter(to)) { /* * Copy to user tends to be so well optimized, but * clear_user() not so much, that it is noticeably From patchwork Wed Jun 22 04:15:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C19EC43334 for ; Wed, 22 Jun 2022 04:16:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356871AbiFVEQM (ORCPT ); Wed, 22 Jun 2022 00:16:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356702AbiFVEP6 (ORCPT ); Wed, 22 Jun 2022 00:15:58 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4BFF655A for ; Tue, 21 Jun 2022 21:15:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=zbtmeSvWooLXXXnLzIrXTw/98z6dwBgai7ozeYmUaHM=; b=P1RCrRctei/y83qI/h/jMT9M6W lal5nQof9ReDomkNoeWa1DDZQ9ZEYyF/XHoSPPcRQDNBmW7ISPi6pzoyagnvtZvaJDaR50gCYFJHP grvGi6AVrHJRUokmp2Tka4oN4I/auSI5r4840JeNS40tXubcAf77hJwNUtIqvfnsAHC5EiFyW9YRD YqsiXDuZHgt/rgnjnhxudYwodWQMsGUhr0Hl2ZPh01c3vc9pB6rQsOyKRaxQLi8ToFzNS811cIacr 82T1arIXUx3vh03jvQ0RQwM33PDagFe6f99502bfMJj2snH0mL34O0JIYYZneOZI6GeOW7iC5b6gG w/uGNaNw==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmU-0035w8-7F; Wed, 22 Jun 2022 04:15:54 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 10/44] switch new_sync_{read,write}() to ITER_UBUF Date: Wed, 22 Jun 2022 05:15:18 +0100 Message-Id: <20220622041552.737754-10-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Reviewed-by: Christian Brauner (Microsoft) Signed-off-by: Al Viro --- fs/read_write.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/fs/read_write.c b/fs/read_write.c index b1b1cdfee9d3..e82e4301cadd 100644 --- a/fs/read_write.c +++ b/fs/read_write.c @@ -389,14 +389,13 @@ EXPORT_SYMBOL(rw_verify_area); static ssize_t new_sync_read(struct file *filp, char __user *buf, size_t len, loff_t *ppos) { - struct iovec iov = { .iov_base = buf, .iov_len = len }; struct kiocb kiocb; struct iov_iter iter; ssize_t ret; init_sync_kiocb(&kiocb, filp); kiocb.ki_pos = (ppos ? *ppos : 0); - iov_iter_init(&iter, READ, &iov, 1, len); + iov_iter_ubuf(&iter, READ, buf, len); ret = call_read_iter(filp, &kiocb, &iter); BUG_ON(ret == -EIOCBQUEUED); @@ -492,14 +491,13 @@ ssize_t vfs_read(struct file *file, char __user *buf, size_t count, loff_t *pos) static ssize_t new_sync_write(struct file *filp, const char __user *buf, size_t len, loff_t *ppos) { - struct iovec iov = { .iov_base = (void __user *)buf, .iov_len = len }; struct kiocb kiocb; struct iov_iter iter; ssize_t ret; init_sync_kiocb(&kiocb, filp); kiocb.ki_pos = (ppos ? *ppos : 0); - iov_iter_init(&iter, WRITE, &iov, 1, len); + iov_iter_ubuf(&iter, WRITE, (void __user *)buf, len); ret = call_write_iter(filp, &kiocb, &iter); BUG_ON(ret == -EIOCBQUEUED); From patchwork Wed Jun 22 04:15:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD177C43334 for ; Wed, 22 Jun 2022 04:16:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356874AbiFVEQP (ORCPT ); Wed, 22 Jun 2022 00:16:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356706AbiFVEP6 (ORCPT ); Wed, 22 Jun 2022 00:15:58 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5D39655C for ; Tue, 21 Jun 2022 21:15:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=6wfIImN1V5bWomaOl/DuvOp7dK2fJI8QCN7xBf4qf0c=; b=aoAjjmIke+y20d4KpZUvJBCgkS VmeKdwy6mXfPOqw2IMJHub178YEsGIMTs7vM/ql4qBNcvbgyy3XbpOeXLZohQaEK6S7IXPGpf4FG6 XTl+M2PkLAEK+DNGplKwTvq/Wg83hQVE261SaE+QvpXEwF3GZW4a5TWpgFYcmeEY6RIyW1jZlwBAI zL3evzMWrx9BOJYoeAyqPfym+S3J5F4CUvDS3EE+zodeOuBJjPMJmrJRyAdVQTgDjMfVS6uey3xnh SuOWsr1gGyVSvn/DMRgn1IghwdkMvPkUeKPCc70zUjGIimJUvJvtjH40hk/Rsn7j5NBcST7DizUGu xiNo2I4g==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmU-0035wC-BL; Wed, 22 Jun 2022 04:15:54 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 11/44] iov_iter_bvec_advance(): don't bother with bvec_iter Date: Wed, 22 Jun 2022 05:15:19 +0100 Message-Id: <20220622041552.737754-11-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org do what we do for iovec/kvec; that ends up generating better code, AFAICS. Signed-off-by: Al Viro Reviewed-by: Jeff Layton Reviewed-by: Christian Brauner (Microsoft) --- lib/iov_iter.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 8275b28e886b..93ceb13ec7b5 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -870,17 +870,22 @@ static void pipe_advance(struct iov_iter *i, size_t size) static void iov_iter_bvec_advance(struct iov_iter *i, size_t size) { - struct bvec_iter bi; + const struct bio_vec *bvec, *end; - bi.bi_size = i->count; - bi.bi_bvec_done = i->iov_offset; - bi.bi_idx = 0; - bvec_iter_advance(i->bvec, &bi, size); + if (!i->count) + return; + i->count -= size; + + size += i->iov_offset; - i->bvec += bi.bi_idx; - i->nr_segs -= bi.bi_idx; - i->count = bi.bi_size; - i->iov_offset = bi.bi_bvec_done; + for (bvec = i->bvec, end = bvec + i->nr_segs; bvec < end; bvec++) { + if (likely(size < bvec->bv_len)) + break; + size -= bvec->bv_len; + } + i->iov_offset = size; + i->nr_segs -= bvec - i->bvec; + i->bvec = bvec; } static void iov_iter_iovec_advance(struct iov_iter *i, size_t size) From patchwork Wed Jun 22 04:15:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E42CDCCA47A for ; Wed, 22 Jun 2022 04:16:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356876AbiFVEQP (ORCPT ); Wed, 22 Jun 2022 00:16:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356707AbiFVEP6 (ORCPT ); Wed, 22 Jun 2022 00:15:58 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 155FA6565 for ; Tue, 21 Jun 2022 21:15:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=2hUtpY7tF+1AULyN9gkMQD7g6cYjYFLHq4WPTwrgyC4=; b=YpT7wBNKlz7lWgcSGpBkV109xp /pJHlZB8tE5elQ0BNhXItGDTtmgrMqxbqYJmA8+0Uyj3vKtlKkwEavCdJWLKdZBoEmwCKNvcDhsGz JIVeWFNhd4GZZehK1czaDrag9Oerv9lbz1/HW7xk4LnB3gfFdPtUZ/0iLn2szNgrjK3o9b7vfcphV Ge3+ifHKfceUuc1UyT+IGyq2QuBcYDy5lXjS+ZXWChHw3l0G0YtDjMlH+dGH+GQFGj+iptGpnQySk gimplZnRrDcl13YOpgNzs5dcQOrl753CAct3Xnkj+WHCl8KAhsD256iYXQfhwnLf7W1yMuT4f1MBC wYhye1iw==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmU-0035wK-Hi; Wed, 22 Jun 2022 04:15:54 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 12/44] fix short copy handling in copy_mc_pipe_to_iter() Date: Wed, 22 Jun 2022 05:15:20 +0100 Message-Id: <20220622041552.737754-12-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Unlike other copying operations on ITER_PIPE, copy_mc_to_iter() can result in a short copy. In that case we need to trim the unused buffers, as well as the length of partially filled one - it's not enough to set ->head, ->iov_offset and ->count to reflect how much had we copied. Not hard to fix, fortunately... I'd put a helper (pipe_discard_from(pipe, head)) into pipe_fs_i.h, rather than iov_iter.c - it has nothing to do with iov_iter and having it will allow us to avoid an ugly kludge in fs/splice.c. We could put it into lib/iov_iter.c for now and move it later, but I don't see the point going that way... Fixes: ca146f6f091e "lib/iov_iter: Fix pipe handling in _copy_to_iter_mcsafe()" Signed-off-by: Al Viro Reviewed-by: Jeff Layton Reviewed-by: Christian Brauner (Microsoft) --- include/linux/pipe_fs_i.h | 9 +++++++++ lib/iov_iter.c | 15 +++++++++++---- 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/include/linux/pipe_fs_i.h b/include/linux/pipe_fs_i.h index cb0fd633a610..4ea496924106 100644 --- a/include/linux/pipe_fs_i.h +++ b/include/linux/pipe_fs_i.h @@ -229,6 +229,15 @@ static inline bool pipe_buf_try_steal(struct pipe_inode_info *pipe, return buf->ops->try_steal(pipe, buf); } +static inline void pipe_discard_from(struct pipe_inode_info *pipe, + unsigned int old_head) +{ + unsigned int mask = pipe->ring_size - 1; + + while (pipe->head > old_head) + pipe_buf_release(pipe, &pipe->bufs[--pipe->head & mask]); +} + /* Differs from PIPE_BUF in that PIPE_SIZE is the length of the actual memory allocation, whereas PIPE_BUF makes atomicity guarantees. */ #define PIPE_SIZE PAGE_SIZE diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 0b64695ab632..2bf20b48a04a 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -689,6 +689,7 @@ static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes, struct pipe_inode_info *pipe = i->pipe; unsigned int p_mask = pipe->ring_size - 1; unsigned int i_head; + unsigned int valid = pipe->head; size_t n, off, xfer = 0; if (!sanity(i)) @@ -702,11 +703,17 @@ static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes, rem = copy_mc_to_kernel(p + off, addr + xfer, chunk); chunk -= rem; kunmap_local(p); - i->head = i_head; - i->iov_offset = off + chunk; - xfer += chunk; - if (rem) + if (chunk) { + i->head = i_head; + i->iov_offset = off + chunk; + xfer += chunk; + valid = i_head + 1; + } + if (rem) { + pipe->bufs[i_head & p_mask].len -= rem; + pipe_discard_from(pipe, valid); break; + } n -= chunk; off = 0; i_head++; From patchwork Wed Jun 22 04:15:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CE98C43334 for ; Wed, 22 Jun 2022 04:16:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356879AbiFVEQS (ORCPT ); Wed, 22 Jun 2022 00:16:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356709AbiFVEP6 (ORCPT ); Wed, 22 Jun 2022 00:15:58 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F65A656E for ; Tue, 21 Jun 2022 21:15:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=NPOwPO2c4uajDyblyCE6/dHQ5kD7t1bOCOYXJLLpL1w=; b=rdcJI37/hMYtodBGotEKwpVlhz +dv2RppEYa0uRaPVBo/9K26GSLBHpxyBgGjny18/PSUT92xLQs90Dwie02L8rv73iwq4iFFB88xZb GnuAZQ8nMoVdtrw0zNYjjX8c01/02Zwz8wJEhLFfIoABKXuwLis4Uxai9EmA+WxKrer+iFz5Owzlq qTIxm9hqhPKdWZlf84yeeD9MF+rB6hmBB5RSuSYTuftr8q8mJG+mPXmeXaTUEz59VEauYGx7MH1S3 4fVsRLUp7D1Ii4CxdudgQr6a2PXdNrJ2wJ9tMZnmtGVFeb3xBZl77UOWsYI4p6ikfrZ9BWSvb3J6K ItqQzATQ==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmU-0035wU-Lp; Wed, 22 Jun 2022 04:15:54 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 13/44] splice: stop abusing iov_iter_advance() to flush a pipe Date: Wed, 22 Jun 2022 05:15:21 +0100 Message-Id: <20220622041552.737754-13-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Use pipe_discard_from() explicitly in generic_file_read_iter(); don't bother with rather non-obvious use of iov_iter_advance() in there. Signed-off-by: Al Viro Reviewed-by: Jeff Layton Reviewed-by: Christian Brauner (Microsoft) --- fs/splice.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 047b79db8eb5..6645b30ec990 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -301,11 +301,9 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, { struct iov_iter to; struct kiocb kiocb; - unsigned int i_head; int ret; iov_iter_pipe(&to, READ, pipe, len); - i_head = to.head; init_sync_kiocb(&kiocb, in); kiocb.ki_pos = *ppos; ret = call_read_iter(in, &kiocb, &to); @@ -313,9 +311,8 @@ ssize_t generic_file_splice_read(struct file *in, loff_t *ppos, *ppos = kiocb.ki_pos; file_accessed(in); } else if (ret < 0) { - to.head = i_head; - to.iov_offset = 0; - iov_iter_advance(&to, 0); /* to free what was emitted */ + /* free what was emitted */ + pipe_discard_from(pipe, to.start_head); /* * callers of ->splice_read() expect -EAGAIN on * "can't put anything in there", rather than -EFAULT. From patchwork Wed Jun 22 04:15:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C8F8C43334 for ; Wed, 22 Jun 2022 04:16:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356878AbiFVEQR (ORCPT ); Wed, 22 Jun 2022 00:16:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356708AbiFVEP6 (ORCPT ); Wed, 22 Jun 2022 00:15:58 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 781776582 for ; Tue, 21 Jun 2022 21:15:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=oWGScIUb1EoaRdhEi06bRMjVnArb0zdP4434VU2a+/c=; b=U6obfarIWWlpDiesiNPkHEn5D5 mcVHnqBeIGccu5voNOeE3FugM7g+E7yBgpkNCpsuDDJPjOhKjyUg7j0sUgjNTSJf3pXDlZtBAIkeW jLjQ76BtdpJGQVA0lZrmfbvZvWfcFG6A164oKOJzAv5sfh+U5qhSx54YgeKRr9XbNg7LIlmrt/qFw IFa+pRxUWr+9IXOe1Udy5Co8UIAyBAoerZYi06nby+8QqCXlUIOqkfItSNTWoMyO9PfQcK0cJ9OtO BPPgBXQiXB1xSh4QbCCGPfEQIstx8H4EezZh8jsxGC16IgaTb7oZE3cIep7By/aQ37m0H0SoAmTMU 278ePCrQ==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmU-0035wa-RD; Wed, 22 Jun 2022 04:15:54 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 14/44] ITER_PIPE: helper for getting pipe buffer by index Date: Wed, 22 Jun 2022 05:15:22 +0100 Message-Id: <20220622041552.737754-14-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org pipe_buffer instances of a pipe are organized as a ring buffer, with power-of-2 size. Indices are kept *not* reduced modulo ring size, so the buffer refered to by index N is pipe->bufs[N & (pipe->ring_size - 1)]. Ring size can change over the lifetime of a pipe, but not while the pipe is locked. So for any iov_iter primitives it's a constant. Original conversion of pipes to this layout went overboard trying to microoptimize that - calculating pipe->ring_size - 1, storing it in a local variable and using through the function. In some cases it might be warranted, but most of the times it only obfuscates what's going on in there. Introduce a helper (pipe_buf(pipe, N)) that would encapsulate that and use it in the obvious cases. More will follow... Signed-off-by: Al Viro Reviewed-by: Jeff Layton Reviewed-by: Christian Brauner (Microsoft) --- lib/iov_iter.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index d00cc8971b5b..08bb393da677 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -183,13 +183,18 @@ static int copyin(void *to, const void __user *from, size_t n) return n; } +static inline struct pipe_buffer *pipe_buf(const struct pipe_inode_info *pipe, + unsigned int slot) +{ + return &pipe->bufs[slot & (pipe->ring_size - 1)]; +} + #ifdef PIPE_PARANOIA static bool sanity(const struct iov_iter *i) { struct pipe_inode_info *pipe = i->pipe; unsigned int p_head = pipe->head; unsigned int p_tail = pipe->tail; - unsigned int p_mask = pipe->ring_size - 1; unsigned int p_occupancy = pipe_occupancy(p_head, p_tail); unsigned int i_head = i->head; unsigned int idx; @@ -201,7 +206,7 @@ static bool sanity(const struct iov_iter *i) if (unlikely(i_head != p_head - 1)) goto Bad; // must be at the last buffer... - p = &pipe->bufs[i_head & p_mask]; + p = pipe_buf(pipe, i_head); if (unlikely(p->offset + p->len != i->iov_offset)) goto Bad; // ... at the end of segment } else { @@ -386,11 +391,10 @@ static inline bool allocated(struct pipe_buffer *buf) static inline void data_start(const struct iov_iter *i, unsigned int *iter_headp, size_t *offp) { - unsigned int p_mask = i->pipe->ring_size - 1; unsigned int iter_head = i->head; size_t off = i->iov_offset; - if (off && (!allocated(&i->pipe->bufs[iter_head & p_mask]) || + if (off && (!allocated(pipe_buf(i->pipe, iter_head)) || off == PAGE_SIZE)) { iter_head++; off = 0; @@ -1180,10 +1184,9 @@ unsigned long iov_iter_alignment(const struct iov_iter *i) return iov_iter_alignment_bvec(i); if (iov_iter_is_pipe(i)) { - unsigned int p_mask = i->pipe->ring_size - 1; size_t size = i->count; - if (size && i->iov_offset && allocated(&i->pipe->bufs[i->head & p_mask])) + if (size && i->iov_offset && allocated(pipe_buf(i->pipe, i->head))) return size | i->iov_offset; return size; } From patchwork Wed Jun 22 04:15:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 543E3C433EF for ; Wed, 22 Jun 2022 04:16:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356882AbiFVEQU (ORCPT ); Wed, 22 Jun 2022 00:16:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356713AbiFVEP7 (ORCPT ); Wed, 22 Jun 2022 00:15:59 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93D5F65B7 for ; Tue, 21 Jun 2022 21:15:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=mLj5PkxHc90os+7X5POLttjxS5yvU/rKhJBnE9f5AP0=; b=ERCJGIzfW42WdNHRCev5awl+WF SnFavVTAjY5nKaoQvLknbbxzghyeUWiiWrFyXi+6CR9Vbh+B2OnSFm6Tcw0vzXGB48gr7gijqCb+0 K9J2Lqssye658lW4RCCkCP7JjjOA1w2uZ+mNHxlsTRpRWgJ1RzJgBk8vMXJBAllWFZK6DW1sU7tyd okvJf+ztQGOg4zcipX8YLGv2EYv2wojheJynSzaloyHM7lcUFpsORtMVTfpOWqnYf7c0SkErYUItp xiJtQwr7eWx2WwNxA9Sm4aoEEW8cX1tmpKSqkvTlycWEUUd2MmhcrJbQtrRStT+UuDeBj1e8cb+BD ZzWO/ifQ==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmU-0035wg-Vr; Wed, 22 Jun 2022 04:15:55 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 15/44] ITER_PIPE: helpers for adding pipe buffers Date: Wed, 22 Jun 2022 05:15:23 +0100 Message-Id: <20220622041552.737754-15-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org There are only two kinds of pipe_buffer in the area used by ITER_PIPE. 1) anonymous - copy_to_iter() et.al. end up creating those and copying data there. They have zero ->offset, and their ->ops points to default_pipe_page_ops. 2) zero-copy ones - those come from copy_page_to_iter(), and page comes from caller. ->offset is also caller-supplied - it might be non-zero. ->ops points to page_cache_pipe_buf_ops. Move creation and insertion of those into helpers - push_anon(pipe, size) and push_page(pipe, page, offset, size) resp., separating them from the "could we avoid creating a new buffer by merging with the current head?" logics. Signed-off-by: Al Viro Acked-by: Jeff Layton --- lib/iov_iter.c | 88 ++++++++++++++++++++++++++------------------------ 1 file changed, 46 insertions(+), 42 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 08bb393da677..924854c2a7ce 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -231,15 +231,39 @@ static bool sanity(const struct iov_iter *i) #define sanity(i) true #endif +static struct page *push_anon(struct pipe_inode_info *pipe, unsigned size) +{ + struct page *page = alloc_page(GFP_USER); + if (page) { + struct pipe_buffer *buf = pipe_buf(pipe, pipe->head++); + *buf = (struct pipe_buffer) { + .ops = &default_pipe_buf_ops, + .page = page, + .offset = 0, + .len = size + }; + } + return page; +} + +static void push_page(struct pipe_inode_info *pipe, struct page *page, + unsigned int offset, unsigned int size) +{ + struct pipe_buffer *buf = pipe_buf(pipe, pipe->head++); + *buf = (struct pipe_buffer) { + .ops = &page_cache_pipe_buf_ops, + .page = page, + .offset = offset, + .len = size + }; + get_page(page); +} + static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { struct pipe_inode_info *pipe = i->pipe; - struct pipe_buffer *buf; - unsigned int p_tail = pipe->tail; - unsigned int p_mask = pipe->ring_size - 1; - unsigned int i_head = i->head; - size_t off; + unsigned int head = pipe->head; if (unlikely(bytes > i->count)) bytes = i->count; @@ -250,32 +274,21 @@ static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t by if (!sanity(i)) return 0; - off = i->iov_offset; - buf = &pipe->bufs[i_head & p_mask]; - if (off) { - if (offset == off && buf->page == page) { - /* merge with the last one */ + if (offset && i->iov_offset == offset) { // could we merge it? + struct pipe_buffer *buf = pipe_buf(pipe, head - 1); + if (buf->page == page) { buf->len += bytes; i->iov_offset += bytes; - goto out; + i->count -= bytes; + return bytes; } - i_head++; - buf = &pipe->bufs[i_head & p_mask]; } - if (pipe_full(i_head, p_tail, pipe->max_usage)) + if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) return 0; - buf->ops = &page_cache_pipe_buf_ops; - buf->flags = 0; - get_page(page); - buf->page = page; - buf->offset = offset; - buf->len = bytes; - - pipe->head = i_head + 1; + push_page(pipe, page, offset, bytes); i->iov_offset = offset + bytes; - i->head = i_head; -out: + i->head = head; i->count -= bytes; return bytes; } @@ -407,8 +420,6 @@ static size_t push_pipe(struct iov_iter *i, size_t size, int *iter_headp, size_t *offp) { struct pipe_inode_info *pipe = i->pipe; - unsigned int p_tail = pipe->tail; - unsigned int p_mask = pipe->ring_size - 1; unsigned int iter_head; size_t off; ssize_t left; @@ -423,30 +434,23 @@ static size_t push_pipe(struct iov_iter *i, size_t size, *iter_headp = iter_head; *offp = off; if (off) { + struct pipe_buffer *buf = pipe_buf(pipe, iter_head); + left -= PAGE_SIZE - off; if (left <= 0) { - pipe->bufs[iter_head & p_mask].len += size; + buf->len += size; return size; } - pipe->bufs[iter_head & p_mask].len = PAGE_SIZE; - iter_head++; + buf->len = PAGE_SIZE; } - while (!pipe_full(iter_head, p_tail, pipe->max_usage)) { - struct pipe_buffer *buf = &pipe->bufs[iter_head & p_mask]; - struct page *page = alloc_page(GFP_USER); + while (!pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { + struct page *page = push_anon(pipe, + min_t(ssize_t, left, PAGE_SIZE)); if (!page) break; - buf->ops = &default_pipe_buf_ops; - buf->flags = 0; - buf->page = page; - buf->offset = 0; - buf->len = min_t(ssize_t, left, PAGE_SIZE); - left -= buf->len; - iter_head++; - pipe->head = iter_head; - - if (left == 0) + left -= PAGE_SIZE; + if (left <= 0) return size; } return size - left; From patchwork Wed Jun 22 04:15:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 990CBC433EF for ; Wed, 22 Jun 2022 04:16:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356893AbiFVEQW (ORCPT ); Wed, 22 Jun 2022 00:16:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356715AbiFVEP7 (ORCPT ); Wed, 22 Jun 2022 00:15:59 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93DC27640 for ; Tue, 21 Jun 2022 21:15:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=nCDb/stb7cqyzIfwc4qSCwYKLlHAIJUuROBEbgZA3HU=; b=Cz9UnkT3koep4YkSOXcTkdlKLQ OxhMP7Z5sdsx9Sb1H0tLv5ODGYnPEvo34csLMlFVxpr6XCDiNIv9hb94VTRKgssRXpHwvUccVlq+B toBdF5xKQy9v5qc7spTGf3nmD1WcbGZk/NvxtjKJx5blKlb1JIk7XiggYeWgic4TIlZVUd7ObnjVr a6lFXY+uCRg7PV/Kg2ojxOhKV5rXr/VSyKCddTeIg9JVpp8BgQ8tR9m0Kgj5WR95FhTOvpGB4izns qILoH4uy2caSpLeda7+XeinYbbK2MjbfeOjMFQubIjeOCYD9vkIC+pxKTD/eF26E4pI9ukZv76TWB 79aUMsnA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmV-0035wq-5a; Wed, 22 Jun 2022 04:15:55 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 16/44] ITER_PIPE: allocate buffers as we go in copy-to-pipe primitives Date: Wed, 22 Jun 2022 05:15:24 +0100 Message-Id: <20220622041552.737754-16-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org New helper: append_pipe(). Extends the last buffer if possible, allocates a new one otherwise. Returns page and offset in it on success, NULL on failure. iov_iter is advanced past the data we've got. Use that instead of push_pipe() in copy-to-pipe primitives; they get simpler that way. Handling of short copy (in "mc" one) is done simply by iov_iter_revert() - iov_iter is in consistent state after that one, so we can use that. Signed-off-by: Al Viro --- lib/iov_iter.c | 159 +++++++++++++++++++++++++++++-------------------- 1 file changed, 93 insertions(+), 66 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 924854c2a7ce..2a445261096e 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -259,6 +259,44 @@ static void push_page(struct pipe_inode_info *pipe, struct page *page, get_page(page); } +static inline bool allocated(struct pipe_buffer *buf) +{ + return buf->ops == &default_pipe_buf_ops; +} + +static struct page *append_pipe(struct iov_iter *i, size_t size, size_t *off) +{ + struct pipe_inode_info *pipe = i->pipe; + size_t offset = i->iov_offset; + struct pipe_buffer *buf; + struct page *page; + + if (offset && offset < PAGE_SIZE) { + // some space in the last buffer; can we add to it? + buf = pipe_buf(pipe, pipe->head - 1); + if (allocated(buf)) { + size = min_t(size_t, size, PAGE_SIZE - offset); + buf->len += size; + i->iov_offset += size; + i->count -= size; + *off = offset; + return buf->page; + } + } + // OK, we need a new buffer + *off = 0; + size = min_t(size_t, size, PAGE_SIZE); + if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) + return NULL; + page = push_anon(pipe, size); + if (!page) + return NULL; + i->head = pipe->head - 1; + i->iov_offset = size; + i->count -= size; + return page; +} + static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { @@ -396,11 +434,6 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, } EXPORT_SYMBOL(iov_iter_init); -static inline bool allocated(struct pipe_buffer *buf) -{ - return buf->ops == &default_pipe_buf_ops; -} - static inline void data_start(const struct iov_iter *i, unsigned int *iter_headp, size_t *offp) { @@ -459,28 +492,26 @@ static size_t push_pipe(struct iov_iter *i, size_t size, static size_t copy_pipe_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { - struct pipe_inode_info *pipe = i->pipe; - unsigned int p_mask = pipe->ring_size - 1; - unsigned int i_head; size_t n, off; - if (!sanity(i)) + if (unlikely(bytes > i->count)) + bytes = i->count; + if (unlikely(!bytes)) return 0; - bytes = n = push_pipe(i, bytes, &i_head, &off); - if (unlikely(!n)) + if (!sanity(i)) return 0; - do { + + n = bytes; + while (n) { + struct page *page = append_pipe(i, n, &off); size_t chunk = min_t(size_t, n, PAGE_SIZE - off); - memcpy_to_page(pipe->bufs[i_head & p_mask].page, off, addr, chunk); - i->head = i_head; - i->iov_offset = off + chunk; - n -= chunk; + if (!page) + break; + memcpy_to_page(page, off, addr, chunk); addr += chunk; - off = 0; - i_head++; - } while (n); - i->count -= bytes; + n -= chunk; + } return bytes; } @@ -494,31 +525,32 @@ static __wsum csum_and_memcpy(void *to, const void *from, size_t len, static size_t csum_and_copy_to_pipe_iter(const void *addr, size_t bytes, struct iov_iter *i, __wsum *sump) { - struct pipe_inode_info *pipe = i->pipe; - unsigned int p_mask = pipe->ring_size - 1; __wsum sum = *sump; size_t off = 0; - unsigned int i_head; size_t r; + if (unlikely(bytes > i->count)) + bytes = i->count; + if (unlikely(!bytes)) + return 0; + if (!sanity(i)) return 0; - bytes = push_pipe(i, bytes, &i_head, &r); while (bytes) { + struct page *page = append_pipe(i, bytes, &r); size_t chunk = min_t(size_t, bytes, PAGE_SIZE - r); - char *p = kmap_local_page(pipe->bufs[i_head & p_mask].page); + char *p; + + if (!page) + break; + p = kmap_local_page(page); sum = csum_and_memcpy(p + r, addr + off, chunk, sum, off); kunmap_local(p); - i->head = i_head; - i->iov_offset = r + chunk; - bytes -= chunk; off += chunk; - r = 0; - i_head++; + bytes -= chunk; } *sump = sum; - i->count -= off; return off; } @@ -550,39 +582,35 @@ static int copyout_mc(void __user *to, const void *from, size_t n) static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { - struct pipe_inode_info *pipe = i->pipe; - unsigned int p_mask = pipe->ring_size - 1; - unsigned int i_head; - unsigned int valid = pipe->head; - size_t n, off, xfer = 0; + size_t off, xfer = 0; + + if (unlikely(bytes > i->count)) + bytes = i->count; + if (unlikely(!bytes)) + return 0; if (!sanity(i)) return 0; - n = push_pipe(i, bytes, &i_head, &off); - while (n) { - size_t chunk = min_t(size_t, n, PAGE_SIZE - off); - char *p = kmap_local_page(pipe->bufs[i_head & p_mask].page); + while (bytes) { + struct page *page = append_pipe(i, bytes, &off); + size_t chunk = min_t(size_t, bytes, PAGE_SIZE - off); unsigned long rem; + char *p; + + if (!page) + break; + p = kmap_local_page(page); rem = copy_mc_to_kernel(p + off, addr + xfer, chunk); chunk -= rem; kunmap_local(p); - if (chunk) { - i->head = i_head; - i->iov_offset = off + chunk; - xfer += chunk; - valid = i_head + 1; - } + xfer += chunk; + bytes -= chunk; if (rem) { - pipe->bufs[i_head & p_mask].len -= rem; - pipe_discard_from(pipe, valid); + iov_iter_revert(i, rem); break; } - n -= chunk; - off = 0; - i_head++; } - i->count -= xfer; return xfer; } @@ -769,30 +797,29 @@ EXPORT_SYMBOL(copy_page_from_iter); static size_t pipe_zero(size_t bytes, struct iov_iter *i) { - struct pipe_inode_info *pipe = i->pipe; - unsigned int p_mask = pipe->ring_size - 1; - unsigned int i_head; size_t n, off; - if (!sanity(i)) + if (unlikely(bytes > i->count)) + bytes = i->count; + if (unlikely(!bytes)) return 0; - bytes = n = push_pipe(i, bytes, &i_head, &off); - if (unlikely(!n)) + if (!sanity(i)) return 0; - do { + n = bytes; + while (n) { + struct page *page = append_pipe(i, n, &off); size_t chunk = min_t(size_t, n, PAGE_SIZE - off); - char *p = kmap_local_page(pipe->bufs[i_head & p_mask].page); + char *p; + + if (!page) + break; + p = kmap_local_page(page); memset(p + off, 0, chunk); kunmap_local(p); - i->head = i_head; - i->iov_offset = off + chunk; n -= chunk; - off = 0; - i_head++; - } while (n); - i->count -= bytes; + } return bytes; } From patchwork Wed Jun 22 04:15:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 704F3C433EF for ; Wed, 22 Jun 2022 04:16:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356900AbiFVEQZ (ORCPT ); Wed, 22 Jun 2022 00:16:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356719AbiFVEP7 (ORCPT ); Wed, 22 Jun 2022 00:15:59 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A9E97649 for ; Tue, 21 Jun 2022 21:15:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=SioEB4rnR/C8LeMY2GFKseUarMvAD/W/8gnG3v9/ARk=; b=jPC9SlWs1uIF8BN4sMfooMAlUL 18jpJiKI4bhxnvCX2H8IAZsY0IBwPIYRmG2CBstoV4Ipdg2YkjRT2OMZmkvroD66H/9WIrprBmiLZ hxqP/1NhAP0BUiEyQ2If5G/4+145vWiRJRSec1lDZVhcBg+DzFVGDQjpSZ3OGdzxzyBoeM8CptuTA WxNWy00PsBgKVAa2lxhcg4SFEnIkUnl66GiRv85z4exn10zOhoWx0RnHhmEC6+NMqdmm+MIZamDPq 2tejWUMH+pMgFTsPtvHNJtz48DEGm0YzqmabZirwpTSkCR7amVnU641IhjNDx7/+hlkAIXv1Qt2u5 xP8qMdJA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmV-0035wu-8e; Wed, 22 Jun 2022 04:15:55 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 17/44] ITER_PIPE: fold push_pipe() into __pipe_get_pages() Date: Wed, 22 Jun 2022 05:15:25 +0100 Message-Id: <20220622041552.737754-17-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Expand the only remaining call of push_pipe() (in __pipe_get_pages()), combine it with the page-collecting loop there. Note that the only reason it's not a loop doing append_pipe() is that append_pipe() is advancing, while iov_iter_get_pages() is not. As soon as it switches to saner semantics, this thing will switch to using append_pipe(). Signed-off-by: Al Viro --- lib/iov_iter.c | 80 ++++++++++++++++---------------------------------- 1 file changed, 25 insertions(+), 55 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 2a445261096e..a507eed67839 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -449,46 +449,6 @@ static inline void data_start(const struct iov_iter *i, *offp = off; } -static size_t push_pipe(struct iov_iter *i, size_t size, - int *iter_headp, size_t *offp) -{ - struct pipe_inode_info *pipe = i->pipe; - unsigned int iter_head; - size_t off; - ssize_t left; - - if (unlikely(size > i->count)) - size = i->count; - if (unlikely(!size)) - return 0; - - left = size; - data_start(i, &iter_head, &off); - *iter_headp = iter_head; - *offp = off; - if (off) { - struct pipe_buffer *buf = pipe_buf(pipe, iter_head); - - left -= PAGE_SIZE - off; - if (left <= 0) { - buf->len += size; - return size; - } - buf->len = PAGE_SIZE; - } - while (!pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { - struct page *page = push_anon(pipe, - min_t(ssize_t, left, PAGE_SIZE)); - if (!page) - break; - - left -= PAGE_SIZE; - if (left <= 0) - return size; - } - return size - left; -} - static size_t copy_pipe_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { @@ -1261,23 +1221,33 @@ static inline ssize_t __pipe_get_pages(struct iov_iter *i, size_t maxsize, struct page **pages, int iter_head, - size_t *start) + size_t off) { struct pipe_inode_info *pipe = i->pipe; - unsigned int p_mask = pipe->ring_size - 1; - ssize_t n = push_pipe(i, maxsize, &iter_head, start); - if (!n) - return -EFAULT; + ssize_t left = maxsize; - maxsize = n; - n += *start; - while (n > 0) { - get_page(*pages++ = pipe->bufs[iter_head & p_mask].page); - iter_head++; - n -= PAGE_SIZE; - } + if (off) { + struct pipe_buffer *buf = pipe_buf(pipe, iter_head); - return maxsize; + get_page(*pages++ = buf->page); + left -= PAGE_SIZE - off; + if (left <= 0) { + buf->len += maxsize; + return maxsize; + } + buf->len = PAGE_SIZE; + } + while (!pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { + struct page *page = push_anon(pipe, + min_t(ssize_t, left, PAGE_SIZE)); + if (!page) + break; + get_page(*pages++ = page); + left -= PAGE_SIZE; + if (left <= 0) + return maxsize; + } + return maxsize - left ? : -EFAULT; } static ssize_t pipe_get_pages(struct iov_iter *i, @@ -1295,7 +1265,7 @@ static ssize_t pipe_get_pages(struct iov_iter *i, npages = pipe_space_for_user(iter_head, i->pipe->tail, i->pipe); capacity = min(npages, maxpages) * PAGE_SIZE - *start; - return __pipe_get_pages(i, min(maxsize, capacity), pages, iter_head, start); + return __pipe_get_pages(i, min(maxsize, capacity), pages, iter_head, *start); } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa, @@ -1491,7 +1461,7 @@ static ssize_t pipe_get_pages_alloc(struct iov_iter *i, p = get_pages_array(npages); if (!p) return -ENOMEM; - n = __pipe_get_pages(i, maxsize, p, iter_head, start); + n = __pipe_get_pages(i, maxsize, p, iter_head, *start); if (n > 0) *pages = p; else From patchwork Wed Jun 22 04:15:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E6AAC433EF for ; Wed, 22 Jun 2022 04:16:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356901AbiFVEQ3 (ORCPT ); Wed, 22 Jun 2022 00:16:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356716AbiFVEP7 (ORCPT ); Wed, 22 Jun 2022 00:15:59 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9283209 for ; Tue, 21 Jun 2022 21:15:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=14wT+3jiMDFp/cm+PVp6c75ud4yITpvG6FiAkIAHu30=; b=sIrp/moA0sej9fVyITkNinoJ2D 8ZJzRR1LARuMLkLGvDfWAnRGKMFTMvUZA9tLC+WAHDhj+uEQ7gEPQV5QEQFPnEqK4y4bMi/HSlqLU pCCOKUGON3I3W3oh80oNcMC4MDNiK9xr/cQBHgwwYdJtvLKybA9tPWDUBpsP5yxQE/21VzR4jK+o/ +/66AtebAy52DhYitiE4E96dz08G73d+0GBE0dYgdab+kHStK9KJxVP349K8yfPlYFlSjRnl+v/jo +3qpmYBFi46bJid/C/S/6ftBILLaFW2XQ//F9Ym+8cUe7f0LIjbXZNdBsHObllfwo1HaPKoehoOdT md0/hFZw==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmV-0035x1-DT; Wed, 22 Jun 2022 04:15:55 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 18/44] ITER_PIPE: lose iter_head argument of __pipe_get_pages() Date: Wed, 22 Jun 2022 05:15:26 +0100 Message-Id: <20220622041552.737754-18-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org it's only used to get to the partial buffer we can add to, and that's always the last one, i.e. pipe->head - 1. Signed-off-by: Al Viro --- lib/iov_iter.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index a507eed67839..4b5a98105547 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1220,14 +1220,13 @@ EXPORT_SYMBOL(iov_iter_gap_alignment); static inline ssize_t __pipe_get_pages(struct iov_iter *i, size_t maxsize, struct page **pages, - int iter_head, size_t off) { struct pipe_inode_info *pipe = i->pipe; ssize_t left = maxsize; if (off) { - struct pipe_buffer *buf = pipe_buf(pipe, iter_head); + struct pipe_buffer *buf = pipe_buf(pipe, pipe->head - 1); get_page(*pages++ = buf->page); left -= PAGE_SIZE - off; @@ -1265,7 +1264,7 @@ static ssize_t pipe_get_pages(struct iov_iter *i, npages = pipe_space_for_user(iter_head, i->pipe->tail, i->pipe); capacity = min(npages, maxpages) * PAGE_SIZE - *start; - return __pipe_get_pages(i, min(maxsize, capacity), pages, iter_head, *start); + return __pipe_get_pages(i, min(maxsize, capacity), pages, *start); } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa, @@ -1461,7 +1460,7 @@ static ssize_t pipe_get_pages_alloc(struct iov_iter *i, p = get_pages_array(npages); if (!p) return -ENOMEM; - n = __pipe_get_pages(i, maxsize, p, iter_head, *start); + n = __pipe_get_pages(i, maxsize, p, *start); if (n > 0) *pages = p; else From patchwork Wed Jun 22 04:15:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32671C43334 for ; Wed, 22 Jun 2022 04:16:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356895AbiFVEQY (ORCPT ); Wed, 22 Jun 2022 00:16:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356739AbiFVEQA (ORCPT ); Wed, 22 Jun 2022 00:16:00 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07C25263 for ; Tue, 21 Jun 2022 21:15:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=MK/QN958MjrQYMSsqbzrMnuTk+uO+BS+5HnTFSy3ka4=; b=g7tICJPbdSIA24vsGhHYqY1x6W 0D1XwkvOOyjAdfUD8IVE1Xu7BRdlwTwZc8zW8SLGO57IUhA32ve0urr6ZNzIjUsToY0hj6K+8/6dH VKcNrd1e3hiQnjyPywtO1dzEAuwRRUTKR7CAGVB237/nI6VEuwXTr/C4I7JZSJiYNXFEImwG5+Bpu HXut0nn3D49HDVLymjdVmaxbGWQRFuzysM680zSxW+aQsUPr3xbOU9F8EezkRMrbUp4UAxTRMUQhU H+aRAOortvnP0n5+0JoUXtnHAElgYuF8GpmEvY8BJFsuFAdY3O58S9ZXaXb3W6WYIMJrmzG7G4/HX sNmaaaJg==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmV-0035x8-Gh; Wed, 22 Jun 2022 04:15:55 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 19/44] ITER_PIPE: clean pipe_advance() up Date: Wed, 22 Jun 2022 05:15:27 +0100 Message-Id: <20220622041552.737754-19-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org instead of setting ->iov_offset for new position and calling pipe_truncate() to adjust ->len of the last buffer and discard everything after it, adjust ->len at the same time we set ->iov_offset and use pipe_discard_from() to deal with buffers past that. Signed-off-by: Al Viro --- lib/iov_iter.c | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 4b5a98105547..6d693c1d189d 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -847,27 +847,27 @@ static inline void pipe_truncate(struct iov_iter *i) static void pipe_advance(struct iov_iter *i, size_t size) { struct pipe_inode_info *pipe = i->pipe; - if (size) { - struct pipe_buffer *buf; - unsigned int p_mask = pipe->ring_size - 1; - unsigned int i_head = i->head; - size_t off = i->iov_offset, left = size; + unsigned int off = i->iov_offset; + if (!off && !size) { + pipe_discard_from(pipe, i->start_head); // discard everything + return; + } + i->count -= size; + while (1) { + struct pipe_buffer *buf = pipe_buf(pipe, i->head); if (off) /* make it relative to the beginning of buffer */ - left += off - pipe->bufs[i_head & p_mask].offset; - while (1) { - buf = &pipe->bufs[i_head & p_mask]; - if (left <= buf->len) - break; - left -= buf->len; - i_head++; + size += off - buf->offset; + if (size <= buf->len) { + buf->len = size; + i->iov_offset = buf->offset + size; + break; } - i->head = i_head; - i->iov_offset = buf->offset + left; + size -= buf->len; + i->head++; + off = 0; } - i->count -= size; - /* ... and discard everything past that point */ - pipe_truncate(i); + pipe_discard_from(pipe, i->head + 1); // discard everything past this one } static void iov_iter_bvec_advance(struct iov_iter *i, size_t size) From patchwork Wed Jun 22 04:15:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890104 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76AE8C433EF for ; Wed, 22 Jun 2022 04:16:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356966AbiFVEQq (ORCPT ); Wed, 22 Jun 2022 00:16:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356782AbiFVEQA (ORCPT ); Wed, 22 Jun 2022 00:16:00 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 356DFB71 for ; Tue, 21 Jun 2022 21:15:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=QgK3buMcOnvqaTWUvnUNevAI1qrMq591PQ5BFWzu1no=; b=Fg6lglL3qpk3dSowdIyD4F21jL 5F9llCksQduzejAHqZKwT7xO8GTyHkJhp6H4uJQwgGZDjj1A/6rx/AcoX1xxTJg5Zr/tGgtC3Qgs5 r6EbeKfy25iGYyx53R/YSIGSI9RyTY0Y8do8CSzkPfQpshBVvdyozNmrklXrYKJGgc5zk1YkoQrtJ mCph1DYEhrf/eh+PxfhaTthdX8oiEc29O6c6oEz8ym5zlcyIXJJO+H8t9Oh3Fo/uBN7lgNgjuklFz gjuFC5gyZ0vDwlS48WPiUIXh9Nv2DVg7FMRLOvbMf7262QCGesxZo3P/MNJji4kMkJxbFhKJVMEEd fDJpk2XA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmV-0035xG-Mx; Wed, 22 Jun 2022 04:15:55 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 20/44] ITER_PIPE: clean iov_iter_revert() Date: Wed, 22 Jun 2022 05:15:28 +0100 Message-Id: <20220622041552.737754-20-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Fold pipe_truncate() into it, clean up. We can release buffers in the same loop where we walk backwards to the iterator beginning looking for the place where the new position will be. Signed-off-by: Al Viro --- lib/iov_iter.c | 60 ++++++++++++-------------------------------------- 1 file changed, 14 insertions(+), 46 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 6d693c1d189d..4e2b000b0466 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -818,32 +818,6 @@ size_t copy_page_from_iter_atomic(struct page *page, unsigned offset, size_t byt } EXPORT_SYMBOL(copy_page_from_iter_atomic); -static inline void pipe_truncate(struct iov_iter *i) -{ - struct pipe_inode_info *pipe = i->pipe; - unsigned int p_tail = pipe->tail; - unsigned int p_head = pipe->head; - unsigned int p_mask = pipe->ring_size - 1; - - if (!pipe_empty(p_head, p_tail)) { - struct pipe_buffer *buf; - unsigned int i_head = i->head; - size_t off = i->iov_offset; - - if (off) { - buf = &pipe->bufs[i_head & p_mask]; - buf->len = off - buf->offset; - i_head++; - } - while (p_head != i_head) { - p_head--; - pipe_buf_release(pipe, &pipe->bufs[p_head & p_mask]); - } - - pipe->head = p_head; - } -} - static void pipe_advance(struct iov_iter *i, size_t size) { struct pipe_inode_info *pipe = i->pipe; @@ -938,28 +912,22 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll) i->count += unroll; if (unlikely(iov_iter_is_pipe(i))) { struct pipe_inode_info *pipe = i->pipe; - unsigned int p_mask = pipe->ring_size - 1; - unsigned int i_head = i->head; - size_t off = i->iov_offset; - while (1) { - struct pipe_buffer *b = &pipe->bufs[i_head & p_mask]; - size_t n = off - b->offset; - if (unroll < n) { - off -= unroll; - break; - } - unroll -= n; - if (!unroll && i_head == i->start_head) { - off = 0; - break; + unsigned int head = pipe->head; + + while (head > i->start_head) { + struct pipe_buffer *b = pipe_buf(pipe, --head); + if (unroll < b->len) { + b->len -= unroll; + i->iov_offset = b->offset + b->len; + i->head = head; + return; } - i_head--; - b = &pipe->bufs[i_head & p_mask]; - off = b->offset + b->len; + unroll -= b->len; + pipe_buf_release(pipe, b); + pipe->head--; } - i->iov_offset = off; - i->head = i_head; - pipe_truncate(i); + i->iov_offset = 0; + i->head = head; return; } if (unlikely(iov_iter_is_discard(i))) From patchwork Wed Jun 22 04:15:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD8B2C433EF for ; Wed, 22 Jun 2022 04:16:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356980AbiFVEQx (ORCPT ); Wed, 22 Jun 2022 00:16:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356827AbiFVEQI (ORCPT ); Wed, 22 Jun 2022 00:16:08 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35E8960E4 for ; Tue, 21 Jun 2022 21:15:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=SmvagF6Vv4nPPylNtEr0z1KTgyZMdUEuo9sZnW9DNAc=; b=sxMhx8urPH3qAK+dOhAYUMb2KL S7wgmFU37ImZWt8/NTauT4GhuLRTzFCShQ4JSuviGKIUKAMJJ8o41Cf3YO9CSCOtJOKDRqZFFrDci 3LyiCh8tgCm3avEvf3u+JBZU9kiQBOPRlbKOWhznhlx2vKY7q+ejQvNBeHqKP+apba+N9reAH+lpc 5k2PkASZLm0uoVxNs/K+1Tn+x0gIsKE5Sd+XcMMtY60LIioPkjAPKLX16C/A98pswDs3AQk53DnIa GtBHhXZD3SAkoFnFOL/VHKPQdmpvN+31UORFBJwXfQk8hJsjskFEA5Ow8LyUngocl2d9pPPcLoQUs Hy5peoOg==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmV-0035xN-TC; Wed, 22 Jun 2022 04:15:55 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 21/44] ITER_PIPE: cache the type of last buffer Date: Wed, 22 Jun 2022 05:15:29 +0100 Message-Id: <20220622041552.737754-21-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org We often need to find whether the last buffer is anon or not, and currently it's rather clumsy: check if ->iov_offset is non-zero (i.e. that pipe is not empty) if so, get the corresponding pipe_buffer and check its ->ops if it's &default_pipe_buf_ops, we have an anon buffer. Let's replace the use of ->iov_offset (which is nowhere near similar to its role for other flavours) with signed field (->last_offset), with the following rules: empty, no buffers occupied: 0 anon, with bytes up to N-1 filled: N zero-copy, with bytes up to N-1 filled: -N That way abs(i->last_offset) is equal to what used to be in i->iov_offset and empty vs. anon vs. zero-copy can be distinguished by the sign of i->last_offset. Checks for "should we extend the last buffer or should we start a new one?" become easier to follow that way. Note that most of the operations can only be done in a sane state - i.e. when the pipe has nothing past the current position of iterator. About the only thing that could be done outside of that state is iov_iter_advance(), which transitions to the sane state by truncating the pipe. There are only two cases where we leave the sane state: 1) iov_iter_get_pages()/iov_iter_get_pages_alloc(). Will be dealt with later, when we make get_pages advancing - the callers are actually happier that way. 2) iov_iter copied, then something is put into the copy. Since they share the underlying pipe, the original gets behind. When we decide that we are done with the copy (original is not usable until then) we advance the original. direct_io used to be done that way; nowadays it operates on the original and we do iov_iter_revert() to discard the excessive data. At the moment there's nothing in the kernel that could do that to ITER_PIPE iterators, so this reason for insane state is theoretical right now. Signed-off-by: Al Viro --- include/linux/uio.h | 5 +++- lib/iov_iter.c | 72 ++++++++++++++++++++++----------------------- 2 files changed, 40 insertions(+), 37 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index 6ab4260c3d6c..d3e13b37ea72 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -40,7 +40,10 @@ struct iov_iter { bool nofault; bool data_source; bool user_backed; - size_t iov_offset; + union { + size_t iov_offset; + int last_offset; + }; size_t count; union { const struct iovec *iov; diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 4e2b000b0466..27ad2ef93dbc 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -199,7 +199,7 @@ static bool sanity(const struct iov_iter *i) unsigned int i_head = i->head; unsigned int idx; - if (i->iov_offset) { + if (i->last_offset) { struct pipe_buffer *p; if (unlikely(p_occupancy == 0)) goto Bad; // pipe must be non-empty @@ -207,7 +207,7 @@ static bool sanity(const struct iov_iter *i) goto Bad; // must be at the last buffer... p = pipe_buf(pipe, i_head); - if (unlikely(p->offset + p->len != i->iov_offset)) + if (unlikely(p->offset + p->len != abs(i->last_offset))) goto Bad; // ... at the end of segment } else { if (i_head != p_head) @@ -215,7 +215,7 @@ static bool sanity(const struct iov_iter *i) } return true; Bad: - printk(KERN_ERR "idx = %d, offset = %zd\n", i_head, i->iov_offset); + printk(KERN_ERR "idx = %d, offset = %d\n", i_head, i->last_offset); printk(KERN_ERR "head = %d, tail = %d, buffers = %d\n", p_head, p_tail, pipe->ring_size); for (idx = 0; idx < pipe->ring_size; idx++) @@ -259,29 +259,30 @@ static void push_page(struct pipe_inode_info *pipe, struct page *page, get_page(page); } -static inline bool allocated(struct pipe_buffer *buf) +static inline int last_offset(const struct pipe_buffer *buf) { - return buf->ops == &default_pipe_buf_ops; + if (buf->ops == &default_pipe_buf_ops) + return buf->len; // buf->offset is 0 for those + else + return -(buf->offset + buf->len); } static struct page *append_pipe(struct iov_iter *i, size_t size, size_t *off) { struct pipe_inode_info *pipe = i->pipe; - size_t offset = i->iov_offset; + int offset = i->last_offset; struct pipe_buffer *buf; struct page *page; - if (offset && offset < PAGE_SIZE) { - // some space in the last buffer; can we add to it? + if (offset > 0 && offset < PAGE_SIZE) { + // some space in the last buffer; add to it buf = pipe_buf(pipe, pipe->head - 1); - if (allocated(buf)) { - size = min_t(size_t, size, PAGE_SIZE - offset); - buf->len += size; - i->iov_offset += size; - i->count -= size; - *off = offset; - return buf->page; - } + size = min_t(size_t, size, PAGE_SIZE - offset); + buf->len += size; + i->last_offset += size; + i->count -= size; + *off = offset; + return buf->page; } // OK, we need a new buffer *off = 0; @@ -292,7 +293,7 @@ static struct page *append_pipe(struct iov_iter *i, size_t size, size_t *off) if (!page) return NULL; i->head = pipe->head - 1; - i->iov_offset = size; + i->last_offset = size; i->count -= size; return page; } @@ -312,11 +313,11 @@ static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t by if (!sanity(i)) return 0; - if (offset && i->iov_offset == offset) { // could we merge it? + if (offset && i->last_offset == -offset) { // could we merge it? struct pipe_buffer *buf = pipe_buf(pipe, head - 1); if (buf->page == page) { buf->len += bytes; - i->iov_offset += bytes; + i->last_offset -= bytes; i->count -= bytes; return bytes; } @@ -325,7 +326,7 @@ static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t by return 0; push_page(pipe, page, offset, bytes); - i->iov_offset = offset + bytes; + i->last_offset = -(offset + bytes); i->head = head; i->count -= bytes; return bytes; @@ -437,16 +438,15 @@ EXPORT_SYMBOL(iov_iter_init); static inline void data_start(const struct iov_iter *i, unsigned int *iter_headp, size_t *offp) { - unsigned int iter_head = i->head; - size_t off = i->iov_offset; + int off = i->last_offset; - if (off && (!allocated(pipe_buf(i->pipe, iter_head)) || - off == PAGE_SIZE)) { - iter_head++; - off = 0; + if (off > 0 && off < PAGE_SIZE) { // anon and not full + *iter_headp = i->pipe->head - 1; + *offp = off; + } else { + *iter_headp = i->pipe->head; + *offp = 0; } - *iter_headp = iter_head; - *offp = off; } static size_t copy_pipe_to_iter(const void *addr, size_t bytes, @@ -821,7 +821,7 @@ EXPORT_SYMBOL(copy_page_from_iter_atomic); static void pipe_advance(struct iov_iter *i, size_t size) { struct pipe_inode_info *pipe = i->pipe; - unsigned int off = i->iov_offset; + int off = i->last_offset; if (!off && !size) { pipe_discard_from(pipe, i->start_head); // discard everything @@ -831,10 +831,10 @@ static void pipe_advance(struct iov_iter *i, size_t size) while (1) { struct pipe_buffer *buf = pipe_buf(pipe, i->head); if (off) /* make it relative to the beginning of buffer */ - size += off - buf->offset; + size += abs(off) - buf->offset; if (size <= buf->len) { buf->len = size; - i->iov_offset = buf->offset + size; + i->last_offset = last_offset(buf); break; } size -= buf->len; @@ -918,7 +918,7 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll) struct pipe_buffer *b = pipe_buf(pipe, --head); if (unroll < b->len) { b->len -= unroll; - i->iov_offset = b->offset + b->len; + i->last_offset = last_offset(b); i->head = head; return; } @@ -926,7 +926,7 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll) pipe_buf_release(pipe, b); pipe->head--; } - i->iov_offset = 0; + i->last_offset = 0; i->head = head; return; } @@ -1029,7 +1029,7 @@ void iov_iter_pipe(struct iov_iter *i, unsigned int direction, .pipe = pipe, .head = pipe->head, .start_head = pipe->head, - .iov_offset = 0, + .last_offset = 0, .count = count }; } @@ -1145,8 +1145,8 @@ unsigned long iov_iter_alignment(const struct iov_iter *i) if (iov_iter_is_pipe(i)) { size_t size = i->count; - if (size && i->iov_offset && allocated(pipe_buf(i->pipe, i->head))) - return size | i->iov_offset; + if (size && i->last_offset > 0) + return size | i->last_offset; return size; } From patchwork Wed Jun 22 04:15:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890096 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3FBAC43334 for ; Wed, 22 Jun 2022 04:16:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356905AbiFVEQg (ORCPT ); Wed, 22 Jun 2022 00:16:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356792AbiFVEQI (ORCPT ); Wed, 22 Jun 2022 00:16:08 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D934655C for ; Tue, 21 Jun 2022 21:15:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=8N29AzBY9Rx4BcEdQmOy+0Rvuc0p6ZRSNSSD/ktkboM=; b=WKy4czSYwkDLFrAKkDmEoulukW aj6u2ImnSx6FLZrAaFYE3tK72J5hjyjN0+ONIZ2OVO66y1V+oTLLuFkZFyXdbn0V1p5nEooUF4jJp OMNzT9eU9bb5QDOUsmfEFZ3mxxIIyAUDmUm3A8nHxh6tK7gBYxFQeL8ePj9WZud0wBTPhbc7wUU1y llTZZHSAZ4twVOT57Tjz9V07ghwKF8EBylYpzgGItgi0nkZqOc6XxS08B3I7x9hP2RtaNsKqOAx8p F70ys05m5s3DM49Jw1lNLWxY6lZbGalefHYS0c97xgHGVQQW+pzIvg207YbclqzDT+IpEsZtwrj5H 4ntcpG8Q==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmW-0035xV-3E; Wed, 22 Jun 2022 04:15:56 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 22/44] ITER_PIPE: fold data_start() and pipe_space_for_user() together Date: Wed, 22 Jun 2022 05:15:30 +0100 Message-Id: <20220622041552.737754-22-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org All their callers are next to each other; all of them want the total amount of pages and, possibly, the offset in the partial final buffer. Combine into a new helper (pipe_npages()), fix the bogosity in pipe_space_for_user(), while we are at it. Signed-off-by: Al Viro --- include/linux/pipe_fs_i.h | 20 ------------------ lib/iov_iter.c | 44 +++++++++++++++++---------------------- 2 files changed, 19 insertions(+), 45 deletions(-) diff --git a/include/linux/pipe_fs_i.h b/include/linux/pipe_fs_i.h index 4ea496924106..6cb65df3e3ba 100644 --- a/include/linux/pipe_fs_i.h +++ b/include/linux/pipe_fs_i.h @@ -156,26 +156,6 @@ static inline bool pipe_full(unsigned int head, unsigned int tail, return pipe_occupancy(head, tail) >= limit; } -/** - * pipe_space_for_user - Return number of slots available to userspace - * @head: The pipe ring head pointer - * @tail: The pipe ring tail pointer - * @pipe: The pipe info structure - */ -static inline unsigned int pipe_space_for_user(unsigned int head, unsigned int tail, - struct pipe_inode_info *pipe) -{ - unsigned int p_occupancy, p_space; - - p_occupancy = pipe_occupancy(head, tail); - if (p_occupancy >= pipe->max_usage) - return 0; - p_space = pipe->ring_size - p_occupancy; - if (p_space > pipe->max_usage) - p_space = pipe->max_usage; - return p_space; -} - /** * pipe_buf_get - get a reference to a pipe_buffer * @pipe: the pipe that the buffer belongs to diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 27ad2ef93dbc..30f4158382d6 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -435,18 +435,20 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, } EXPORT_SYMBOL(iov_iter_init); -static inline void data_start(const struct iov_iter *i, - unsigned int *iter_headp, size_t *offp) +// returns the offset in partial buffer (if any) +static inline unsigned int pipe_npages(const struct iov_iter *i, int *npages) { + struct pipe_inode_info *pipe = i->pipe; + int used = pipe->head - pipe->tail; int off = i->last_offset; + *npages = max((int)pipe->max_usage - used, 0); + if (off > 0 && off < PAGE_SIZE) { // anon and not full - *iter_headp = i->pipe->head - 1; - *offp = off; - } else { - *iter_headp = i->pipe->head; - *offp = 0; + (*npages)++; + return off; } + return 0; } static size_t copy_pipe_to_iter(const void *addr, size_t bytes, @@ -1221,18 +1223,16 @@ static ssize_t pipe_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start) { - unsigned int iter_head, npages; + unsigned int npages, off; size_t capacity; if (!sanity(i)) return -EFAULT; - data_start(i, &iter_head, start); - /* Amount of free space: some of this one + all after this one */ - npages = pipe_space_for_user(iter_head, i->pipe->tail, i->pipe); - capacity = min(npages, maxpages) * PAGE_SIZE - *start; + *start = off = pipe_npages(i, &npages); + capacity = min(npages, maxpages) * PAGE_SIZE - off; - return __pipe_get_pages(i, min(maxsize, capacity), pages, *start); + return __pipe_get_pages(i, min(maxsize, capacity), pages, off); } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa, @@ -1411,24 +1411,22 @@ static ssize_t pipe_get_pages_alloc(struct iov_iter *i, size_t *start) { struct page **p; - unsigned int iter_head, npages; + unsigned int npages, off; ssize_t n; if (!sanity(i)) return -EFAULT; - data_start(i, &iter_head, start); - /* Amount of free space: some of this one + all after this one */ - npages = pipe_space_for_user(iter_head, i->pipe->tail, i->pipe); - n = npages * PAGE_SIZE - *start; + *start = off = pipe_npages(i, &npages); + n = npages * PAGE_SIZE - off; if (maxsize > n) maxsize = n; else - npages = DIV_ROUND_UP(maxsize + *start, PAGE_SIZE); + npages = DIV_ROUND_UP(maxsize + off, PAGE_SIZE); p = get_pages_array(npages); if (!p) return -ENOMEM; - n = __pipe_get_pages(i, maxsize, p, *start); + n = __pipe_get_pages(i, maxsize, p, off); if (n > 0) *pages = p; else @@ -1653,16 +1651,12 @@ int iov_iter_npages(const struct iov_iter *i, int maxpages) if (iov_iter_is_bvec(i)) return bvec_npages(i, maxpages); if (iov_iter_is_pipe(i)) { - unsigned int iter_head; int npages; - size_t off; if (!sanity(i)) return 0; - data_start(i, &iter_head, &off); - /* some of this one + all after this one */ - npages = pipe_space_for_user(iter_head, i->pipe->tail, i->pipe); + pipe_npages(i, &npages); return min(npages, maxpages); } if (iov_iter_is_xarray(i)) { From patchwork Wed Jun 22 04:15:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890094 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5FEDC43334 for ; Wed, 22 Jun 2022 04:16:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356825AbiFVEQ1 (ORCPT ); Wed, 22 Jun 2022 00:16:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356757AbiFVEQA (ORCPT ); Wed, 22 Jun 2022 00:16:00 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3565D270 for ; Tue, 21 Jun 2022 21:15:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=seWHHlYvuiRkYMdFe+STiBfjmOWR+hFiyvE5vuqSB6Q=; b=gUfV5iT289q9ehqNWBCgt3zj+G G+IVR5jpVVlGXQry4K0sFctwgHLPl+wLUIKqiOkcoQLa/H1MkP6zCjsqKxysKUNkgP0WnBDWLiidt GYcVltWWPPFTcv2K3P8Syg1LbWZgBdWJSylEBAZ9pNu2coD5yAAeGKTDzcapgcrQEb8m4DETpfUtr gPp/jXCH+e3ddw1zNwg7wUz/SQxXovoZq8+cRXwJzLctT3AmB60hxrXmQLLjgD5EEE50dolAk6m8Q 9cIiOAA6byq9tUwrFpLdEfoK+PmnRW+MRd9Bzm122+/J8gzOP63bTfubAq7lRdvnVFz1WrZF6wCGH KUEuwbZw==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmW-0035xf-Bn; Wed, 22 Jun 2022 04:15:56 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 23/44] iov_iter_get_pages{,_alloc}(): cap the maxsize with MAX_RW_COUNT Date: Wed, 22 Jun 2022 05:15:31 +0100 Message-Id: <20220622041552.737754-23-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org All callers can and should handle iov_iter_get_pages() returning fewer pages than requested. All in-kernel ones do. And it makes the arithmetical overflow analysis much simpler... Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 30f4158382d6..c3fb7853dbe8 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1367,6 +1367,8 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, maxsize = i->count; if (!maxsize) return 0; + if (maxsize > MAX_RW_COUNT) + maxsize = MAX_RW_COUNT; if (likely(user_backed_iter(i))) { unsigned int gup_flags = 0; @@ -1485,6 +1487,8 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, maxsize = i->count; if (!maxsize) return 0; + if (maxsize > MAX_RW_COUNT) + maxsize = MAX_RW_COUNT; if (likely(user_backed_iter(i))) { unsigned int gup_flags = 0; From patchwork Wed Jun 22 04:15:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46A97C433EF for ; Wed, 22 Jun 2022 04:16:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356866AbiFVEQt (ORCPT ); Wed, 22 Jun 2022 00:16:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356773AbiFVEQA (ORCPT ); Wed, 22 Jun 2022 00:16:00 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3575AB7F for ; Tue, 21 Jun 2022 21:15:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=dzXKtRiyP3JgvScP2PDu57QL9iKNSgv5vG2wxukWCBw=; b=fI6yVun9khy0KkwMMLvGBF3yUy rKiYzrQgOBKRNROhSh0QtOKCC+QAkOyuyWstLlPsMo9thUuelccdFYCOc97jmfc3JRzONuMm1xUia Q8WVVC1ek7fW3IwEioTzD009x5KPShQOUIVNoAj6Poq5oxDuf2NC0osn3NALW0/18CGBBA7Nr5Bw9 vleWx/baP8HgeOGun1SLKSM/ZfAXSv7YW5RsgN5XvNxhNnKpQxqsFwUNpjx7yLBmYNQk0jFupNXKr 5/+csgbA5qDmsg8d1J7u8FEX/1nmHwnyoE5TN01p+TOD30P/Jhwu31HpIjTjqBH+ZWTQBBfAfmMzr pH+dxhsA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmW-0035xm-HI; Wed, 22 Jun 2022 04:15:56 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 24/44] iov_iter_get_pages_alloc(): lift freeing pages array on failure exits into wrapper Date: Wed, 22 Jun 2022 05:15:32 +0100 Message-Id: <20220622041552.737754-24-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Incidentally, ITER_XARRAY did *not* free the sucker in case when iter_xarray_populate_pages() returned 0... Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 38 ++++++++++++++++++++++---------------- 1 file changed, 22 insertions(+), 16 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index c3fb7853dbe8..9c25661684c6 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1425,15 +1425,10 @@ static ssize_t pipe_get_pages_alloc(struct iov_iter *i, maxsize = n; else npages = DIV_ROUND_UP(maxsize + off, PAGE_SIZE); - p = get_pages_array(npages); + *pages = p = get_pages_array(npages); if (!p) return -ENOMEM; - n = __pipe_get_pages(i, maxsize, p, off); - if (n > 0) - *pages = p; - else - kvfree(p); - return n; + return __pipe_get_pages(i, maxsize, p, off); } static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i, @@ -1463,10 +1458,9 @@ static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i, count++; } - p = get_pages_array(count); + *pages = p = get_pages_array(count); if (!p) return -ENOMEM; - *pages = p; nr = iter_xarray_populate_pages(p, i->xarray, index, count); if (nr == 0) @@ -1475,7 +1469,7 @@ static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i, return min_t(size_t, nr * PAGE_SIZE - offset, maxsize); } -ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, +static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start) { @@ -1501,16 +1495,12 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, addr = first_iovec_segment(i, &len, start, maxsize, ~0U); n = DIV_ROUND_UP(len, PAGE_SIZE); - p = get_pages_array(n); + *pages = p = get_pages_array(n); if (!p) return -ENOMEM; res = get_user_pages_fast(addr, n, gup_flags, p); - if (unlikely(res <= 0)) { - kvfree(p); - *pages = NULL; + if (unlikely(res <= 0)) return res; - } - *pages = p; return (res == n ? len : res * PAGE_SIZE) - *start; } if (iov_iter_is_bvec(i)) { @@ -1531,6 +1521,22 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, return iter_xarray_get_pages_alloc(i, pages, maxsize, start); return -EFAULT; } + +ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, + struct page ***pages, size_t maxsize, + size_t *start) +{ + ssize_t len; + + *pages = NULL; + + len = __iov_iter_get_pages_alloc(i, pages, maxsize, start); + if (len <= 0) { + kvfree(*pages); + *pages = NULL; + } + return len; +} EXPORT_SYMBOL(iov_iter_get_pages_alloc); size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, From patchwork Wed Jun 22 04:15:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890105 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99AFCC43334 for ; Wed, 22 Jun 2022 04:16:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356970AbiFVEQr (ORCPT ); Wed, 22 Jun 2022 00:16:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48640 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356779AbiFVEQA (ORCPT ); Wed, 22 Jun 2022 00:16:00 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35DDB10C4 for ; Tue, 21 Jun 2022 21:15:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=FxLpAQp33qMHhiVHeDBym+xIzy9dgSEuFPNXXyxK8aM=; b=GAUvgMWyrnSJpgz6iF78J6Q+eL ENx1Bdcbf0viXEVDO83Y8js2HgNmdSZ2/ruguGl5bt0mk8cA5hyf9ZUpVdxFTNvQAbwjLLrkpLb9x P2VuS0UTCZU3PuGEEpNqf6i5AoaGJpth0wpq+KLQkTXqw7cCZZ6MhwPx0rXsQVWvV97Wi54y2S/HU blvir2gY0XJufRvCbNyy3ML5ed2U9SMxZkbn3F4nR/NW35SMhQq2Xn2mQGMtZmvyJnXWrNDVT01oU x7mtQ4CMnc2VgFOGKPs8kKhOO7mGHtdtc0TiwGz76cmyvgP87uv/d6NeHMc+0Je2O/uz7UMspKzak hDdOVGfA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmW-0035xv-MB; Wed, 22 Jun 2022 04:15:56 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 25/44] iov_iter_get_pages(): sanity-check arguments Date: Wed, 22 Jun 2022 05:15:33 +0100 Message-Id: <20220622041552.737754-25-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org zero maxpages is bogus, but best treated as "just return 0"; NULL pages, OTOH, should be treated as a hard bug. get rid of now completely useless checks in xarray_get_pages{,_alloc}(). Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 9c25661684c6..5c985cf2858e 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1271,9 +1271,6 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, size_t size = maxsize; loff_t pos; - if (!size || !maxpages) - return 0; - pos = i->xarray_start + i->iov_offset; index = pos >> PAGE_SHIFT; offset = pos & ~PAGE_MASK; @@ -1365,10 +1362,11 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, if (maxsize > i->count) maxsize = i->count; - if (!maxsize) + if (!maxsize || !maxpages) return 0; if (maxsize > MAX_RW_COUNT) maxsize = MAX_RW_COUNT; + BUG_ON(!pages); if (likely(user_backed_iter(i))) { unsigned int gup_flags = 0; @@ -1441,9 +1439,6 @@ static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i, size_t size = maxsize; loff_t pos; - if (!size) - return 0; - pos = i->xarray_start + i->iov_offset; index = pos >> PAGE_SHIFT; offset = pos & ~PAGE_MASK; From patchwork Wed Jun 22 04:15:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81327C433EF for ; Wed, 22 Jun 2022 04:16:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356911AbiFVEQh (ORCPT ); Wed, 22 Jun 2022 00:16:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356790AbiFVEQI (ORCPT ); Wed, 22 Jun 2022 00:16:08 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 364BB64F2 for ; Tue, 21 Jun 2022 21:15:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=JV0OnyBdMZZfXf+wleA06sFMCnIGJlM1+pK3Xfzmzao=; b=m3c8SzUOemipZk9cmrmGAqn9Br LrXmxxEnQBk7843okcy28/n9gLYhMqmbaGTVei/o6VHo4qdR+dDBmGad0YwNowgMclNVFqGXzlZP4 BUOn99ejpEi60Gu/LAfc/2YVXpVXwBzP9uelNFV6ZZaaoDh0YS48pCdn9qwZkTqvDl/R5eFV0XVUG qAn8TFTIyHOE3tPSXI0qwjTSMe2spiHH00tXic428JLw4qvz/cszzR1LDKhx9pEdOR8fDtjy9OWyh Pt9BcbOcp7NpV9BG3DQdibwTFRzcWZCQw1OgozC1bhRXcm89N7lFrZB6yi6JAz/vYHj8zmOqRFfat YitQWodw==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmW-0035y3-UQ; Wed, 22 Jun 2022 04:15:57 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 26/44] unify pipe_get_pages() and pipe_get_pages_alloc() Date: Wed, 22 Jun 2022 05:15:34 +0100 Message-Id: <20220622041552.737754-26-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The differences between those two are * pipe_get_pages() gets a non-NULL struct page ** value pointing to preallocated array + array size. * pipe_get_pages_alloc() gets an address of struct page ** variable that contains NULL, allocates the array and (on success) stores its address in that variable. Not hard to combine - always pass struct page ***, have the previous pipe_get_pages_alloc() caller pass ~0U as cap for array size. Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 49 +++++++++++++++++-------------------------------- 1 file changed, 17 insertions(+), 32 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 5c985cf2858e..1c98f2f3a581 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1187,6 +1187,11 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i) } EXPORT_SYMBOL(iov_iter_gap_alignment); +static struct page **get_pages_array(size_t n) +{ + return kvmalloc_array(n, sizeof(struct page *), GFP_KERNEL); +} + static inline ssize_t __pipe_get_pages(struct iov_iter *i, size_t maxsize, struct page **pages, @@ -1220,10 +1225,11 @@ static inline ssize_t __pipe_get_pages(struct iov_iter *i, } static ssize_t pipe_get_pages(struct iov_iter *i, - struct page **pages, size_t maxsize, unsigned maxpages, + struct page ***pages, size_t maxsize, unsigned maxpages, size_t *start) { unsigned int npages, off; + struct page **p; size_t capacity; if (!sanity(i)) @@ -1231,8 +1237,15 @@ static ssize_t pipe_get_pages(struct iov_iter *i, *start = off = pipe_npages(i, &npages); capacity = min(npages, maxpages) * PAGE_SIZE - off; + maxsize = min(maxsize, capacity); + p = *pages; + if (!p) { + *pages = p = get_pages_array(DIV_ROUND_UP(maxsize + off, PAGE_SIZE)); + if (!p) + return -ENOMEM; + } - return __pipe_get_pages(i, min(maxsize, capacity), pages, off); + return __pipe_get_pages(i, maxsize, p, off); } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa, @@ -1394,41 +1407,13 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, return len - *start; } if (iov_iter_is_pipe(i)) - return pipe_get_pages(i, pages, maxsize, maxpages, start); + return pipe_get_pages(i, &pages, maxsize, maxpages, start); if (iov_iter_is_xarray(i)) return iter_xarray_get_pages(i, pages, maxsize, maxpages, start); return -EFAULT; } EXPORT_SYMBOL(iov_iter_get_pages); -static struct page **get_pages_array(size_t n) -{ - return kvmalloc_array(n, sizeof(struct page *), GFP_KERNEL); -} - -static ssize_t pipe_get_pages_alloc(struct iov_iter *i, - struct page ***pages, size_t maxsize, - size_t *start) -{ - struct page **p; - unsigned int npages, off; - ssize_t n; - - if (!sanity(i)) - return -EFAULT; - - *start = off = pipe_npages(i, &npages); - n = npages * PAGE_SIZE - off; - if (maxsize > n) - maxsize = n; - else - npages = DIV_ROUND_UP(maxsize + off, PAGE_SIZE); - *pages = p = get_pages_array(npages); - if (!p) - return -ENOMEM; - return __pipe_get_pages(i, maxsize, p, off); -} - static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *_start_offset) @@ -1511,7 +1496,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, return len - *start; } if (iov_iter_is_pipe(i)) - return pipe_get_pages_alloc(i, pages, maxsize, start); + return pipe_get_pages(i, pages, maxsize, ~0U, start); if (iov_iter_is_xarray(i)) return iter_xarray_get_pages_alloc(i, pages, maxsize, start); return -EFAULT; From patchwork Wed Jun 22 04:15:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AA95C43334 for ; Wed, 22 Jun 2022 04:16:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356939AbiFVEQo (ORCPT ); Wed, 22 Jun 2022 00:16:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356796AbiFVEQI (ORCPT ); Wed, 22 Jun 2022 00:16:08 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E22D764C for ; Tue, 21 Jun 2022 21:15:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=t+ZhgCyfjPqW81eYAJou5jvuXDlrzrNGX9oqhs/L/dg=; b=KzSVaG30mWIDlNSlGlbZ5+X3MW L1i8B0Z3UUaPvb3cmLzJbpAve9c8zqYRHkerL1A4z2vm9wL9XDfRmT6xgE30/BC6BZUKiqDhPT/02 4GLMpx9b4sI3+KSqTLaFBW7Z3dFU5kp3HsMu+qKVr8LaZvh/PM4cnPL8KaAP8Iqk2BMLCdEMxP1rM YjWMNfo/3UkuNUhhnpQ7EpRhRNLATpPmbs0MYcZVW+Ju3AEtSkEinziLZf4lRnrYVzF3TnGyA7rp0 p9J1zKGmAtJLWXBEmKszRkgNjWVRZRkEqcCLAlMCAQXdPv7t+fcuDAPyccP61Oh2UcUzRLDgLAW8z is5uhj1g==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmX-0035yA-3t; Wed, 22 Jun 2022 04:15:57 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 27/44] unify xarray_get_pages() and xarray_get_pages_alloc() Date: Wed, 22 Jun 2022 05:15:35 +0100 Message-Id: <20220622041552.737754-27-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org same as for pipes Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 49 ++++++++++--------------------------------------- 1 file changed, 10 insertions(+), 39 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 1c98f2f3a581..07dacb274ba5 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1276,7 +1276,7 @@ static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa } static ssize_t iter_xarray_get_pages(struct iov_iter *i, - struct page **pages, size_t maxsize, + struct page ***pages, size_t maxsize, unsigned maxpages, size_t *_start_offset) { unsigned nr, offset; @@ -1301,7 +1301,13 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, if (count > maxpages) count = maxpages; - nr = iter_xarray_populate_pages(pages, i->xarray, index, count); + if (!*pages) { + *pages = get_pages_array(count); + if (!*pages) + return -ENOMEM; + } + + nr = iter_xarray_populate_pages(*pages, i->xarray, index, count); if (nr == 0) return 0; @@ -1409,46 +1415,11 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, if (iov_iter_is_pipe(i)) return pipe_get_pages(i, &pages, maxsize, maxpages, start); if (iov_iter_is_xarray(i)) - return iter_xarray_get_pages(i, pages, maxsize, maxpages, start); + return iter_xarray_get_pages(i, &pages, maxsize, maxpages, start); return -EFAULT; } EXPORT_SYMBOL(iov_iter_get_pages); -static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i, - struct page ***pages, size_t maxsize, - size_t *_start_offset) -{ - struct page **p; - unsigned nr, offset; - pgoff_t index, count; - size_t size = maxsize; - loff_t pos; - - pos = i->xarray_start + i->iov_offset; - index = pos >> PAGE_SHIFT; - offset = pos & ~PAGE_MASK; - *_start_offset = offset; - - count = 1; - if (size > PAGE_SIZE - offset) { - size -= PAGE_SIZE - offset; - count += size >> PAGE_SHIFT; - size &= ~PAGE_MASK; - if (size) - count++; - } - - *pages = p = get_pages_array(count); - if (!p) - return -ENOMEM; - - nr = iter_xarray_populate_pages(p, i->xarray, index, count); - if (nr == 0) - return 0; - - return min_t(size_t, nr * PAGE_SIZE - offset, maxsize); -} - static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start) @@ -1498,7 +1469,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, if (iov_iter_is_pipe(i)) return pipe_get_pages(i, pages, maxsize, ~0U, start); if (iov_iter_is_xarray(i)) - return iter_xarray_get_pages_alloc(i, pages, maxsize, start); + return iter_xarray_get_pages(i, pages, maxsize, ~0U, start); return -EFAULT; } From patchwork Wed Jun 22 04:15:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890098 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C2EACCA47A for ; Wed, 22 Jun 2022 04:16:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356913AbiFVEQi (ORCPT ); Wed, 22 Jun 2022 00:16:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356687AbiFVEQI (ORCPT ); Wed, 22 Jun 2022 00:16:08 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E1B86582 for ; Tue, 21 Jun 2022 21:15:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=VH22792t7ZdKST0qLwcvc1oYMXr0Krbc1eewtKWvlyw=; b=wuSVrjO6ntzs5hFs7Xwz+2zObK xRSp/1KIMP7/AA2SMRXlRCcO36MlKIlNls9L9Q34cAPLzwM88sFT8jp7/UrHvSr220OMEBoosfDeh jyissQN7EmUxeB5PA7gA3Wpvo0mXDDy61xKVwhSzUDsYDbq3e/l/IiGnVn4XE124cD66ex9zpRRCe b+pkt4diSWYWFDdgRZI+H8jSoPlx2Zg+BFxy1+2C8qcIop/L8O4LdO5FIXcaW5Znov7GenZcUb23p SRiiUB2oTN/SYa8CBAvfOgYBJTdkLRHiFchVyct+7I8KNTgZNFxSoqFMqn4mNNPSwf03TWd4AtBWW S2k2vEmA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmX-0035yI-9B; Wed, 22 Jun 2022 04:15:57 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 28/44] unify the rest of iov_iter_get_pages()/iov_iter_get_pages_alloc() guts Date: Wed, 22 Jun 2022 05:15:36 +0100 Message-Id: <20220622041552.737754-28-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org same as for pipes and xarrays; after that iov_iter_get_pages() becomes a wrapper for __iov_iter_get_pages_alloc(). Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 86 ++++++++++++++++---------------------------------- 1 file changed, 28 insertions(+), 58 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 07dacb274ba5..811fa09515d8 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1372,20 +1372,19 @@ static struct page *first_bvec_segment(const struct iov_iter *i, return page; } -ssize_t iov_iter_get_pages(struct iov_iter *i, - struct page **pages, size_t maxsize, unsigned maxpages, - size_t *start) +static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, + struct page ***pages, size_t maxsize, + unsigned int maxpages, size_t *start) { size_t len; int n, res; if (maxsize > i->count) maxsize = i->count; - if (!maxsize || !maxpages) + if (!maxsize) return 0; if (maxsize > MAX_RW_COUNT) maxsize = MAX_RW_COUNT; - BUG_ON(!pages); if (likely(user_backed_iter(i))) { unsigned int gup_flags = 0; @@ -1398,80 +1397,51 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, addr = first_iovec_segment(i, &len, start, maxsize, maxpages); n = DIV_ROUND_UP(len, PAGE_SIZE); - res = get_user_pages_fast(addr, n, gup_flags, pages); + if (!*pages) { + *pages = get_pages_array(n); + if (!*pages) + return -ENOMEM; + } + res = get_user_pages_fast(addr, n, gup_flags, *pages); if (unlikely(res <= 0)) return res; return (res == n ? len : res * PAGE_SIZE) - *start; } if (iov_iter_is_bvec(i)) { + struct page **p; struct page *page; page = first_bvec_segment(i, &len, start, maxsize, maxpages); n = DIV_ROUND_UP(len, PAGE_SIZE); + p = *pages; + if (!p) { + *pages = p = get_pages_array(n); + if (!p) + return -ENOMEM; + } while (n--) - get_page(*pages++ = page++); + get_page(*p++ = page++); return len - *start; } if (iov_iter_is_pipe(i)) - return pipe_get_pages(i, &pages, maxsize, maxpages, start); + return pipe_get_pages(i, pages, maxsize, maxpages, start); if (iov_iter_is_xarray(i)) - return iter_xarray_get_pages(i, &pages, maxsize, maxpages, start); + return iter_xarray_get_pages(i, pages, maxsize, maxpages, + start); return -EFAULT; } -EXPORT_SYMBOL(iov_iter_get_pages); -static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, - struct page ***pages, size_t maxsize, +ssize_t iov_iter_get_pages(struct iov_iter *i, + struct page **pages, size_t maxsize, unsigned maxpages, size_t *start) { - struct page **p; - size_t len; - int n, res; - - if (maxsize > i->count) - maxsize = i->count; - if (!maxsize) + if (!maxpages) return 0; - if (maxsize > MAX_RW_COUNT) - maxsize = MAX_RW_COUNT; - - if (likely(user_backed_iter(i))) { - unsigned int gup_flags = 0; - unsigned long addr; - - if (iov_iter_rw(i) != WRITE) - gup_flags |= FOLL_WRITE; - if (i->nofault) - gup_flags |= FOLL_NOFAULT; - - addr = first_iovec_segment(i, &len, start, maxsize, ~0U); - n = DIV_ROUND_UP(len, PAGE_SIZE); - *pages = p = get_pages_array(n); - if (!p) - return -ENOMEM; - res = get_user_pages_fast(addr, n, gup_flags, p); - if (unlikely(res <= 0)) - return res; - return (res == n ? len : res * PAGE_SIZE) - *start; - } - if (iov_iter_is_bvec(i)) { - struct page *page; + BUG_ON(!pages); - page = first_bvec_segment(i, &len, start, maxsize, ~0U); - n = DIV_ROUND_UP(len, PAGE_SIZE); - *pages = p = get_pages_array(n); - if (!p) - return -ENOMEM; - while (n--) - get_page(*p++ = page++); - return len - *start; - } - if (iov_iter_is_pipe(i)) - return pipe_get_pages(i, pages, maxsize, ~0U, start); - if (iov_iter_is_xarray(i)) - return iter_xarray_get_pages(i, pages, maxsize, ~0U, start); - return -EFAULT; + return __iov_iter_get_pages_alloc(i, &pages, maxsize, maxpages, start); } +EXPORT_SYMBOL(iov_iter_get_pages); ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, @@ -1481,7 +1451,7 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, *pages = NULL; - len = __iov_iter_get_pages_alloc(i, pages, maxsize, start); + len = __iov_iter_get_pages_alloc(i, pages, maxsize, ~0U, start); if (len <= 0) { kvfree(*pages); *pages = NULL; From patchwork Wed Jun 22 04:15:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890106 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 918CEC43334 for ; Wed, 22 Jun 2022 04:16:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356973AbiFVEQs (ORCPT ); Wed, 22 Jun 2022 00:16:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48596 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356766AbiFVEQA (ORCPT ); Wed, 22 Jun 2022 00:16:00 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E05C6565 for ; Tue, 21 Jun 2022 21:15:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=E8nxIBtz4fKhoonh4VerKpPAu/2XUoouy95LORnjtug=; b=UkzwhRuvLPguJ1zKWS5P+8tkAN c3AyUjTZiFEgUtuOx7Ag9yalu5I0S5q8UjTHz12len+7E8nGOJygeAD+5sR9KZ62wFS93fmqRxf3u I9h4wbe9bZMvEHDIASuMEVF2ieTElLYB38os5JjkQPFokM74lI7pIrnr/aPAdvzpCDH0TFmBDB+ez ndaRs+f0FjOpxsLMKgM/FzAIsJhYoWtBaNuAV0RtR6BfKJImA7f+l56AU1m7OdenCF5Aj9SFtLujj 2n20ul+M503/LW+rHy2hFJXGt09BjG7Uqi1JpV2e8glKEEJtXB2ql8BdmXjlddWtgRSLSONn8XpVF yvyke8Gg==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmX-0035yQ-GS; Wed, 22 Jun 2022 04:15:57 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 29/44] ITER_XARRAY: don't open-code DIV_ROUND_UP() Date: Wed, 22 Jun 2022 05:15:37 +0100 Message-Id: <20220622041552.737754-29-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 811fa09515d8..92a566f839f9 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1289,15 +1289,7 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, offset = pos & ~PAGE_MASK; *_start_offset = offset; - count = 1; - if (size > PAGE_SIZE - offset) { - size -= PAGE_SIZE - offset; - count += size >> PAGE_SHIFT; - size &= ~PAGE_MASK; - if (size) - count++; - } - + count = DIV_ROUND_UP(size + offset, PAGE_SIZE); if (count > maxpages) count = maxpages; From patchwork Wed Jun 22 04:15:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E963C43334 for ; Wed, 22 Jun 2022 04:16:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356916AbiFVEQj (ORCPT ); Wed, 22 Jun 2022 00:16:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356799AbiFVEQI (ORCPT ); Wed, 22 Jun 2022 00:16:08 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 784AA65B7 for ; Tue, 21 Jun 2022 21:15:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=YOpUgORcpbiNEH9/73ZqIGTUJ1PVoFu8RHS9BVtmW1Y=; b=rdI5/mfYCInMp+FFE6D9DAmAFW d/2Lp7JQgJMS5j1lBWE7kxUXqY7Fy0kmwV1l51rVWxG0br6e2yA2NlGmNwT2ACsxhrqx/F1yZ6CJ5 9Ny4nXo2y0xJOsSwmmNxUDL6r16lcd82GN7LAtz9bh1TyZ8t9gYj0+wiQWOPkS9OBv9F/NVxpDDjP hvPVTMU4E/DynGQNgTz7HNTqYHBmqMHQcZItbZzaX/Gy6cJyL+JF2H0z54SHPNLrZJsdypuIQGv8l D3qiDY7NNTNA5NbHA9xgZPGcmj5yh2dGH/YVVE7uR/pJqS4ecmFgJJVwuWNe9VaRKPLWOdP+Y/BM1 aCozWJlA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmX-0035yX-Lw; Wed, 22 Jun 2022 04:15:57 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 30/44] iov_iter: lift dealing with maxpages out of first_{iovec,bvec}_segment() Date: Wed, 22 Jun 2022 05:15:38 +0100 Message-Id: <20220622041552.737754-30-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 92a566f839f9..9ef671b101dc 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1308,12 +1308,9 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, static unsigned long found_ubuf_segment(unsigned long addr, size_t len, - size_t *size, size_t *start, - unsigned maxpages) + size_t *size, size_t *start) { len += (*start = addr % PAGE_SIZE); - if (len > maxpages * PAGE_SIZE) - len = maxpages * PAGE_SIZE; *size = len; return addr & PAGE_MASK; } @@ -1321,14 +1318,14 @@ static unsigned long found_ubuf_segment(unsigned long addr, /* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ static unsigned long first_iovec_segment(const struct iov_iter *i, size_t *size, size_t *start, - size_t maxsize, unsigned maxpages) + size_t maxsize) { size_t skip; long k; if (iter_is_ubuf(i)) { unsigned long addr = (unsigned long)i->ubuf + i->iov_offset; - return found_ubuf_segment(addr, maxsize, size, start, maxpages); + return found_ubuf_segment(addr, maxsize, size, start); } for (k = 0, skip = i->iov_offset; k < i->nr_segs; k++, skip = 0) { @@ -1339,7 +1336,7 @@ static unsigned long first_iovec_segment(const struct iov_iter *i, continue; if (len > maxsize) len = maxsize; - return found_ubuf_segment(addr, len, size, start, maxpages); + return found_ubuf_segment(addr, len, size, start); } BUG(); // if it had been empty, we wouldn't get called } @@ -1347,7 +1344,7 @@ static unsigned long first_iovec_segment(const struct iov_iter *i, /* must be done on non-empty ITER_BVEC one */ static struct page *first_bvec_segment(const struct iov_iter *i, size_t *size, size_t *start, - size_t maxsize, unsigned maxpages) + size_t maxsize) { struct page *page; size_t skip = i->iov_offset, len; @@ -1358,8 +1355,6 @@ static struct page *first_bvec_segment(const struct iov_iter *i, skip += i->bvec->bv_offset; page = i->bvec->bv_page + skip / PAGE_SIZE; len += (*start = skip % PAGE_SIZE); - if (len > maxpages * PAGE_SIZE) - len = maxpages * PAGE_SIZE; *size = len; return page; } @@ -1387,7 +1382,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, if (i->nofault) gup_flags |= FOLL_NOFAULT; - addr = first_iovec_segment(i, &len, start, maxsize, maxpages); + addr = first_iovec_segment(i, &len, start, maxsize); + if (len > maxpages * PAGE_SIZE) + len = maxpages * PAGE_SIZE; n = DIV_ROUND_UP(len, PAGE_SIZE); if (!*pages) { *pages = get_pages_array(n); @@ -1403,7 +1400,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page **p; struct page *page; - page = first_bvec_segment(i, &len, start, maxsize, maxpages); + page = first_bvec_segment(i, &len, start, maxsize); + if (len > maxpages * PAGE_SIZE) + len = maxpages * PAGE_SIZE; n = DIV_ROUND_UP(len, PAGE_SIZE); p = *pages; if (!p) { From patchwork Wed Jun 22 04:15:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57DE3C43334 for ; Wed, 22 Jun 2022 04:16:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356975AbiFVEQv (ORCPT ); Wed, 22 Jun 2022 00:16:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356822AbiFVEQI (ORCPT ); Wed, 22 Jun 2022 00:16:08 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CC769FEC for ; Tue, 21 Jun 2022 21:15:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=7IzDSm8tRO3+5esG+C7nReWEXcnk7GKepoB+cwWuyTE=; b=gIU4wl2LI7qwYqjplX6Bb+hzdE jctKsyyl9M4lzRd8Qr8sucsJswj/aQqLK6sGOWFuJnhqZMXK7Sa8GWreIXEH3k78vx5ny8+M8Ju5N Ie5f+U0x+MU/oixRKpjgLE2mgBOMlEFidZ3aix6VCEv8releOJEYL1DIw+GR/yUCsMsCTTwl0ONTP Kdhadpx2+CiC2+xUBpWvZXKNNbr7H9I9htS+8GOipwWaeBc0CPngJh5cpMI9AkF9u+FEVxv9bLIxV 7tc4MzkzSAJvO1ZlKPFCoPDq/0mpbXNLd/d30RoR8MKEcY5e7Tlj7f3pFdL2Rkys8GXxrJABvbfzP gvVZbsbw==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmX-0035yg-Vj; Wed, 22 Jun 2022 04:15:58 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 31/44] iov_iter: first_{iovec,bvec}_segment() - simplify a bit Date: Wed, 22 Jun 2022 05:15:39 +0100 Message-Id: <20220622041552.737754-31-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org We return length + offset in page via *size. Don't bother - the caller can do that arithmetics just as well; just report the length to it. Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 9ef671b101dc..0bed684d91d0 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1310,7 +1310,7 @@ static unsigned long found_ubuf_segment(unsigned long addr, size_t len, size_t *size, size_t *start) { - len += (*start = addr % PAGE_SIZE); + *start = addr % PAGE_SIZE; *size = len; return addr & PAGE_MASK; } @@ -1354,7 +1354,7 @@ static struct page *first_bvec_segment(const struct iov_iter *i, len = maxsize; skip += i->bvec->bv_offset; page = i->bvec->bv_page + skip / PAGE_SIZE; - len += (*start = skip % PAGE_SIZE); + *start = skip % PAGE_SIZE; *size = len; return page; } @@ -1383,9 +1383,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, gup_flags |= FOLL_NOFAULT; addr = first_iovec_segment(i, &len, start, maxsize); - if (len > maxpages * PAGE_SIZE) - len = maxpages * PAGE_SIZE; - n = DIV_ROUND_UP(len, PAGE_SIZE); + n = DIV_ROUND_UP(len + *start, PAGE_SIZE); + if (n > maxpages) + n = maxpages; if (!*pages) { *pages = get_pages_array(n); if (!*pages) @@ -1394,25 +1394,25 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, res = get_user_pages_fast(addr, n, gup_flags, *pages); if (unlikely(res <= 0)) return res; - return (res == n ? len : res * PAGE_SIZE) - *start; + return min_t(size_t, len, res * PAGE_SIZE - *start); } if (iov_iter_is_bvec(i)) { struct page **p; struct page *page; page = first_bvec_segment(i, &len, start, maxsize); - if (len > maxpages * PAGE_SIZE) - len = maxpages * PAGE_SIZE; - n = DIV_ROUND_UP(len, PAGE_SIZE); + n = DIV_ROUND_UP(len + *start, PAGE_SIZE); + if (n > maxpages) + n = maxpages; p = *pages; if (!p) { *pages = p = get_pages_array(n); if (!p) return -ENOMEM; } - while (n--) + for (int k = 0; k < n; k++) get_page(*p++ = page++); - return len - *start; + return min_t(size_t, len, n * PAGE_SIZE - *start); } if (iov_iter_is_pipe(i)) return pipe_get_pages(i, pages, maxsize, maxpages, start); From patchwork Wed Jun 22 04:15:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890100 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 189A0C433EF for ; Wed, 22 Jun 2022 04:16:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356920AbiFVEQk (ORCPT ); Wed, 22 Jun 2022 00:16:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356821AbiFVEQI (ORCPT ); Wed, 22 Jun 2022 00:16:08 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01BAEA469 for ; Tue, 21 Jun 2022 21:15:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=maVM+hWdRPpz8NIDpl66Gu92HL5HhGcpaGz+BHhMEqk=; b=KgvM9wL55NwD7arjz2377WijqV xH7/zzLOx+tW2S7imH4kmHOYt4c3aofvwcqbAD3RC1ywUaKdOK/DH6CPykpQ8GoVfCbmW5XXsHByX 4r9MWk8RpWQpYcjcROCydDcbXzrUPdMx8xWfGaWT7Fnpv9O4n1EvDbma820mLkpWy/ZqrONUK5xRx ccrzkjvjzsrR0HxrkDW1UWCQ3cYXGEr9GMirGutXSjnTy7A1HgwHvipr0ZqzLcK6Ums/zoXXDOLpn 2HxE7n6ZyUcKUox/mUfb+qSzonbQhXAhQfRKQqNHDr1RTC6I5TSy5pLuH3ZfvYfniCHmTg3Rjo+jX Xpc+CgKA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmY-0035yp-6V; Wed, 22 Jun 2022 04:15:58 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 32/44] iov_iter: massage calling conventions for first_{iovec,bvec}_segment() Date: Wed, 22 Jun 2022 05:15:40 +0100 Message-Id: <20220622041552.737754-32-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Pass maxsize by reference, return length via the same. Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 0bed684d91d0..fca66ecce7a0 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1306,26 +1306,22 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, return min_t(size_t, nr * PAGE_SIZE - offset, maxsize); } -static unsigned long found_ubuf_segment(unsigned long addr, - size_t len, - size_t *size, size_t *start) +static unsigned long found_ubuf_segment(unsigned long addr, size_t *start) { *start = addr % PAGE_SIZE; - *size = len; return addr & PAGE_MASK; } /* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ static unsigned long first_iovec_segment(const struct iov_iter *i, - size_t *size, size_t *start, - size_t maxsize) + size_t *size, size_t *start) { size_t skip; long k; if (iter_is_ubuf(i)) { unsigned long addr = (unsigned long)i->ubuf + i->iov_offset; - return found_ubuf_segment(addr, maxsize, size, start); + return found_ubuf_segment(addr, start); } for (k = 0, skip = i->iov_offset; k < i->nr_segs; k++, skip = 0) { @@ -1334,28 +1330,26 @@ static unsigned long first_iovec_segment(const struct iov_iter *i, if (unlikely(!len)) continue; - if (len > maxsize) - len = maxsize; - return found_ubuf_segment(addr, len, size, start); + if (*size > len) + *size = len; + return found_ubuf_segment(addr, start); } BUG(); // if it had been empty, we wouldn't get called } /* must be done on non-empty ITER_BVEC one */ static struct page *first_bvec_segment(const struct iov_iter *i, - size_t *size, size_t *start, - size_t maxsize) + size_t *size, size_t *start) { struct page *page; size_t skip = i->iov_offset, len; len = i->bvec->bv_len - skip; - if (len > maxsize) - len = maxsize; + if (*size > len) + *size = len; skip += i->bvec->bv_offset; page = i->bvec->bv_page + skip / PAGE_SIZE; *start = skip % PAGE_SIZE; - *size = len; return page; } @@ -1363,7 +1357,6 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned int maxpages, size_t *start) { - size_t len; int n, res; if (maxsize > i->count) @@ -1382,8 +1375,8 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, if (i->nofault) gup_flags |= FOLL_NOFAULT; - addr = first_iovec_segment(i, &len, start, maxsize); - n = DIV_ROUND_UP(len + *start, PAGE_SIZE); + addr = first_iovec_segment(i, &maxsize, start); + n = DIV_ROUND_UP(maxsize + *start, PAGE_SIZE); if (n > maxpages) n = maxpages; if (!*pages) { @@ -1394,14 +1387,14 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, res = get_user_pages_fast(addr, n, gup_flags, *pages); if (unlikely(res <= 0)) return res; - return min_t(size_t, len, res * PAGE_SIZE - *start); + return min_t(size_t, maxsize, res * PAGE_SIZE - *start); } if (iov_iter_is_bvec(i)) { struct page **p; struct page *page; - page = first_bvec_segment(i, &len, start, maxsize); - n = DIV_ROUND_UP(len + *start, PAGE_SIZE); + page = first_bvec_segment(i, &maxsize, start); + n = DIV_ROUND_UP(maxsize + *start, PAGE_SIZE); if (n > maxpages) n = maxpages; p = *pages; @@ -1412,7 +1405,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, } for (int k = 0; k < n; k++) get_page(*p++ = page++); - return min_t(size_t, len, n * PAGE_SIZE - *start); + return min_t(size_t, maxsize, n * PAGE_SIZE - *start); } if (iov_iter_is_pipe(i)) return pipe_get_pages(i, pages, maxsize, maxpages, start); From patchwork Wed Jun 22 04:15:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14BFEC43334 for ; Wed, 22 Jun 2022 04:16:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356922AbiFVEQl (ORCPT ); Wed, 22 Jun 2022 00:16:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48786 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356835AbiFVEQI (ORCPT ); Wed, 22 Jun 2022 00:16:08 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B78EB1C0 for ; Tue, 21 Jun 2022 21:15:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=6mqxTMoe5FWZMpdVAJw5cA30N8xjSVczJ7xJ4VOscU0=; b=Gckg4A67efp2WoqiO4wz7Ms6iy OCXLWHqB20bX3j3YH8TkiHwOVHadQ5MY5Yg67fHGKQhghX11RfZaGOUJS7v7ut2p1mQUoW5X2t65Q txuO3mtELr1Wsyo15MsrAUm9kNYvjcfE8zL2XQKxnBO5eRf38EVRRadCK4AlmSK4laRo6xjuaVy4j 0npBIJExuLp1zQPst6GxK4yKiqPAubAIVMZwuf7dQ2Cyybx/N2ZEma03vr0C9oJT23jxB+EHakMW3 Ylw0ljHApntfVWjOJc5S6MjgwGSjuiRojKUtZptlCvo7+pQugJk/ukdF1V/gYwqjo2jdj7kNBtHr2 Inv4jBiQ==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmY-0035yz-BN; Wed, 22 Jun 2022 04:15:58 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 33/44] found_iovec_segment(): just return address Date: Wed, 22 Jun 2022 05:15:41 +0100 Message-Id: <20220622041552.737754-33-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org ... and calculate the offset in the caller Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 22 +++++++--------------- 1 file changed, 7 insertions(+), 15 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index fca66ecce7a0..f455b8ee0d76 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1306,33 +1306,23 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, return min_t(size_t, nr * PAGE_SIZE - offset, maxsize); } -static unsigned long found_ubuf_segment(unsigned long addr, size_t *start) -{ - *start = addr % PAGE_SIZE; - return addr & PAGE_MASK; -} - /* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ static unsigned long first_iovec_segment(const struct iov_iter *i, - size_t *size, size_t *start) + size_t *size) { size_t skip; long k; - if (iter_is_ubuf(i)) { - unsigned long addr = (unsigned long)i->ubuf + i->iov_offset; - return found_ubuf_segment(addr, start); - } + if (iter_is_ubuf(i)) + return (unsigned long)i->ubuf + i->iov_offset; for (k = 0, skip = i->iov_offset; k < i->nr_segs; k++, skip = 0) { - unsigned long addr = (unsigned long)i->iov[k].iov_base + skip; size_t len = i->iov[k].iov_len - skip; - if (unlikely(!len)) continue; if (*size > len) *size = len; - return found_ubuf_segment(addr, start); + return (unsigned long)i->iov[k].iov_base + skip; } BUG(); // if it had been empty, we wouldn't get called } @@ -1375,7 +1365,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, if (i->nofault) gup_flags |= FOLL_NOFAULT; - addr = first_iovec_segment(i, &maxsize, start); + addr = first_iovec_segment(i, &maxsize); + *start = addr % PAGE_SIZE; + addr &= PAGE_MASK; n = DIV_ROUND_UP(maxsize + *start, PAGE_SIZE); if (n > maxpages) n = maxpages; From patchwork Wed Jun 22 04:15:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890102 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FD1ECCA47A for ; Wed, 22 Jun 2022 04:16:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356934AbiFVEQn (ORCPT ); Wed, 22 Jun 2022 00:16:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356818AbiFVEQI (ORCPT ); Wed, 22 Jun 2022 00:16:08 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B609AE75 for ; Tue, 21 Jun 2022 21:16:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=OEd90yREgJWELU2w7cWXJBVJE39zUZSsOryGxe4Q4OQ=; b=ibihlEmo6jUNN/KlWvPzuNjBhs hwbAvzzPlOIAdoKxWkkMlkdlkyHMnspJibZkLOThwWKh4pz9YEbhjSbRjOODT3tYk17Q9j43EG91n BwuUXbjoZ4Lgkd2U+XFNOeuzz2RxplTJDZOfirW8z6HCDeUftyBab51d3vXrweGMuDY2ZY8IYdDFV /bPsn2NFtlhNKBkv7LWUH9rKvwiJSuiuvgiKKSPSArREz0n05auOTLD2VWCnbDCMxIPJ9sqb4BNyA th0wuJhTgE2StZaOZARUrfWZNpaZwVqkKW5FbkZl6Mg4oHewY/EFx0ZZuuFeVETIzRRgv3l3F2jXv 0ZRqjBNQ==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmY-0035z3-FY; Wed, 22 Jun 2022 04:15:58 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 34/44] fold __pipe_get_pages() into pipe_get_pages() Date: Wed, 22 Jun 2022 05:15:42 +0100 Message-Id: <20220622041552.737754-34-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org ... and don't mangle maxsize there - turn the loop into counting one instead. Easier to see that we won't run out of array that way. Note that special treatment of the partial buffer in that thing is an artifact of the non-advancing semantics of iov_iter_get_pages() - if not for that, it would be append_pipe(), same as the body of the loop that follows it. IOW, once we make iov_iter_get_pages() advancing, the whole thing will turn into calculate how many pages do we want allocate an array (if needed) call append_pipe() that many times. Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 75 +++++++++++++++++++++++++------------------------- 1 file changed, 38 insertions(+), 37 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index f455b8ee0d76..9280f865fd6a 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1192,60 +1192,61 @@ static struct page **get_pages_array(size_t n) return kvmalloc_array(n, sizeof(struct page *), GFP_KERNEL); } -static inline ssize_t __pipe_get_pages(struct iov_iter *i, - size_t maxsize, - struct page **pages, - size_t off) -{ - struct pipe_inode_info *pipe = i->pipe; - ssize_t left = maxsize; - - if (off) { - struct pipe_buffer *buf = pipe_buf(pipe, pipe->head - 1); - - get_page(*pages++ = buf->page); - left -= PAGE_SIZE - off; - if (left <= 0) { - buf->len += maxsize; - return maxsize; - } - buf->len = PAGE_SIZE; - } - while (!pipe_full(pipe->head, pipe->tail, pipe->max_usage)) { - struct page *page = push_anon(pipe, - min_t(ssize_t, left, PAGE_SIZE)); - if (!page) - break; - get_page(*pages++ = page); - left -= PAGE_SIZE; - if (left <= 0) - return maxsize; - } - return maxsize - left ? : -EFAULT; -} - static ssize_t pipe_get_pages(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned maxpages, size_t *start) { + struct pipe_inode_info *pipe = i->pipe; unsigned int npages, off; struct page **p; - size_t capacity; + ssize_t left; + int count; if (!sanity(i)) return -EFAULT; *start = off = pipe_npages(i, &npages); - capacity = min(npages, maxpages) * PAGE_SIZE - off; - maxsize = min(maxsize, capacity); + count = DIV_ROUND_UP(maxsize + off, PAGE_SIZE); + if (count > npages) + count = npages; + if (count > maxpages) + count = maxpages; p = *pages; if (!p) { - *pages = p = get_pages_array(DIV_ROUND_UP(maxsize + off, PAGE_SIZE)); + *pages = p = get_pages_array(count); if (!p) return -ENOMEM; } - return __pipe_get_pages(i, maxsize, p, off); + left = maxsize; + npages = 0; + if (off) { + struct pipe_buffer *buf = pipe_buf(pipe, pipe->head - 1); + + get_page(*p++ = buf->page); + left -= PAGE_SIZE - off; + if (left <= 0) { + buf->len += maxsize; + return maxsize; + } + buf->len = PAGE_SIZE; + npages = 1; + } + for ( ; npages < count; npages++) { + struct page *page; + unsigned int size = min_t(ssize_t, left, PAGE_SIZE); + + if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) + break; + page = push_anon(pipe, size); + if (!page) + break; + get_page(*p++ = page); + left -= size; + } + if (!npages) + return -EFAULT; + return maxsize - left; } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa, From patchwork Wed Jun 22 04:15:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890110 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C33FC433EF for ; Wed, 22 Jun 2022 04:16:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356981AbiFVEQz (ORCPT ); Wed, 22 Jun 2022 00:16:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356836AbiFVEQI (ORCPT ); Wed, 22 Jun 2022 00:16:08 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B7C1B1E5 for ; Tue, 21 Jun 2022 21:16:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=gJcrpECXmawhpmZvu+Z8NVIT9Mj7s3PfGZqh6o7f49U=; b=h3ONxjxIJYt0nchSvzr+BU8xpD sencD4znlpBqB2ZhbYB5PuYHxu5oiv+DtPlTBSpLNtCq4Cmu+Ss+OBKhdzj5igeeOjAtLFpyfnRSN 17PvPvQiX7+h68EW7e+RLsEIhMIl3ZzCTZg3KAe5paIOxrV2k9nBiWj6ddG4RDfTyu+U1Tg9rYm28 bIbViG70TK7lvPVW55NVPR7dGuuSt6dM9zpFLyJP60yL8+LdntkoRNcjema4hhjnBak5/TVyXKpH3 8rqOWYwrgUS6uYB9XBuh2i1z+c2yKXgtOAvtQFwFjzFyImEAS8IsWEuM2c/2CrIkJNLpfCee6iTZH V5jPFDnw==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmY-0035zF-Mb; Wed, 22 Jun 2022 04:15:58 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 35/44] iov_iter: saner helper for page array allocation Date: Wed, 22 Jun 2022 05:15:43 +0100 Message-Id: <20220622041552.737754-35-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org All call sites of get_pages_array() are essenitally identical now. Replace with common helper... Returns number of slots available in resulting array or 0 on OOM; it's up to the caller to make sure it doesn't ask to zero-entry array (i.e. neither maxpages nor size are allowed to be zero). Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 77 +++++++++++++++++++++----------------------------- 1 file changed, 32 insertions(+), 45 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 9280f865fd6a..1c744f0c0b2c 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1187,9 +1187,20 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i) } EXPORT_SYMBOL(iov_iter_gap_alignment); -static struct page **get_pages_array(size_t n) +static int want_pages_array(struct page ***res, size_t size, + size_t start, unsigned int maxpages) { - return kvmalloc_array(n, sizeof(struct page *), GFP_KERNEL); + unsigned int count = DIV_ROUND_UP(size + start, PAGE_SIZE); + + if (count > maxpages) + count = maxpages; + WARN_ON(!count); // caller should've prevented that + if (!*res) { + *res = kvmalloc_array(count, sizeof(struct page *), GFP_KERNEL); + if (!*res) + return 0; + } + return count; } static ssize_t pipe_get_pages(struct iov_iter *i, @@ -1197,27 +1208,20 @@ static ssize_t pipe_get_pages(struct iov_iter *i, size_t *start) { struct pipe_inode_info *pipe = i->pipe; - unsigned int npages, off; + unsigned int npages, off, count; struct page **p; ssize_t left; - int count; if (!sanity(i)) return -EFAULT; *start = off = pipe_npages(i, &npages); - count = DIV_ROUND_UP(maxsize + off, PAGE_SIZE); - if (count > npages) - count = npages; - if (count > maxpages) - count = maxpages; + if (!npages) + return -EFAULT; + count = want_pages_array(pages, maxsize, off, min(npages, maxpages)); + if (!count) + return -ENOMEM; p = *pages; - if (!p) { - *pages = p = get_pages_array(count); - if (!p) - return -ENOMEM; - } - left = maxsize; npages = 0; if (off) { @@ -1280,9 +1284,8 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned maxpages, size_t *_start_offset) { - unsigned nr, offset; - pgoff_t index, count; - size_t size = maxsize; + unsigned nr, offset, count; + pgoff_t index; loff_t pos; pos = i->xarray_start + i->iov_offset; @@ -1290,16 +1293,9 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, offset = pos & ~PAGE_MASK; *_start_offset = offset; - count = DIV_ROUND_UP(size + offset, PAGE_SIZE); - if (count > maxpages) - count = maxpages; - - if (!*pages) { - *pages = get_pages_array(count); - if (!*pages) - return -ENOMEM; - } - + count = want_pages_array(pages, maxsize, offset, maxpages); + if (!count) + return -ENOMEM; nr = iter_xarray_populate_pages(*pages, i->xarray, index, count); if (nr == 0) return 0; @@ -1348,7 +1344,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned int maxpages, size_t *start) { - int n, res; + unsigned int n; if (maxsize > i->count) maxsize = i->count; @@ -1360,6 +1356,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, if (likely(user_backed_iter(i))) { unsigned int gup_flags = 0; unsigned long addr; + int res; if (iov_iter_rw(i) != WRITE) gup_flags |= FOLL_WRITE; @@ -1369,14 +1366,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, addr = first_iovec_segment(i, &maxsize); *start = addr % PAGE_SIZE; addr &= PAGE_MASK; - n = DIV_ROUND_UP(maxsize + *start, PAGE_SIZE); - if (n > maxpages) - n = maxpages; - if (!*pages) { - *pages = get_pages_array(n); - if (!*pages) - return -ENOMEM; - } + n = want_pages_array(pages, maxsize, *start, maxpages); + if (!n) + return -ENOMEM; res = get_user_pages_fast(addr, n, gup_flags, *pages); if (unlikely(res <= 0)) return res; @@ -1387,15 +1379,10 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, struct page *page; page = first_bvec_segment(i, &maxsize, start); - n = DIV_ROUND_UP(maxsize + *start, PAGE_SIZE); - if (n > maxpages) - n = maxpages; + n = want_pages_array(pages, maxsize, *start, maxpages); + if (!n) + return -ENOMEM; p = *pages; - if (!p) { - *pages = p = get_pages_array(n); - if (!p) - return -ENOMEM; - } for (int k = 0; k < n; k++) get_page(*p++ = page++); return min_t(size_t, maxsize, n * PAGE_SIZE - *start); From patchwork Wed Jun 22 04:15:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EBA3C43334 for ; Wed, 22 Jun 2022 04:17:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356897AbiFVEQ6 (ORCPT ); Wed, 22 Jun 2022 00:16:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356839AbiFVEQJ (ORCPT ); Wed, 22 Jun 2022 00:16:09 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4375B1EE for ; Tue, 21 Jun 2022 21:16:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=rJnvd3gxx0323F5XV/MaNwH0y9IWGe1FKJo6f70OOso=; b=Egta4K+ieLhb46OB33MSA+kDOH guVQg5pWDDZAe9Uu1m6igfnm0osy7YfRstvdYpiRQRAFT8/aORqUhF4r54SZUToOe9PlYIDPXF7lW vTTbUD/zSFbRevOtcv6l8SqUJI6/Tig30jRsDwOIhi/do4VZVHCsl1C6IR3GUd+4tTAoYJlgIOBv4 VIdo9f0argReiHlwezGftgqfCF+QAgKRyj0N4qxQBd3R8rXwZSJHxK2CajbtXgC0hZxYYNHrgcyne DUunS0yyKwsBLJNKaxZeNsq5AiXkhKPY1KlZxhLjPFeZl4fEy3Vj63BsK7Z4iugg7vUwUmlSP22AP Ttpwou+A==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmY-0035zK-QX; Wed, 22 Jun 2022 04:15:58 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 36/44] iov_iter: advancing variants of iov_iter_get_pages{,_alloc}() Date: Wed, 22 Jun 2022 05:15:44 +0100 Message-Id: <20220622041552.737754-36-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Most of the users immediately follow successful iov_iter_get_pages() with advancing by the amount it had returned. Provide inline wrappers doing that, convert trivial open-coded uses of those. BTW, iov_iter_get_pages() never returns more than it had been asked to; such checks in cifs ought to be removed someday... Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- drivers/vhost/scsi.c | 4 +--- fs/ceph/file.c | 3 +-- fs/cifs/file.c | 6 ++---- fs/cifs/misc.c | 3 +-- fs/direct-io.c | 3 +-- fs/fuse/dev.c | 3 +-- fs/fuse/file.c | 3 +-- fs/nfs/direct.c | 6 ++---- include/linux/uio.h | 20 ++++++++++++++++++++ net/core/datagram.c | 3 +-- net/core/skmsg.c | 3 +-- net/rds/message.c | 3 +-- net/tls/tls_sw.c | 4 +--- 13 files changed, 34 insertions(+), 30 deletions(-) diff --git a/drivers/vhost/scsi.c b/drivers/vhost/scsi.c index ffd9e6c2ffc1..9b65509424dc 100644 --- a/drivers/vhost/scsi.c +++ b/drivers/vhost/scsi.c @@ -643,14 +643,12 @@ vhost_scsi_map_to_sgl(struct vhost_scsi_cmd *cmd, size_t offset; unsigned int npages = 0; - bytes = iov_iter_get_pages(iter, pages, LONG_MAX, + bytes = iov_iter_get_pages2(iter, pages, LONG_MAX, VHOST_SCSI_PREALLOC_UPAGES, &offset); /* No pages were pinned */ if (bytes <= 0) return bytes < 0 ? bytes : -EFAULT; - iov_iter_advance(iter, bytes); - while (bytes) { unsigned n = min_t(unsigned, PAGE_SIZE - offset, bytes); sg_set_page(sg++, pages[npages++], n, offset); diff --git a/fs/ceph/file.c b/fs/ceph/file.c index c535de5852bf..8fab5db16c73 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -95,12 +95,11 @@ static ssize_t __iter_get_bvecs(struct iov_iter *iter, size_t maxsize, size_t start; int idx = 0; - bytes = iov_iter_get_pages(iter, pages, maxsize - size, + bytes = iov_iter_get_pages2(iter, pages, maxsize - size, ITER_GET_BVECS_PAGES, &start); if (bytes < 0) return size ?: bytes; - iov_iter_advance(iter, bytes); size += bytes; for ( ; bytes; idx++, bvec_idx++) { diff --git a/fs/cifs/file.c b/fs/cifs/file.c index e1e05b253daa..3ba013e2987f 100644 --- a/fs/cifs/file.c +++ b/fs/cifs/file.c @@ -3022,7 +3022,7 @@ cifs_write_from_iter(loff_t offset, size_t len, struct iov_iter *from, if (ctx->direct_io) { ssize_t result; - result = iov_iter_get_pages_alloc( + result = iov_iter_get_pages_alloc2( from, &pagevec, cur_len, &start); if (result < 0) { cifs_dbg(VFS, @@ -3036,7 +3036,6 @@ cifs_write_from_iter(loff_t offset, size_t len, struct iov_iter *from, break; } cur_len = (size_t)result; - iov_iter_advance(from, cur_len); nr_pages = (cur_len + start + PAGE_SIZE - 1) / PAGE_SIZE; @@ -3758,7 +3757,7 @@ cifs_send_async_read(loff_t offset, size_t len, struct cifsFileInfo *open_file, if (ctx->direct_io) { ssize_t result; - result = iov_iter_get_pages_alloc( + result = iov_iter_get_pages_alloc2( &direct_iov, &pagevec, cur_len, &start); if (result < 0) { @@ -3774,7 +3773,6 @@ cifs_send_async_read(loff_t offset, size_t len, struct cifsFileInfo *open_file, break; } cur_len = (size_t)result; - iov_iter_advance(&direct_iov, cur_len); rdata = cifs_readdata_direct_alloc( pagevec, cifs_uncached_readv_complete); diff --git a/fs/cifs/misc.c b/fs/cifs/misc.c index c69e1240d730..37493118fb72 100644 --- a/fs/cifs/misc.c +++ b/fs/cifs/misc.c @@ -1022,7 +1022,7 @@ setup_aio_ctx_iter(struct cifs_aio_ctx *ctx, struct iov_iter *iter, int rw) saved_len = count; while (count && npages < max_pages) { - rc = iov_iter_get_pages(iter, pages, count, max_pages, &start); + rc = iov_iter_get_pages2(iter, pages, count, max_pages, &start); if (rc < 0) { cifs_dbg(VFS, "Couldn't get user pages (rc=%zd)\n", rc); break; @@ -1034,7 +1034,6 @@ setup_aio_ctx_iter(struct cifs_aio_ctx *ctx, struct iov_iter *iter, int rw) break; } - iov_iter_advance(iter, rc); count -= rc; rc += start; cur_npages = DIV_ROUND_UP(rc, PAGE_SIZE); diff --git a/fs/direct-io.c b/fs/direct-io.c index 72237f49ad94..9724244f12ce 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -169,7 +169,7 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) { ssize_t ret; - ret = iov_iter_get_pages(sdio->iter, dio->pages, LONG_MAX, DIO_PAGES, + ret = iov_iter_get_pages2(sdio->iter, dio->pages, LONG_MAX, DIO_PAGES, &sdio->from); if (ret < 0 && sdio->blocks_available && (dio->op == REQ_OP_WRITE)) { @@ -191,7 +191,6 @@ static inline int dio_refill_pages(struct dio *dio, struct dio_submit *sdio) } if (ret >= 0) { - iov_iter_advance(sdio->iter, ret); ret += sdio->from; sdio->head = 0; sdio->tail = (ret + PAGE_SIZE - 1) / PAGE_SIZE; diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 8d657c2cd6f7..51897427a534 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -730,14 +730,13 @@ static int fuse_copy_fill(struct fuse_copy_state *cs) } } else { size_t off; - err = iov_iter_get_pages(cs->iter, &page, PAGE_SIZE, 1, &off); + err = iov_iter_get_pages2(cs->iter, &page, PAGE_SIZE, 1, &off); if (err < 0) return err; BUG_ON(!err); cs->len = err; cs->offset = off; cs->pg = page; - iov_iter_advance(cs->iter, err); } return lock_request(cs->req); diff --git a/fs/fuse/file.c b/fs/fuse/file.c index c982e3afe3b4..69e19fc0afc1 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -1401,14 +1401,13 @@ static int fuse_get_user_pages(struct fuse_args_pages *ap, struct iov_iter *ii, while (nbytes < *nbytesp && ap->num_pages < max_pages) { unsigned npages; size_t start; - ret = iov_iter_get_pages(ii, &ap->pages[ap->num_pages], + ret = iov_iter_get_pages2(ii, &ap->pages[ap->num_pages], *nbytesp - nbytes, max_pages - ap->num_pages, &start); if (ret < 0) break; - iov_iter_advance(ii, ret); nbytes += ret; ret += start; diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c index 022e1ce63e62..c275c83f0aef 100644 --- a/fs/nfs/direct.c +++ b/fs/nfs/direct.c @@ -364,13 +364,12 @@ static ssize_t nfs_direct_read_schedule_iovec(struct nfs_direct_req *dreq, size_t pgbase; unsigned npages, i; - result = iov_iter_get_pages_alloc(iter, &pagevec, + result = iov_iter_get_pages_alloc2(iter, &pagevec, rsize, &pgbase); if (result < 0) break; bytes = result; - iov_iter_advance(iter, bytes); npages = (result + pgbase + PAGE_SIZE - 1) / PAGE_SIZE; for (i = 0; i < npages; i++) { struct nfs_page *req; @@ -812,13 +811,12 @@ static ssize_t nfs_direct_write_schedule_iovec(struct nfs_direct_req *dreq, size_t pgbase; unsigned npages, i; - result = iov_iter_get_pages_alloc(iter, &pagevec, + result = iov_iter_get_pages_alloc2(iter, &pagevec, wsize, &pgbase); if (result < 0) break; bytes = result; - iov_iter_advance(iter, bytes); npages = (result + pgbase + PAGE_SIZE - 1) / PAGE_SIZE; for (i = 0; i < npages; i++) { struct nfs_page *req; diff --git a/include/linux/uio.h b/include/linux/uio.h index d3e13b37ea72..ab1cc218b9de 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -349,4 +349,24 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } +static inline ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages, + size_t maxsize, unsigned maxpages, size_t *start) +{ + ssize_t res = iov_iter_get_pages(i, pages, maxsize, maxpages, start); + + if (res >= 0) + iov_iter_advance(i, res); + return res; +} + +static inline ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, + size_t maxsize, size_t *start) +{ + ssize_t res = iov_iter_get_pages_alloc(i, pages, maxsize, start); + + if (res >= 0) + iov_iter_advance(i, res); + return res; +} + #endif diff --git a/net/core/datagram.c b/net/core/datagram.c index 50f4faeea76c..344b4c5791ac 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -629,12 +629,11 @@ int __zerocopy_sg_from_iter(struct sock *sk, struct sk_buff *skb, if (frag == MAX_SKB_FRAGS) return -EMSGSIZE; - copied = iov_iter_get_pages(from, pages, length, + copied = iov_iter_get_pages2(from, pages, length, MAX_SKB_FRAGS - frag, &start); if (copied < 0) return -EFAULT; - iov_iter_advance(from, copied); length -= copied; truesize = PAGE_ALIGN(copied + start); diff --git a/net/core/skmsg.c b/net/core/skmsg.c index 22b983ade0e7..662151678f20 100644 --- a/net/core/skmsg.c +++ b/net/core/skmsg.c @@ -324,14 +324,13 @@ int sk_msg_zerocopy_from_iter(struct sock *sk, struct iov_iter *from, goto out; } - copied = iov_iter_get_pages(from, pages, bytes, maxpages, + copied = iov_iter_get_pages2(from, pages, bytes, maxpages, &offset); if (copied <= 0) { ret = -EFAULT; goto out; } - iov_iter_advance(from, copied); bytes -= copied; msg->sg.size += copied; diff --git a/net/rds/message.c b/net/rds/message.c index 799034e0f513..d74be4e3f3fa 100644 --- a/net/rds/message.c +++ b/net/rds/message.c @@ -391,7 +391,7 @@ static int rds_message_zcopy_from_user(struct rds_message *rm, struct iov_iter * size_t start; ssize_t copied; - copied = iov_iter_get_pages(from, &pages, PAGE_SIZE, + copied = iov_iter_get_pages2(from, &pages, PAGE_SIZE, 1, &start); if (copied < 0) { struct mmpin *mmp; @@ -405,7 +405,6 @@ static int rds_message_zcopy_from_user(struct rds_message *rm, struct iov_iter * goto err; } total_copied += copied; - iov_iter_advance(from, copied); length -= copied; sg_set_page(sg, pages, copied, start); rm->data.op_nents++; diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 0513f82b8537..b1406c60f8df 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -1361,7 +1361,7 @@ static int tls_setup_from_iter(struct iov_iter *from, rc = -EFAULT; goto out; } - copied = iov_iter_get_pages(from, pages, + copied = iov_iter_get_pages2(from, pages, length, maxpages, &offset); if (copied <= 0) { @@ -1369,8 +1369,6 @@ static int tls_setup_from_iter(struct iov_iter *from, goto out; } - iov_iter_advance(from, copied); - length -= copied; size += copied; while (copied) { From patchwork Wed Jun 22 04:15:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890113 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61D1EC433EF for ; Wed, 22 Jun 2022 04:17:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356903AbiFVERB (ORCPT ); Wed, 22 Jun 2022 00:17:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48794 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356842AbiFVEQJ (ORCPT ); Wed, 22 Jun 2022 00:16:09 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C43C6B1FB for ; Tue, 21 Jun 2022 21:16:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=ji4s7IcChyTmimSd4SUpg7WzF88FA4cY7ZpcuIiuIos=; b=Vdk+9DjwMbYBdCzzkBdlV0tBi6 Yhv/76jtsznxbtU4qkZOzactwSaFSe7zmo45Q3vMR9McmsrKh8C9gizzg2qiFrDUF4Lmhogyr/Y4V uPy06mx1pHe1RnYNX1jXMZ26NgTI11H9sBOGNC7b4z7xA1H8zh93QoRffV+CW7gV56yg8ibmsv983 vMvDOUE74rEOa1DmumB+fw4lW7jXuT+mNmT1yQuTv6xj2DycXh2hOWpp9M3L5C8WobkGLLge2gJYh g2WcfiYm9lYmwRp/Vn6alGPCYqnmapiK1JISV0RatPhM8biFquPtOdipapOdQepSmDKc1ZFObu9gf YXYaPw9g==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmZ-0035zS-3q; Wed, 22 Jun 2022 04:15:59 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 37/44] block: convert to advancing variants of iov_iter_get_pages{,_alloc}() Date: Wed, 22 Jun 2022 05:15:45 +0100 Message-Id: <20220622041552.737754-37-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org ... doing revert if we end up not using some pages Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- block/bio.c | 15 ++++++--------- block/blk-map.c | 7 ++++--- 2 files changed, 10 insertions(+), 12 deletions(-) diff --git a/block/bio.c b/block/bio.c index 51c99f2c5c90..01ab683e67be 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1190,7 +1190,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2); pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); - size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset); + size = iov_iter_get_pages2(iter, pages, LONG_MAX, nr_pages, &offset); if (unlikely(size <= 0)) return size ? size : -EFAULT; @@ -1205,6 +1205,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) } else { if (WARN_ON_ONCE(bio_full(bio, len))) { bio_put_pages(pages + i, left, offset); + iov_iter_revert(iter, left); return -EINVAL; } __bio_add_page(bio, page, len, offset); @@ -1212,7 +1213,6 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) offset = 0; } - iov_iter_advance(iter, size); return 0; } @@ -1227,7 +1227,6 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) ssize_t size, left; unsigned len, i; size_t offset; - int ret = 0; if (WARN_ON_ONCE(!max_append_sectors)) return 0; @@ -1240,7 +1239,7 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2); pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); - size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset); + size = iov_iter_get_pages2(iter, pages, LONG_MAX, nr_pages, &offset); if (unlikely(size <= 0)) return size ? size : -EFAULT; @@ -1252,16 +1251,14 @@ static int __bio_iov_append_get_pages(struct bio *bio, struct iov_iter *iter) if (bio_add_hw_page(q, bio, page, len, offset, max_append_sectors, &same_page) != len) { bio_put_pages(pages + i, left, offset); - ret = -EINVAL; - break; + iov_iter_revert(iter, left); + return -EINVAL; } if (same_page) put_page(page); offset = 0; } - - iov_iter_advance(iter, size - left); - return ret; + return 0; } /** diff --git a/block/blk-map.c b/block/blk-map.c index df8b066cd548..7196a6b64c80 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -254,7 +254,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, size_t offs, added = 0; int npages; - bytes = iov_iter_get_pages_alloc(iter, &pages, LONG_MAX, &offs); + bytes = iov_iter_get_pages_alloc2(iter, &pages, LONG_MAX, &offs); if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; goto out_unmap; @@ -284,7 +284,6 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, bytes -= n; offs = 0; } - iov_iter_advance(iter, added); } /* * release the pages we didn't map into the bio, if any @@ -293,8 +292,10 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, put_page(pages[j++]); kvfree(pages); /* couldn't stuff something into bio? */ - if (bytes) + if (bytes) { + iov_iter_revert(iter, bytes); break; + } } ret = blk_rq_append_bio(rq, bio); From patchwork Wed Jun 22 04:15:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890114 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F24BC43334 for ; Wed, 22 Jun 2022 04:17:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356910AbiFVERC (ORCPT ); Wed, 22 Jun 2022 00:17:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48596 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356846AbiFVEQK (ORCPT ); Wed, 22 Jun 2022 00:16:10 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87453B7D9 for ; Tue, 21 Jun 2022 21:16:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=kBA4jm/ByDgMz+ovb1bPQYkTGj/XoEeGjXJPyhPI5/k=; b=aGivSm3YkaWm6Eof5gz4pXymlC xXx0WmbWyVOtURl5lNklDgQyzsMLj2592IMCplZ/oVXIC/HnpRmcFcmGsc2K9HejNsRCiNTZIUftW 3zOlrQFcw7t3djfQvChh9TD4WVnSxUyImREsJnPIGff7tMAvPIIdulXkTnzvYqFDNjA2eXGvrvmru Zh6KkcVKuGqRhsHM1XzLiDxHPsZ3r7ffWnBPn7Asu3dHVuylaSeKVKhIJs6uVQpNgbCDbfe1HHTcs +5SyiClk3lHErRJFKn6rHGAccuTCRUb4AnxMWMO9J63JzhD9hWfZyzlBsJc8x7vhG9ClxEgYTvtmI ByzMBIiA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmZ-0035zh-Ax; Wed, 22 Jun 2022 04:15:59 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 38/44] iter_to_pipe(): switch to advancing variant of iov_iter_get_pages() Date: Wed, 22 Jun 2022 05:15:46 +0100 Message-Id: <20220622041552.737754-38-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org ... and untangle the cleanup on failure to add into pipe. Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- fs/splice.c | 47 ++++++++++++++++++++++++----------------------- 1 file changed, 24 insertions(+), 23 deletions(-) diff --git a/fs/splice.c b/fs/splice.c index 6645b30ec990..9f84bd21f64c 100644 --- a/fs/splice.c +++ b/fs/splice.c @@ -1160,39 +1160,40 @@ static int iter_to_pipe(struct iov_iter *from, }; size_t total = 0; int ret = 0; - bool failed = false; - while (iov_iter_count(from) && !failed) { + while (iov_iter_count(from)) { struct page *pages[16]; - ssize_t copied; + ssize_t left; size_t start; - int n; + int i, n; - copied = iov_iter_get_pages(from, pages, ~0UL, 16, &start); - if (copied <= 0) { - ret = copied; + left = iov_iter_get_pages2(from, pages, ~0UL, 16, &start); + if (left <= 0) { + ret = left; break; } - for (n = 0; copied; n++, start = 0) { - int size = min_t(int, copied, PAGE_SIZE - start); - if (!failed) { - buf.page = pages[n]; - buf.offset = start; - buf.len = size; - ret = add_to_pipe(pipe, &buf); - if (unlikely(ret < 0)) { - failed = true; - } else { - iov_iter_advance(from, ret); - total += ret; - } - } else { - put_page(pages[n]); + n = DIV_ROUND_UP(left + start, PAGE_SIZE); + for (i = 0; i < n; i++) { + int size = min_t(int, left, PAGE_SIZE - start); + + buf.page = pages[i]; + buf.offset = start; + buf.len = size; + ret = add_to_pipe(pipe, &buf); + if (unlikely(ret < 0)) { + iov_iter_revert(from, left); + // this one got dropped by add_to_pipe() + while (++i < n) + put_page(pages[i]); + goto out; } - copied -= size; + total += ret; + left -= size; + start = 0; } } +out: return total ? total : ret; } From patchwork Wed Jun 22 04:15:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890118 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54C21C43334 for ; Wed, 22 Jun 2022 04:17:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356925AbiFVERK (ORCPT ); Wed, 22 Jun 2022 00:17:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356863AbiFVEQK (ORCPT ); Wed, 22 Jun 2022 00:16:10 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5B86B85E for ; Tue, 21 Jun 2022 21:16:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=BNnRC2cBRpd7fh9UVBlc5dtw8QKktwDKtprwB9veQ4M=; b=ZWIP+OTwKfATJ1kr96g3RRsE/i x/rJcPwWgA/EwDmW/OL500owX+ZqauBE0dLn71/pgRT33mfPt4ECro/neT7qKk8lY/942x4qNmm4X SQM0Svs9VFrTj3y5J98qBF+xo3o2iVc7YtsZ7DUVH2vEecjOJqxF4XqSiHh3wSQ53gIzIfnXdMDXj 3WgnCL8GGZXfZahNaFm87hTW3fmZxs7CAtIHXkuG4ED9Cuk/qp9r6wKV/bWDRaTosPFO5/+ZrDVlZ j3kRGwBQmlrcCOQVO6j9yVHw49NRYKf3HSyOeLNinOVL3xdLaRSHprpZVFQrJOohXvd/oAMLBOyu+ eGySC7mw==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmZ-0035zs-Nn; Wed, 22 Jun 2022 04:15:59 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 39/44] af_alg_make_sg(): switch to advancing variant of iov_iter_get_pages() Date: Wed, 22 Jun 2022 05:15:47 +0100 Message-Id: <20220622041552.737754-39-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org ... and adjust the callers Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- crypto/af_alg.c | 3 +-- crypto/algif_hash.c | 5 +++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/crypto/af_alg.c b/crypto/af_alg.c index c8289b7a85ba..e893c0f6c879 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -404,7 +404,7 @@ int af_alg_make_sg(struct af_alg_sgl *sgl, struct iov_iter *iter, int len) ssize_t n; int npages, i; - n = iov_iter_get_pages(iter, sgl->pages, len, ALG_MAX_PAGES, &off); + n = iov_iter_get_pages2(iter, sgl->pages, len, ALG_MAX_PAGES, &off); if (n < 0) return n; @@ -1191,7 +1191,6 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, len += err; atomic_add(err, &ctx->rcvused); rsgl->sg_num_bytes = err; - iov_iter_advance(&msg->msg_iter, err); } *outlen = len; diff --git a/crypto/algif_hash.c b/crypto/algif_hash.c index 50f7b22f1b48..1d017ec5c63c 100644 --- a/crypto/algif_hash.c +++ b/crypto/algif_hash.c @@ -102,11 +102,12 @@ static int hash_sendmsg(struct socket *sock, struct msghdr *msg, err = crypto_wait_req(crypto_ahash_update(&ctx->req), &ctx->wait); af_alg_free_sg(&ctx->sgl); - if (err) + if (err) { + iov_iter_revert(&msg->msg_iter, len); goto unlock; + } copied += len; - iov_iter_advance(&msg->msg_iter, len); } err = 0; From patchwork Wed Jun 22 04:15:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890116 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09F73C43334 for ; Wed, 22 Jun 2022 04:17:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356921AbiFVERF (ORCPT ); Wed, 22 Jun 2022 00:17:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356852AbiFVEQK (ORCPT ); Wed, 22 Jun 2022 00:16:10 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A058FB7DB for ; Tue, 21 Jun 2022 21:16:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=FOfPqUwHsOIPaCpLaB+QW6WYhqelX7/QZQoEcNKj3Bw=; b=ES1XOW6QRtvDGQvpNgs0obMPQ1 vSWer81TLAEBsTfh/jbS139HhQjhjWVJf1KF9zZ3bq0HdOcyNv0FBNVcPz4qP3ZYry3s4XYv8eb7Q rMhsj4HrMm6hzTdHbxVO9JJmYCwIuDum07g3esbY2/PrZEY1tsJN2JE9XrGz1Ty0kVJjDZrMnNnwI fJVz0UO1dLQpgfv1slmaTDgT76ttfGxvLA+7e5QQWK20kbHxZlR8reG5faJ/MIAsTjuISKA3dU4fS vHGNZWZBPMqgdv3LaY6RzVCtlm6IrVr2xPI22bPHl6M6MTFkMHCPvnTNse3RAlGCU8NhxrqXrLip9 fJFWPVHg==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rmZ-003604-Uv; Wed, 22 Jun 2022 04:15:59 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 40/44] 9p: convert to advancing variant of iov_iter_get_pages_alloc() Date: Wed, 22 Jun 2022 05:15:48 +0100 Message-Id: <20220622041552.737754-40-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org that one is somewhat clumsier than usual and needs serious testing. Signed-off-by: Al Viro Reviewed-by: Dominique Martinet Reviewed-by: Christian Schoenebeck Tested-by: Christian Schoenebeck --- net/9p/client.c | 39 +++++++++++++++++++++++---------------- net/9p/protocol.c | 3 +-- net/9p/trans_virtio.c | 3 ++- 3 files changed, 26 insertions(+), 19 deletions(-) diff --git a/net/9p/client.c b/net/9p/client.c index d403085b9ef5..cb4324211561 100644 --- a/net/9p/client.c +++ b/net/9p/client.c @@ -1491,7 +1491,7 @@ p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to, struct p9_client *clnt = fid->clnt; struct p9_req_t *req; int count = iov_iter_count(to); - int rsize, non_zc = 0; + int rsize, received, non_zc = 0; char *dataptr; *err = 0; @@ -1520,36 +1520,40 @@ p9_client_read_once(struct p9_fid *fid, u64 offset, struct iov_iter *to, } if (IS_ERR(req)) { *err = PTR_ERR(req); + if (!non_zc) + iov_iter_revert(to, count - iov_iter_count(to)); return 0; } *err = p9pdu_readf(&req->rc, clnt->proto_version, - "D", &count, &dataptr); + "D", &received, &dataptr); if (*err) { + if (!non_zc) + iov_iter_revert(to, count - iov_iter_count(to)); trace_9p_protocol_dump(clnt, &req->rc); p9_tag_remove(clnt, req); return 0; } - if (rsize < count) { - pr_err("bogus RREAD count (%d > %d)\n", count, rsize); - count = rsize; + if (rsize < received) { + pr_err("bogus RREAD count (%d > %d)\n", received, rsize); + received = rsize; } p9_debug(P9_DEBUG_9P, "<<< RREAD count %d\n", count); if (non_zc) { - int n = copy_to_iter(dataptr, count, to); + int n = copy_to_iter(dataptr, received, to); - if (n != count) { + if (n != received) { *err = -EFAULT; p9_tag_remove(clnt, req); return n; } } else { - iov_iter_advance(to, count); + iov_iter_revert(to, count - received - iov_iter_count(to)); } p9_tag_remove(clnt, req); - return count; + return received; } EXPORT_SYMBOL(p9_client_read_once); @@ -1567,6 +1571,7 @@ p9_client_write(struct p9_fid *fid, u64 offset, struct iov_iter *from, int *err) while (iov_iter_count(from)) { int count = iov_iter_count(from); int rsize = fid->iounit; + int written; if (!rsize || rsize > clnt->msize - P9_IOHDRSZ) rsize = clnt->msize - P9_IOHDRSZ; @@ -1584,27 +1589,29 @@ p9_client_write(struct p9_fid *fid, u64 offset, struct iov_iter *from, int *err) offset, rsize, from); } if (IS_ERR(req)) { + iov_iter_revert(from, count - iov_iter_count(from)); *err = PTR_ERR(req); break; } - *err = p9pdu_readf(&req->rc, clnt->proto_version, "d", &count); + *err = p9pdu_readf(&req->rc, clnt->proto_version, "d", &written); if (*err) { + iov_iter_revert(from, count - iov_iter_count(from)); trace_9p_protocol_dump(clnt, &req->rc); p9_tag_remove(clnt, req); break; } - if (rsize < count) { - pr_err("bogus RWRITE count (%d > %d)\n", count, rsize); - count = rsize; + if (rsize < written) { + pr_err("bogus RWRITE count (%d > %d)\n", written, rsize); + written = rsize; } p9_debug(P9_DEBUG_9P, "<<< RWRITE count %d\n", count); p9_tag_remove(clnt, req); - iov_iter_advance(from, count); - total += count; - offset += count; + iov_iter_revert(from, count - written - iov_iter_count(from)); + total += written; + offset += written; } return total; } diff --git a/net/9p/protocol.c b/net/9p/protocol.c index 3754c33e2974..83694c631989 100644 --- a/net/9p/protocol.c +++ b/net/9p/protocol.c @@ -63,9 +63,8 @@ static size_t pdu_write_u(struct p9_fcall *pdu, struct iov_iter *from, size_t size) { size_t len = min(pdu->capacity - pdu->size, size); - struct iov_iter i = *from; - if (!copy_from_iter_full(&pdu->sdata[pdu->size], len, &i)) + if (!copy_from_iter_full(&pdu->sdata[pdu->size], len, from)) len = 0; pdu->size += len; diff --git a/net/9p/trans_virtio.c b/net/9p/trans_virtio.c index 2a210c2f8e40..1977d33475fe 100644 --- a/net/9p/trans_virtio.c +++ b/net/9p/trans_virtio.c @@ -331,7 +331,7 @@ static int p9_get_mapped_pages(struct virtio_chan *chan, if (err == -ERESTARTSYS) return err; } - n = iov_iter_get_pages_alloc(data, pages, count, offs); + n = iov_iter_get_pages_alloc2(data, pages, count, offs); if (n < 0) return n; *need_drop = 1; @@ -373,6 +373,7 @@ static int p9_get_mapped_pages(struct virtio_chan *chan, (*pages)[index] = kmap_to_page(p); p += PAGE_SIZE; } + iov_iter_advance(data, len); return len; } } From patchwork Wed Jun 22 04:15:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890112 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F226CCA47A for ; Wed, 22 Jun 2022 04:17:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356902AbiFVERA (ORCPT ); Wed, 22 Jun 2022 00:17:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356845AbiFVEQK (ORCPT ); Wed, 22 Jun 2022 00:16:10 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5A4FB7EA for ; Tue, 21 Jun 2022 21:16:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=XO8RatToLOYlx5LqtT10Kw6Qkwp7D31l2wdQV5VDJCQ=; b=Jx4GAwsit5tbwMiF8FH0gLGTDH +HQ1G71/iYAK7tnLy1ASylqDrgXQMvrkL44b0HGXeUDUuG7PM+uuJMRM+zT6gAROclL53sbK2kbEw v1K6BTv40EFLg0kF7AcM/1K1Lq3s+XVmZ90NoaW+iUiyIC3tpclxeialYKFTCAlTl7oLuj1IJamKm lYxxvEnj2z4rtVhNCKDjxJUQ6wR/aOxdTXmwLdBP1lnkNIlF2oPgPqN1Ns3DzLuMPplaj3Y98wnxi 2IOdabpo1uu1soQGZjnuZ05WL3R+y45iIHUT769HmyCtNDSG0BRjSofqb4TgxMlqygrCtjtdVahgj sakWBEyQ==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rma-003608-3G; Wed, 22 Jun 2022 04:16:00 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 41/44] ceph: switch the last caller of iov_iter_get_pages_alloc() Date: Wed, 22 Jun 2022 05:15:49 +0100 Message-Id: <20220622041552.737754-41-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org here nothing even looks at the iov_iter after the call, so we couldn't care less whether it advances or not. Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- fs/ceph/addr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c index 6dee88815491..3c8a7cf19e5d 100644 --- a/fs/ceph/addr.c +++ b/fs/ceph/addr.c @@ -329,7 +329,7 @@ static void ceph_netfs_issue_read(struct netfs_io_subrequest *subreq) dout("%s: pos=%llu orig_len=%zu len=%llu\n", __func__, subreq->start, subreq->len, len); iov_iter_xarray(&iter, READ, &rreq->mapping->i_pages, subreq->start, len); - err = iov_iter_get_pages_alloc(&iter, &pages, len, &page_off); + err = iov_iter_get_pages_alloc2(&iter, &pages, len, &page_off); if (err < 0) { dout("%s: iov_ter_get_pages_alloc returned %d\n", __func__, err); goto out; From patchwork Wed Jun 22 04:15:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890119 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DBC9C433EF for ; Wed, 22 Jun 2022 04:17:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357004AbiFVERO (ORCPT ); Wed, 22 Jun 2022 00:17:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356864AbiFVEQK (ORCPT ); Wed, 22 Jun 2022 00:16:10 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A840BC86 for ; Tue, 21 Jun 2022 21:16:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=BeoB4cjTFciz31d7M7XFM7Xq9Q0tWo2tlx8HPa0iYL0=; b=eJSQcJkjXQqcnY8UPd1OT3g4Mc XJFbZ6NV10qiSLGyQ7LYViPBYpVwWRut5xT+BM2LzUf7JE64qxwg4HXqFpkN+e7u6kh3N9N3AvkjL DMIftoW3CerUK+24zHqTOSvEYQYSwDRiRWd2WQ3IYUl9YxDC4NQDxTqcJ6sCkHUtbaBAzaNpFlOz9 0g/sC9DTy0PlJrqxRSnwSpNJ6uSIZgz4GPDbIPzqvRusj3DmTRuqjfYJXySGC+3ll1OVI8ePR91vn eO1sTPGM7K8K/66s6GfYb8dvBKd4DZWPPNcspNnFs6GKg3VEVV25P8njA7AHAqaf2cKA2/mOoG20C rRFPa7OA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rma-00360C-7n; Wed, 22 Jun 2022 04:16:00 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 42/44] get rid of non-advancing variants Date: Wed, 22 Jun 2022 05:15:50 +0100 Message-Id: <20220622041552.737754-42-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org mechanical change; will be further massaged in subsequent commits Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- include/linux/uio.h | 24 ++---------------------- lib/iov_iter.c | 27 ++++++++++++++++++--------- 2 files changed, 20 insertions(+), 31 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index ab1cc218b9de..f2fc55f88e45 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -245,9 +245,9 @@ void iov_iter_pipe(struct iov_iter *i, unsigned int direction, struct pipe_inode void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count); void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray *xarray, loff_t start, size_t count); -ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, +ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start); -ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, +ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start); int iov_iter_npages(const struct iov_iter *i, int maxpages); void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state); @@ -349,24 +349,4 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, }; } -static inline ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages, - size_t maxsize, unsigned maxpages, size_t *start) -{ - ssize_t res = iov_iter_get_pages(i, pages, maxsize, maxpages, start); - - if (res >= 0) - iov_iter_advance(i, res); - return res; -} - -static inline ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, - size_t maxsize, size_t *start) -{ - ssize_t res = iov_iter_get_pages_alloc(i, pages, maxsize, start); - - if (res >= 0) - iov_iter_advance(i, res); - return res; -} - #endif diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 1c744f0c0b2c..70736b3e07c5 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1231,6 +1231,7 @@ static ssize_t pipe_get_pages(struct iov_iter *i, left -= PAGE_SIZE - off; if (left <= 0) { buf->len += maxsize; + iov_iter_advance(i, maxsize); return maxsize; } buf->len = PAGE_SIZE; @@ -1250,7 +1251,9 @@ static ssize_t pipe_get_pages(struct iov_iter *i, } if (!npages) return -EFAULT; - return maxsize - left; + maxsize -= left; + iov_iter_advance(i, maxsize); + return maxsize; } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa, @@ -1300,7 +1303,9 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, if (nr == 0) return 0; - return min_t(size_t, nr * PAGE_SIZE - offset, maxsize); + maxsize = min_t(size_t, nr * PAGE_SIZE - offset, maxsize); + iov_iter_advance(i, maxsize); + return maxsize; } /* must be done on non-empty ITER_UBUF or ITER_IOVEC one */ @@ -1372,7 +1377,9 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, res = get_user_pages_fast(addr, n, gup_flags, *pages); if (unlikely(res <= 0)) return res; - return min_t(size_t, maxsize, res * PAGE_SIZE - *start); + maxsize = min_t(size_t, maxsize, res * PAGE_SIZE - *start); + iov_iter_advance(i, maxsize); + return maxsize; } if (iov_iter_is_bvec(i)) { struct page **p; @@ -1384,8 +1391,10 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, return -ENOMEM; p = *pages; for (int k = 0; k < n; k++) - get_page(*p++ = page++); - return min_t(size_t, maxsize, n * PAGE_SIZE - *start); + get_page(p[k] = page + k); + maxsize = min_t(size_t, maxsize, n * PAGE_SIZE - *start); + iov_iter_advance(i, maxsize); + return maxsize; } if (iov_iter_is_pipe(i)) return pipe_get_pages(i, pages, maxsize, maxpages, start); @@ -1395,7 +1404,7 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, return -EFAULT; } -ssize_t iov_iter_get_pages(struct iov_iter *i, +ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start) { @@ -1405,9 +1414,9 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, return __iov_iter_get_pages_alloc(i, &pages, maxsize, maxpages, start); } -EXPORT_SYMBOL(iov_iter_get_pages); +EXPORT_SYMBOL(iov_iter_get_pages2); -ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, +ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start) { @@ -1422,7 +1431,7 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, } return len; } -EXPORT_SYMBOL(iov_iter_get_pages_alloc); +EXPORT_SYMBOL(iov_iter_get_pages_alloc2); size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) From patchwork Wed Jun 22 04:15:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890115 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D9D5CCA47A for ; Wed, 22 Jun 2022 04:17:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356915AbiFVERE (ORCPT ); Wed, 22 Jun 2022 00:17:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356847AbiFVEQK (ORCPT ); Wed, 22 Jun 2022 00:16:10 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CE9513EB8 for ; Tue, 21 Jun 2022 21:16:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=TvO9AxhuUqm3giUpUDkx42XZvSxm/hz1ZoCEeXDw8Tc=; b=v4gH8oftfVeyxzNlzbN6Til65f qg28Qi1O6esKO+eFE/HAXleLj4fO+MlWlEYk/OFraRsr7KGOE+s1eBd57Zthuf5IW6CGcLNN4Juz6 uNHxf+LmrXDEGut9VQf8veBjp/kEcKEu+pm+tJe2icDth6uItihMgR3SUclevO2Pqrw7VIV4w/2UP 8BthXSZgTW93km5A2CL0hXY4uf740KjJRz7KK7vMoHlkzSX3uliIBs/jSfdA31Q2sehwviCRc3AR7 HB7WqRCvVUOXo5ZSy+nUi6sjMv+0AI7TM3+tMOkL0tlbQL/yL5PMRp2oon1vQ3fM4ZnWxPFcGcVzo AXnQCvfA==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rma-00360L-HZ; Wed, 22 Jun 2022 04:16:00 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 43/44] pipe_get_pages(): switch to append_pipe() Date: Wed, 22 Jun 2022 05:15:51 +0100 Message-Id: <20220622041552.737754-43-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org now that we are advancing the iterator, there's no need to treat the first page separately - just call append_pipe() in a loop. Signed-off-by: Al Viro Reviewed-by: Jeff Layton --- lib/iov_iter.c | 36 ++++++++---------------------------- 1 file changed, 8 insertions(+), 28 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 70736b3e07c5..a8045c97b975 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1207,10 +1207,10 @@ static ssize_t pipe_get_pages(struct iov_iter *i, struct page ***pages, size_t maxsize, unsigned maxpages, size_t *start) { - struct pipe_inode_info *pipe = i->pipe; - unsigned int npages, off, count; + unsigned int npages, count; struct page **p; ssize_t left; + size_t off; if (!sanity(i)) return -EFAULT; @@ -1222,38 +1222,18 @@ static ssize_t pipe_get_pages(struct iov_iter *i, if (!count) return -ENOMEM; p = *pages; - left = maxsize; - npages = 0; - if (off) { - struct pipe_buffer *buf = pipe_buf(pipe, pipe->head - 1); - - get_page(*p++ = buf->page); - left -= PAGE_SIZE - off; - if (left <= 0) { - buf->len += maxsize; - iov_iter_advance(i, maxsize); - return maxsize; - } - buf->len = PAGE_SIZE; - npages = 1; - } - for ( ; npages < count; npages++) { - struct page *page; - unsigned int size = min_t(ssize_t, left, PAGE_SIZE); - - if (pipe_full(pipe->head, pipe->tail, pipe->max_usage)) - break; - page = push_anon(pipe, size); + for (npages = 0, left = maxsize ; npages < count; npages++) { + struct page *page = append_pipe(i, left, &off); if (!page) break; get_page(*p++ = page); - left -= size; + if (left <= PAGE_SIZE - off) + return maxsize; + left -= PAGE_SIZE - off; } if (!npages) return -EFAULT; - maxsize -= left; - iov_iter_advance(i, maxsize); - return maxsize; + return maxsize - left; } static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa, From patchwork Wed Jun 22 04:15:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Al Viro X-Patchwork-Id: 12890117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8B15CCA47A for ; Wed, 22 Jun 2022 04:17:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356907AbiFVERH (ORCPT ); Wed, 22 Jun 2022 00:17:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48640 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356851AbiFVEQK (ORCPT ); Wed, 22 Jun 2022 00:16:10 -0400 Received: from zeniv.linux.org.uk (zeniv.linux.org.uk [IPv6:2a03:a000:7:0:5054:ff:fe1c:15ff]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4556165A8 for ; Tue, 21 Jun 2022 21:16:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=linux.org.uk; s=zeniv-20220401; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=U4EtewnBFbTPEF12aprvcdvs76IRTF0XhXYK8SHtVQg=; b=gGubJ94ov4565wjcpkPfTWqPrq a0WAgUADlYT0dPHu+OUelNyZSQ1qwKMqhx4FlhlFPkyeqfYd8UrVbLmacxkuGqs0wGh48wq0PAry7 LohvWnBzODELjclGsHNtKK9Nv5J7bOGbJU5PZ/J06/Ar1pwbmYZ50bU0uS7zkqONYFah0+pZFuX+T UfhuS5sc79oit78p27R9Fw/7mVg/c6SnYHSs6deN1Jo1A6hzDrooIhcCIc/O6zlSNHpXtUlaDuPxY +HQzrurMm/0HNdKkiGbA7H93wZAPkF3tb44APaMpfLwcPgnBdtg3vkYudK0S0jpqU78wA1fAtQZNU AK3FM9vg==; Received: from viro by zeniv.linux.org.uk with local (Exim 4.95 #2 (Red Hat Linux)) id 1o3rma-00360c-Qr; Wed, 22 Jun 2022 04:16:00 +0000 From: Al Viro To: linux-fsdevel@vger.kernel.org Cc: Linus Torvalds , Jens Axboe , Christoph Hellwig , Matthew Wilcox , David Howells , Dominique Martinet , Christian Brauner Subject: [PATCH 44/44] expand those iov_iter_advance()... Date: Wed, 22 Jun 2022 05:15:52 +0100 Message-Id: <20220622041552.737754-44-viro@zeniv.linux.org.uk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220622041552.737754-1-viro@zeniv.linux.org.uk> References: <20220622041552.737754-1-viro@zeniv.linux.org.uk> MIME-Version: 1.0 Sender: Al Viro Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Signed-off-by: Al Viro --- lib/iov_iter.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index a8045c97b975..79c86add8dea 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1284,7 +1284,8 @@ static ssize_t iter_xarray_get_pages(struct iov_iter *i, return 0; maxsize = min_t(size_t, nr * PAGE_SIZE - offset, maxsize); - iov_iter_advance(i, maxsize); + i->iov_offset += maxsize; + i->count -= maxsize; return maxsize; } @@ -1373,7 +1374,13 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i, for (int k = 0; k < n; k++) get_page(p[k] = page + k); maxsize = min_t(size_t, maxsize, n * PAGE_SIZE - *start); - iov_iter_advance(i, maxsize); + i->count -= maxsize; + i->iov_offset += maxsize; + if (i->iov_offset == i->bvec->bv_len) { + i->iov_offset = 0; + i->bvec++; + i->nr_segs--; + } return maxsize; } if (iov_iter_is_pipe(i))