From patchwork Sat Nov 21 14:13:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923401 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 518BFC63697 for ; Sat, 21 Nov 2020 14:13:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DAEA022226 for ; Sat, 21 Nov 2020 14:13:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="YVk6GZqk" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727909AbgKUONo (ORCPT ); Sat, 21 Nov 2020 09:13:44 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:36935 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727912AbgKUONo (ORCPT ); Sat, 21 Nov 2020 09:13:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Lfv4FQ56Ng1mDXK+3kEJa6lpcMG3pXvv44BndwILziA=; b=YVk6GZqkUyyNrk8MdXauR/gPt79YAVMA/j1cNEjRVCVdzARCAf2wjSfmV2i9uPrn0OvMtF zahhXmkF8lkD9mlVZgwbBSN7I66jGpGyP3GeqvI2bRru9Yoy3wFeQxUaglnT2gfQ0nWj7w Jx2YYO08w0B9/dNh107j8b9bxgZpocE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-533-l7J7gu7oNPG5NISzJ38DoA-1; Sat, 21 Nov 2020 09:13:35 -0500 X-MC-Unique: l7J7gu7oNPG5NISzJ38DoA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 355D9804759; Sat, 21 Nov 2020 14:13:33 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0FA2210016DB; Sat, 21 Nov 2020 14:13:30 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 01/29] iov_iter: Switch to using a table of operations From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:13:30 +0000 Message-ID: <160596801020.154728.15935034745159191564.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Switch to using a table of operations. In a future patch the individual methods will be split up by type. For the moment, however, the ops tables just jump directly to the old functions - which are now static. Inline wrappers are provided to jump through the hooks. Signed-off-by: David Howells Reported-by: kernel test robot --- fs/io_uring.c | 2 include/linux/uio.h | 241 ++++++++++++++++++++++++++++++++++-------- lib/iov_iter.c | 293 +++++++++++++++++++++++++++++++++++++++------------ 3 files changed, 422 insertions(+), 114 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 4ead291b2976..baa78f58ae5c 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -3192,7 +3192,7 @@ static void io_req_map_rw(struct io_kiocb *req, const struct iovec *iovec, rw->free_iovec = iovec; rw->bytes_done = 0; /* can only be fixed buffers, no need to do anything */ - if (iter->type == ITER_BVEC) + if (iov_iter_is_bvec(iter)) return; if (!iovec) { unsigned iov_off = 0; diff --git a/include/linux/uio.h b/include/linux/uio.h index 72d88566694e..45ee087f8c43 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -32,9 +32,10 @@ struct iov_iter { * Bit 1 is the BVEC_FLAG_NO_REF bit, set if type is a bvec and * the caller isn't expecting to drop a page reference when done. */ - unsigned int type; + unsigned int flags; size_t iov_offset; size_t count; + const struct iov_iter_ops *ops; union { const struct iovec *iov; const struct kvec *kvec; @@ -50,9 +51,63 @@ struct iov_iter { }; }; +void iov_iter_init(struct iov_iter *i, unsigned int direction, const struct iovec *iov, + unsigned long nr_segs, size_t count); +void iov_iter_kvec(struct iov_iter *i, unsigned int direction, const struct kvec *kvec, + unsigned long nr_segs, size_t count); +void iov_iter_bvec(struct iov_iter *i, unsigned int direction, const struct bio_vec *bvec, + unsigned long nr_segs, size_t count); +void iov_iter_pipe(struct iov_iter *i, unsigned int direction, struct pipe_inode_info *pipe, + size_t count); +void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count); + +struct iov_iter_ops { + enum iter_type type; + size_t (*copy_from_user_atomic)(struct page *page, struct iov_iter *i, + unsigned long offset, size_t bytes); + void (*advance)(struct iov_iter *i, size_t bytes); + void (*revert)(struct iov_iter *i, size_t bytes); + int (*fault_in_readable)(struct iov_iter *i, size_t bytes); + size_t (*single_seg_count)(const struct iov_iter *i); + size_t (*copy_page_to_iter)(struct page *page, size_t offset, size_t bytes, + struct iov_iter *i); + size_t (*copy_page_from_iter)(struct page *page, size_t offset, size_t bytes, + struct iov_iter *i); + size_t (*copy_to_iter)(const void *addr, size_t bytes, struct iov_iter *i); + size_t (*copy_from_iter)(void *addr, size_t bytes, struct iov_iter *i); + bool (*copy_from_iter_full)(void *addr, size_t bytes, struct iov_iter *i); + size_t (*copy_from_iter_nocache)(void *addr, size_t bytes, struct iov_iter *i); + bool (*copy_from_iter_full_nocache)(void *addr, size_t bytes, struct iov_iter *i); +#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE + size_t (*copy_from_iter_flushcache)(void *addr, size_t bytes, struct iov_iter *i); +#endif +#ifdef CONFIG_ARCH_HAS_COPY_MC + size_t (*copy_mc_to_iter)(const void *addr, size_t bytes, struct iov_iter *i); +#endif + size_t (*csum_and_copy_to_iter)(const void *addr, size_t bytes, void *csump, + struct iov_iter *i); + size_t (*csum_and_copy_from_iter)(void *addr, size_t bytes, __wsum *csum, + struct iov_iter *i); + bool (*csum_and_copy_from_iter_full)(void *addr, size_t bytes, __wsum *csum, + struct iov_iter *i); + + size_t (*zero)(size_t bytes, struct iov_iter *i); + unsigned long (*alignment)(const struct iov_iter *i); + unsigned long (*gap_alignment)(const struct iov_iter *i); + ssize_t (*get_pages)(struct iov_iter *i, struct page **pages, + size_t maxsize, unsigned maxpages, size_t *start); + ssize_t (*get_pages_alloc)(struct iov_iter *i, struct page ***pages, + size_t maxsize, size_t *start); + int (*npages)(const struct iov_iter *i, int maxpages); + const void *(*dup_iter)(struct iov_iter *new, struct iov_iter *old, gfp_t flags); + int (*for_each_range)(struct iov_iter *i, size_t bytes, + int (*f)(struct kvec *vec, void *context), + void *context); +}; + static inline enum iter_type iov_iter_type(const struct iov_iter *i) { - return i->type & ~(READ | WRITE); + return i->ops->type; } static inline bool iter_is_iovec(const struct iov_iter *i) @@ -82,7 +137,7 @@ static inline bool iov_iter_is_discard(const struct iov_iter *i) static inline unsigned char iov_iter_rw(const struct iov_iter *i) { - return i->type & (READ | WRITE); + return i->flags & (READ | WRITE); } /* @@ -111,22 +166,71 @@ static inline struct iovec iov_iter_iovec(const struct iov_iter *iter) }; } -size_t iov_iter_copy_from_user_atomic(struct page *page, - struct iov_iter *i, unsigned long offset, size_t bytes); -void iov_iter_advance(struct iov_iter *i, size_t bytes); -void iov_iter_revert(struct iov_iter *i, size_t bytes); -int iov_iter_fault_in_readable(struct iov_iter *i, size_t bytes); -size_t iov_iter_single_seg_count(const struct iov_iter *i); +static inline +size_t iov_iter_copy_from_user_atomic(struct page *page, struct iov_iter *i, + unsigned long offset, size_t bytes) +{ + return i->ops->copy_from_user_atomic(page, i, offset, bytes); +} +static inline +void iov_iter_advance(struct iov_iter *i, size_t bytes) +{ + return i->ops->advance(i, bytes); +} +static inline +void iov_iter_revert(struct iov_iter *i, size_t bytes) +{ + return i->ops->revert(i, bytes); +} +static inline +int iov_iter_fault_in_readable(struct iov_iter *i, size_t bytes) +{ + return i->ops->fault_in_readable(i, bytes); +} +static inline +size_t iov_iter_single_seg_count(const struct iov_iter *i) +{ + return i->ops->single_seg_count(i); +} + +static inline size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes, - struct iov_iter *i); + struct iov_iter *i) +{ + return i->ops->copy_page_to_iter(page, offset, bytes, i); +} +static inline size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, - struct iov_iter *i); + struct iov_iter *i) +{ + return i->ops->copy_page_from_iter(page, offset, bytes, i); +} -size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i); -size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i); -bool _copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i); -size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i); -bool _copy_from_iter_full_nocache(void *addr, size_t bytes, struct iov_iter *i); +static __always_inline __must_check +size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +{ + return i->ops->copy_to_iter(addr, bytes, i); +} +static __always_inline __must_check +size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) +{ + return i->ops->copy_from_iter(addr, bytes, i); +} +static __always_inline __must_check +bool _copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i) +{ + return i->ops->copy_from_iter_full(addr, bytes, i); +} +static __always_inline __must_check +size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) +{ + return i->ops->copy_from_iter_nocache(addr, bytes, i); +} +static __always_inline __must_check +bool _copy_from_iter_full_nocache(void *addr, size_t bytes, struct iov_iter *i) +{ + return i->ops->copy_from_iter_full_nocache(addr, bytes, i); +} static __always_inline __must_check size_t copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) @@ -173,23 +277,21 @@ bool copy_from_iter_full_nocache(void *addr, size_t bytes, struct iov_iter *i) return _copy_from_iter_full_nocache(addr, bytes, i); } -#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE /* * Note, users like pmem that depend on the stricter semantics of * copy_from_iter_flushcache() than copy_from_iter_nocache() must check for * IS_ENABLED(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) before assuming that the * destination is flushed from the cache on return. */ -size_t _copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i); -#else -#define _copy_from_iter_flushcache _copy_from_iter_nocache -#endif - -#ifdef CONFIG_ARCH_HAS_COPY_MC -size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i); +static __always_inline __must_check +size_t _copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) +{ +#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE + return i->ops->copy_from_iter_flushcache(addr, bytes, i); #else -#define _copy_mc_to_iter _copy_to_iter + return i->ops->copy_from_iter_nocache(addr, bytes, i); #endif +} static __always_inline __must_check size_t copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) @@ -200,6 +302,16 @@ size_t copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) return _copy_from_iter_flushcache(addr, bytes, i); } +static __always_inline __must_check +size_t _copy_mc_to_iter(void *addr, size_t bytes, struct iov_iter *i) +{ +#ifdef CONFIG_ARCH_HAS_COPY_MC + return i->ops->copy_mc_to_iter(addr, bytes, i); +#else + return i->ops->copy_to_iter(addr, bytes, i); +#endif +} + static __always_inline __must_check size_t copy_mc_to_iter(void *addr, size_t bytes, struct iov_iter *i) { @@ -209,25 +321,47 @@ size_t copy_mc_to_iter(void *addr, size_t bytes, struct iov_iter *i) return _copy_mc_to_iter(addr, bytes, i); } -size_t iov_iter_zero(size_t bytes, struct iov_iter *); -unsigned long iov_iter_alignment(const struct iov_iter *i); -unsigned long iov_iter_gap_alignment(const struct iov_iter *i); -void iov_iter_init(struct iov_iter *i, unsigned int direction, const struct iovec *iov, - unsigned long nr_segs, size_t count); -void iov_iter_kvec(struct iov_iter *i, unsigned int direction, const struct kvec *kvec, - unsigned long nr_segs, size_t count); -void iov_iter_bvec(struct iov_iter *i, unsigned int direction, const struct bio_vec *bvec, - unsigned long nr_segs, size_t count); -void iov_iter_pipe(struct iov_iter *i, unsigned int direction, struct pipe_inode_info *pipe, - size_t count); -void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count); +static inline +size_t iov_iter_zero(size_t bytes, struct iov_iter *i) +{ + return i->ops->zero(bytes, i); +} +static inline +unsigned long iov_iter_alignment(const struct iov_iter *i) +{ + return i->ops->alignment(i); +} +static inline +unsigned long iov_iter_gap_alignment(const struct iov_iter *i) +{ + return i->ops->gap_alignment(i); +} + +static inline ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, - size_t maxsize, unsigned maxpages, size_t *start); + size_t maxsize, unsigned maxpages, size_t *start) +{ + return i->ops->get_pages(i, pages, maxsize, maxpages, start); +} + +static inline ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, - size_t maxsize, size_t *start); -int iov_iter_npages(const struct iov_iter *i, int maxpages); + size_t maxsize, size_t *start) +{ + return i->ops->get_pages_alloc(i, pages, maxsize, start); +} + +static inline +int iov_iter_npages(const struct iov_iter *i, int maxpages) +{ + return i->ops->npages(i, maxpages); +} -const void *dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags); +static inline +const void *dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) +{ + return old->ops->dup_iter(new, old, flags); +} static inline size_t iov_iter_count(const struct iov_iter *i) { @@ -260,9 +394,22 @@ static inline void iov_iter_reexpand(struct iov_iter *i, size_t count) { i->count = count; } -size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump, struct iov_iter *i); -size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i); -bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i); + +static inline +size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump, struct iov_iter *i) +{ + return i->ops->csum_and_copy_to_iter(addr, bytes, csump, i); +} +static inline +size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) +{ + return i->ops->csum_and_copy_from_iter(addr, bytes, csum, i); +} +static inline +bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) +{ + return i->ops->csum_and_copy_from_iter_full(addr, bytes, csum, i); +} size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, struct iov_iter *i); @@ -278,8 +425,12 @@ ssize_t __import_iovec(int type, const struct iovec __user *uvec, int import_single_range(int type, void __user *buf, size_t len, struct iovec *iov, struct iov_iter *i); +static inline int iov_iter_for_each_range(struct iov_iter *i, size_t bytes, int (*f)(struct kvec *vec, void *context), - void *context); + void *context) +{ + return i->ops->for_each_range(i, bytes, f, context); +} #endif diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 1635111c5bd2..e403d524c797 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -13,6 +13,12 @@ #include #include +static const struct iov_iter_ops iovec_iter_ops; +static const struct iov_iter_ops kvec_iter_ops; +static const struct iov_iter_ops bvec_iter_ops; +static const struct iov_iter_ops pipe_iter_ops; +static const struct iov_iter_ops discard_iter_ops; + #define PIPE_PARANOIA /* for now */ #define iterate_iovec(i, n, __v, __p, skip, STEP) { \ @@ -81,15 +87,15 @@ #define iterate_all_kinds(i, n, v, I, B, K) { \ if (likely(n)) { \ size_t skip = i->iov_offset; \ - if (unlikely(i->type & ITER_BVEC)) { \ + if (unlikely(iov_iter_type(i) & ITER_BVEC)) { \ struct bio_vec v; \ struct bvec_iter __bi; \ iterate_bvec(i, n, v, __bi, skip, (B)) \ - } else if (unlikely(i->type & ITER_KVEC)) { \ + } else if (unlikely(iov_iter_type(i) & ITER_KVEC)) { \ const struct kvec *kvec; \ struct kvec v; \ iterate_kvec(i, n, v, kvec, skip, (K)) \ - } else if (unlikely(i->type & ITER_DISCARD)) { \ + } else if (unlikely(iov_iter_type(i) & ITER_DISCARD)) { \ } else { \ const struct iovec *iov; \ struct iovec v; \ @@ -103,7 +109,7 @@ n = i->count; \ if (i->count) { \ size_t skip = i->iov_offset; \ - if (unlikely(i->type & ITER_BVEC)) { \ + if (unlikely(iov_iter_type(i) & ITER_BVEC)) { \ const struct bio_vec *bvec = i->bvec; \ struct bio_vec v; \ struct bvec_iter __bi; \ @@ -111,7 +117,7 @@ i->bvec = __bvec_iter_bvec(i->bvec, __bi); \ i->nr_segs -= i->bvec - bvec; \ skip = __bi.bi_bvec_done; \ - } else if (unlikely(i->type & ITER_KVEC)) { \ + } else if (unlikely(iov_iter_type(i) & ITER_KVEC)) { \ const struct kvec *kvec; \ struct kvec v; \ iterate_kvec(i, n, v, kvec, skip, (K)) \ @@ -121,7 +127,7 @@ } \ i->nr_segs -= kvec - i->kvec; \ i->kvec = kvec; \ - } else if (unlikely(i->type & ITER_DISCARD)) { \ + } else if (unlikely(iov_iter_type(i) & ITER_DISCARD)) { \ skip += n; \ } else { \ const struct iovec *iov; \ @@ -427,14 +433,14 @@ static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t by * Return 0 on success, or non-zero if the memory could not be accessed (i.e. * because it is an invalid address). */ -int iov_iter_fault_in_readable(struct iov_iter *i, size_t bytes) +static int xxx_fault_in_readable(struct iov_iter *i, size_t bytes) { size_t skip = i->iov_offset; const struct iovec *iov; int err; struct iovec v; - if (!(i->type & (ITER_BVEC|ITER_KVEC))) { + if (!(iov_iter_type(i) & (ITER_BVEC|ITER_KVEC))) { iterate_iovec(i, bytes, v, iov, skip, ({ err = fault_in_pages_readable(v.iov_base, v.iov_len); if (unlikely(err)) @@ -443,7 +449,6 @@ int iov_iter_fault_in_readable(struct iov_iter *i, size_t bytes) } return 0; } -EXPORT_SYMBOL(iov_iter_fault_in_readable); void iov_iter_init(struct iov_iter *i, unsigned int direction, const struct iovec *iov, unsigned long nr_segs, @@ -454,10 +459,12 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, /* It will get better. Eventually... */ if (uaccess_kernel()) { - i->type = ITER_KVEC | direction; + i->ops = &kvec_iter_ops; + i->flags = direction; i->kvec = (struct kvec *)iov; } else { - i->type = ITER_IOVEC | direction; + i->ops = &iovec_iter_ops; + i->flags = direction; i->iov = iov; } i->nr_segs = nr_segs; @@ -625,7 +632,7 @@ static size_t csum_and_copy_to_pipe_iter(const void *addr, size_t bytes, return bytes; } -size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +static size_t xxx_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { const char *from = addr; if (unlikely(iov_iter_is_pipe(i))) @@ -641,7 +648,6 @@ size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) return bytes; } -EXPORT_SYMBOL(_copy_to_iter); #ifdef CONFIG_ARCH_HAS_COPY_MC static int copyout_mc(void __user *to, const void *from, size_t n) @@ -723,7 +729,7 @@ static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes, * Compare to copy_to_iter() where only ITER_IOVEC attempts might return * a short copy. */ -size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +static size_t xxx_copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { const char *from = addr; unsigned long rem, curr_addr, s_addr = (unsigned long) addr; @@ -757,10 +763,9 @@ size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) return bytes; } -EXPORT_SYMBOL_GPL(_copy_mc_to_iter); #endif /* CONFIG_ARCH_HAS_COPY_MC */ -size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) +static size_t xxx_copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) { char *to = addr; if (unlikely(iov_iter_is_pipe(i))) { @@ -778,9 +783,8 @@ size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) return bytes; } -EXPORT_SYMBOL(_copy_from_iter); -bool _copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i) +static bool xxx_copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i) { char *to = addr; if (unlikely(iov_iter_is_pipe(i))) { @@ -805,9 +809,8 @@ bool _copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i) iov_iter_advance(i, bytes); return true; } -EXPORT_SYMBOL(_copy_from_iter_full); -size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) +static size_t xxx_copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) { char *to = addr; if (unlikely(iov_iter_is_pipe(i))) { @@ -824,7 +827,6 @@ size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) return bytes; } -EXPORT_SYMBOL(_copy_from_iter_nocache); #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE /** @@ -841,7 +843,7 @@ EXPORT_SYMBOL(_copy_from_iter_nocache); * bypass the cache for the ITER_IOVEC case, and on some archs may use * instructions that strand dirty-data in the cache. */ -size_t _copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) +static size_t xxx_copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) { char *to = addr; if (unlikely(iov_iter_is_pipe(i))) { @@ -859,10 +861,9 @@ size_t _copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) return bytes; } -EXPORT_SYMBOL_GPL(_copy_from_iter_flushcache); #endif -bool _copy_from_iter_full_nocache(void *addr, size_t bytes, struct iov_iter *i) +static bool xxx_copy_from_iter_full_nocache(void *addr, size_t bytes, struct iov_iter *i) { char *to = addr; if (unlikely(iov_iter_is_pipe(i))) { @@ -884,7 +885,6 @@ bool _copy_from_iter_full_nocache(void *addr, size_t bytes, struct iov_iter *i) iov_iter_advance(i, bytes); return true; } -EXPORT_SYMBOL(_copy_from_iter_full_nocache); static inline bool page_copy_sane(struct page *page, size_t offset, size_t n) { @@ -910,12 +910,12 @@ static inline bool page_copy_sane(struct page *page, size_t offset, size_t n) return false; } -size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes, +static size_t xxx_copy_page_to_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { if (unlikely(!page_copy_sane(page, offset, bytes))) return 0; - if (i->type & (ITER_BVEC|ITER_KVEC)) { + if (iov_iter_type(i) & (ITER_BVEC|ITER_KVEC)) { void *kaddr = kmap_atomic(page); size_t wanted = copy_to_iter(kaddr + offset, bytes, i); kunmap_atomic(kaddr); @@ -927,9 +927,8 @@ size_t copy_page_to_iter(struct page *page, size_t offset, size_t bytes, else return copy_page_to_iter_pipe(page, offset, bytes, i); } -EXPORT_SYMBOL(copy_page_to_iter); -size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, +static size_t xxx_copy_page_from_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { if (unlikely(!page_copy_sane(page, offset, bytes))) @@ -938,15 +937,14 @@ size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, WARN_ON(1); return 0; } - if (i->type & (ITER_BVEC|ITER_KVEC)) { + if (iov_iter_type(i) & (ITER_BVEC|ITER_KVEC)) { void *kaddr = kmap_atomic(page); - size_t wanted = _copy_from_iter(kaddr + offset, bytes, i); + size_t wanted = xxx_copy_from_iter(kaddr + offset, bytes, i); kunmap_atomic(kaddr); return wanted; } else return copy_page_from_iter_iovec(page, offset, bytes, i); } -EXPORT_SYMBOL(copy_page_from_iter); static size_t pipe_zero(size_t bytes, struct iov_iter *i) { @@ -975,7 +973,7 @@ static size_t pipe_zero(size_t bytes, struct iov_iter *i) return bytes; } -size_t iov_iter_zero(size_t bytes, struct iov_iter *i) +static size_t xxx_zero(size_t bytes, struct iov_iter *i) { if (unlikely(iov_iter_is_pipe(i))) return pipe_zero(bytes, i); @@ -987,9 +985,8 @@ size_t iov_iter_zero(size_t bytes, struct iov_iter *i) return bytes; } -EXPORT_SYMBOL(iov_iter_zero); -size_t iov_iter_copy_from_user_atomic(struct page *page, +static size_t xxx_copy_from_user_atomic(struct page *page, struct iov_iter *i, unsigned long offset, size_t bytes) { char *kaddr = kmap_atomic(page), *p = kaddr + offset; @@ -1011,7 +1008,6 @@ size_t iov_iter_copy_from_user_atomic(struct page *page, kunmap_atomic(kaddr); return bytes; } -EXPORT_SYMBOL(iov_iter_copy_from_user_atomic); static inline void pipe_truncate(struct iov_iter *i) { @@ -1067,7 +1063,7 @@ static void pipe_advance(struct iov_iter *i, size_t size) pipe_truncate(i); } -void iov_iter_advance(struct iov_iter *i, size_t size) +static void xxx_advance(struct iov_iter *i, size_t size) { if (unlikely(iov_iter_is_pipe(i))) { pipe_advance(i, size); @@ -1079,9 +1075,8 @@ void iov_iter_advance(struct iov_iter *i, size_t size) } iterate_and_advance(i, size, v, 0, 0, 0) } -EXPORT_SYMBOL(iov_iter_advance); -void iov_iter_revert(struct iov_iter *i, size_t unroll) +static void xxx_revert(struct iov_iter *i, size_t unroll) { if (!unroll) return; @@ -1147,12 +1142,11 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll) } } } -EXPORT_SYMBOL(iov_iter_revert); /* * Return the count of just the current iov_iter segment. */ -size_t iov_iter_single_seg_count(const struct iov_iter *i) +static size_t xxx_single_seg_count(const struct iov_iter *i) { if (unlikely(iov_iter_is_pipe(i))) return i->count; // it is a silly place, anyway @@ -1165,14 +1159,14 @@ size_t iov_iter_single_seg_count(const struct iov_iter *i) else return min(i->count, i->iov->iov_len - i->iov_offset); } -EXPORT_SYMBOL(iov_iter_single_seg_count); void iov_iter_kvec(struct iov_iter *i, unsigned int direction, - const struct kvec *kvec, unsigned long nr_segs, - size_t count) + const struct kvec *kvec, unsigned long nr_segs, + size_t count) { WARN_ON(direction & ~(READ | WRITE)); - i->type = ITER_KVEC | (direction & (READ | WRITE)); + i->ops = &kvec_iter_ops; + i->flags = direction & (READ | WRITE); i->kvec = kvec; i->nr_segs = nr_segs; i->iov_offset = 0; @@ -1185,7 +1179,8 @@ void iov_iter_bvec(struct iov_iter *i, unsigned int direction, size_t count) { WARN_ON(direction & ~(READ | WRITE)); - i->type = ITER_BVEC | (direction & (READ | WRITE)); + i->ops = &bvec_iter_ops; + i->flags = direction & (READ | WRITE); i->bvec = bvec; i->nr_segs = nr_segs; i->iov_offset = 0; @@ -1199,7 +1194,8 @@ void iov_iter_pipe(struct iov_iter *i, unsigned int direction, { BUG_ON(direction != READ); WARN_ON(pipe_full(pipe->head, pipe->tail, pipe->ring_size)); - i->type = ITER_PIPE | READ; + i->ops = &pipe_iter_ops; + i->flags = READ; i->pipe = pipe; i->head = pipe->head; i->iov_offset = 0; @@ -1220,13 +1216,14 @@ EXPORT_SYMBOL(iov_iter_pipe); void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count) { BUG_ON(direction != READ); - i->type = ITER_DISCARD | READ; + i->ops = &discard_iter_ops; + i->flags = READ; i->count = count; i->iov_offset = 0; } EXPORT_SYMBOL(iov_iter_discard); -unsigned long iov_iter_alignment(const struct iov_iter *i) +static unsigned long xxx_alignment(const struct iov_iter *i) { unsigned long res = 0; size_t size = i->count; @@ -1245,9 +1242,8 @@ unsigned long iov_iter_alignment(const struct iov_iter *i) ) return res; } -EXPORT_SYMBOL(iov_iter_alignment); -unsigned long iov_iter_gap_alignment(const struct iov_iter *i) +static unsigned long xxx_gap_alignment(const struct iov_iter *i) { unsigned long res = 0; size_t size = i->count; @@ -1267,7 +1263,6 @@ unsigned long iov_iter_gap_alignment(const struct iov_iter *i) ); return res; } -EXPORT_SYMBOL(iov_iter_gap_alignment); static inline ssize_t __pipe_get_pages(struct iov_iter *i, size_t maxsize, @@ -1313,7 +1308,7 @@ static ssize_t pipe_get_pages(struct iov_iter *i, return __pipe_get_pages(i, min(maxsize, capacity), pages, iter_head, start); } -ssize_t iov_iter_get_pages(struct iov_iter *i, +static ssize_t xxx_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start) { @@ -1352,7 +1347,6 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, ) return 0; } -EXPORT_SYMBOL(iov_iter_get_pages); static struct page **get_pages_array(size_t n) { @@ -1392,7 +1386,7 @@ static ssize_t pipe_get_pages_alloc(struct iov_iter *i, return n; } -ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, +static ssize_t xxx_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start) { @@ -1439,9 +1433,8 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, ) return 0; } -EXPORT_SYMBOL(iov_iter_get_pages_alloc); -size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, +static size_t xxx_csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) { char *to = addr; @@ -1478,9 +1471,8 @@ size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, *csum = sum; return bytes; } -EXPORT_SYMBOL(csum_and_copy_from_iter); -bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, +static bool xxx_csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) { char *to = addr; @@ -1520,9 +1512,8 @@ bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, iov_iter_advance(i, bytes); return true; } -EXPORT_SYMBOL(csum_and_copy_from_iter_full); -size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump, +static size_t xxx_csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump, struct iov_iter *i) { const char *from = addr; @@ -1564,7 +1555,6 @@ size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump, *csum = sum; return bytes; } -EXPORT_SYMBOL(csum_and_copy_to_iter); size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, struct iov_iter *i) @@ -1585,7 +1575,7 @@ size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, } EXPORT_SYMBOL(hash_and_copy_to_iter); -int iov_iter_npages(const struct iov_iter *i, int maxpages) +static int xxx_npages(const struct iov_iter *i, int maxpages) { size_t size = i->count; int npages = 0; @@ -1628,9 +1618,8 @@ int iov_iter_npages(const struct iov_iter *i, int maxpages) ) return npages; } -EXPORT_SYMBOL(iov_iter_npages); -const void *dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) +static const void *xxx_dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) { *new = *old; if (unlikely(iov_iter_is_pipe(new))) { @@ -1649,7 +1638,6 @@ const void *dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) new->nr_segs * sizeof(struct iovec), flags); } -EXPORT_SYMBOL(dup_iter); static int copy_compat_iovec_from_user(struct iovec *iov, const struct iovec __user *uvec, unsigned long nr_segs) @@ -1826,7 +1814,7 @@ int import_single_range(int rw, void __user *buf, size_t len, } EXPORT_SYMBOL(import_single_range); -int iov_iter_for_each_range(struct iov_iter *i, size_t bytes, +static int xxx_for_each_range(struct iov_iter *i, size_t bytes, int (*f)(struct kvec *vec, void *context), void *context) { @@ -1846,4 +1834,173 @@ int iov_iter_for_each_range(struct iov_iter *i, size_t bytes, ) return err; } -EXPORT_SYMBOL(iov_iter_for_each_range); + +static const struct iov_iter_ops iovec_iter_ops = { + .type = ITER_IOVEC, + .copy_from_user_atomic = xxx_copy_from_user_atomic, + .advance = xxx_advance, + .revert = xxx_revert, + .fault_in_readable = xxx_fault_in_readable, + .single_seg_count = xxx_single_seg_count, + .copy_page_to_iter = xxx_copy_page_to_iter, + .copy_page_from_iter = xxx_copy_page_from_iter, + .copy_to_iter = xxx_copy_to_iter, + .copy_from_iter = xxx_copy_from_iter, + .copy_from_iter_full = xxx_copy_from_iter_full, + .copy_from_iter_nocache = xxx_copy_from_iter_nocache, + .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, +#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE + .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, +#endif +#ifdef CONFIG_ARCH_HAS_COPY_MC + .copy_mc_to_iter = xxx_copy_mc_to_iter, +#endif + .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, + .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, + .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, + + .zero = xxx_zero, + .alignment = xxx_alignment, + .gap_alignment = xxx_gap_alignment, + .get_pages = xxx_get_pages, + .get_pages_alloc = xxx_get_pages_alloc, + .npages = xxx_npages, + .dup_iter = xxx_dup_iter, + .for_each_range = xxx_for_each_range, +}; + +static const struct iov_iter_ops kvec_iter_ops = { + .type = ITER_KVEC, + .copy_from_user_atomic = xxx_copy_from_user_atomic, + .advance = xxx_advance, + .revert = xxx_revert, + .fault_in_readable = xxx_fault_in_readable, + .single_seg_count = xxx_single_seg_count, + .copy_page_to_iter = xxx_copy_page_to_iter, + .copy_page_from_iter = xxx_copy_page_from_iter, + .copy_to_iter = xxx_copy_to_iter, + .copy_from_iter = xxx_copy_from_iter, + .copy_from_iter_full = xxx_copy_from_iter_full, + .copy_from_iter_nocache = xxx_copy_from_iter_nocache, + .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, +#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE + .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, +#endif +#ifdef CONFIG_ARCH_HAS_COPY_MC + .copy_mc_to_iter = xxx_copy_mc_to_iter, +#endif + .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, + .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, + .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, + + .zero = xxx_zero, + .alignment = xxx_alignment, + .gap_alignment = xxx_gap_alignment, + .get_pages = xxx_get_pages, + .get_pages_alloc = xxx_get_pages_alloc, + .npages = xxx_npages, + .dup_iter = xxx_dup_iter, + .for_each_range = xxx_for_each_range, +}; + +static const struct iov_iter_ops bvec_iter_ops = { + .type = ITER_BVEC, + .copy_from_user_atomic = xxx_copy_from_user_atomic, + .advance = xxx_advance, + .revert = xxx_revert, + .fault_in_readable = xxx_fault_in_readable, + .single_seg_count = xxx_single_seg_count, + .copy_page_to_iter = xxx_copy_page_to_iter, + .copy_page_from_iter = xxx_copy_page_from_iter, + .copy_to_iter = xxx_copy_to_iter, + .copy_from_iter = xxx_copy_from_iter, + .copy_from_iter_full = xxx_copy_from_iter_full, + .copy_from_iter_nocache = xxx_copy_from_iter_nocache, + .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, +#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE + .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, +#endif +#ifdef CONFIG_ARCH_HAS_COPY_MC + .copy_mc_to_iter = xxx_copy_mc_to_iter, +#endif + .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, + .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, + .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, + + .zero = xxx_zero, + .alignment = xxx_alignment, + .gap_alignment = xxx_gap_alignment, + .get_pages = xxx_get_pages, + .get_pages_alloc = xxx_get_pages_alloc, + .npages = xxx_npages, + .dup_iter = xxx_dup_iter, + .for_each_range = xxx_for_each_range, +}; + +static const struct iov_iter_ops pipe_iter_ops = { + .type = ITER_PIPE, + .copy_from_user_atomic = xxx_copy_from_user_atomic, + .advance = xxx_advance, + .revert = xxx_revert, + .fault_in_readable = xxx_fault_in_readable, + .single_seg_count = xxx_single_seg_count, + .copy_page_to_iter = xxx_copy_page_to_iter, + .copy_page_from_iter = xxx_copy_page_from_iter, + .copy_to_iter = xxx_copy_to_iter, + .copy_from_iter = xxx_copy_from_iter, + .copy_from_iter_full = xxx_copy_from_iter_full, + .copy_from_iter_nocache = xxx_copy_from_iter_nocache, + .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, +#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE + .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, +#endif +#ifdef CONFIG_ARCH_HAS_COPY_MC + .copy_mc_to_iter = xxx_copy_mc_to_iter, +#endif + .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, + .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, + .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, + + .zero = xxx_zero, + .alignment = xxx_alignment, + .gap_alignment = xxx_gap_alignment, + .get_pages = xxx_get_pages, + .get_pages_alloc = xxx_get_pages_alloc, + .npages = xxx_npages, + .dup_iter = xxx_dup_iter, + .for_each_range = xxx_for_each_range, +}; + +static const struct iov_iter_ops discard_iter_ops = { + .type = ITER_DISCARD, + .copy_from_user_atomic = xxx_copy_from_user_atomic, + .advance = xxx_advance, + .revert = xxx_revert, + .fault_in_readable = xxx_fault_in_readable, + .single_seg_count = xxx_single_seg_count, + .copy_page_to_iter = xxx_copy_page_to_iter, + .copy_page_from_iter = xxx_copy_page_from_iter, + .copy_to_iter = xxx_copy_to_iter, + .copy_from_iter = xxx_copy_from_iter, + .copy_from_iter_full = xxx_copy_from_iter_full, + .copy_from_iter_nocache = xxx_copy_from_iter_nocache, + .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, +#ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE + .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, +#endif +#ifdef CONFIG_ARCH_HAS_COPY_MC + .copy_mc_to_iter = xxx_copy_mc_to_iter, +#endif + .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, + .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, + .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, + + .zero = xxx_zero, + .alignment = xxx_alignment, + .gap_alignment = xxx_gap_alignment, + .get_pages = xxx_get_pages, + .get_pages_alloc = xxx_get_pages_alloc, + .npages = xxx_npages, + .dup_iter = xxx_dup_iter, + .for_each_range = xxx_for_each_range, +}; From patchwork Sat Nov 21 14:13:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923403 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38A54C64E75 for ; Sat, 21 Nov 2020 14:13:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F02E522261 for ; Sat, 21 Nov 2020 14:13:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="SrXwtr6u" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727948AbgKUONs (ORCPT ); Sat, 21 Nov 2020 09:13:48 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:37328 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727920AbgKUONq (ORCPT ); Sat, 21 Nov 2020 09:13:46 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JcWyfdmDr6kJ/GdHDBbHFKPN7p1aKQ7b6+idcK3zWxo=; b=SrXwtr6uXmOTaD5PKVbYGHsyTW1cFfp4qWcGRGS+s7wuBjL532xsFa2yrAtwEU9QY0CkCO sWyzYXl2J2/YDz7s2e0LTLBqzwABKGn0HZ7SX8jN/qogB2hxoxxP5o0cklMRcrDKagt+Pi rdch+QmTNY+GqRv+TDjs9ZvqgMENwu8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-290-vJNJYYrXOdWQ0EgNvmQI8g-1; Sat, 21 Nov 2020 09:13:42 -0500 X-MC-Unique: vJNJYYrXOdWQ0EgNvmQI8g-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 18DF1107AD44; Sat, 21 Nov 2020 14:13:41 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 483236064B; Sat, 21 Nov 2020 14:13:39 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 02/29] iov_iter: Split copy_page_to_iter() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:13:38 +0000 Message-ID: <160596801844.154728.6494357560297311234.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split copy_page_to_iter() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index e403d524c797..fee8e99fbb9c 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -19,6 +19,8 @@ static const struct iov_iter_ops bvec_iter_ops; static const struct iov_iter_ops pipe_iter_ops; static const struct iov_iter_ops discard_iter_ops; +static inline bool page_copy_sane(struct page *page, size_t offset, size_t n); + #define PIPE_PARANOIA /* for now */ #define iterate_iovec(i, n, __v, __p, skip, STEP) { \ @@ -167,7 +169,7 @@ static int copyin(void *to, const void __user *from, size_t n) return n; } -static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t bytes, +static size_t iovec_copy_page_to_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { size_t skip, copy, left, wanted; @@ -175,6 +177,8 @@ static size_t copy_page_to_iter_iovec(struct page *page, size_t offset, size_t b char __user *buf; void *kaddr, *from; + if (unlikely(!page_copy_sane(page, offset, bytes))) + return 0; if (unlikely(bytes > i->count)) bytes = i->count; @@ -378,7 +382,7 @@ static bool sanity(const struct iov_iter *i) #define sanity(i) true #endif -static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t bytes, +static size_t pipe_copy_page_to_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { struct pipe_inode_info *pipe = i->pipe; @@ -388,6 +392,8 @@ static size_t copy_page_to_iter_pipe(struct page *page, size_t offset, size_t by unsigned int i_head = i->head; size_t off; + if (unlikely(!page_copy_sane(page, offset, bytes))) + return 0; if (unlikely(bytes > i->count)) bytes = i->count; @@ -910,22 +916,22 @@ static inline bool page_copy_sane(struct page *page, size_t offset, size_t n) return false; } -static size_t xxx_copy_page_to_iter(struct page *page, size_t offset, size_t bytes, +static size_t bkvec_copy_page_to_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { - if (unlikely(!page_copy_sane(page, offset, bytes))) - return 0; - if (iov_iter_type(i) & (ITER_BVEC|ITER_KVEC)) { + size_t wanted = 0; + if (likely(page_copy_sane(page, offset, bytes))) { void *kaddr = kmap_atomic(page); - size_t wanted = copy_to_iter(kaddr + offset, bytes, i); + wanted = copy_to_iter(kaddr + offset, bytes, i); kunmap_atomic(kaddr); - return wanted; - } else if (unlikely(iov_iter_is_discard(i))) - return bytes; - else if (likely(!iov_iter_is_pipe(i))) - return copy_page_to_iter_iovec(page, offset, bytes, i); - else - return copy_page_to_iter_pipe(page, offset, bytes, i); + } + return wanted; +} + +static size_t discard_copy_page_to_iter(struct page *page, size_t offset, size_t bytes, + struct iov_iter *i) +{ + return bytes; } static size_t xxx_copy_page_from_iter(struct page *page, size_t offset, size_t bytes, @@ -1842,7 +1848,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .revert = xxx_revert, .fault_in_readable = xxx_fault_in_readable, .single_seg_count = xxx_single_seg_count, - .copy_page_to_iter = xxx_copy_page_to_iter, + .copy_page_to_iter = iovec_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = xxx_copy_to_iter, .copy_from_iter = xxx_copy_from_iter, @@ -1876,7 +1882,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .revert = xxx_revert, .fault_in_readable = xxx_fault_in_readable, .single_seg_count = xxx_single_seg_count, - .copy_page_to_iter = xxx_copy_page_to_iter, + .copy_page_to_iter = bkvec_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = xxx_copy_to_iter, .copy_from_iter = xxx_copy_from_iter, @@ -1910,7 +1916,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .revert = xxx_revert, .fault_in_readable = xxx_fault_in_readable, .single_seg_count = xxx_single_seg_count, - .copy_page_to_iter = xxx_copy_page_to_iter, + .copy_page_to_iter = bkvec_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = xxx_copy_to_iter, .copy_from_iter = xxx_copy_from_iter, @@ -1944,7 +1950,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .revert = xxx_revert, .fault_in_readable = xxx_fault_in_readable, .single_seg_count = xxx_single_seg_count, - .copy_page_to_iter = xxx_copy_page_to_iter, + .copy_page_to_iter = pipe_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = xxx_copy_to_iter, .copy_from_iter = xxx_copy_from_iter, @@ -1978,7 +1984,7 @@ static const struct iov_iter_ops discard_iter_ops = { .revert = xxx_revert, .fault_in_readable = xxx_fault_in_readable, .single_seg_count = xxx_single_seg_count, - .copy_page_to_iter = xxx_copy_page_to_iter, + .copy_page_to_iter = discard_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = xxx_copy_to_iter, .copy_from_iter = xxx_copy_from_iter, From patchwork Sat Nov 21 14:13:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923405 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9417C64E7B for ; Sat, 21 Nov 2020 14:13:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7517E22226 for ; Sat, 21 Nov 2020 14:13:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="DldCBxFs" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727996AbgKUON4 (ORCPT ); Sat, 21 Nov 2020 09:13:56 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:45644 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727974AbgKUONz (ORCPT ); Sat, 21 Nov 2020 09:13:55 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968034; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Brwyd9luXNZHwUZnWyh55ctEF8pacMKJRPmRg6XxmTU=; b=DldCBxFspXmol8aP+1S7HZH8p/aAYdpz5tFhSfB6UeCexRzgn0ohfsqjuPywSOxC8wcRsX J+uHzsQUsYRP+qtuiGpRuK9Tb/eopNNtvLumLteO2WbJpR557Ggpf0nxp/6AleBv6Gluse kVJaikY8mF6UWwr2gQzSVJShjA+htK0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-196-taPHaKzFMImi3BiULlzl2A-1; Sat, 21 Nov 2020 09:13:50 -0500 X-MC-Unique: taPHaKzFMImi3BiULlzl2A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A92D91005D67; Sat, 21 Nov 2020 14:13:48 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 160005C22B; Sat, 21 Nov 2020 14:13:46 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 03/29] iov_iter: Split iov_iter_fault_in_readable From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:13:46 +0000 Message-ID: <160596802633.154728.149371155811186006.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split iov_iter_fault_in_readable() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index fee8e99fbb9c..280b5c9c9a9c 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -439,20 +439,23 @@ static size_t pipe_copy_page_to_iter(struct page *page, size_t offset, size_t by * Return 0 on success, or non-zero if the memory could not be accessed (i.e. * because it is an invalid address). */ -static int xxx_fault_in_readable(struct iov_iter *i, size_t bytes) +static int iovec_fault_in_readable(struct iov_iter *i, size_t bytes) { size_t skip = i->iov_offset; const struct iovec *iov; int err; struct iovec v; - if (!(iov_iter_type(i) & (ITER_BVEC|ITER_KVEC))) { - iterate_iovec(i, bytes, v, iov, skip, ({ - err = fault_in_pages_readable(v.iov_base, v.iov_len); - if (unlikely(err)) - return err; - 0;})) - } + iterate_iovec(i, bytes, v, iov, skip, ({ + err = fault_in_pages_readable(v.iov_base, v.iov_len); + if (unlikely(err)) + return err; + 0;})) + return 0; +} + +static int no_fault_in_readable(struct iov_iter *i, size_t bytes) +{ return 0; } @@ -1846,7 +1849,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .copy_from_user_atomic = xxx_copy_from_user_atomic, .advance = xxx_advance, .revert = xxx_revert, - .fault_in_readable = xxx_fault_in_readable, + .fault_in_readable = iovec_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = iovec_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, @@ -1880,7 +1883,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .copy_from_user_atomic = xxx_copy_from_user_atomic, .advance = xxx_advance, .revert = xxx_revert, - .fault_in_readable = xxx_fault_in_readable, + .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = bkvec_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, @@ -1914,7 +1917,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .copy_from_user_atomic = xxx_copy_from_user_atomic, .advance = xxx_advance, .revert = xxx_revert, - .fault_in_readable = xxx_fault_in_readable, + .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = bkvec_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, @@ -1948,7 +1951,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .copy_from_user_atomic = xxx_copy_from_user_atomic, .advance = xxx_advance, .revert = xxx_revert, - .fault_in_readable = xxx_fault_in_readable, + .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = pipe_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, @@ -1982,7 +1985,7 @@ static const struct iov_iter_ops discard_iter_ops = { .copy_from_user_atomic = xxx_copy_from_user_atomic, .advance = xxx_advance, .revert = xxx_revert, - .fault_in_readable = xxx_fault_in_readable, + .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = discard_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, From patchwork Sat Nov 21 14:13:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923407 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41BEBC56202 for ; Sat, 21 Nov 2020 14:14:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0E0D82237B for ; Sat, 21 Nov 2020 14:14:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="hyL+7+zQ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728037AbgKUOOE (ORCPT ); Sat, 21 Nov 2020 09:14:04 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:47169 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728006AbgKUOOD (ORCPT ); Sat, 21 Nov 2020 09:14:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968041; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4SrNEeqWVEcvXCl0seBrseTcT0IMzwDCjOyOwZ0vCNA=; b=hyL+7+zQdJxFsiMViiYyiT928rfjAbFs59BbBY2+vDtZP9tPxOa38S650qKAjXP1I60DyQ HxfKp6vs3BEaOjnXo56Xw+8QsRdPhc1ZypLL45J0MACuDq4mCqlS/ipb3rCjRfhQLRy/9u /0vcUBoAqWHpIm7xaZEHu60v9EBjqok= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-516-bgXRmFisNpCoZrNH8mhsRQ-1; Sat, 21 Nov 2020 09:13:57 -0500 X-MC-Unique: bgXRmFisNpCoZrNH8mhsRQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 60D23804741; Sat, 21 Nov 2020 14:13:56 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id B39F710016DB; Sat, 21 Nov 2020 14:13:54 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 04/29] iov_iter: Split the iterate_and_advance() macro From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:13:53 +0000 Message-ID: <160596803388.154728.17090770115276211012.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split the iterate_and_advance() macro into iovec, bvec, kvec and discard variants. It doesn't handle pipes. Signed-off-by: David Howells --- lib/iov_iter.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 280b5c9c9a9c..a221e7771201 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -147,6 +147,68 @@ static inline bool page_copy_sane(struct page *page, size_t offset, size_t n); } \ } +#define iterate_and_advance_iovec(i, n, v, CMD) { \ + if (unlikely(i->count < n)) \ + n = i->count; \ + if (i->count) { \ + size_t skip = i->iov_offset; \ + const struct iovec *iov; \ + struct iovec v; \ + iterate_iovec(i, n, v, iov, skip, (CMD)) \ + if (skip == iov->iov_len) { \ + iov++; \ + skip = 0; \ + } \ + i->nr_segs -= iov - i->iov; \ + i->iov = iov; \ + i->count -= n; \ + i->iov_offset = skip; \ + } \ +} + +#define iterate_and_advance_bvec(i, n, v, CMD) { \ + if (unlikely(i->count < n)) \ + n = i->count; \ + if (i->count) { \ + size_t skip = i->iov_offset; \ + const struct bio_vec *bvec = i->bvec; \ + struct bio_vec v; \ + struct bvec_iter __bi; \ + iterate_bvec(i, n, v, __bi, skip, (CMD)) \ + i->bvec = __bvec_iter_bvec(i->bvec, __bi); \ + i->nr_segs -= i->bvec - bvec; \ + skip = __bi.bi_bvec_done; \ + i->count -= n; \ + i->iov_offset = skip; \ + } \ +} + +#define iterate_and_advance_kvec(i, n, v, CMD) { \ + if (unlikely(i->count < n)) \ + n = i->count; \ + if (i->count) { \ + size_t skip = i->iov_offset; \ + const struct kvec *kvec; \ + struct kvec v; \ + iterate_kvec(i, n, v, kvec, skip, (CMD)) \ + if (skip == kvec->iov_len) { \ + kvec++; \ + skip = 0; \ + } \ + i->nr_segs -= kvec - i->kvec; \ + i->kvec = kvec; \ + i->count -= n; \ + i->iov_offset = skip; \ + } \ +} + +#define iterate_and_advance_discard(i, n) { \ + if (unlikely(i->count < n)) \ + n = i->count; \ + i->count -= n; \ + i->iov_offset += n; \ +} + static int copyout(void __user *to, const void *from, size_t n) { if (should_fail_usercopy()) From patchwork Sat Nov 21 14:14:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923409 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D7A8C64E7A for ; Sat, 21 Nov 2020 14:14:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CA6A622403 for ; Sat, 21 Nov 2020 14:14:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="fQ0jq0Sf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728053AbgKUOOK (ORCPT ); Sat, 21 Nov 2020 09:14:10 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:41340 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727974AbgKUOOK (ORCPT ); Sat, 21 Nov 2020 09:14:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968048; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lMVXr0SmpHGBj3khS+Fv45yL6polUCT+1h37uSQqPAQ=; b=fQ0jq0SfQe9Vx0RKnrAxosnaC8zTLbl6Geqdn+kzgP/zeqAGRUGCNzb54gLqQKDbmtU4Dz 1NQjTU/bTRwZmZ39hJygPdyYoA2VqqtZGDcAygFEArIbv9bost84zfFyc+KpIMnK0DhHiH aFkmvzA0vSMunKKXS17G/vpiueIiv28= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-277-9caVioyUN0WqeRNE9gUNAg-1; Sat, 21 Nov 2020 09:14:05 -0500 X-MC-Unique: 9caVioyUN0WqeRNE9gUNAg-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4712E80EDAA; Sat, 21 Nov 2020 14:14:04 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 731CD60C15; Sat, 21 Nov 2020 14:14:02 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 05/29] iov_iter: Split copy_to_iter() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:14:01 +0000 Message-ID: <160596804162.154728.9403561513393584180.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split copy_to_iter() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 47 +++++++++++++++++++++++++++++++---------------- 1 file changed, 31 insertions(+), 16 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index a221e7771201..0865e0b6eee9 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -634,7 +634,7 @@ static size_t push_pipe(struct iov_iter *i, size_t size, return size - left; } -static size_t copy_pipe_to_iter(const void *addr, size_t bytes, +static size_t pipe_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { struct pipe_inode_info *pipe = i->pipe; @@ -703,20 +703,35 @@ static size_t csum_and_copy_to_pipe_iter(const void *addr, size_t bytes, return bytes; } -static size_t xxx_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +static size_t iovec_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { const char *from = addr; - if (unlikely(iov_iter_is_pipe(i))) - return copy_pipe_to_iter(addr, bytes, i); - if (iter_is_iovec(i)) - might_fault(); - iterate_and_advance(i, bytes, v, - copyout(v.iov_base, (from += v.iov_len) - v.iov_len, v.iov_len), + might_fault(); + iterate_and_advance_iovec(i, bytes, v, + copyout(v.iov_base, (from += v.iov_len) - v.iov_len, v.iov_len)); + return bytes; +} + +static size_t bvec_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +{ + const char *from = addr; + iterate_and_advance_bvec(i, bytes, v, memcpy_to_page(v.bv_page, v.bv_offset, - (from += v.bv_len) - v.bv_len, v.bv_len), - memcpy(v.iov_base, (from += v.iov_len) - v.iov_len, v.iov_len) - ) + (from += v.bv_len) - v.bv_len, v.bv_len)); + return bytes; +} +static size_t kvec_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +{ + const char *from = addr; + iterate_and_advance_kvec(i, bytes, v, + memcpy(v.iov_base, (from += v.iov_len) - v.iov_len, v.iov_len)); + return bytes; +} + +static size_t discard_copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +{ + iterate_and_advance_discard(i, bytes); return bytes; } @@ -1915,7 +1930,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = iovec_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, - .copy_to_iter = xxx_copy_to_iter, + .copy_to_iter = iovec_copy_to_iter, .copy_from_iter = xxx_copy_from_iter, .copy_from_iter_full = xxx_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, @@ -1949,7 +1964,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = bkvec_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, - .copy_to_iter = xxx_copy_to_iter, + .copy_to_iter = kvec_copy_to_iter, .copy_from_iter = xxx_copy_from_iter, .copy_from_iter_full = xxx_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, @@ -1983,7 +1998,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = bkvec_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, - .copy_to_iter = xxx_copy_to_iter, + .copy_to_iter = bvec_copy_to_iter, .copy_from_iter = xxx_copy_from_iter, .copy_from_iter_full = xxx_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, @@ -2017,7 +2032,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = pipe_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, - .copy_to_iter = xxx_copy_to_iter, + .copy_to_iter = pipe_copy_to_iter, .copy_from_iter = xxx_copy_from_iter, .copy_from_iter_full = xxx_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, @@ -2051,7 +2066,7 @@ static const struct iov_iter_ops discard_iter_ops = { .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = discard_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, - .copy_to_iter = xxx_copy_to_iter, + .copy_to_iter = discard_copy_to_iter, .copy_from_iter = xxx_copy_from_iter, .copy_from_iter_full = xxx_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, From patchwork Sat Nov 21 14:14:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51724C64E69 for ; Sat, 21 Nov 2020 14:14:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 23B4D22264 for ; Sat, 21 Nov 2020 14:14:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="BlVA+lnG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728076AbgKUOOU (ORCPT ); Sat, 21 Nov 2020 09:14:20 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:31702 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728040AbgKUOOT (ORCPT ); Sat, 21 Nov 2020 09:14:19 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968057; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cts/UysbCeke0SRrY+6cxZefIvVJUV/6NiBfhTcZI1c=; b=BlVA+lnGqGffc6nIFQ/igEg8utlPIK9OhyA5WDt4jwRua3eRP3WcXbKu+7gH715pLuHBik inr2AKj2JuDfy/6SPh7jYL6bBq1RpVyNMQfTMbUxKXUvk/v6GDM0GYsagZkgRwjYm5R3XH r2e2bput9vEi/6z5jr+loPh2sO/QOCw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-592-05YIL5MQOp25jyuwwWcgug-1; Sat, 21 Nov 2020 09:14:13 -0500 X-MC-Unique: 05YIL5MQOp25jyuwwWcgug-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2BD94804766; Sat, 21 Nov 2020 14:14:12 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 59C9D5C253; Sat, 21 Nov 2020 14:14:10 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 06/29] iov_iter: Split copy_mc_to_iter() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:14:09 +0000 Message-ID: <160596804951.154728.12388891412575059303.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split copy_mc_to_iter() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 54 +++++++++++++++++++++++++++++++++--------------------- 1 file changed, 33 insertions(+), 21 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 0865e0b6eee9..7c1d92f7d020 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -758,7 +758,7 @@ static unsigned long copy_mc_to_page(struct page *page, size_t offset, return ret; } -static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes, +static size_t pipe_copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { struct pipe_inode_info *pipe = i->pipe; @@ -815,18 +815,23 @@ static size_t copy_mc_pipe_to_iter(const void *addr, size_t bytes, * Compare to copy_to_iter() where only ITER_IOVEC attempts might return * a short copy. */ -static size_t xxx_copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +static size_t iovec_copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { const char *from = addr; - unsigned long rem, curr_addr, s_addr = (unsigned long) addr; - if (unlikely(iov_iter_is_pipe(i))) - return copy_mc_pipe_to_iter(addr, bytes, i); - if (iter_is_iovec(i)) - might_fault(); - iterate_and_advance(i, bytes, v, + might_fault(); + iterate_and_advance_iovec(i, bytes, v, copyout_mc(v.iov_base, (from += v.iov_len) - v.iov_len, - v.iov_len), + v.iov_len)); + return bytes; +} + +static size_t bvec_copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +{ + const char *from = addr; + unsigned long rem, curr_addr, s_addr = (unsigned long) addr; + + iterate_and_advance_bvec(i, bytes, v, ({ rem = copy_mc_to_page(v.bv_page, v.bv_offset, (from += v.bv_len) - v.bv_len, v.bv_len); @@ -835,18 +840,25 @@ static size_t xxx_copy_mc_to_iter(const void *addr, size_t bytes, struct iov_ite bytes = curr_addr - s_addr - rem; return bytes; } - }), - ({ - rem = copy_mc_to_kernel(v.iov_base, (from += v.iov_len) - - v.iov_len, v.iov_len); + })) + return bytes; +} + +static size_t kvec_copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) +{ + const char *from = addr; + unsigned long rem, curr_addr, s_addr = (unsigned long) addr; + + iterate_and_advance_kvec(i, bytes, v, ({ + rem = copy_mc_to_kernel(v.iov_base, + (from += v.iov_len) - v.iov_len, + v.iov_len); if (rem) { curr_addr = (unsigned long) from; bytes = curr_addr - s_addr - rem; return bytes; } - }) - ) - + })); return bytes; } #endif /* CONFIG_ARCH_HAS_COPY_MC */ @@ -1939,7 +1951,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, #endif #ifdef CONFIG_ARCH_HAS_COPY_MC - .copy_mc_to_iter = xxx_copy_mc_to_iter, + .copy_mc_to_iter = iovec_copy_mc_to_iter, #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, @@ -1973,7 +1985,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, #endif #ifdef CONFIG_ARCH_HAS_COPY_MC - .copy_mc_to_iter = xxx_copy_mc_to_iter, + .copy_mc_to_iter = kvec_copy_mc_to_iter, #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, @@ -2007,7 +2019,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, #endif #ifdef CONFIG_ARCH_HAS_COPY_MC - .copy_mc_to_iter = xxx_copy_mc_to_iter, + .copy_mc_to_iter = bvec_copy_mc_to_iter, #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, @@ -2041,7 +2053,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, #endif #ifdef CONFIG_ARCH_HAS_COPY_MC - .copy_mc_to_iter = xxx_copy_mc_to_iter, + .copy_mc_to_iter = pipe_copy_mc_to_iter, #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, @@ -2075,7 +2087,7 @@ static const struct iov_iter_ops discard_iter_ops = { .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, #endif #ifdef CONFIG_ARCH_HAS_COPY_MC - .copy_mc_to_iter = xxx_copy_mc_to_iter, + .copy_mc_to_iter = discard_copy_to_iter, #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, From patchwork Sat Nov 21 14:14:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05681C388F9 for ; Sat, 21 Nov 2020 14:14:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C27D02245F for ; Sat, 21 Nov 2020 14:14:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XFSKZimY" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728046AbgKUOO2 (ORCPT ); Sat, 21 Nov 2020 09:14:28 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:48233 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728077AbgKUOO2 (ORCPT ); Sat, 21 Nov 2020 09:14:28 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968066; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=io9b5GXM3K4SaRCqdO0RNvHKl1y+Lut3sf8uCdt9yCQ=; b=XFSKZimYvEsY7uaBXAdbb+ckT32s/Rdh/PA6pwdvNLBNgY3hU6xewYNzYhuEKy46aYlVPQ 9PhRh39+qEX0MxFk74CtpWBVSguTlq61qPDahhkQNVPTPqLFUzfB6vl+jY5vTXKWIEb9O3 KqhKby8KJNmN4kxSItVNHJDEMV45OlM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-141-n32Qrx4qM7KKa3j2SHR7Jg-1; Sat, 21 Nov 2020 09:14:21 -0500 X-MC-Unique: n32Qrx4qM7KKa3j2SHR7Jg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 17701107AD44; Sat, 21 Nov 2020 14:14:20 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 329005C22B; Sat, 21 Nov 2020 14:14:18 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 07/29] iov_iter: Split copy_from_iter() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:14:17 +0000 Message-ID: <160596805741.154728.12190794089892831917.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split copy_from_iter() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 50 ++++++++++++++++++++++++++++++++------------------ 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 7c1d92f7d020..5b18dfe0dcc7 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -863,22 +863,36 @@ static size_t kvec_copy_mc_to_iter(const void *addr, size_t bytes, struct iov_it } #endif /* CONFIG_ARCH_HAS_COPY_MC */ -static size_t xxx_copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) +static size_t iovec_copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) { char *to = addr; - if (unlikely(iov_iter_is_pipe(i))) { - WARN_ON(1); - return 0; - } - if (iter_is_iovec(i)) - might_fault(); - iterate_and_advance(i, bytes, v, - copyin((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len), + might_fault(); + iterate_and_advance_iovec(i, bytes, v, + copyin((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len)); + + return bytes; +} + +static size_t bvec_copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) +{ + char *to = addr; + iterate_and_advance_bvec(i, bytes, v, memcpy_from_page((to += v.bv_len) - v.bv_len, v.bv_page, - v.bv_offset, v.bv_len), - memcpy((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len) - ) + v.bv_offset, v.bv_len)); + return bytes; +} + +static size_t kvec_copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) +{ + char *to = addr; + iterate_and_advance_kvec(i, bytes, v, + memcpy((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len)); + return bytes; +} +static size_t no_copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) +{ + WARN_ON(1); return bytes; } @@ -1037,7 +1051,7 @@ static size_t xxx_copy_page_from_iter(struct page *page, size_t offset, size_t b } if (iov_iter_type(i) & (ITER_BVEC|ITER_KVEC)) { void *kaddr = kmap_atomic(page); - size_t wanted = xxx_copy_from_iter(kaddr + offset, bytes, i); + size_t wanted = copy_from_iter(kaddr + offset, bytes, i); kunmap_atomic(kaddr); return wanted; } else @@ -1943,7 +1957,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .copy_page_to_iter = iovec_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = iovec_copy_to_iter, - .copy_from_iter = xxx_copy_from_iter, + .copy_from_iter = iovec_copy_from_iter, .copy_from_iter_full = xxx_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, @@ -1977,7 +1991,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .copy_page_to_iter = bkvec_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = kvec_copy_to_iter, - .copy_from_iter = xxx_copy_from_iter, + .copy_from_iter = kvec_copy_from_iter, .copy_from_iter_full = xxx_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, @@ -2011,7 +2025,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .copy_page_to_iter = bkvec_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = bvec_copy_to_iter, - .copy_from_iter = xxx_copy_from_iter, + .copy_from_iter = bvec_copy_from_iter, .copy_from_iter_full = xxx_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, @@ -2045,7 +2059,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .copy_page_to_iter = pipe_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = pipe_copy_to_iter, - .copy_from_iter = xxx_copy_from_iter, + .copy_from_iter = no_copy_from_iter, .copy_from_iter_full = xxx_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, @@ -2079,7 +2093,7 @@ static const struct iov_iter_ops discard_iter_ops = { .copy_page_to_iter = discard_copy_page_to_iter, .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = discard_copy_to_iter, - .copy_from_iter = xxx_copy_from_iter, + .copy_from_iter = no_copy_from_iter, .copy_from_iter_full = xxx_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, From patchwork Sat Nov 21 14:14:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5928AC388F9 for ; Sat, 21 Nov 2020 14:15:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1BC692224A for ; Sat, 21 Nov 2020 14:15:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="L77E540W" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728092AbgKUOOd (ORCPT ); Sat, 21 Nov 2020 09:14:33 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:22070 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728040AbgKUOOc (ORCPT ); Sat, 21 Nov 2020 09:14:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968071; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MOxzf6AmZSNKfEnpa+Mn4aEC9Xg4TZA2VqmfqOCv8zg=; b=L77E540W9mg+s/OBDskT/IDXbbQ4CbzltHSUnV5GZP6fZb8wpfGF8hgxAR2kdsrGryWh12 RdBjDvGj3JBKJOXqL6yXu0nSp1PIkxpyioWfoZ7QTADruirODRznK/bzyiZ0t6NcTEJTKC F4Q/JCO5R2ilqzcUWRqsdgubXMpRxTc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-163-EkozzKeyPzuaN7Np9QIgOw-1; Sat, 21 Nov 2020 09:14:29 -0500 X-MC-Unique: EkozzKeyPzuaN7Np9QIgOw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A458780ED8E; Sat, 21 Nov 2020 14:14:27 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1DBE35D6BA; Sat, 21 Nov 2020 14:14:25 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 08/29] iov_iter: Split the iterate_all_kinds() macro From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:14:25 +0000 Message-ID: <160596806531.154728.6113611661561602930.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split the iterate_all_kinds() macro into iovec, bvec and kvec variants. It doesn't handle pipes and the discard variant is a no-op and can be built in directly. Signed-off-by: David Howells --- lib/iov_iter.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 5b18dfe0dcc7..934193627540 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -106,6 +106,33 @@ static inline bool page_copy_sane(struct page *page, size_t offset, size_t n); } \ } +#define iterate_over_iovec(i, n, v, CMD) { \ + if (likely(n)) { \ + size_t skip = i->iov_offset; \ + const struct iovec *iov; \ + struct iovec v; \ + iterate_iovec(i, n, v, iov, skip, (CMD)) \ + } \ +} + +#define iterate_over_bvec(i, n, v, CMD) { \ + if (likely(n)) { \ + size_t skip = i->iov_offset; \ + struct bio_vec v; \ + struct bvec_iter __bi; \ + iterate_bvec(i, n, v, __bi, skip, (CMD)) \ + } \ +} + +#define iterate_over_kvec(i, n, v, CMD) { \ + if (likely(n)) { \ + size_t skip = i->iov_offset; \ + const struct kvec *kvec; \ + struct kvec v; \ + iterate_kvec(i, n, v, kvec, skip, (CMD)) \ + } \ +} + #define iterate_and_advance(i, n, v, I, B, K) { \ if (unlikely(i->count < n)) \ n = i->count; \ From patchwork Sat Nov 21 14:14:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B240CC63798 for ; Sat, 21 Nov 2020 14:15:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8127122264 for ; Sat, 21 Nov 2020 14:15:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="TazDcVbO" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728121AbgKUOOl (ORCPT ); Sat, 21 Nov 2020 09:14:41 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:20850 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728112AbgKUOOk (ORCPT ); Sat, 21 Nov 2020 09:14:40 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968079; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nYwhg3i3DgX8/KOErE9/+meufmLn4/JOQzdQe5HA0zA=; b=TazDcVbO5evOtOl3WAqdSm+O7RA9ordXSHiVpPQ1fl+AQdymhKgHbXtWzkUUR75+zNeHTQ eLWwn1lUuMl6kCjC0gh6t8KmFL1EzBuou55nL8PijbTAnpsFueXsMDSKka4G3MEhjxOPjq wXq4LO4xO6YHY+cdwLHTR+ksplOWil8= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-295-NJtFoOwfOlS7D78FCgVuVw-1; Sat, 21 Nov 2020 09:14:37 -0500 X-MC-Unique: NJtFoOwfOlS7D78FCgVuVw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8494C1005D65; Sat, 21 Nov 2020 14:14:35 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id B1CAA6085D; Sat, 21 Nov 2020 14:14:33 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 09/29] iov_iter: Split copy_from_iter_full() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:14:32 +0000 Message-ID: <160596807288.154728.10950334371240472423.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split copy_from_iter_full() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 59 +++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 41 insertions(+), 18 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 934193627540..3dba665a1ee9 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -923,32 +923,55 @@ static size_t no_copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) return bytes; } -static bool xxx_copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i) +static bool iovec_copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i) { char *to = addr; - if (unlikely(iov_iter_is_pipe(i))) { - WARN_ON(1); - return false; - } + if (unlikely(i->count < bytes)) return false; - if (iter_is_iovec(i)) - might_fault(); - iterate_all_kinds(i, bytes, v, ({ + might_fault(); + iterate_over_iovec(i, bytes, v, ({ if (copyin((to += v.iov_len) - v.iov_len, - v.iov_base, v.iov_len)) + v.iov_base, v.iov_len)) return false; - 0;}), + 0;})); + iov_iter_advance(i, bytes); + return true; +} + +static bool bvec_copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i) +{ + char *to = addr; + + if (unlikely(i->count < bytes)) + return false; + iterate_over_bvec(i, bytes, v, memcpy_from_page((to += v.bv_len) - v.bv_len, v.bv_page, - v.bv_offset, v.bv_len), - memcpy((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len) - ) + v.bv_offset, v.bv_len)); + iov_iter_advance(i, bytes); + return true; +} + +static bool kvec_copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i) +{ + char *to = addr; + if (unlikely(i->count < bytes)) + return false; + + iterate_over_kvec(i, bytes, v, + memcpy((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len)); iov_iter_advance(i, bytes); return true; } +static bool no_copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i) +{ + WARN_ON(1); + return false; +} + static size_t xxx_copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) { char *to = addr; @@ -1985,7 +2008,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = iovec_copy_to_iter, .copy_from_iter = iovec_copy_from_iter, - .copy_from_iter_full = xxx_copy_from_iter_full, + .copy_from_iter_full = iovec_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE @@ -2019,7 +2042,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = kvec_copy_to_iter, .copy_from_iter = kvec_copy_from_iter, - .copy_from_iter_full = xxx_copy_from_iter_full, + .copy_from_iter_full = kvec_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE @@ -2053,7 +2076,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = bvec_copy_to_iter, .copy_from_iter = bvec_copy_from_iter, - .copy_from_iter_full = xxx_copy_from_iter_full, + .copy_from_iter_full = bvec_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE @@ -2087,7 +2110,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = pipe_copy_to_iter, .copy_from_iter = no_copy_from_iter, - .copy_from_iter_full = xxx_copy_from_iter_full, + .copy_from_iter_full = no_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE @@ -2121,7 +2144,7 @@ static const struct iov_iter_ops discard_iter_ops = { .copy_page_from_iter = xxx_copy_page_from_iter, .copy_to_iter = discard_copy_to_iter, .copy_from_iter = no_copy_from_iter, - .copy_from_iter_full = xxx_copy_from_iter_full, + .copy_from_iter_full = no_copy_from_iter_full, .copy_from_iter_nocache = xxx_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE From patchwork Sat Nov 21 14:14:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81DAAC64E7A for ; Sat, 21 Nov 2020 14:15:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3656F22403 for ; Sat, 21 Nov 2020 14:15:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="R7Ziw8r6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728138AbgKUOOv (ORCPT ); Sat, 21 Nov 2020 09:14:51 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:34897 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728078AbgKUOOu (ORCPT ); Sat, 21 Nov 2020 09:14:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968088; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aRdtaSmCdug5+jFBzzaX1HskS8uwScH3nKxj3/t7xpo=; b=R7Ziw8r6x8YROxYVvpse4xBadCbZ1a9k8MBm8hL3DZ6DbCpbpB03RREocDbJHSDCAEYtGO GxawUo51XBL/BhcGexQ1v82g1ve9VTxSwijEUmt99eHecW7VhlS/VTHUHrnSbX68487kLn E10yqJoYGvm9p79tI41hdr8M6w/klcw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-234-jgjAXMVQNoecXTGmFFrtug-1; Sat, 21 Nov 2020 09:14:44 -0500 X-MC-Unique: jgjAXMVQNoecXTGmFFrtug-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 41F721DDE7; Sat, 21 Nov 2020 14:14:43 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8AD4C5C22B; Sat, 21 Nov 2020 14:14:41 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 10/29] iov_iter: Split copy_from_iter_nocache() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:14:40 +0000 Message-ID: <160596808077.154728.7004427791292645021.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split copy_from_iter_nocache() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 3dba665a1ee9..c57c2171f730 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -972,21 +972,29 @@ static bool no_copy_from_iter_full(void *addr, size_t bytes, struct iov_iter *i) return false; } -static size_t xxx_copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) +static size_t iovec_copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) { char *to = addr; - if (unlikely(iov_iter_is_pipe(i))) { - WARN_ON(1); - return 0; - } - iterate_and_advance(i, bytes, v, + iterate_and_advance_iovec(i, bytes, v, __copy_from_user_inatomic_nocache((to += v.iov_len) - v.iov_len, - v.iov_base, v.iov_len), + v.iov_base, v.iov_len)); + return bytes; +} + +static size_t bvec_copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) +{ + char *to = addr; + iterate_and_advance_bvec(i, bytes, v, memcpy_from_page((to += v.bv_len) - v.bv_len, v.bv_page, - v.bv_offset, v.bv_len), - memcpy((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len) - ) + v.bv_offset, v.bv_len)); + return bytes; +} +static size_t kvec_copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) +{ + char *to = addr; + iterate_and_advance_kvec(i, bytes, v, + memcpy((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len)); return bytes; } @@ -2009,7 +2017,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .copy_to_iter = iovec_copy_to_iter, .copy_from_iter = iovec_copy_from_iter, .copy_from_iter_full = iovec_copy_from_iter_full, - .copy_from_iter_nocache = xxx_copy_from_iter_nocache, + .copy_from_iter_nocache = iovec_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, @@ -2043,7 +2051,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .copy_to_iter = kvec_copy_to_iter, .copy_from_iter = kvec_copy_from_iter, .copy_from_iter_full = kvec_copy_from_iter_full, - .copy_from_iter_nocache = xxx_copy_from_iter_nocache, + .copy_from_iter_nocache = kvec_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, @@ -2077,7 +2085,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .copy_to_iter = bvec_copy_to_iter, .copy_from_iter = bvec_copy_from_iter, .copy_from_iter_full = bvec_copy_from_iter_full, - .copy_from_iter_nocache = xxx_copy_from_iter_nocache, + .copy_from_iter_nocache = bvec_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, @@ -2111,7 +2119,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .copy_to_iter = pipe_copy_to_iter, .copy_from_iter = no_copy_from_iter, .copy_from_iter_full = no_copy_from_iter_full, - .copy_from_iter_nocache = xxx_copy_from_iter_nocache, + .copy_from_iter_nocache = no_copy_from_iter, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, @@ -2145,7 +2153,7 @@ static const struct iov_iter_ops discard_iter_ops = { .copy_to_iter = discard_copy_to_iter, .copy_from_iter = no_copy_from_iter, .copy_from_iter_full = no_copy_from_iter_full, - .copy_from_iter_nocache = xxx_copy_from_iter_nocache, + .copy_from_iter_nocache = no_copy_from_iter, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, From patchwork Sat Nov 21 14:14:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0F7EC64E7B for ; Sat, 21 Nov 2020 14:15:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 89C1D22261 for ; Sat, 21 Nov 2020 14:15:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="gHWyXz+T" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728155AbgKUOO5 (ORCPT ); Sat, 21 Nov 2020 09:14:57 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:22071 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728145AbgKUOO4 (ORCPT ); Sat, 21 Nov 2020 09:14:56 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968094; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ni5MPjwKtqOf/JECYK0nOWDI9zN4ZAqVX24jMSlVJjE=; b=gHWyXz+Tq07ksNJR4b3WtxH5QboTW7quBb6jw686/QVfq4yy+UHEP1yR6mfXeu4DJGskKT 8pn0H1hxgfwutsu7mpCRz3tuGiLHwwOk0d+pv4fmsqjRXz4J3b4YxN+bY2/1o/Hu8r184R KJy0JFXYqAul4mnpwuFoR3gdyuxEyGY= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-524-z7_-QuhqN5ODuAEHuaWKyg-1; Sat, 21 Nov 2020 09:14:52 -0500 X-MC-Unique: z7_-QuhqN5ODuAEHuaWKyg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CE5521DDE7; Sat, 21 Nov 2020 14:14:50 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3C85C5D6BA; Sat, 21 Nov 2020 14:14:49 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 11/29] iov_iter: Split copy_from_iter_flushcache() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:14:48 +0000 Message-ID: <160596808849.154728.14090146839666312433.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split copy_from_iter_flushcache() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 42 +++++++++++++++++++++++++----------------- 1 file changed, 25 insertions(+), 17 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index c57c2171f730..6b4739d7dd9a 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1000,7 +1000,7 @@ static size_t kvec_copy_from_iter_nocache(void *addr, size_t bytes, struct iov_i #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE /** - * _copy_from_iter_flushcache - write destination through cpu cache + * copy_from_iter_flushcache - write destination through cpu cache * @addr: destination kernel address * @bytes: total transfer length * @iter: source iterator @@ -1013,22 +1013,30 @@ static size_t kvec_copy_from_iter_nocache(void *addr, size_t bytes, struct iov_i * bypass the cache for the ITER_IOVEC case, and on some archs may use * instructions that strand dirty-data in the cache. */ -static size_t xxx_copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) +static size_t iovec_copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) { char *to = addr; - if (unlikely(iov_iter_is_pipe(i))) { - WARN_ON(1); - return 0; - } - iterate_and_advance(i, bytes, v, + iterate_and_advance_iovec(i, bytes, v, __copy_from_user_flushcache((to += v.iov_len) - v.iov_len, - v.iov_base, v.iov_len), + v.iov_base, v.iov_len)); + return bytes; +} + +static size_t bvec_copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) +{ + char *to = addr; + iterate_and_advance_bvec(i, bytes, v, memcpy_page_flushcache((to += v.bv_len) - v.bv_len, v.bv_page, - v.bv_offset, v.bv_len), - memcpy_flushcache((to += v.iov_len) - v.iov_len, v.iov_base, - v.iov_len) - ) + v.bv_offset, v.bv_len)); + return bytes; +} +static size_t kvec_copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) +{ + char *to = addr; + iterate_and_advance_kvec(i, bytes, v, + memcpy_flushcache((to += v.iov_len) - v.iov_len, v.iov_base, + v.iov_len)); return bytes; } #endif @@ -2020,7 +2028,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .copy_from_iter_nocache = iovec_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE - .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, + .copy_from_iter_flushcache = iovec_copy_from_iter_flushcache, #endif #ifdef CONFIG_ARCH_HAS_COPY_MC .copy_mc_to_iter = iovec_copy_mc_to_iter, @@ -2054,7 +2062,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .copy_from_iter_nocache = kvec_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE - .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, + .copy_from_iter_flushcache = kvec_copy_from_iter_flushcache, #endif #ifdef CONFIG_ARCH_HAS_COPY_MC .copy_mc_to_iter = kvec_copy_mc_to_iter, @@ -2088,7 +2096,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .copy_from_iter_nocache = bvec_copy_from_iter_nocache, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE - .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, + .copy_from_iter_flushcache = bvec_copy_from_iter_flushcache, #endif #ifdef CONFIG_ARCH_HAS_COPY_MC .copy_mc_to_iter = bvec_copy_mc_to_iter, @@ -2122,7 +2130,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .copy_from_iter_nocache = no_copy_from_iter, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE - .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, + .copy_from_iter_flushcache = no_copy_from_iter, #endif #ifdef CONFIG_ARCH_HAS_COPY_MC .copy_mc_to_iter = pipe_copy_mc_to_iter, @@ -2156,7 +2164,7 @@ static const struct iov_iter_ops discard_iter_ops = { .copy_from_iter_nocache = no_copy_from_iter, .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE - .copy_from_iter_flushcache = xxx_copy_from_iter_flushcache, + .copy_from_iter_flushcache = no_copy_from_iter, #endif #ifdef CONFIG_ARCH_HAS_COPY_MC .copy_mc_to_iter = discard_copy_to_iter, From patchwork Sat Nov 21 14:14:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96A98C388F9 for ; Sat, 21 Nov 2020 14:15:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4C1AD22226 for ; Sat, 21 Nov 2020 14:15:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="FQJszw0i" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728166AbgKUOPE (ORCPT ); Sat, 21 Nov 2020 09:15:04 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:58733 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728146AbgKUOPD (ORCPT ); Sat, 21 Nov 2020 09:15:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968102; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CVU2Fb/xVpWFVdtrZFqElVvemnq4TZD7WPoTZ9/Kuds=; b=FQJszw0iyfkMO1rPGEqpu441PMG4wZ3fUtlsRk2ikeGQ9oKWz8193hbhwb0onyOBnNUkOD Ps8inJaflh1lAaxTQW1qfJHIpmyNKlBm5N3y5LvAQNG/4hSuovdvLTQ6ODnB7O0gpZuX1v kizBpQnGc01Gr7xTR7/WYFGnCcqEEdc= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-494-4Q0AbEC-Pj2Mpja_toWsPA-1; Sat, 21 Nov 2020 09:15:00 -0500 X-MC-Unique: 4Q0AbEC-Pj2Mpja_toWsPA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 889C0107AD46; Sat, 21 Nov 2020 14:14:58 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id CE2C51975F; Sat, 21 Nov 2020 14:14:56 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 12/29] iov_iter: Split copy_from_iter_full_nocache() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:14:56 +0000 Message-ID: <160596809604.154728.18003023672281785295.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split copy_from_iter_full_nocache() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 44 +++++++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 6b4739d7dd9a..544e532e3e9f 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1041,25 +1041,39 @@ static size_t kvec_copy_from_iter_flushcache(void *addr, size_t bytes, struct io } #endif -static bool xxx_copy_from_iter_full_nocache(void *addr, size_t bytes, struct iov_iter *i) +static bool iovec_copy_from_iter_full_nocache(void *addr, size_t bytes, struct iov_iter *i) { char *to = addr; - if (unlikely(iov_iter_is_pipe(i))) { - WARN_ON(1); - return false; - } if (unlikely(i->count < bytes)) return false; - iterate_all_kinds(i, bytes, v, ({ + iterate_over_iovec(i, bytes, v, ({ if (__copy_from_user_inatomic_nocache((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len)) return false; - 0;}), + 0;})); + iov_iter_advance(i, bytes); + return true; +} + +static bool bvec_copy_from_iter_full_nocache(void *addr, size_t bytes, struct iov_iter *i) +{ + char *to = addr; + if (unlikely(i->count < bytes)) + return false; + iterate_over_bvec(i, bytes, v, memcpy_from_page((to += v.bv_len) - v.bv_len, v.bv_page, - v.bv_offset, v.bv_len), - memcpy((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len) - ) + v.bv_offset, v.bv_len)); + iov_iter_advance(i, bytes); + return true; +} +static bool kvec_copy_from_iter_full_nocache(void *addr, size_t bytes, struct iov_iter *i) +{ + char *to = addr; + if (unlikely(i->count < bytes)) + return false; + iterate_over_kvec(i, bytes, v, + memcpy((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len)); iov_iter_advance(i, bytes); return true; } @@ -2026,7 +2040,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .copy_from_iter = iovec_copy_from_iter, .copy_from_iter_full = iovec_copy_from_iter_full, .copy_from_iter_nocache = iovec_copy_from_iter_nocache, - .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, + .copy_from_iter_full_nocache = iovec_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE .copy_from_iter_flushcache = iovec_copy_from_iter_flushcache, #endif @@ -2060,7 +2074,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .copy_from_iter = kvec_copy_from_iter, .copy_from_iter_full = kvec_copy_from_iter_full, .copy_from_iter_nocache = kvec_copy_from_iter_nocache, - .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, + .copy_from_iter_full_nocache = kvec_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE .copy_from_iter_flushcache = kvec_copy_from_iter_flushcache, #endif @@ -2094,7 +2108,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .copy_from_iter = bvec_copy_from_iter, .copy_from_iter_full = bvec_copy_from_iter_full, .copy_from_iter_nocache = bvec_copy_from_iter_nocache, - .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, + .copy_from_iter_full_nocache = bvec_copy_from_iter_full_nocache, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE .copy_from_iter_flushcache = bvec_copy_from_iter_flushcache, #endif @@ -2128,7 +2142,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .copy_from_iter = no_copy_from_iter, .copy_from_iter_full = no_copy_from_iter_full, .copy_from_iter_nocache = no_copy_from_iter, - .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, + .copy_from_iter_full_nocache = no_copy_from_iter_full, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE .copy_from_iter_flushcache = no_copy_from_iter, #endif @@ -2162,7 +2176,7 @@ static const struct iov_iter_ops discard_iter_ops = { .copy_from_iter = no_copy_from_iter, .copy_from_iter_full = no_copy_from_iter_full, .copy_from_iter_nocache = no_copy_from_iter, - .copy_from_iter_full_nocache = xxx_copy_from_iter_full_nocache, + .copy_from_iter_full_nocache = no_copy_from_iter_full, #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE .copy_from_iter_flushcache = no_copy_from_iter, #endif From patchwork Sat Nov 21 14:15:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19001C64EBC for ; Sat, 21 Nov 2020 14:15:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D3CF022261 for ; Sat, 21 Nov 2020 14:15:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="QZV70xkO" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728193AbgKUOPQ (ORCPT ); Sat, 21 Nov 2020 09:15:16 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:25686 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728178AbgKUOPO (ORCPT ); Sat, 21 Nov 2020 09:15:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968113; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ImAvip0pcYDz2Ziq1ISGpj9wOLsuHLQraCd9SE2xGcg=; b=QZV70xkOqXL2eEizlQip/PF8zU431M2TE7maDZHLGkO7JLYaxVlISmfWgkEEi2bZoWruNn rBr34oknDMhaarQS4T3WV3OT+/AZDvvW21X8tFPWvrsZQf79mUUU1ktVrBaCP/386Fell8 9NgAHcBq8zlgiKZGHC3+yvMbbz/BMv0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-163-1ygIhq0NPK2LnOwvAFWPPw-1; Sat, 21 Nov 2020 09:15:08 -0500 X-MC-Unique: 1ygIhq0NPK2LnOwvAFWPPw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BD69B814266; Sat, 21 Nov 2020 14:15:06 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 985E65C234; Sat, 21 Nov 2020 14:15:04 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 13/29] iov_iter: Split copy_page_from_iter() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:15:03 +0000 Message-ID: <160596810377.154728.16198787469062466076.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split copy_page_from_iter() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 544e532e3e9f..54029aeab3ec 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -344,7 +344,7 @@ static size_t iovec_copy_page_to_iter(struct page *page, size_t offset, size_t b return wanted - bytes; } -static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t bytes, +static size_t iovec_copy_page_from_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { size_t skip, copy, left, wanted; @@ -352,6 +352,8 @@ static size_t copy_page_from_iter_iovec(struct page *page, size_t offset, size_t char __user *buf; void *kaddr, *to; + if (unlikely(!page_copy_sane(page, offset, bytes))) + return 0; if (unlikely(bytes > i->count)) bytes = i->count; @@ -1120,22 +1122,23 @@ static size_t discard_copy_page_to_iter(struct page *page, size_t offset, size_t return bytes; } -static size_t xxx_copy_page_from_iter(struct page *page, size_t offset, size_t bytes, +static size_t bkvec_copy_page_from_iter(struct page *page, size_t offset, size_t bytes, struct iov_iter *i) { - if (unlikely(!page_copy_sane(page, offset, bytes))) - return 0; - if (unlikely(iov_iter_is_pipe(i) || iov_iter_is_discard(i))) { - WARN_ON(1); - return 0; - } - if (iov_iter_type(i) & (ITER_BVEC|ITER_KVEC)) { + size_t wanted = 0; + if (likely(page_copy_sane(page, offset, bytes))) { void *kaddr = kmap_atomic(page); - size_t wanted = copy_from_iter(kaddr + offset, bytes, i); + wanted = copy_from_iter(kaddr + offset, bytes, i); kunmap_atomic(kaddr); - return wanted; - } else - return copy_page_from_iter_iovec(page, offset, bytes, i); + } + return wanted; +} + +static size_t no_copy_page_from_iter(struct page *page, size_t offset, size_t bytes, + struct iov_iter *i) +{ + WARN_ON(1); + return 0; } static size_t pipe_zero(size_t bytes, struct iov_iter *i) @@ -2035,7 +2038,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .fault_in_readable = iovec_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = iovec_copy_page_to_iter, - .copy_page_from_iter = xxx_copy_page_from_iter, + .copy_page_from_iter = iovec_copy_page_from_iter, .copy_to_iter = iovec_copy_to_iter, .copy_from_iter = iovec_copy_from_iter, .copy_from_iter_full = iovec_copy_from_iter_full, @@ -2069,7 +2072,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = bkvec_copy_page_to_iter, - .copy_page_from_iter = xxx_copy_page_from_iter, + .copy_page_from_iter = bkvec_copy_page_from_iter, .copy_to_iter = kvec_copy_to_iter, .copy_from_iter = kvec_copy_from_iter, .copy_from_iter_full = kvec_copy_from_iter_full, @@ -2103,7 +2106,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = bkvec_copy_page_to_iter, - .copy_page_from_iter = xxx_copy_page_from_iter, + .copy_page_from_iter = bkvec_copy_page_from_iter, .copy_to_iter = bvec_copy_to_iter, .copy_from_iter = bvec_copy_from_iter, .copy_from_iter_full = bvec_copy_from_iter_full, @@ -2137,7 +2140,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = pipe_copy_page_to_iter, - .copy_page_from_iter = xxx_copy_page_from_iter, + .copy_page_from_iter = no_copy_page_from_iter, .copy_to_iter = pipe_copy_to_iter, .copy_from_iter = no_copy_from_iter, .copy_from_iter_full = no_copy_from_iter_full, @@ -2171,7 +2174,7 @@ static const struct iov_iter_ops discard_iter_ops = { .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = discard_copy_page_to_iter, - .copy_page_from_iter = xxx_copy_page_from_iter, + .copy_page_from_iter = no_copy_page_from_iter, .copy_to_iter = discard_copy_to_iter, .copy_from_iter = no_copy_from_iter, .copy_from_iter_full = no_copy_from_iter_full, From patchwork Sat Nov 21 14:15:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73725C63697 for ; Sat, 21 Nov 2020 14:15:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 32AF222226 for ; Sat, 21 Nov 2020 14:15:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Fr5Gd+U8" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728196AbgKUOPW (ORCPT ); Sat, 21 Nov 2020 09:15:22 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:26845 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728197AbgKUOPV (ORCPT ); Sat, 21 Nov 2020 09:15:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968119; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=24lXXSwLkr6iyJf1PKmI6Pby4jsOo+WBOkmoFqT5dpA=; b=Fr5Gd+U8vWrL7aE5oh4HAGx9eRXzjqQrnsCaXdYRsBryviH9Mb8k7YJ48gdcbWUxkRptUF 1asF9ov9/a3XxwtPgBY7GGeLLqI/XLB0VUTedFKDLbLhprhrtsxs5b9KH45KtqR3Seiord i1EB+aZlMvtZUm15GQa4IpvQV4xoe+U= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-456-sZqBLL8mPSWhZLjZCn95iQ-1; Sat, 21 Nov 2020 09:15:15 -0500 X-MC-Unique: sZqBLL8mPSWhZLjZCn95iQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5C4B31005D65; Sat, 21 Nov 2020 14:15:14 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id C6F915D6BA; Sat, 21 Nov 2020 14:15:12 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 14/29] iov_iter: Split iov_iter_zero() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:15:12 +0000 Message-ID: <160596811200.154728.9144901939425608415.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split iov_iter_zero() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 40 +++++++++++++++++++++++++++------------- 1 file changed, 27 insertions(+), 13 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 54029aeab3ec..9a167f53ecff 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1168,16 +1168,30 @@ static size_t pipe_zero(size_t bytes, struct iov_iter *i) return bytes; } -static size_t xxx_zero(size_t bytes, struct iov_iter *i) +static size_t iovec_zero(size_t bytes, struct iov_iter *i) { - if (unlikely(iov_iter_is_pipe(i))) - return pipe_zero(bytes, i); - iterate_and_advance(i, bytes, v, - clear_user(v.iov_base, v.iov_len), - memzero_page(v.bv_page, v.bv_offset, v.bv_len), - memset(v.iov_base, 0, v.iov_len) - ) + iterate_and_advance_iovec(i, bytes, v, + clear_user(v.iov_base, v.iov_len)); + return bytes; +} +static size_t bvec_zero(size_t bytes, struct iov_iter *i) +{ + iterate_and_advance_bvec(i, bytes, v, + memzero_page(v.bv_page, v.bv_offset, v.bv_len)); + return bytes; +} + +static size_t kvec_zero(size_t bytes, struct iov_iter *i) +{ + iterate_and_advance_kvec(i, bytes, v, + memset(v.iov_base, 0, v.iov_len)); + return bytes; +} + +static size_t discard_zero(size_t bytes, struct iov_iter *i) +{ + iterate_and_advance_discard(i, bytes); return bytes; } @@ -2054,7 +2068,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, - .zero = xxx_zero, + .zero = iovec_zero, .alignment = xxx_alignment, .gap_alignment = xxx_gap_alignment, .get_pages = xxx_get_pages, @@ -2088,7 +2102,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, - .zero = xxx_zero, + .zero = kvec_zero, .alignment = xxx_alignment, .gap_alignment = xxx_gap_alignment, .get_pages = xxx_get_pages, @@ -2122,7 +2136,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, - .zero = xxx_zero, + .zero = bvec_zero, .alignment = xxx_alignment, .gap_alignment = xxx_gap_alignment, .get_pages = xxx_get_pages, @@ -2156,7 +2170,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, - .zero = xxx_zero, + .zero = pipe_zero, .alignment = xxx_alignment, .gap_alignment = xxx_gap_alignment, .get_pages = xxx_get_pages, @@ -2190,7 +2204,7 @@ static const struct iov_iter_ops discard_iter_ops = { .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, - .zero = xxx_zero, + .zero = discard_zero, .alignment = xxx_alignment, .gap_alignment = xxx_gap_alignment, .get_pages = xxx_get_pages, From patchwork Sat Nov 21 14:15:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC0FFC6379D for ; Sat, 21 Nov 2020 14:15:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A78482224A for ; Sat, 21 Nov 2020 14:15:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="gUcOGfz5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728224AbgKUOPc (ORCPT ); Sat, 21 Nov 2020 09:15:32 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:42612 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728215AbgKUOP3 (ORCPT ); Sat, 21 Nov 2020 09:15:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968127; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S9kgWJW9dIIPW9GuwP3JfJltGkxfQ81N7Gf5cW28glQ=; b=gUcOGfz5qNUV587QhAQ8B7N+ISUJ2JUIRxJWKt3AUQIIoCNJIVZQ6xsQOZQYrU8oZCfRcL ywLfM0A5dYNsHAvd7iuAU+ptYAkBpXV3/nffQQqQUYOFleC5vCkkv8gXa2JS686HFYnmbF lbAasVqDns6mRz5AupP4ewtogZdC+nw= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-279-Eo_K1U2FO26W8OddVM7xzQ-1; Sat, 21 Nov 2020 09:15:23 -0500 X-MC-Unique: Eo_K1U2FO26W8OddVM7xzQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 168ED1005D6B; Sat, 21 Nov 2020 14:15:22 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5E5165D6BA; Sat, 21 Nov 2020 14:15:20 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 15/29] iov_iter: Split copy_from_user_atomic() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:15:19 +0000 Message-ID: <160596811958.154728.14803339635754597304.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split copy_from_user_atomic() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 53 ++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 40 insertions(+), 13 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 9a167f53ecff..a626d41fef72 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1195,7 +1195,7 @@ static size_t discard_zero(size_t bytes, struct iov_iter *i) return bytes; } -static size_t xxx_copy_from_user_atomic(struct page *page, +static size_t iovec_copy_from_user_atomic(struct page *page, struct iov_iter *i, unsigned long offset, size_t bytes) { char *kaddr = kmap_atomic(page), *p = kaddr + offset; @@ -1203,21 +1203,48 @@ static size_t xxx_copy_from_user_atomic(struct page *page, kunmap_atomic(kaddr); return 0; } - if (unlikely(iov_iter_is_pipe(i) || iov_iter_is_discard(i))) { + iterate_over_iovec(i, bytes, v, + copyin((p += v.iov_len) - v.iov_len, v.iov_base, v.iov_len)); + kunmap_atomic(kaddr); + return bytes; +} + +static size_t bvec_copy_from_user_atomic(struct page *page, + struct iov_iter *i, unsigned long offset, size_t bytes) +{ + char *kaddr = kmap_atomic(page), *p = kaddr + offset; + if (unlikely(!page_copy_sane(page, offset, bytes))) { kunmap_atomic(kaddr); - WARN_ON(1); return 0; } - iterate_all_kinds(i, bytes, v, - copyin((p += v.iov_len) - v.iov_len, v.iov_base, v.iov_len), + iterate_over_bvec(i, bytes, v, memcpy_from_page((p += v.bv_len) - v.bv_len, v.bv_page, - v.bv_offset, v.bv_len), - memcpy((p += v.iov_len) - v.iov_len, v.iov_base, v.iov_len) - ) + v.bv_offset, v.bv_len)); kunmap_atomic(kaddr); return bytes; } +static size_t kvec_copy_from_user_atomic(struct page *page, + struct iov_iter *i, unsigned long offset, size_t bytes) +{ + char *kaddr = kmap_atomic(page), *p = kaddr + offset; + if (unlikely(!page_copy_sane(page, offset, bytes))) { + kunmap_atomic(kaddr); + return 0; + } + iterate_over_kvec(i, bytes, v, + memcpy((p += v.iov_len) - v.iov_len, v.iov_base, v.iov_len)); + kunmap_atomic(kaddr); + return bytes; +} + +static size_t no_copy_from_user_atomic(struct page *page, + struct iov_iter *i, unsigned long offset, size_t bytes) +{ + WARN_ON(1); + return 0; +} + static inline void pipe_truncate(struct iov_iter *i) { struct pipe_inode_info *pipe = i->pipe; @@ -2046,7 +2073,7 @@ static int xxx_for_each_range(struct iov_iter *i, size_t bytes, static const struct iov_iter_ops iovec_iter_ops = { .type = ITER_IOVEC, - .copy_from_user_atomic = xxx_copy_from_user_atomic, + .copy_from_user_atomic = iovec_copy_from_user_atomic, .advance = xxx_advance, .revert = xxx_revert, .fault_in_readable = iovec_fault_in_readable, @@ -2080,7 +2107,7 @@ static const struct iov_iter_ops iovec_iter_ops = { static const struct iov_iter_ops kvec_iter_ops = { .type = ITER_KVEC, - .copy_from_user_atomic = xxx_copy_from_user_atomic, + .copy_from_user_atomic = kvec_copy_from_user_atomic, .advance = xxx_advance, .revert = xxx_revert, .fault_in_readable = no_fault_in_readable, @@ -2114,7 +2141,7 @@ static const struct iov_iter_ops kvec_iter_ops = { static const struct iov_iter_ops bvec_iter_ops = { .type = ITER_BVEC, - .copy_from_user_atomic = xxx_copy_from_user_atomic, + .copy_from_user_atomic = bvec_copy_from_user_atomic, .advance = xxx_advance, .revert = xxx_revert, .fault_in_readable = no_fault_in_readable, @@ -2148,7 +2175,7 @@ static const struct iov_iter_ops bvec_iter_ops = { static const struct iov_iter_ops pipe_iter_ops = { .type = ITER_PIPE, - .copy_from_user_atomic = xxx_copy_from_user_atomic, + .copy_from_user_atomic = no_copy_from_user_atomic, .advance = xxx_advance, .revert = xxx_revert, .fault_in_readable = no_fault_in_readable, @@ -2182,7 +2209,7 @@ static const struct iov_iter_ops pipe_iter_ops = { static const struct iov_iter_ops discard_iter_ops = { .type = ITER_DISCARD, - .copy_from_user_atomic = xxx_copy_from_user_atomic, + .copy_from_user_atomic = no_copy_from_user_atomic, .advance = xxx_advance, .revert = xxx_revert, .fault_in_readable = no_fault_in_readable, From patchwork Sat Nov 21 14:15:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39AF6C64E75 for ; Sat, 21 Nov 2020 14:15:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 040A92224A for ; Sat, 21 Nov 2020 14:15:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="gja71lf9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728216AbgKUOPf (ORCPT ); Sat, 21 Nov 2020 09:15:35 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:47057 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728225AbgKUOPf (ORCPT ); Sat, 21 Nov 2020 09:15:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968133; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lIjo+vSpe3gII8dHaOlAjbgozxM0tOye0nfuL6GBd1k=; b=gja71lf9NWMvMDzt0LIJbwstOx+mvINGNuHQJuocWZV3gHGEBlimeyZr/1iQOz/H1JqJU3 hHgwqqGdZgDZ99XwzcM7aKQjBLLgcnTJ7mpNxy/KkQwWEf0Kusqzrno3LtTJVYdYWPomll KuQ9dPgCkConMs9ARwG2JzBvA6dpOx4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-11-MI51AxE4MIayROV3LgzCHw-1; Sat, 21 Nov 2020 09:15:31 -0500 X-MC-Unique: MI51AxE4MIayROV3LgzCHw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A3C531DDE7; Sat, 21 Nov 2020 14:15:29 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1BE7A60C15; Sat, 21 Nov 2020 14:15:27 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 16/29] iov_iter: Split iov_iter_advance() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:15:27 +0000 Message-ID: <160596812731.154728.10053807137767432607.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split iov_iter_advance() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index a626d41fef72..9859b4b8a116 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1299,17 +1299,24 @@ static void pipe_advance(struct iov_iter *i, size_t size) pipe_truncate(i); } -static void xxx_advance(struct iov_iter *i, size_t size) +static void iovec_advance(struct iov_iter *i, size_t size) { - if (unlikely(iov_iter_is_pipe(i))) { - pipe_advance(i, size); - return; - } - if (unlikely(iov_iter_is_discard(i))) { - i->count -= size; - return; - } - iterate_and_advance(i, size, v, 0, 0, 0) + iterate_and_advance_iovec(i, size, v, 0) +} + +static void bvec_iov_advance(struct iov_iter *i, size_t size) +{ + iterate_and_advance_bvec(i, size, v, 0) +} + +static void kvec_advance(struct iov_iter *i, size_t size) +{ + iterate_and_advance_kvec(i, size, v, 0) +} + +static void discard_advance(struct iov_iter *i, size_t size) +{ + i->count -= size; } static void xxx_revert(struct iov_iter *i, size_t unroll) @@ -2074,7 +2081,7 @@ static int xxx_for_each_range(struct iov_iter *i, size_t bytes, static const struct iov_iter_ops iovec_iter_ops = { .type = ITER_IOVEC, .copy_from_user_atomic = iovec_copy_from_user_atomic, - .advance = xxx_advance, + .advance = iovec_advance, .revert = xxx_revert, .fault_in_readable = iovec_fault_in_readable, .single_seg_count = xxx_single_seg_count, @@ -2108,7 +2115,7 @@ static const struct iov_iter_ops iovec_iter_ops = { static const struct iov_iter_ops kvec_iter_ops = { .type = ITER_KVEC, .copy_from_user_atomic = kvec_copy_from_user_atomic, - .advance = xxx_advance, + .advance = kvec_advance, .revert = xxx_revert, .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, @@ -2142,7 +2149,7 @@ static const struct iov_iter_ops kvec_iter_ops = { static const struct iov_iter_ops bvec_iter_ops = { .type = ITER_BVEC, .copy_from_user_atomic = bvec_copy_from_user_atomic, - .advance = xxx_advance, + .advance = bvec_iov_advance, .revert = xxx_revert, .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, @@ -2176,7 +2183,7 @@ static const struct iov_iter_ops bvec_iter_ops = { static const struct iov_iter_ops pipe_iter_ops = { .type = ITER_PIPE, .copy_from_user_atomic = no_copy_from_user_atomic, - .advance = xxx_advance, + .advance = pipe_advance, .revert = xxx_revert, .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, @@ -2210,7 +2217,7 @@ static const struct iov_iter_ops pipe_iter_ops = { static const struct iov_iter_ops discard_iter_ops = { .type = ITER_DISCARD, .copy_from_user_atomic = no_copy_from_user_atomic, - .advance = xxx_advance, + .advance = discard_advance, .revert = xxx_revert, .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, From patchwork Sat Nov 21 14:15:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D380C64E7C for ; Sat, 21 Nov 2020 14:15:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5C4A72224A for ; Sat, 21 Nov 2020 14:15:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="RjFVRAlb" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728247AbgKUOPo (ORCPT ); Sat, 21 Nov 2020 09:15:44 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:52858 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727862AbgKUOPn (ORCPT ); Sat, 21 Nov 2020 09:15:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968141; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yGrPe4o/S5aCZvKVL3hw7DCeW/9Xw/NFeCgsf4eSSA0=; b=RjFVRAlbuU1uM5B6qc3b/d2e9kBj0EnXYfffwhPqFa0eRs/RQKjT+4PkujNA9WHi2pdbmH 4kY64QFrTBvSTprAFz9IGD543KW+RZk4uc8B5KPu1dms1gb4SeiYL/49cD3poXsAtkZhZJ FFYC7ny/wjpyYUuAWdG+ZeG9NJkqhhg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-269-jHd7vtpgMDuuCO6mnqNbTQ-1; Sat, 21 Nov 2020 09:15:38 -0500 X-MC-Unique: jHd7vtpgMDuuCO6mnqNbTQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5D43A1005D65; Sat, 21 Nov 2020 14:15:37 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id A55CA5D9D7; Sat, 21 Nov 2020 14:15:35 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 17/29] iov_iter: Split iov_iter_revert() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:15:34 +0000 Message-ID: <160596813488.154728.6654456287509529880.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split iov_iter_revert() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 132 ++++++++++++++++++++++++++++++++++---------------------- 1 file changed, 79 insertions(+), 53 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 9859b4b8a116..b8e3da20547e 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1319,71 +1319,97 @@ static void discard_advance(struct iov_iter *i, size_t size) i->count -= size; } -static void xxx_revert(struct iov_iter *i, size_t unroll) +static void iovec_kvec_revert(struct iov_iter *i, size_t unroll) { + const struct iovec *iov = i->iov; if (!unroll) return; if (WARN_ON(unroll > MAX_RW_COUNT)) return; i->count += unroll; - if (unlikely(iov_iter_is_pipe(i))) { - struct pipe_inode_info *pipe = i->pipe; - unsigned int p_mask = pipe->ring_size - 1; - unsigned int i_head = i->head; - size_t off = i->iov_offset; - while (1) { - struct pipe_buffer *b = &pipe->bufs[i_head & p_mask]; - size_t n = off - b->offset; - if (unroll < n) { - off -= unroll; - break; - } - unroll -= n; - if (!unroll && i_head == i->start_head) { - off = 0; - break; - } - i_head--; - b = &pipe->bufs[i_head & p_mask]; - off = b->offset + b->len; - } - i->iov_offset = off; - i->head = i_head; - pipe_truncate(i); + if (unroll <= i->iov_offset) { + i->iov_offset -= unroll; return; } - if (unlikely(iov_iter_is_discard(i))) + unroll -= i->iov_offset; + while (1) { + size_t n = (--iov)->iov_len; + i->nr_segs++; + if (unroll <= n) { + i->iov = iov; + i->iov_offset = n - unroll; + return; + } + unroll -= n; + } +} + +static void bvec_revert(struct iov_iter *i, size_t unroll) +{ + const struct bio_vec *bvec = i->bvec; + + if (!unroll) return; + if (WARN_ON(unroll > MAX_RW_COUNT)) + return; + i->count += unroll; if (unroll <= i->iov_offset) { i->iov_offset -= unroll; return; } unroll -= i->iov_offset; - if (iov_iter_is_bvec(i)) { - const struct bio_vec *bvec = i->bvec; - while (1) { - size_t n = (--bvec)->bv_len; - i->nr_segs++; - if (unroll <= n) { - i->bvec = bvec; - i->iov_offset = n - unroll; - return; - } - unroll -= n; + while (1) { + size_t n = (--bvec)->bv_len; + i->nr_segs++; + if (unroll <= n) { + i->bvec = bvec; + i->iov_offset = n - unroll; + return; } - } else { /* same logics for iovec and kvec */ - const struct iovec *iov = i->iov; - while (1) { - size_t n = (--iov)->iov_len; - i->nr_segs++; - if (unroll <= n) { - i->iov = iov; - i->iov_offset = n - unroll; - return; - } - unroll -= n; + unroll -= n; + } +} + +static void pipe_revert(struct iov_iter *i, size_t unroll) +{ + struct pipe_inode_info *pipe = i->pipe; + unsigned int p_mask = pipe->ring_size - 1; + unsigned int i_head = i->head; + size_t off = i->iov_offset; + + if (!unroll) + return; + if (WARN_ON(unroll > MAX_RW_COUNT)) + return; + + while (1) { + struct pipe_buffer *b = &pipe->bufs[i_head & p_mask]; + size_t n = off - b->offset; + if (unroll < n) { + off -= unroll; + break; + } + unroll -= n; + if (!unroll && i_head == i->start_head) { + off = 0; + break; } + i_head--; + b = &pipe->bufs[i_head & p_mask]; + off = b->offset + b->len; } + i->iov_offset = off; + i->head = i_head; + pipe_truncate(i); +} + +static void discard_revert(struct iov_iter *i, size_t unroll) +{ + if (!unroll) + return; + if (WARN_ON(unroll > MAX_RW_COUNT)) + return; + i->count += unroll; } /* @@ -2082,7 +2108,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .type = ITER_IOVEC, .copy_from_user_atomic = iovec_copy_from_user_atomic, .advance = iovec_advance, - .revert = xxx_revert, + .revert = iovec_kvec_revert, .fault_in_readable = iovec_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = iovec_copy_page_to_iter, @@ -2116,7 +2142,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .type = ITER_KVEC, .copy_from_user_atomic = kvec_copy_from_user_atomic, .advance = kvec_advance, - .revert = xxx_revert, + .revert = iovec_kvec_revert, .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = bkvec_copy_page_to_iter, @@ -2150,7 +2176,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .type = ITER_BVEC, .copy_from_user_atomic = bvec_copy_from_user_atomic, .advance = bvec_iov_advance, - .revert = xxx_revert, + .revert = bvec_revert, .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = bkvec_copy_page_to_iter, @@ -2184,7 +2210,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .type = ITER_PIPE, .copy_from_user_atomic = no_copy_from_user_atomic, .advance = pipe_advance, - .revert = xxx_revert, + .revert = pipe_revert, .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = pipe_copy_page_to_iter, @@ -2218,7 +2244,7 @@ static const struct iov_iter_ops discard_iter_ops = { .type = ITER_DISCARD, .copy_from_user_atomic = no_copy_from_user_atomic, .advance = discard_advance, - .revert = xxx_revert, + .revert = discard_revert, .fault_in_readable = no_fault_in_readable, .single_seg_count = xxx_single_seg_count, .copy_page_to_iter = discard_copy_page_to_iter, From patchwork Sat Nov 21 14:15:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A621C64E90 for ; Sat, 21 Nov 2020 14:15:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 05CDD2224A for ; Sat, 21 Nov 2020 14:15:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VM/OBmsR" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728266AbgKUOPw (ORCPT ); Sat, 21 Nov 2020 09:15:52 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:22359 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728258AbgKUOPv (ORCPT ); Sat, 21 Nov 2020 09:15:51 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968149; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XP3mnUJc4mxTb6vHs3+R4NsGbX5N4I46o5CgY2C0v48=; b=VM/OBmsRXfpXfA0JdrlOCSqd3RRT25BIWRiG9Yl6LY7YSl4NfEgOgAjMUtcZWrYKjPLFWH uxyNoiYUj4l6G6wyhQFIguf7pXtR4ye2pOc98f+1vlIGbkISjkv4pcu4QsBeVPGbt67Usn 5uj18nl2F/8426cmDlTHH1FgoNfoKQU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-253-W_XtH6zlM8uyYu3poeiBQw-1; Sat, 21 Nov 2020 09:15:46 -0500 X-MC-Unique: W_XtH6zlM8uyYu3poeiBQw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 469471DDE8; Sat, 21 Nov 2020 14:15:45 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 70A4C5D6CF; Sat, 21 Nov 2020 14:15:43 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 18/29] iov_iter: Split iov_iter_single_seg_count() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:15:42 +0000 Message-ID: <160596814260.154728.8650477861968611110.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split iov_iter_single_seg_count() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index b8e3da20547e..90291188ace5 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1415,18 +1415,23 @@ static void discard_revert(struct iov_iter *i, size_t unroll) /* * Return the count of just the current iov_iter segment. */ -static size_t xxx_single_seg_count(const struct iov_iter *i) +static size_t iovec_kvec_single_seg_count(const struct iov_iter *i) { - if (unlikely(iov_iter_is_pipe(i))) - return i->count; // it is a silly place, anyway if (i->nr_segs == 1) return i->count; - if (unlikely(iov_iter_is_discard(i))) + return min(i->count, i->iov->iov_len - i->iov_offset); +} + +static size_t bvec_single_seg_count(const struct iov_iter *i) +{ + if (i->nr_segs == 1) return i->count; - else if (iov_iter_is_bvec(i)) - return min(i->count, i->bvec->bv_len - i->iov_offset); - else - return min(i->count, i->iov->iov_len - i->iov_offset); + return min(i->count, i->bvec->bv_len - i->iov_offset); +} + +static size_t simple_single_seg_count(const struct iov_iter *i) +{ + return i->count; } void iov_iter_kvec(struct iov_iter *i, unsigned int direction, @@ -2110,7 +2115,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .advance = iovec_advance, .revert = iovec_kvec_revert, .fault_in_readable = iovec_fault_in_readable, - .single_seg_count = xxx_single_seg_count, + .single_seg_count = iovec_kvec_single_seg_count, .copy_page_to_iter = iovec_copy_page_to_iter, .copy_page_from_iter = iovec_copy_page_from_iter, .copy_to_iter = iovec_copy_to_iter, @@ -2144,7 +2149,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .advance = kvec_advance, .revert = iovec_kvec_revert, .fault_in_readable = no_fault_in_readable, - .single_seg_count = xxx_single_seg_count, + .single_seg_count = iovec_kvec_single_seg_count, .copy_page_to_iter = bkvec_copy_page_to_iter, .copy_page_from_iter = bkvec_copy_page_from_iter, .copy_to_iter = kvec_copy_to_iter, @@ -2178,7 +2183,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .advance = bvec_iov_advance, .revert = bvec_revert, .fault_in_readable = no_fault_in_readable, - .single_seg_count = xxx_single_seg_count, + .single_seg_count = bvec_single_seg_count, .copy_page_to_iter = bkvec_copy_page_to_iter, .copy_page_from_iter = bkvec_copy_page_from_iter, .copy_to_iter = bvec_copy_to_iter, @@ -2212,7 +2217,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .advance = pipe_advance, .revert = pipe_revert, .fault_in_readable = no_fault_in_readable, - .single_seg_count = xxx_single_seg_count, + .single_seg_count = simple_single_seg_count, .copy_page_to_iter = pipe_copy_page_to_iter, .copy_page_from_iter = no_copy_page_from_iter, .copy_to_iter = pipe_copy_to_iter, @@ -2246,7 +2251,7 @@ static const struct iov_iter_ops discard_iter_ops = { .advance = discard_advance, .revert = discard_revert, .fault_in_readable = no_fault_in_readable, - .single_seg_count = xxx_single_seg_count, + .single_seg_count = simple_single_seg_count, .copy_page_to_iter = discard_copy_page_to_iter, .copy_page_from_iter = no_copy_page_from_iter, .copy_to_iter = discard_copy_to_iter, From patchwork Sat Nov 21 14:15:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 329E0C63697 for ; Sat, 21 Nov 2020 14:16:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EAEFC22226 for ; Sat, 21 Nov 2020 14:16:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="MvB9o7kA" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728225AbgKUOQB (ORCPT ); Sat, 21 Nov 2020 09:16:01 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:29764 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728286AbgKUOQA (ORCPT ); Sat, 21 Nov 2020 09:16:00 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968158; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=h4NEsUyJ94oX0w8CnfwZML3FMbcbOmVC869wx5bM3lA=; b=MvB9o7kAMpzKHSKwG2fzW+67E7R5uBcuaVezrHok96lk4MYH+TKyFTH81P2AuhrOf9R1V4 xFIMZyNLG+3BiRLfHXKS7mqOzea8ORPLOTciGQwI6lA/AJLMmCbT2bRTR8KBTLeU379Fny Uih1OqapZHHDXX2LpRtPWFv/NAyZG3g= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-305-WRSH_fD4PlCHu0TtuMO4hQ-1; Sat, 21 Nov 2020 09:15:54 -0500 X-MC-Unique: WRSH_fD4PlCHu0TtuMO4hQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 029F61842142; Sat, 21 Nov 2020 14:15:53 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4BBB66085D; Sat, 21 Nov 2020 14:15:51 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 19/29] iov_iter: Split iov_iter_alignment() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:15:50 +0000 Message-ID: <160596815051.154728.15434909811290333829.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split iov_iter_alignment() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 59 ++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 42 insertions(+), 17 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 90291188ace5..d2a66e951995 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1497,26 +1497,51 @@ void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count) } EXPORT_SYMBOL(iov_iter_discard); -static unsigned long xxx_alignment(const struct iov_iter *i) +static unsigned long iovec_alignment(const struct iov_iter *i) { unsigned long res = 0; size_t size = i->count; - if (unlikely(iov_iter_is_pipe(i))) { - unsigned int p_mask = i->pipe->ring_size - 1; + iterate_over_iovec(i, size, v, + (res |= (unsigned long)v.iov_base | v.iov_len, 0)); + return res; +} - if (size && i->iov_offset && allocated(&i->pipe->bufs[i->head & p_mask])) - return size | i->iov_offset; - return size; - } - iterate_all_kinds(i, size, v, - (res |= (unsigned long)v.iov_base | v.iov_len, 0), - res |= v.bv_offset | v.bv_len, - res |= (unsigned long)v.iov_base | v.iov_len - ) +static unsigned long bvec_alignment(const struct iov_iter *i) +{ + unsigned long res = 0; + size_t size = i->count; + + iterate_over_bvec(i, size, v, + res |= v.bv_offset | v.bv_len); return res; } +static unsigned long kvec_alignment(const struct iov_iter *i) +{ + unsigned long res = 0; + size_t size = i->count; + + iterate_over_kvec(i, size, v, + res |= (unsigned long)v.iov_base | v.iov_len); + return res; +} + +static unsigned long pipe_alignment(const struct iov_iter *i) +{ + size_t size = i->count; + unsigned int p_mask = i->pipe->ring_size - 1; + + if (size && i->iov_offset && allocated(&i->pipe->bufs[i->head & p_mask])) + return size | i->iov_offset; + return size; +} + +static unsigned long no_alignment(const struct iov_iter *i) +{ + return 0; +} + static unsigned long xxx_gap_alignment(const struct iov_iter *i) { unsigned long res = 0; @@ -2134,7 +2159,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, .zero = iovec_zero, - .alignment = xxx_alignment, + .alignment = iovec_alignment, .gap_alignment = xxx_gap_alignment, .get_pages = xxx_get_pages, .get_pages_alloc = xxx_get_pages_alloc, @@ -2168,7 +2193,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, .zero = kvec_zero, - .alignment = xxx_alignment, + .alignment = kvec_alignment, .gap_alignment = xxx_gap_alignment, .get_pages = xxx_get_pages, .get_pages_alloc = xxx_get_pages_alloc, @@ -2202,7 +2227,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, .zero = bvec_zero, - .alignment = xxx_alignment, + .alignment = bvec_alignment, .gap_alignment = xxx_gap_alignment, .get_pages = xxx_get_pages, .get_pages_alloc = xxx_get_pages_alloc, @@ -2236,7 +2261,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, .zero = pipe_zero, - .alignment = xxx_alignment, + .alignment = pipe_alignment, .gap_alignment = xxx_gap_alignment, .get_pages = xxx_get_pages, .get_pages_alloc = xxx_get_pages_alloc, @@ -2270,7 +2295,7 @@ static const struct iov_iter_ops discard_iter_ops = { .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, .zero = discard_zero, - .alignment = xxx_alignment, + .alignment = no_alignment, .gap_alignment = xxx_gap_alignment, .get_pages = xxx_get_pages, .get_pages_alloc = xxx_get_pages_alloc, From patchwork Sat Nov 21 14:15:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89FC8C6379F for ; Sat, 21 Nov 2020 14:16:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4E12F22226 for ; Sat, 21 Nov 2020 14:16:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="FDnj/pId" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728314AbgKUOQJ (ORCPT ); Sat, 21 Nov 2020 09:16:09 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:37850 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728295AbgKUOQI (ORCPT ); Sat, 21 Nov 2020 09:16:08 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O9qDylx4ewpBGOJoY7+LpQLGp85UBysg0NqjPw7tJrk=; b=FDnj/pIdnd4aneh+mZg7IPbDBuvgN6Y98tOIgA9cihaanSrlsuers/9dkq7EH2Dp3FLl12 G6jrBFQH8AapYzSFjMRZKKepVlQe5p7S58VxXO+aQ7u77piexbOAmEFe52q/kMYLjfs+CK lawq9D6WdFQ1IXBD6/prsXxfzsK3Ko4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-426-HeJl7aVjNTez9AGzqE1P7w-1; Sat, 21 Nov 2020 09:16:02 -0500 X-MC-Unique: HeJl7aVjNTez9AGzqE1P7w-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 929DB1DDE7; Sat, 21 Nov 2020 14:16:00 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id F15F55D6BA; Sat, 21 Nov 2020 14:15:58 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 20/29] iov_iter: Split iov_iter_gap_alignment() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:15:58 +0000 Message-ID: <160596815823.154728.8595962159705739709.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split iov_iter_gap_alignment() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 50 ++++++++++++++++++++++++++++++++++---------------- 1 file changed, 34 insertions(+), 16 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index d2a66e951995..5744ddec854f 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1542,27 +1542,45 @@ static unsigned long no_alignment(const struct iov_iter *i) return 0; } -static unsigned long xxx_gap_alignment(const struct iov_iter *i) +static unsigned long iovec_gap_alignment(const struct iov_iter *i) { unsigned long res = 0; size_t size = i->count; - if (unlikely(iov_iter_is_pipe(i) || iov_iter_is_discard(i))) { - WARN_ON(1); - return ~0U; - } - - iterate_all_kinds(i, size, v, + iterate_over_iovec(i, size, v, (res |= (!res ? 0 : (unsigned long)v.iov_base) | - (size != v.iov_len ? size : 0), 0), + (size != v.iov_len ? size : 0), 0)); + return res; +} + +static unsigned long bvec_gap_alignment(const struct iov_iter *i) +{ + unsigned long res = 0; + size_t size = i->count; + + iterate_over_bvec(i, size, v, (res |= (!res ? 0 : (unsigned long)v.bv_offset) | - (size != v.bv_len ? size : 0)), + (size != v.bv_len ? size : 0))); + return res; +} + +static unsigned long kvec_gap_alignment(const struct iov_iter *i) +{ + unsigned long res = 0; + size_t size = i->count; + + iterate_over_kvec(i, size, v, (res |= (!res ? 0 : (unsigned long)v.iov_base) | - (size != v.iov_len ? size : 0)) - ); + (size != v.iov_len ? size : 0))); return res; } +static unsigned long no_gap_alignment(const struct iov_iter *i) +{ + WARN_ON(1); + return ~0U; +} + static inline ssize_t __pipe_get_pages(struct iov_iter *i, size_t maxsize, struct page **pages, @@ -2160,7 +2178,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .zero = iovec_zero, .alignment = iovec_alignment, - .gap_alignment = xxx_gap_alignment, + .gap_alignment = iovec_gap_alignment, .get_pages = xxx_get_pages, .get_pages_alloc = xxx_get_pages_alloc, .npages = xxx_npages, @@ -2194,7 +2212,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .zero = kvec_zero, .alignment = kvec_alignment, - .gap_alignment = xxx_gap_alignment, + .gap_alignment = kvec_gap_alignment, .get_pages = xxx_get_pages, .get_pages_alloc = xxx_get_pages_alloc, .npages = xxx_npages, @@ -2228,7 +2246,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .zero = bvec_zero, .alignment = bvec_alignment, - .gap_alignment = xxx_gap_alignment, + .gap_alignment = bvec_gap_alignment, .get_pages = xxx_get_pages, .get_pages_alloc = xxx_get_pages_alloc, .npages = xxx_npages, @@ -2262,7 +2280,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .zero = pipe_zero, .alignment = pipe_alignment, - .gap_alignment = xxx_gap_alignment, + .gap_alignment = no_gap_alignment, .get_pages = xxx_get_pages, .get_pages_alloc = xxx_get_pages_alloc, .npages = xxx_npages, @@ -2296,7 +2314,7 @@ static const struct iov_iter_ops discard_iter_ops = { .zero = discard_zero, .alignment = no_alignment, - .gap_alignment = xxx_gap_alignment, + .gap_alignment = no_gap_alignment, .get_pages = xxx_get_pages, .get_pages_alloc = xxx_get_pages_alloc, .npages = xxx_npages, From patchwork Sat Nov 21 14:16:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4697AC64E7A for ; Sat, 21 Nov 2020 14:16:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0FB7F2224C for ; Sat, 21 Nov 2020 14:16:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="XE3kgxLo" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728321AbgKUOQO (ORCPT ); Sat, 21 Nov 2020 09:16:14 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:55437 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728299AbgKUOQO (ORCPT ); Sat, 21 Nov 2020 09:16:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968172; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bw9ipJWjx8AimfhLoz84wN1/imOFmg/xGA0Qomc1kTU=; b=XE3kgxLo7vLZ769FdDzmVGibdQRXsPLf1/wWWj2bGULBbrvZK3HyBWQgmqwqzN2JLljChU 3XteXmPwaIrDL3++hIzKc8ApdD0saUvR2B4k+R/Z1uG6vns3BFtfkurdlrjeqoeeLa2Itl QScSm/yz5gqjvd3Asnhl7WIeIKXKzvA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-284-bkI6hkIANsCc6_3NDXBAfg-1; Sat, 21 Nov 2020 09:16:09 -0500 X-MC-Unique: bkI6hkIANsCc6_3NDXBAfg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 718DF814409; Sat, 21 Nov 2020 14:16:08 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id B7B305C22B; Sat, 21 Nov 2020 14:16:06 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 21/29] iov_iter: Split iov_iter_get_pages() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:16:05 +0000 Message-ID: <160596816579.154728.8100196610696982437.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split iov_iter_get_pages() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 46 +++++++++++++++++++++++++++++----------------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 5744ddec854f..a2de201b947f 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1611,6 +1611,8 @@ static ssize_t pipe_get_pages(struct iov_iter *i, unsigned int iter_head, npages; size_t capacity; + if (maxsize > i->count) + maxsize = i->count; if (!maxsize) return 0; @@ -1625,19 +1627,14 @@ static ssize_t pipe_get_pages(struct iov_iter *i, return __pipe_get_pages(i, min(maxsize, capacity), pages, iter_head, start); } -static ssize_t xxx_get_pages(struct iov_iter *i, +static ssize_t iovec_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start) { if (maxsize > i->count) maxsize = i->count; - if (unlikely(iov_iter_is_pipe(i))) - return pipe_get_pages(i, pages, maxsize, maxpages, start); - if (unlikely(iov_iter_is_discard(i))) - return -EFAULT; - - iterate_all_kinds(i, maxsize, v, ({ + iterate_over_iovec(i, maxsize, v, ({ unsigned long addr = (unsigned long)v.iov_base; size_t len = v.iov_len + (*start = addr & (PAGE_SIZE - 1)); int n; @@ -1653,18 +1650,33 @@ static ssize_t xxx_get_pages(struct iov_iter *i, if (unlikely(res < 0)) return res; return (res == n ? len : res * PAGE_SIZE) - *start; - 0;}),({ + 0;})); + return 0; +} + +static ssize_t bvec_get_pages(struct iov_iter *i, + struct page **pages, size_t maxsize, unsigned maxpages, + size_t *start) +{ + if (maxsize > i->count) + maxsize = i->count; + + iterate_over_bvec(i, maxsize, v, ({ /* can't be more than PAGE_SIZE */ *start = v.bv_offset; get_page(*pages = v.bv_page); return v.bv_len; - }),({ - return -EFAULT; - }) - ) + })); return 0; } +static ssize_t no_get_pages(struct iov_iter *i, + struct page **pages, size_t maxsize, unsigned maxpages, + size_t *start) +{ + return -EFAULT; +} + static struct page **get_pages_array(size_t n) { return kvmalloc_array(n, sizeof(struct page *), GFP_KERNEL); @@ -2179,7 +2191,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .zero = iovec_zero, .alignment = iovec_alignment, .gap_alignment = iovec_gap_alignment, - .get_pages = xxx_get_pages, + .get_pages = iovec_get_pages, .get_pages_alloc = xxx_get_pages_alloc, .npages = xxx_npages, .dup_iter = xxx_dup_iter, @@ -2213,7 +2225,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .zero = kvec_zero, .alignment = kvec_alignment, .gap_alignment = kvec_gap_alignment, - .get_pages = xxx_get_pages, + .get_pages = no_get_pages, .get_pages_alloc = xxx_get_pages_alloc, .npages = xxx_npages, .dup_iter = xxx_dup_iter, @@ -2247,7 +2259,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .zero = bvec_zero, .alignment = bvec_alignment, .gap_alignment = bvec_gap_alignment, - .get_pages = xxx_get_pages, + .get_pages = bvec_get_pages, .get_pages_alloc = xxx_get_pages_alloc, .npages = xxx_npages, .dup_iter = xxx_dup_iter, @@ -2281,7 +2293,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .zero = pipe_zero, .alignment = pipe_alignment, .gap_alignment = no_gap_alignment, - .get_pages = xxx_get_pages, + .get_pages = pipe_get_pages, .get_pages_alloc = xxx_get_pages_alloc, .npages = xxx_npages, .dup_iter = xxx_dup_iter, @@ -2315,7 +2327,7 @@ static const struct iov_iter_ops discard_iter_ops = { .zero = discard_zero, .alignment = no_alignment, .gap_alignment = no_gap_alignment, - .get_pages = xxx_get_pages, + .get_pages = no_get_pages, .get_pages_alloc = xxx_get_pages_alloc, .npages = xxx_npages, .dup_iter = xxx_dup_iter, From patchwork Sat Nov 21 14:16:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFCB4C64E7D for ; Sat, 21 Nov 2020 14:16:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7B86022264 for ; Sat, 21 Nov 2020 14:16:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Sqed8zDW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728332AbgKUOQZ (ORCPT ); Sat, 21 Nov 2020 09:16:25 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:58923 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728330AbgKUOQY (ORCPT ); Sat, 21 Nov 2020 09:16:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968183; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=42Modz9lzLK9PgNIzmKcsfAc59EtrxVIzllSBW53iIs=; b=Sqed8zDWESOw4gH1iTyS1uUALcnP+8r4/ffUPwqW9aIsuS7HhCNRdHb+ywIPCAqU2AvPEX U+XuNM6QZgTSqQe0+whVBWRSnJ1UxKctdA523YAuKr6MhEBNWXAIM0B3iuh33cJ/DPEznJ clFviZiwSUBurSnSwHoJszCHsl+/C6o= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-495-zJzT5r59Ohu6bqsLNumyOA-1; Sat, 21 Nov 2020 09:16:17 -0500 X-MC-Unique: zJzT5r59Ohu6bqsLNumyOA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4435F1005D6D; Sat, 21 Nov 2020 14:16:16 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 758F56085D; Sat, 21 Nov 2020 14:16:14 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 22/29] iov_iter: Split iov_iter_get_pages_alloc() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:16:13 +0000 Message-ID: <160596817368.154728.12568762587413499390.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split iov_iter_get_pages_alloc() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 48 +++++++++++++++++++++++++++++++----------------- 1 file changed, 31 insertions(+), 17 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index a2de201b947f..a038bfbbbd53 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1690,6 +1690,8 @@ static ssize_t pipe_get_pages_alloc(struct iov_iter *i, unsigned int iter_head, npages; ssize_t n; + if (maxsize > i->count) + maxsize = i->count; if (!maxsize) return 0; @@ -1715,7 +1717,7 @@ static ssize_t pipe_get_pages_alloc(struct iov_iter *i, return n; } -static ssize_t xxx_get_pages_alloc(struct iov_iter *i, +static ssize_t iovec_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start) { @@ -1724,12 +1726,7 @@ static ssize_t xxx_get_pages_alloc(struct iov_iter *i, if (maxsize > i->count) maxsize = i->count; - if (unlikely(iov_iter_is_pipe(i))) - return pipe_get_pages_alloc(i, pages, maxsize, start); - if (unlikely(iov_iter_is_discard(i))) - return -EFAULT; - - iterate_all_kinds(i, maxsize, v, ({ + iterate_over_iovec(i, maxsize, v, ({ unsigned long addr = (unsigned long)v.iov_base; size_t len = v.iov_len + (*start = addr & (PAGE_SIZE - 1)); int n; @@ -1748,7 +1745,20 @@ static ssize_t xxx_get_pages_alloc(struct iov_iter *i, } *pages = p; return (res == n ? len : res * PAGE_SIZE) - *start; - 0;}),({ + 0;})); + return 0; +} + +static ssize_t bvec_get_pages_alloc(struct iov_iter *i, + struct page ***pages, size_t maxsize, + size_t *start) +{ + struct page **p; + + if (maxsize > i->count) + maxsize = i->count; + + iterate_over_bvec(i, maxsize, v, ({ /* can't be more than PAGE_SIZE */ *start = v.bv_offset; *pages = p = get_pages_array(1); @@ -1756,13 +1766,17 @@ static ssize_t xxx_get_pages_alloc(struct iov_iter *i, return -ENOMEM; get_page(*p = v.bv_page); return v.bv_len; - }),({ - return -EFAULT; - }) - ) + })); return 0; } +static ssize_t no_get_pages_alloc(struct iov_iter *i, + struct page ***pages, size_t maxsize, + size_t *start) +{ + return -EFAULT; +} + static size_t xxx_csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) { @@ -2192,7 +2206,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .alignment = iovec_alignment, .gap_alignment = iovec_gap_alignment, .get_pages = iovec_get_pages, - .get_pages_alloc = xxx_get_pages_alloc, + .get_pages_alloc = iovec_get_pages_alloc, .npages = xxx_npages, .dup_iter = xxx_dup_iter, .for_each_range = xxx_for_each_range, @@ -2226,7 +2240,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .alignment = kvec_alignment, .gap_alignment = kvec_gap_alignment, .get_pages = no_get_pages, - .get_pages_alloc = xxx_get_pages_alloc, + .get_pages_alloc = no_get_pages_alloc, .npages = xxx_npages, .dup_iter = xxx_dup_iter, .for_each_range = xxx_for_each_range, @@ -2260,7 +2274,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .alignment = bvec_alignment, .gap_alignment = bvec_gap_alignment, .get_pages = bvec_get_pages, - .get_pages_alloc = xxx_get_pages_alloc, + .get_pages_alloc = bvec_get_pages_alloc, .npages = xxx_npages, .dup_iter = xxx_dup_iter, .for_each_range = xxx_for_each_range, @@ -2294,7 +2308,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .alignment = pipe_alignment, .gap_alignment = no_gap_alignment, .get_pages = pipe_get_pages, - .get_pages_alloc = xxx_get_pages_alloc, + .get_pages_alloc = pipe_get_pages_alloc, .npages = xxx_npages, .dup_iter = xxx_dup_iter, .for_each_range = xxx_for_each_range, @@ -2328,7 +2342,7 @@ static const struct iov_iter_ops discard_iter_ops = { .alignment = no_alignment, .gap_alignment = no_gap_alignment, .get_pages = no_get_pages, - .get_pages_alloc = xxx_get_pages_alloc, + .get_pages_alloc = no_get_pages_alloc, .npages = xxx_npages, .dup_iter = xxx_dup_iter, .for_each_range = xxx_for_each_range, From patchwork Sat Nov 21 14:16:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0E2AC64E8A for ; Sat, 21 Nov 2020 14:16:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C579222226 for ; Sat, 21 Nov 2020 14:16:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="J3JmIKqo" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728356AbgKUOQd (ORCPT ); Sat, 21 Nov 2020 09:16:33 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:31126 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728346AbgKUOQb (ORCPT ); Sat, 21 Nov 2020 09:16:31 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968189; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hvV2XUFZAv7awKYqds0unEsSCOnFk/jwB0nBnZEA1To=; b=J3JmIKqo2Ug/OEEdhBR9Z/zC8iZjUGydK2glhqK1DHGbDeBL/n96KK/ydw3MkE7BD9gTKw PsNdIccKCpINnJ6jRUMxzfAAs0JWFhkXxpJ8xJJF7/Qlrl/augfQHy98Q6tkVbAqMLZ2fH QRLr44xiJqWi/bfBbjkAI3Yj6W908ww= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-503-v2QWFXBSP2q7L4tE7stg2w-1; Sat, 21 Nov 2020 09:16:25 -0500 X-MC-Unique: v2QWFXBSP2q7L4tE7stg2w-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 10F6C1005D65; Sat, 21 Nov 2020 14:16:24 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3B6A15D9D7; Sat, 21 Nov 2020 14:16:22 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 23/29] iov_iter: Split csum_and_copy_from_iter() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:16:21 +0000 Message-ID: <160596818140.154728.302284980617377681.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split csum_and_copy_from_iter() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 56 +++++++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 41 insertions(+), 15 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index a038bfbbbd53..1f596cffddf9 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1777,18 +1777,14 @@ static ssize_t no_get_pages_alloc(struct iov_iter *i, return -EFAULT; } -static size_t xxx_csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, +static size_t iovec_csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) { char *to = addr; __wsum sum, next; size_t off = 0; sum = *csum; - if (unlikely(iov_iter_is_pipe(i) || iov_iter_is_discard(i))) { - WARN_ON(1); - return 0; - } - iterate_and_advance(i, bytes, v, ({ + iterate_and_advance_iovec(i, bytes, v, ({ next = csum_and_copy_from_user(v.iov_base, (to += v.iov_len) - v.iov_len, v.iov_len); @@ -1797,24 +1793,54 @@ static size_t xxx_csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum off += v.iov_len; } next ? 0 : v.iov_len; - }), ({ + })); + *csum = sum; + return bytes; +} + +static size_t bvec_csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, + struct iov_iter *i) +{ + char *to = addr; + __wsum sum; + size_t off = 0; + sum = *csum; + iterate_and_advance_bvec(i, bytes, v, ({ char *p = kmap_atomic(v.bv_page); sum = csum_and_memcpy((to += v.bv_len) - v.bv_len, p + v.bv_offset, v.bv_len, sum, off); kunmap_atomic(p); off += v.bv_len; - }),({ + })); + *csum = sum; + return bytes; +} + +static size_t kvec_csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, + struct iov_iter *i) +{ + char *to = addr; + __wsum sum; + size_t off = 0; + sum = *csum; + iterate_and_advance_kvec(i, bytes, v, ({ sum = csum_and_memcpy((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len, sum, off); off += v.iov_len; - }) - ) + })); *csum = sum; return bytes; } +static size_t no_csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, + struct iov_iter *i) +{ + WARN_ON(1); + return 0; +} + static bool xxx_csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) { @@ -2199,7 +2225,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .copy_mc_to_iter = iovec_copy_mc_to_iter, #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, - .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, + .csum_and_copy_from_iter = iovec_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, .zero = iovec_zero, @@ -2233,7 +2259,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .copy_mc_to_iter = kvec_copy_mc_to_iter, #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, - .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, + .csum_and_copy_from_iter = kvec_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, .zero = kvec_zero, @@ -2267,7 +2293,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .copy_mc_to_iter = bvec_copy_mc_to_iter, #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, - .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, + .csum_and_copy_from_iter = bvec_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, .zero = bvec_zero, @@ -2301,7 +2327,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .copy_mc_to_iter = pipe_copy_mc_to_iter, #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, - .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, + .csum_and_copy_from_iter = no_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, .zero = pipe_zero, @@ -2335,7 +2361,7 @@ static const struct iov_iter_ops discard_iter_ops = { .copy_mc_to_iter = discard_copy_to_iter, #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, - .csum_and_copy_from_iter = xxx_csum_and_copy_from_iter, + .csum_and_copy_from_iter = no_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, .zero = discard_zero, From patchwork Sat Nov 21 14:16:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2BC9C63697 for ; Sat, 21 Nov 2020 14:17:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6BA7C2224A for ; Sat, 21 Nov 2020 14:17:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="SPdpu5eX" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728362AbgKUOQi (ORCPT ); Sat, 21 Nov 2020 09:16:38 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:38414 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728346AbgKUOQh (ORCPT ); Sat, 21 Nov 2020 09:16:37 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968195; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lc3ApZ/hH2TB+Llf7riKZKHlOr661a/omnTxq1itTBI=; b=SPdpu5eX7/brlzdaDogNgWpvdbje45RsQNydzRvz700ne/56AjFXZ8SSNHv2x7pEmQ/8kX 0Z0GO9ZP5Xh15R2AS4V9bL563/tS9nXrNScPrgDyx3chSzzdKc4LxoZLL3nluMM6qeO4pp 7RXry6agjafHFv0nxLGzrtJ9w6NxghE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-268-oHw2WpuYMEmsy3PV_OL-lA-1; Sat, 21 Nov 2020 09:16:33 -0500 X-MC-Unique: oHw2WpuYMEmsy3PV_OL-lA-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C4F801842165; Sat, 21 Nov 2020 14:16:31 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 175781975F; Sat, 21 Nov 2020 14:16:29 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 24/29] iov_iter: Split csum_and_copy_from_iter_full() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:16:29 +0000 Message-ID: <160596818929.154728.728490523100444099.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split csum_and_copy_from_iter_full() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 62 ++++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 47 insertions(+), 15 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 1f596cffddf9..8820a9e72815 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1841,20 +1841,16 @@ static size_t no_csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, return 0; } -static bool xxx_csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, +static bool iovec_csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) { char *to = addr; __wsum sum, next; size_t off = 0; sum = *csum; - if (unlikely(iov_iter_is_pipe(i) || iov_iter_is_discard(i))) { - WARN_ON(1); - return false; - } if (unlikely(i->count < bytes)) return false; - iterate_all_kinds(i, bytes, v, ({ + iterate_over_iovec(i, bytes, v, ({ next = csum_and_copy_from_user(v.iov_base, (to += v.iov_len) - v.iov_len, v.iov_len); @@ -1863,25 +1859,61 @@ static bool xxx_csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *c sum = csum_block_add(sum, next, off); off += v.iov_len; 0; - }), ({ + })); + *csum = sum; + iov_iter_advance(i, bytes); + return true; +} + +static bool bvec_csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, + struct iov_iter *i) +{ + char *to = addr; + __wsum sum; + size_t off = 0; + sum = *csum; + if (unlikely(i->count < bytes)) + return false; + iterate_over_bvec(i, bytes, v, ({ char *p = kmap_atomic(v.bv_page); sum = csum_and_memcpy((to += v.bv_len) - v.bv_len, p + v.bv_offset, v.bv_len, sum, off); kunmap_atomic(p); off += v.bv_len; - }),({ + })); + *csum = sum; + iov_iter_advance(i, bytes); + return true; +} + +static bool kvec_csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, + struct iov_iter *i) +{ + char *to = addr; + __wsum sum; + size_t off = 0; + sum = *csum; + if (unlikely(i->count < bytes)) + return false; + iterate_over_kvec(i, bytes, v, ({ sum = csum_and_memcpy((to += v.iov_len) - v.iov_len, v.iov_base, v.iov_len, sum, off); off += v.iov_len; - }) - ) + })); *csum = sum; iov_iter_advance(i, bytes); return true; } +static bool no_csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, + struct iov_iter *i) +{ + WARN_ON(1); + return false; +} + static size_t xxx_csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump, struct iov_iter *i) { @@ -2226,7 +2258,7 @@ static const struct iov_iter_ops iovec_iter_ops = { #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, .csum_and_copy_from_iter = iovec_csum_and_copy_from_iter, - .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, + .csum_and_copy_from_iter_full = iovec_csum_and_copy_from_iter_full, .zero = iovec_zero, .alignment = iovec_alignment, @@ -2260,7 +2292,7 @@ static const struct iov_iter_ops kvec_iter_ops = { #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, .csum_and_copy_from_iter = kvec_csum_and_copy_from_iter, - .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, + .csum_and_copy_from_iter_full = kvec_csum_and_copy_from_iter_full, .zero = kvec_zero, .alignment = kvec_alignment, @@ -2294,7 +2326,7 @@ static const struct iov_iter_ops bvec_iter_ops = { #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, .csum_and_copy_from_iter = bvec_csum_and_copy_from_iter, - .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, + .csum_and_copy_from_iter_full = bvec_csum_and_copy_from_iter_full, .zero = bvec_zero, .alignment = bvec_alignment, @@ -2328,7 +2360,7 @@ static const struct iov_iter_ops pipe_iter_ops = { #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, .csum_and_copy_from_iter = no_csum_and_copy_from_iter, - .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, + .csum_and_copy_from_iter_full = no_csum_and_copy_from_iter_full, .zero = pipe_zero, .alignment = pipe_alignment, @@ -2362,7 +2394,7 @@ static const struct iov_iter_ops discard_iter_ops = { #endif .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, .csum_and_copy_from_iter = no_csum_and_copy_from_iter, - .csum_and_copy_from_iter_full = xxx_csum_and_copy_from_iter_full, + .csum_and_copy_from_iter_full = no_csum_and_copy_from_iter_full, .zero = discard_zero, .alignment = no_alignment, From patchwork Sat Nov 21 14:16:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33C07C6379D for ; Sat, 21 Nov 2020 14:17:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 00FC0222B8 for ; Sat, 21 Nov 2020 14:17:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="W66qt2Pc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728381AbgKUOQs (ORCPT ); Sat, 21 Nov 2020 09:16:48 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:60016 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728374AbgKUOQr (ORCPT ); Sat, 21 Nov 2020 09:16:47 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968205; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WeE4/rk1/OfbaMSMAIeE8ipLTBFgO9CU38gcibjW7Y0=; b=W66qt2Pc4QZnAUlXIAlEn9C0DkhfgOqkq9vkEDyTlxbT2yDeck+f889PWFTPXe0j9fTOge lYIg0CfG0pXTp/lBfZC6aTlL6TUP1Gpmsz0MNlf6klzNLZrnLfHiZ9snwmYwDO0vmldTud I5UyxDb9p0hnqB8EEhlSI0LCGzgfoJo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-523-Zn_niTc9NVKbsbADixy2Mg-1; Sat, 21 Nov 2020 09:16:41 -0500 X-MC-Unique: Zn_niTc9NVKbsbADixy2Mg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7FC781005D67; Sat, 21 Nov 2020 14:16:39 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id C8A016085D; Sat, 21 Nov 2020 14:16:37 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 25/29] iov_iter: Split csum_and_copy_to_iter() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:16:37 +0000 Message-ID: <160596819702.154728.12031895151454434193.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split csum_and_copy_to_iter() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 68 ++++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 48 insertions(+), 20 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 8820a9e72815..2f8019e3b09a 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -698,14 +698,15 @@ static __wsum csum_and_memcpy(void *to, const void *from, size_t len, return csum_block_add(sum, next, off); } -static size_t csum_and_copy_to_pipe_iter(const void *addr, size_t bytes, - __wsum *csum, struct iov_iter *i) +static size_t pipe_csum_and_copy_to_iter(const void *addr, size_t bytes, + void *csump, struct iov_iter *i) { struct pipe_inode_info *pipe = i->pipe; unsigned int p_mask = pipe->ring_size - 1; unsigned int i_head; size_t n, r; size_t off = 0; + __wsum *csum = csump; __wsum sum = *csum; if (!sanity(i)) @@ -1914,7 +1915,7 @@ static bool no_csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *cs return false; } -static size_t xxx_csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump, +static size_t iovec_csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump, struct iov_iter *i) { const char *from = addr; @@ -1922,15 +1923,8 @@ static size_t xxx_csum_and_copy_to_iter(const void *addr, size_t bytes, void *cs __wsum sum, next; size_t off = 0; - if (unlikely(iov_iter_is_pipe(i))) - return csum_and_copy_to_pipe_iter(addr, bytes, csum, i); - sum = *csum; - if (unlikely(iov_iter_is_discard(i))) { - WARN_ON(1); /* for now */ - return 0; - } - iterate_and_advance(i, bytes, v, ({ + iterate_and_advance_iovec(i, bytes, v, ({ next = csum_and_copy_to_user((from += v.iov_len) - v.iov_len, v.iov_base, v.iov_len); @@ -1939,24 +1933,58 @@ static size_t xxx_csum_and_copy_to_iter(const void *addr, size_t bytes, void *cs off += v.iov_len; } next ? 0 : v.iov_len; - }), ({ + })); + *csum = sum; + return bytes; +} + +static size_t bvec_csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump, + struct iov_iter *i) +{ + const char *from = addr; + __wsum *csum = csump; + __wsum sum; + size_t off = 0; + + sum = *csum; + iterate_and_advance_bvec(i, bytes, v, ({ char *p = kmap_atomic(v.bv_page); sum = csum_and_memcpy(p + v.bv_offset, (from += v.bv_len) - v.bv_len, v.bv_len, sum, off); kunmap_atomic(p); off += v.bv_len; - }),({ + })); + *csum = sum; + return bytes; +} + +static size_t kvec_csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump, + struct iov_iter *i) +{ + const char *from = addr; + __wsum *csum = csump; + __wsum sum; + size_t off = 0; + + sum = *csum; + iterate_and_advance_kvec(i, bytes, v, ({ sum = csum_and_memcpy(v.iov_base, (from += v.iov_len) - v.iov_len, v.iov_len, sum, off); off += v.iov_len; - }) - ) + })); *csum = sum; return bytes; } +static size_t discard_csum_and_copy_to_iter(const void *addr, size_t bytes, void *csump, + struct iov_iter *i) +{ + WARN_ON(1); /* for now */ + return 0; +} + size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, struct iov_iter *i) { @@ -2256,7 +2284,7 @@ static const struct iov_iter_ops iovec_iter_ops = { #ifdef CONFIG_ARCH_HAS_COPY_MC .copy_mc_to_iter = iovec_copy_mc_to_iter, #endif - .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, + .csum_and_copy_to_iter = iovec_csum_and_copy_to_iter, .csum_and_copy_from_iter = iovec_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = iovec_csum_and_copy_from_iter_full, @@ -2290,7 +2318,7 @@ static const struct iov_iter_ops kvec_iter_ops = { #ifdef CONFIG_ARCH_HAS_COPY_MC .copy_mc_to_iter = kvec_copy_mc_to_iter, #endif - .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, + .csum_and_copy_to_iter = kvec_csum_and_copy_to_iter, .csum_and_copy_from_iter = kvec_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = kvec_csum_and_copy_from_iter_full, @@ -2324,7 +2352,7 @@ static const struct iov_iter_ops bvec_iter_ops = { #ifdef CONFIG_ARCH_HAS_COPY_MC .copy_mc_to_iter = bvec_copy_mc_to_iter, #endif - .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, + .csum_and_copy_to_iter = bvec_csum_and_copy_to_iter, .csum_and_copy_from_iter = bvec_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = bvec_csum_and_copy_from_iter_full, @@ -2358,7 +2386,7 @@ static const struct iov_iter_ops pipe_iter_ops = { #ifdef CONFIG_ARCH_HAS_COPY_MC .copy_mc_to_iter = pipe_copy_mc_to_iter, #endif - .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, + .csum_and_copy_to_iter = pipe_csum_and_copy_to_iter, .csum_and_copy_from_iter = no_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = no_csum_and_copy_from_iter_full, @@ -2392,7 +2420,7 @@ static const struct iov_iter_ops discard_iter_ops = { #ifdef CONFIG_ARCH_HAS_COPY_MC .copy_mc_to_iter = discard_copy_to_iter, #endif - .csum_and_copy_to_iter = xxx_csum_and_copy_to_iter, + .csum_and_copy_to_iter = discard_csum_and_copy_to_iter, .csum_and_copy_from_iter = no_csum_and_copy_from_iter, .csum_and_copy_from_iter_full = no_csum_and_copy_from_iter_full, From patchwork Sat Nov 21 14:16:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96C9FC64E7A for ; Sat, 21 Nov 2020 14:17:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 640F62224C for ; Sat, 21 Nov 2020 14:17:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="DQAcCDpd" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727848AbgKUOQ5 (ORCPT ); Sat, 21 Nov 2020 09:16:57 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:42795 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728400AbgKUOQz (ORCPT ); Sat, 21 Nov 2020 09:16:55 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968213; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HFapy1GTMY1jnDDBsU1AjiSd9A2Mok+7FVSYnlNcoE0=; b=DQAcCDpdU/E7kr+XZIflEI7IWBdQ3D5+8z6YUPYB9jSYpHaiU5WwD1auD5XTMxSMg3zxzJ uekSV1XCVUh/C35WaRL3Vc0I57aA9J5K+A3VhtSm2wcrsHerlgYfMYZKBDoijFzDYkKuWZ 0A8bswZm4MxX6LI2g6ccY6i8DDTdFxE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-24-Jce-_24YOKG1wx3Hb_Vh7g-1; Sat, 21 Nov 2020 09:16:49 -0500 X-MC-Unique: Jce-_24YOKG1wx3Hb_Vh7g-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8DEA51005D6D; Sat, 21 Nov 2020 14:16:47 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 903985C22B; Sat, 21 Nov 2020 14:16:45 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 26/29] iov_iter: Split iov_iter_npages() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:16:44 +0000 Message-ID: <160596820474.154728.13702168133845850499.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split iov_iter_npages() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 84 ++++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 57 insertions(+), 27 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 2f8019e3b09a..d8ef6c81c55f 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -2004,50 +2004,80 @@ size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, } EXPORT_SYMBOL(hash_and_copy_to_iter); -static int xxx_npages(const struct iov_iter *i, int maxpages) +static int iovec_npages(const struct iov_iter *i, int maxpages) { size_t size = i->count; int npages = 0; if (!size) return 0; - if (unlikely(iov_iter_is_discard(i))) - return 0; - - if (unlikely(iov_iter_is_pipe(i))) { - struct pipe_inode_info *pipe = i->pipe; - unsigned int iter_head; - size_t off; - - if (!sanity(i)) - return 0; - - data_start(i, &iter_head, &off); - /* some of this one + all after this one */ - npages = pipe_space_for_user(iter_head, pipe->tail, pipe); - if (npages >= maxpages) - return maxpages; - } else iterate_all_kinds(i, size, v, ({ + iterate_over_iovec(i, size, v, ({ unsigned long p = (unsigned long)v.iov_base; npages += DIV_ROUND_UP(p + v.iov_len, PAGE_SIZE) - p / PAGE_SIZE; if (npages >= maxpages) return maxpages; - 0;}),({ + 0;})); + return npages; +} + +static int bvec_npages(const struct iov_iter *i, int maxpages) +{ + size_t size = i->count; + int npages = 0; + + if (!size) + return 0; + iterate_over_bvec(i, size, v, ({ npages++; if (npages >= maxpages) return maxpages; - }),({ + })); + return npages; +} + +static int kvec_npages(const struct iov_iter *i, int maxpages) +{ + size_t size = i->count; + int npages = 0; + + if (!size) + return 0; + iterate_over_kvec(i, size, v, ({ unsigned long p = (unsigned long)v.iov_base; npages += DIV_ROUND_UP(p + v.iov_len, PAGE_SIZE) - p / PAGE_SIZE; if (npages >= maxpages) return maxpages; - }) - ) + })); return npages; } +static int pipe_npages(const struct iov_iter *i, int maxpages) +{ + struct pipe_inode_info *pipe = i->pipe; + size_t size = i->count, off; + unsigned int iter_head; + int npages = 0; + + if (!size) + return 0; + if (!sanity(i)) + return 0; + + data_start(i, &iter_head, &off); + /* some of this one + all after this one */ + npages = pipe_space_for_user(iter_head, pipe->tail, pipe); + if (npages >= maxpages) + return maxpages; + return npages; +} + +static int discard_npages(const struct iov_iter *i, int maxpages) +{ + return 0; +} + static const void *xxx_dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) { *new = *old; @@ -2293,7 +2323,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .gap_alignment = iovec_gap_alignment, .get_pages = iovec_get_pages, .get_pages_alloc = iovec_get_pages_alloc, - .npages = xxx_npages, + .npages = iovec_npages, .dup_iter = xxx_dup_iter, .for_each_range = xxx_for_each_range, }; @@ -2327,7 +2357,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .gap_alignment = kvec_gap_alignment, .get_pages = no_get_pages, .get_pages_alloc = no_get_pages_alloc, - .npages = xxx_npages, + .npages = kvec_npages, .dup_iter = xxx_dup_iter, .for_each_range = xxx_for_each_range, }; @@ -2361,7 +2391,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .gap_alignment = bvec_gap_alignment, .get_pages = bvec_get_pages, .get_pages_alloc = bvec_get_pages_alloc, - .npages = xxx_npages, + .npages = bvec_npages, .dup_iter = xxx_dup_iter, .for_each_range = xxx_for_each_range, }; @@ -2395,7 +2425,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .gap_alignment = no_gap_alignment, .get_pages = pipe_get_pages, .get_pages_alloc = pipe_get_pages_alloc, - .npages = xxx_npages, + .npages = pipe_npages, .dup_iter = xxx_dup_iter, .for_each_range = xxx_for_each_range, }; @@ -2429,7 +2459,7 @@ static const struct iov_iter_ops discard_iter_ops = { .gap_alignment = no_gap_alignment, .get_pages = no_get_pages, .get_pages_alloc = no_get_pages_alloc, - .npages = xxx_npages, + .npages = discard_npages, .dup_iter = xxx_dup_iter, .for_each_range = xxx_for_each_range, }; From patchwork Sat Nov 21 14:16:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923491 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3E09C64E7C for ; Sat, 21 Nov 2020 14:17:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 88458222B8 for ; Sat, 21 Nov 2020 14:17:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="OqJDEQD0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728407AbgKUORA (ORCPT ); Sat, 21 Nov 2020 09:17:00 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:59510 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727892AbgKUOQ7 (ORCPT ); Sat, 21 Nov 2020 09:16:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968218; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GnsHwUz2owKZw0pxEz2d8pMfFj7z+dxze69mcs+0QuY=; b=OqJDEQD0x85brkq8/gauMZQdfHXVmsCiLiQ+MSmJ+QxDSSzvFlctylGwe780uo3LVtpFvL n1pJxV1Pen85aBFugoq+FaICBzNGUm7HnRIsW0DEY7Mt3c4PlFP46xZklD9SKTF0kRKbkA L9IiQUn+MZeMg2eaAuguDcpCatsynX0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-418-oO_5MCQAMwSJuSmormMs6g-1; Sat, 21 Nov 2020 09:16:56 -0500 X-MC-Unique: oO_5MCQAMwSJuSmormMs6g-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2A1B11842155; Sat, 21 Nov 2020 14:16:55 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 96FA85C22B; Sat, 21 Nov 2020 14:16:53 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 27/29] iov_iter: Split dup_iter() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:16:52 +0000 Message-ID: <160596821281.154728.8509303947333815047.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split dup_iter() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 49 +++++++++++++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 20 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index d8ef6c81c55f..ca0e94596eda 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -2078,26 +2078,35 @@ static int discard_npages(const struct iov_iter *i, int maxpages) return 0; } -static const void *xxx_dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) +static const void *iovec_kvec_dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) { *new = *old; - if (unlikely(iov_iter_is_pipe(new))) { - WARN_ON(1); - return NULL; - } - if (unlikely(iov_iter_is_discard(new))) - return NULL; - if (iov_iter_is_bvec(new)) - return new->bvec = kmemdup(new->bvec, - new->nr_segs * sizeof(struct bio_vec), - flags); - else - /* iovec and kvec have identical layout */ - return new->iov = kmemdup(new->iov, - new->nr_segs * sizeof(struct iovec), + /* iovec and kvec have identical layout */ + return new->iov = kmemdup(new->iov, + new->nr_segs * sizeof(struct iovec), + flags); +} + +static const void *bvec_dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) +{ + *new = *old; + return new->bvec = kmemdup(new->bvec, + new->nr_segs * sizeof(struct bio_vec), flags); } +static const void *discard_dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) +{ + *new = *old; + return NULL; +} + +static const void *no_dup_iter(struct iov_iter *new, struct iov_iter *old, gfp_t flags) +{ + WARN_ON(1); + return NULL; +} + static int copy_compat_iovec_from_user(struct iovec *iov, const struct iovec __user *uvec, unsigned long nr_segs) { @@ -2324,7 +2333,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .get_pages = iovec_get_pages, .get_pages_alloc = iovec_get_pages_alloc, .npages = iovec_npages, - .dup_iter = xxx_dup_iter, + .dup_iter = iovec_kvec_dup_iter, .for_each_range = xxx_for_each_range, }; @@ -2358,7 +2367,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .get_pages = no_get_pages, .get_pages_alloc = no_get_pages_alloc, .npages = kvec_npages, - .dup_iter = xxx_dup_iter, + .dup_iter = iovec_kvec_dup_iter, .for_each_range = xxx_for_each_range, }; @@ -2392,7 +2401,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .get_pages = bvec_get_pages, .get_pages_alloc = bvec_get_pages_alloc, .npages = bvec_npages, - .dup_iter = xxx_dup_iter, + .dup_iter = bvec_dup_iter, .for_each_range = xxx_for_each_range, }; @@ -2426,7 +2435,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .get_pages = pipe_get_pages, .get_pages_alloc = pipe_get_pages_alloc, .npages = pipe_npages, - .dup_iter = xxx_dup_iter, + .dup_iter = no_dup_iter, .for_each_range = xxx_for_each_range, }; @@ -2460,6 +2469,6 @@ static const struct iov_iter_ops discard_iter_ops = { .get_pages = no_get_pages, .get_pages_alloc = no_get_pages_alloc, .npages = discard_npages, - .dup_iter = xxx_dup_iter, + .dup_iter = discard_dup_iter, .for_each_range = xxx_for_each_range, }; From patchwork Sat Nov 21 14:17:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28964C63697 for ; Sat, 21 Nov 2020 14:17:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E8F482224A for ; Sat, 21 Nov 2020 14:17:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="PofveTjn" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727943AbgKUORK (ORCPT ); Sat, 21 Nov 2020 09:17:10 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:53516 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728399AbgKUORI (ORCPT ); Sat, 21 Nov 2020 09:17:08 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968226; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d87qXzclWdjG5Yvx06kiTnj6d7IqmRDOIVlZdO+mdY0=; b=PofveTjnytimYUIVC87JLHMEiM9t+4qozd8ojf2/C73MIBhxfA41dA/NZZ6CNguJG7RO4C hYxe6TKtryvd4SbUDH2SF6Fjb4vj2uIm+t6TyB+xK+EyCvttLhFU6IdcJJO0gQcuZ1xmQB NFM84MieTefy4ULewGyYKpTgZHgNmLk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-564-1lZL9I3jOhi78lr1D53rOw-1; Sat, 21 Nov 2020 09:17:04 -0500 X-MC-Unique: 1lZL9I3jOhi78lr1D53rOw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D66688042C6; Sat, 21 Nov 2020 14:17:02 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id 35AD65C22B; Sat, 21 Nov 2020 14:17:01 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 28/29] iov_iter: Split iov_iter_for_each_range() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:17:00 +0000 Message-ID: <160596822038.154728.6350968091336040046.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Split iov_iter_for_each_range() by type. Signed-off-by: David Howells --- lib/iov_iter.c | 41 +++++++++++++++++++++++++++++++---------- 1 file changed, 31 insertions(+), 10 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index ca0e94596eda..db798966823e 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -2282,7 +2282,7 @@ int import_single_range(int rw, void __user *buf, size_t len, } EXPORT_SYMBOL(import_single_range); -static int xxx_for_each_range(struct iov_iter *i, size_t bytes, +static int bvec_for_each_range(struct iov_iter *i, size_t bytes, int (*f)(struct kvec *vec, void *context), void *context) { @@ -2291,18 +2291,39 @@ static int xxx_for_each_range(struct iov_iter *i, size_t bytes, if (!bytes) return 0; - iterate_all_kinds(i, bytes, v, -EINVAL, ({ + iterate_over_bvec(i, bytes, v, ({ w.iov_base = kmap(v.bv_page) + v.bv_offset; w.iov_len = v.bv_len; err = f(&w, context); kunmap(v.bv_page); - err;}), ({ + err; + })); + return err; +} + +static int kvec_for_each_range(struct iov_iter *i, size_t bytes, + int (*f)(struct kvec *vec, void *context), + void *context) +{ + struct kvec w; + int err = -EINVAL; + if (!bytes) + return 0; + + iterate_over_kvec(i, bytes, v, ({ w = v; - err = f(&w, context);}) - ) + err = f(&w, context); + })); return err; } +static int no_for_each_range(struct iov_iter *i, size_t bytes, + int (*f)(struct kvec *vec, void *context), + void *context) +{ + return !bytes ? 0 : -EINVAL; +} + static const struct iov_iter_ops iovec_iter_ops = { .type = ITER_IOVEC, .copy_from_user_atomic = iovec_copy_from_user_atomic, @@ -2334,7 +2355,7 @@ static const struct iov_iter_ops iovec_iter_ops = { .get_pages_alloc = iovec_get_pages_alloc, .npages = iovec_npages, .dup_iter = iovec_kvec_dup_iter, - .for_each_range = xxx_for_each_range, + .for_each_range = no_for_each_range, }; static const struct iov_iter_ops kvec_iter_ops = { @@ -2368,7 +2389,7 @@ static const struct iov_iter_ops kvec_iter_ops = { .get_pages_alloc = no_get_pages_alloc, .npages = kvec_npages, .dup_iter = iovec_kvec_dup_iter, - .for_each_range = xxx_for_each_range, + .for_each_range = kvec_for_each_range, }; static const struct iov_iter_ops bvec_iter_ops = { @@ -2402,7 +2423,7 @@ static const struct iov_iter_ops bvec_iter_ops = { .get_pages_alloc = bvec_get_pages_alloc, .npages = bvec_npages, .dup_iter = bvec_dup_iter, - .for_each_range = xxx_for_each_range, + .for_each_range = bvec_for_each_range, }; static const struct iov_iter_ops pipe_iter_ops = { @@ -2436,7 +2457,7 @@ static const struct iov_iter_ops pipe_iter_ops = { .get_pages_alloc = pipe_get_pages_alloc, .npages = pipe_npages, .dup_iter = no_dup_iter, - .for_each_range = xxx_for_each_range, + .for_each_range = no_for_each_range, }; static const struct iov_iter_ops discard_iter_ops = { @@ -2470,5 +2491,5 @@ static const struct iov_iter_ops discard_iter_ops = { .get_pages_alloc = no_get_pages_alloc, .npages = discard_npages, .dup_iter = discard_dup_iter, - .for_each_range = xxx_for_each_range, + .for_each_range = no_for_each_range, }; From patchwork Sat Nov 21 14:17:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 11923495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1C15C56202 for ; Sat, 21 Nov 2020 14:17:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6D0042224A for ; Sat, 21 Nov 2020 14:17:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="A+DIxU1A" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728438AbgKUORQ (ORCPT ); Sat, 21 Nov 2020 09:17:16 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:55332 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728435AbgKUORQ (ORCPT ); Sat, 21 Nov 2020 09:17:16 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605968234; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qj6g0VELT0qjcnm2/ASGN+ZeoPWRa3oiC5gcLS/+yjQ=; b=A+DIxU1AhBEAYC7MjRjFL80eff8wOjrqbXAfCTfW/yCkblpgC7ulf2Dr4TiHhQjvh0uTb8 mRpmDZknaEtgrQsZpbRJ36HuoPclqUpP0ZROaEyySteOGeTYtmNCUFwNu3GCpAxjZrP9i1 98/A7w6HJFEBp54BmgoaRH+7Ul9Dl7E= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-291-7wUiTP2RMgKzm_8_wma7kw-1; Sat, 21 Nov 2020 09:17:12 -0500 X-MC-Unique: 7wUiTP2RMgKzm_8_wma7kw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 93A2B80EF8B; Sat, 21 Nov 2020 14:17:10 +0000 (UTC) Received: from warthog.procyon.org.uk (ovpn-112-246.rdu2.redhat.com [10.10.112.246]) by smtp.corp.redhat.com (Postfix) with ESMTP id E83885C22B; Sat, 21 Nov 2020 14:17:08 +0000 (UTC) Organization: Red Hat UK Ltd. Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SI4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 3798903 Subject: [PATCH 29/29] iov_iter: Remove iterate_all_kinds() and iterate_and_advance() From: David Howells To: Pavel Begunkov , Matthew Wilcox , Jens Axboe , Alexander Viro Cc: dhowells@redhat.com, Linus Torvalds , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org Date: Sat, 21 Nov 2020 14:17:08 +0000 Message-ID: <160596822810.154728.8582333072148760464.stgit@warthog.procyon.org.uk> In-Reply-To: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> References: <160596800145.154728.7192318545120181269.stgit@warthog.procyon.org.uk> User-Agent: StGit/0.23 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Remove iterate_all_kinds() and iterate_and_advance() as they're no longer used, having been split. Signed-off-by: David Howells --- lib/iov_iter.c | 61 -------------------------------------------------------- 1 file changed, 61 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index db798966823e..ba6b60c45103 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -86,26 +86,6 @@ static inline bool page_copy_sane(struct page *page, size_t offset, size_t n); } \ } -#define iterate_all_kinds(i, n, v, I, B, K) { \ - if (likely(n)) { \ - size_t skip = i->iov_offset; \ - if (unlikely(iov_iter_type(i) & ITER_BVEC)) { \ - struct bio_vec v; \ - struct bvec_iter __bi; \ - iterate_bvec(i, n, v, __bi, skip, (B)) \ - } else if (unlikely(iov_iter_type(i) & ITER_KVEC)) { \ - const struct kvec *kvec; \ - struct kvec v; \ - iterate_kvec(i, n, v, kvec, skip, (K)) \ - } else if (unlikely(iov_iter_type(i) & ITER_DISCARD)) { \ - } else { \ - const struct iovec *iov; \ - struct iovec v; \ - iterate_iovec(i, n, v, iov, skip, (I)) \ - } \ - } \ -} - #define iterate_over_iovec(i, n, v, CMD) { \ if (likely(n)) { \ size_t skip = i->iov_offset; \ @@ -133,47 +113,6 @@ static inline bool page_copy_sane(struct page *page, size_t offset, size_t n); } \ } -#define iterate_and_advance(i, n, v, I, B, K) { \ - if (unlikely(i->count < n)) \ - n = i->count; \ - if (i->count) { \ - size_t skip = i->iov_offset; \ - if (unlikely(iov_iter_type(i) & ITER_BVEC)) { \ - const struct bio_vec *bvec = i->bvec; \ - struct bio_vec v; \ - struct bvec_iter __bi; \ - iterate_bvec(i, n, v, __bi, skip, (B)) \ - i->bvec = __bvec_iter_bvec(i->bvec, __bi); \ - i->nr_segs -= i->bvec - bvec; \ - skip = __bi.bi_bvec_done; \ - } else if (unlikely(iov_iter_type(i) & ITER_KVEC)) { \ - const struct kvec *kvec; \ - struct kvec v; \ - iterate_kvec(i, n, v, kvec, skip, (K)) \ - if (skip == kvec->iov_len) { \ - kvec++; \ - skip = 0; \ - } \ - i->nr_segs -= kvec - i->kvec; \ - i->kvec = kvec; \ - } else if (unlikely(iov_iter_type(i) & ITER_DISCARD)) { \ - skip += n; \ - } else { \ - const struct iovec *iov; \ - struct iovec v; \ - iterate_iovec(i, n, v, iov, skip, (I)) \ - if (skip == iov->iov_len) { \ - iov++; \ - skip = 0; \ - } \ - i->nr_segs -= iov - i->iov; \ - i->iov = iov; \ - } \ - i->count -= n; \ - i->iov_offset = skip; \ - } \ -} - #define iterate_and_advance_iovec(i, n, v, CMD) { \ if (unlikely(i->count < n)) \ n = i->count; \