From patchwork Fri Sep 22 12:02:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13395698 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 859EDCD4F5B for ; Fri, 22 Sep 2023 12:02:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 117576B02BC; Fri, 22 Sep 2023 08:02:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AB0C6B02BD; Fri, 22 Sep 2023 08:02:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EAAFF6B02BF; Fri, 22 Sep 2023 08:02:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D676F6B02BC for ; Fri, 22 Sep 2023 08:02:38 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A8207B456F for ; Fri, 22 Sep 2023 12:02:38 +0000 (UTC) X-FDA: 81264096396.09.2486843 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id E094C2003A for ; Fri, 22 Sep 2023 12:02:36 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=a4F0OQbZ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695384157; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=UkL6HEMM59VaR1MHXCusr0awnLTOfiBRcRLWSZ6/5jE=; b=zO8aDoz/PAFNv5m0Kctf5YyHDhOqAgp5p3XRKttkUHsRtK4vRUcik77rDo9LUX7HPLx0lM aLKFXXX6lB74vMG1OGcgsoQIZ0g3EtUQ9TZ4elnJUczCoIKIx/F2OxhQQ9dO9QgkJJ7Vh3 FkDv49TEAKZNZtgHNwdXI8B1KO3FRvg= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=a4F0OQbZ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695384157; a=rsa-sha256; cv=none; b=XAe/1eCa1nvlsPY0rsVBcMAOfaeNpRpwsHJzs+y7Eeia+W2yXUlY8h1KmPSO9j/dAFVxWL J5IEyliaf12S04NglGJQmG8/vb5kVuDQioIQBBpIrvMJkYRqSBLU0qRULAb4CKHZbO1sWq O6J9jdVhDyfPPOfvuLw25Ng2gzdRwVE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695384156; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=UkL6HEMM59VaR1MHXCusr0awnLTOfiBRcRLWSZ6/5jE=; b=a4F0OQbZ+GTPvaycuFUURBOak9CicHI9BckYNcHC8bb+gklHNPlWFlzt3S0X6vS9HF6ZGO N25mqG0Q/L18BbP8txNfAL7x83DtmIvFiOWXTOxxRuQDrATgJalBkcUbO4Has3gO7morxq mgK0Edzq7XtmEY/ZcLlcUiyGpAVGqPk= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-511-WK1RsznjOcehA0svLM3e2w-1; Fri, 22 Sep 2023 08:02:33 -0400 X-MC-Unique: WK1RsznjOcehA0svLM3e2w-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0439E3C1350C; Fri, 22 Sep 2023 12:02:32 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3CA7449BB9A; Fri, 22 Sep 2023 12:02:30 +0000 (UTC) From: David Howells To: Jens Axboe Cc: David Howells , Al Viro , Linus Torvalds , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 00/13] iov_iter: Convert the iterator macros into inline funcs Date: Fri, 22 Sep 2023 13:02:14 +0100 Message-ID: <20230922120227.1173720-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: E094C2003A X-Stat-Signature: po7bwjopdmyc9sf7gdgomxzkfymg6gut X-Rspam-User: X-HE-Tag: 1695384156-447440 X-HE-Meta: U2FsdGVkX1/C9Rm5gLXtzmXb6YNXa7j4ymuYHKP+9T3wMfF/epjO8ZvXMbq7h66OO+Hce6V7nXh5kHJv05L+2YnkMPJuqK34MrLVoZxxPrZTS7EPtTVfSvSjXrxAM8KiP9SXpGsgUjzC6uOEMHTtt1WJfHnmPKWbLQvdZxrC+009R0HWMAnqXrBD3d0eIgwzRjcyi0Z4aLoocXYj8JhfmsErGsQsyIfGWP1AzxyowhUO5kAVS5vnLhrS96mPe0vGa3yga2OVjWXoLxul+UsAnnylR+SfVHZlpZ/K/X8PE0ZlzM6I7xTLjDZ4cYtAjYSeDAbXpnDydVuPrc0b87To/Y7GOThZMASzas+rObLiTOejmGdcGVVYzN+PvYBxf0KbHTA4RhflH7VvRC8KKdopo1WJd2Ad3ifL6rMKmZPzDSNz3g7I5UEIKAd/JDiCcWaxgAqlKxtvobXQwnQM8xi8MetqsZL1ZN0WEKz1Faxot2uWQX7dxWkmRD/mpnRlmOlrvUpezVpXSGFY0hAPS1B7lOV+91lsT4Q9hMWV/X/jOYVU1VjVpTqiRlZGvabSFaW8zsp7atZ9xQp8RUJnoDCMVXivfn8k5QJ7OBxzVsQ/ZciTCjrI4Aus7p4PBGlLWyKi7AtH9sI5VxOOJx5FZmolPI4AZEOzrGVYhCXXS2qokDQKxwmlRignUd4lBSf+fRNEZZvh6Il9Sf6bUWXUjgbqqmF7mKO0jYhuXZEvRorYACBPyFsLji1kTCRAmj22p+UrPyiqaEce0D48v6cfM4iKZdmOSH0rUWCvhH7uhyTsHOrCPn8GvvhhWtxPIgXPnEsobdqbeD2t59CksC7JLksC3ibAdcieZXxGdrEGch0YOvLrt/ESvO5L/XxSMUlsOTFMcMiSOQElE+u7g8WVIpztP+qtSNWSC99MBiEaMp71xpKdb98HlTtE3MbsN34GHF+HRCvRdbsnnzW5+z1+NoG Ti+zYfrG f9i0/r2E/pvKVaZVDNuk7asS0u58snnvF7eWzSiqE0Uh1diCdJeEO2XU6f8NrZsGsUaPlFUMfOaCV/3qImGZDxP41pBtmjrh8Q7Aso9b+fMByEdqH5+1UPnWP11aCheeYiSr+PglSxp+QPilosCAiPxBK0vMc3tRUsQ34Yx/+k+ky39nrEZHwfzMeLoTIBOQJ0gB2hhf36XdQ3tNHqioQoAFcwlGt3oy2pG6RYPV0QFhASvdRYHisuHL1FIkUenGzEzNPPay0aa5En7YFmuG7OEwuFHkMu6Y0pOS7eqnwiLQPCCBj470k0jDVfGa1R1uDfqwqFk60j3EufHpG5xEBtcvXkA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Jens, Could you take these patches into the block tree? The patches convert the iov_iter iteration macros to be inline functions. (1) Remove last_offset from iov_iter as it was only used by ITER_PIPE. (2) Add a __user tag on copy_mc_to_user()'s dst argument on x86 to match that on powerpc and get rid of a sparse warning. (3) Convert iter->user_backed to user_backed_iter() in the sound PCM driver. (4) Convert iter->user_backed to user_backed_iter() in a couple of infiniband drivers. (5) Renumber the type enum so that the ITER_* constants match the order in iterate_and_advance*(). (6) Since the preceding patch puts UBUF and IOVEC at 0 and 1, change user_backed_iter() to just use the type value and get rid of the extra flag. (7) Convert the iov_iter iteration macros to always-inline functions to make the code easier to follow. It uses function pointers, but they get optimised away. The priv2 argument likewise gets optimised away if unused. The functions are placed into a header file so that bespoke iterators can be created elsewhere. For instance, rbd has an optimisation that requires it to scan to the buffer it is given to see if it is all zeros. It would be nice if this could use iterate_and_advance() - but that's currently buried inside lib/iov_iter.c. Whilst we could provide a generic iteration function that takes a pair of function pointers instead, it does add around 16% overhead in the framework, presumably from the combination of function pointers and various mitigations. Further, if it is known that just a particular iterator-type is in play, just that iteration function can be used. (8) Move the check for ->copy_mc to _copy_from_iter() and copy_page_from_iter_atomic() rather than in memcpy_from_iter_mc() where it gets repeated for every segment. Instead, we check once and invoke a side function that can use iterate_bvec() rather than iterate_and_advance() and supply a different step function. (9) Provide a cut-down iteration function that only handles kernel-backed iterators (ie. BVEC, KVEC, XARRAY and DISCARD) for situations where we know that we won't see UBUF/IOVEC. This crops up in a number of filesystems where we decant a UBUF/IOVEC iter into a series of BVEC iters. (10) Move the copy-and-csum code to net/ where it can be in proximity with the code that uses it. This eliminates the code if CONFIG_NET=n and allows for the slim possibility of it being inlined. (11) Fold memcpy_and_csum() in to its two users. (12) Move csum_and_copy_from_iter_full() out of line and merge in csum_and_copy_from_iter() since the former is the only caller of the latter. (13) Move hash_and_copy_to_iter() to net/ where it can be with its only caller. Anyway, the changes in compiled function size either side of patches (5)+(6) on x86_64 look like: __copy_from_iter_mc new 0xe8 _copy_from_iter inc 0x360 -> 0x3a7 +0x47 _copy_from_iter_flushcache inc 0x34c -> 0x38e +0x42 _copy_from_iter_nocache inc 0x354 -> 0x378 +0x24 _copy_mc_to_iter inc 0x396 -> 0x3f1 +0x5b _copy_to_iter inc 0x33b -> 0x385 +0x4a copy_page_from_iter_atomic.part.0 inc 0x393 -> 0x3e0 +0x4d copy_page_to_iter_nofault.part.0 inc 0x3de -> 0x3eb +0xd copyin del 0x30 copyout del 0x2d copyout_mc del 0x2b csum_and_copy_from_iter inc 0x3db -> 0x41d +0x42 csum_and_copy_to_iter inc 0x45d -> 0x48d +0x30 iov_iter_zero inc 0x34a -> 0x36a +0x20 memcpy_from_iter.isra.0 del 0x1f Note that there's a noticeable expansion on some of the main functions because a number of the helpers get inlined instead of being called. In terms of benchmarking patch (5)+(6), three runs without them: iov_kunit_benchmark_ubuf: avg 3955 uS, stddev 169 uS iov_kunit_benchmark_ubuf: avg 4122 uS, stddev 1292 uS iov_kunit_benchmark_ubuf: avg 4451 uS, stddev 1362 uS iov_kunit_benchmark_iovec: avg 6607 uS, stddev 22 uS iov_kunit_benchmark_iovec: avg 6608 uS, stddev 19 uS iov_kunit_benchmark_iovec: avg 6609 uS, stddev 24 uS iov_kunit_benchmark_bvec: avg 3166 uS, stddev 11 uS iov_kunit_benchmark_bvec: avg 3167 uS, stddev 13 uS iov_kunit_benchmark_bvec: avg 3170 uS, stddev 16 uS iov_kunit_benchmark_bvec_split: avg 3394 uS, stddev 12 uS iov_kunit_benchmark_bvec_split: avg 3394 uS, stddev 20 uS iov_kunit_benchmark_bvec_split: avg 3395 uS, stddev 19 uS iov_kunit_benchmark_kvec: avg 2672 uS, stddev 12 uS iov_kunit_benchmark_kvec: avg 2672 uS, stddev 12 uS iov_kunit_benchmark_kvec: avg 2672 uS, stddev 9 uS iov_kunit_benchmark_xarray: avg 3719 uS, stddev 9 uS iov_kunit_benchmark_xarray: avg 3719 uS, stddev 9 uS iov_kunit_benchmark_xarray: avg 3721 uS, stddev 24 uS and three runs with them: iov_kunit_benchmark_ubuf: avg 4110 uS, stddev 1254 uS iov_kunit_benchmark_ubuf: avg 4141 uS, stddev 1411 uS iov_kunit_benchmark_ubuf: avg 4572 uS, stddev 1889 uS iov_kunit_benchmark_iovec: avg 6582 uS, stddev 27 uS iov_kunit_benchmark_iovec: avg 6585 uS, stddev 25 uS iov_kunit_benchmark_iovec: avg 6586 uS, stddev 48 uS iov_kunit_benchmark_bvec: avg 3175 uS, stddev 13 uS iov_kunit_benchmark_bvec: avg 3177 uS, stddev 12 uS iov_kunit_benchmark_bvec: avg 3178 uS, stddev 12 uS iov_kunit_benchmark_bvec_split: avg 3380 uS, stddev 20 uS iov_kunit_benchmark_bvec_split: avg 3384 uS, stddev 15 uS iov_kunit_benchmark_bvec_split: avg 3386 uS, stddev 25 uS iov_kunit_benchmark_kvec: avg 2671 uS, stddev 11 uS iov_kunit_benchmark_kvec: avg 2672 uS, stddev 12 uS iov_kunit_benchmark_kvec: avg 2677 uS, stddev 20 uS iov_kunit_benchmark_xarray: avg 3599 uS, stddev 20 uS iov_kunit_benchmark_xarray: avg 3603 uS, stddev 8 uS iov_kunit_benchmark_xarray: avg 3610 uS, stddev 16 uS I've pushed the patches here also: https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/linux-fs.git/log/?h=iov-cleanup David Changes ======= ver #6) - Rebase on v6.6-rc2 rather than on my iov-kunit branch. - Add a patch to remove last_offset from iov_iter. - Add a patch to tag copy_mc_to_user() with __user on x86. - Document the priv2 arg of iterate_and_advance_kernel(). ver #5) - Moved kunit test patches out to a separate set. - Moved "iter->count - progress" into individual iteration subfunctions. - Fix iterate_iovec() and to update iter->__iovec after subtracting it to calculate iter->nr_segs. - Merged the function inlining patch and the move-to-header patch. - Rearranged the patch order slightly to put the patches to move networking stuff to net/ last. - Went back to a simpler patch to special-case ->copy_mc in _copy_from_iter() and copy_page_from_iter_atomic() before we get to call iterate_and_advance() so as to reduce the number of checks for this. ver #4) - Fix iterate_bvec() and iterate_kvec() to update iter->bvec and iter->kvec after subtracting it to calculate iter->nr_segs. - Change iterate_xarray() to use start+progress rather than increasing start to reduce code size. - Added patches to move some iteration functions over to net/ as the files there can #include the iterator framework. - Added a patch to benchmark the iteration. - Tried an experimental patch to make copy_from_iter() and similar always catch MCE. ver #3) - Use min_t(size_t,) not min() to avoid a warning on Hexagon. - Inline all the step functions. - Added a patch to better handle copy_mc. ver #2) - Rebased on top of Willy's changes in linux-next. - Change the checksum argument to the iteration functions to be a general void* and use it to pass iter->copy_mc flag to memcpy_from_iter_mc() to avoid using a function pointer. - Arrange the end of the iterate_*() functions to look the same to give the optimiser the best chance. - Make iterate_and_advance() a wrapper around iterate_and_advance2(). - Adjust iterate_and_advance2() to use if-else-if-else-if-else rather than switch(), to put ITER_BVEC before KVEC and to mark UBUF and IOVEC as likely(). - Move "iter->count += progress" into iterate_and_advance2() from the iterate functions. - Mark a number of the iterator helpers with __always_inline. - Fix _copy_from_iter_flushcache() to use memcpy_from_iter_flushcache() not memcpy_from_iter(). Link: https://lore.kernel.org/r/3710261.1691764329@warthog.procyon.org.uk/ # v1 Link: https://lore.kernel.org/r/855.1692047347@warthog.procyon.org.uk/ # v2 Link: https://lore.kernel.org/r/20230816120741.534415-1-dhowells@redhat.com/ # v3 Link: https://lore.kernel.org/r/20230913165648.2570623-1-dhowells@redhat.com/ # v4 Link: https://lore.kernel.org/r/20230920222231.686275-1-dhowells@redhat.com/ # v5 David Howells (13): iov_iter: Remove last_offset from iov_iter as it was for ITER_PIPE iov_iter: Be consistent about the __user tag on copy_mc_to_user() sound: Fix snd_pcm_readv()/writev() to use iov access functions infiniband: Use user_backed_iter() to see if iterator is UBUF/IOVEC iov_iter: Renumber ITER_* constants iov_iter: Derive user-backedness from the iterator type iov_iter: Convert iterate*() to inline funcs iov_iter: Don't deal with iter->copy_mc in memcpy_from_iter_mc() iov_iter: Add a kernel-type iterator-only iteration function iov_iter, net: Move csum_and_copy_to/from_iter() to net/ iov_iter, net: Fold in csum_and_memcpy() iov_iter, net: Merge csum_and_copy_from_iter{,_full}() together iov_iter, net: Move hash_and_copy_to_iter() to net/ arch/x86/include/asm/uaccess.h | 2 +- arch/x86/lib/copy_mc.c | 8 +- drivers/infiniband/hw/hfi1/file_ops.c | 2 +- drivers/infiniband/hw/qib/qib_file_ops.c | 2 +- include/linux/iov_iter.h | 306 ++++++++++++++++ include/linux/skbuff.h | 3 + include/linux/uio.h | 34 +- lib/iov_iter.c | 441 +++++++---------------- net/core/datagram.c | 75 +++- net/core/skbuff.c | 40 ++ sound/core/pcm_native.c | 4 +- 11 files changed, 576 insertions(+), 341 deletions(-) create mode 100644 include/linux/iov_iter.h