From patchwork Wed Sep 13 16:56:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C314EE01E0 for ; Wed, 13 Sep 2023 16:57:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D1D8D6B01F2; Wed, 13 Sep 2023 12:57:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCC736B01F3; Wed, 13 Sep 2023 12:57:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B94F96B01F4; Wed, 13 Sep 2023 12:57:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A52D86B01F2 for ; Wed, 13 Sep 2023 12:57:04 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 63B9D1404C7 for ; Wed, 13 Sep 2023 16:57:04 +0000 (UTC) X-FDA: 81232179168.14.345E286 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id A649B12001B for ; Wed, 13 Sep 2023 16:57:02 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IVV5jXBj; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624222; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iCJtvcwek/PzSsuqar23vXotUlaq5CuwRpVtwMRiY9g=; b=sCnzoYuFggWVdzX92ghhk1jI5IVpsD1peDZyUmTDHHtKayWbce9nffNqeFXxRETCQZ/Fxf dLONj/MGk0Dv6Kp115p8fSiN5N9JKDC1TPouHfpWn09F0QEmW4Szplvcuybr6S1iN7UPwk 1At4DLcLXqKP5cZnptZp1YRsjajv0xw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624222; a=rsa-sha256; cv=none; b=Ju52/a4we6GLAkTvQ7kgdOXfWC6DZNEyQdKkS/Ds6e0eZsvszm/KNnwIoua/WFXI5mZf6T f0UpkHljbeotCXsp5eUqjlYAXcQfHW7F0uw2dJfJVWuVT/d/5AWJkLLtNwEPOd6DFVfGea CpZhYj86e9sBr8w7t8OaFhTUJmDckqs= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IVV5jXBj; spf=pass (imf29.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624222; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iCJtvcwek/PzSsuqar23vXotUlaq5CuwRpVtwMRiY9g=; b=IVV5jXBjjY2Q0i1MmIoZpQkTxsy71CmhlIPVFmxgLxgmi1puahc6w9/sDiuVlT6wVGE3Gc 6ZAcYD8FGGwFzeMTdazrha2qs/sf4AbrSbQcvgS9DIhWwR4MkZ0OHsE/evZx+WzGIODiJt kMvzIX6H1lsEO9ua75+x1Lvu94vGitM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-674-3UG6PQn5O3qmluWTH-Ob_w-1; Wed, 13 Sep 2023 12:56:56 -0400 X-MC-Unique: 3UG6PQn5O3qmluWTH-Ob_w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EA3DC185A79B; Wed, 13 Sep 2023 16:56:55 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4D05340C6EA8; Wed, 13 Sep 2023 16:56:54 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 01/13] iov_iter: Add a benchmarking kunit test Date: Wed, 13 Sep 2023 17:56:36 +0100 Message-ID: <20230913165648.2570623-2-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: A649B12001B X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: bgxojosdxtdccenit4igu11feehyist1 X-HE-Tag: 1694624222-306430 X-HE-Meta: U2FsdGVkX19xGbKToS9zswI4hkr+pFGmJ5lkkJO5iZPgjjO1R/eRLAz5Yuatm9xXEQScbYH2csaJLLv63vviVO5NGqvjisObcFkZVBUH3/wPwuRsR0Rnjv22J0SvwdYxAbpSIysu/SWGueoMOQbufM91RUS9QqBb0//ENShhPIiFU6KF6QNuzl1Yo9QCiSfqooPSAY0Fp1P6zDn2JwL/xJ12ZVb4SUJcfKiE1yfXs/Eqz6gJsU68MGiehfNXeoET4CiFKdbV9uRCrng414AHtZsbzo1OvJQmkiRbX88fVHcsen2gZn6Y6mQJEMuYamfFD6WwSbZTOhi9zlEWS0WlfV+0bRL1edy5EVZ/8eQuRdDkK86olGTEE4RfIkC5a+j3RWdgM1+781jc+aGlbWZpebgrZSr6jH/ESLvaQgX1ChbNWoA066EpgyLafbW77N2A9Muk6mspwSmy2EwwflcFdjBBrkuO0EPp3IdmlGIphMDxFPGdK1c7mmTEs+h3BKmNHxC1n1kx4kixETuPT04ZUMJcFKl9O0z5FYIf3J/P9/ZRvN3hkM8YMDg5MbG+wrGEUheulCKd81PwtDPKavFuQzdBJvKreLULX8T+hM/mH7PoWCB9DMmkxdq64JX+kDzmOMXylel5yVuVxyclhNPgOH92Dv3mW5v5EAAV8bt1wAkjwCKnU9h+5cB9tDhVdCH3oW0lwKhE7bCrfKovjB0csoL8AgCtBdFUBU2QPwHP1gp4Oi7CO5ucVdUYkSnJw3fa5LMKytbjXTxagWEf9I3sMEHMwr6q2AFMSQurQDnCAfJbo8VSXkeE3gOIsZalPp+EDXi4mB3XAQIWu9ziOD1qtvGyzDvnbeNax7JcTrPECFIKGgGX3NdJDwkvZVytSQtgosdARC9MTFSVncT/LtNZce3wilHgI5peE3tV+qO/2QM5yp5bzLVmJrx/tCqKWGsHAUdqLutQcAay8BtL9Em ZKq5p8vE 4kDd01eIruKdw7GpG6sfD6fz9VELQebuFXArHskiLfS/tUJed841WgDiS6wPwptwiI4l8vVQatcDoE7xWLcZyUVclEsVycHpR+tGk1ujLmG+QfebPlbIHEJq4QwUrEAKgqOlEH0mjtqtCS9ZVSeZVVPZ+mX+KckJOFbtrx4UQhznawhoGr2hivADifyBa2fObTi52f5E8Nu/qUL4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --- lib/kunit_iov_iter.c | 181 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) diff --git a/lib/kunit_iov_iter.c b/lib/kunit_iov_iter.c index 859b67c4d697..478fea956f58 100644 --- a/lib/kunit_iov_iter.c +++ b/lib/kunit_iov_iter.c @@ -756,6 +756,184 @@ static void __init iov_kunit_extract_pages_xarray(struct kunit *test) KUNIT_SUCCEED(); } +static void iov_kunit_free_page(void *data) +{ + __free_page(data); +} + +static void __init iov_kunit_benchmark_print_stats(struct kunit *test, + unsigned int *samples) +{ + unsigned long total = 0; + int i; + + for (i = 0; i < 16; i++) { + total += samples[i]; + kunit_info(test, "run %x: %u uS\n", i, samples[i]); + } + + kunit_info(test, "avg %lu uS\n", total / 16); +} + +/* + * Time copying 256MiB through an ITER_BVEC. + */ +static void __init iov_kunit_benchmark_bvec(struct kunit *test) +{ + struct iov_iter iter; + struct bio_vec *bvec; + struct page *page, **pages; + unsigned int samples[16]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE; + void *scratch; + int i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + bvec = kunit_kmalloc_array(test, npages, sizeof(bvec[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, bvec); + for (i = 0; i < npages; i++) + bvec_set_page(&bvec[i], page, PAGE_SIZE, 0); + + /* Create a single large buffer to copy to/from. */ + pages = kunit_kmalloc_array(test, npages, sizeof(pages[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pages); + for (i = 0; i < npages; i++) + pages[i] = page; + + scratch = vmap(pages, npages, VM_MAP | VM_MAP_PUT_PAGES, PAGE_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, scratch); + kunit_add_action_or_reset(test, iov_kunit_unmap, scratch); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over BVEC:\n"); + for (i = 0; i < 16; i++) { + iov_iter_bvec(&iter, ITER_DEST, bvec, npages, size); + a = ktime_get_real(); + copied = copy_to_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_BVEC in 256 page chunks. + */ +static void __init iov_kunit_benchmark_bvec_split(struct kunit *test) +{ + struct iov_iter iter; + struct bio_vec *bvec; + struct page *page, **pages; + unsigned int samples[16]; + ktime_t a, b; + ssize_t copied; + size_t size, npages = 64; + void *scratch; + int i, j; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + /* Create a single large buffer to copy to/from. */ + pages = kunit_kmalloc_array(test, npages, sizeof(pages[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pages); + for (i = 0; i < npages; i++) + pages[i] = page; + + scratch = vmap(pages, npages, VM_MAP | VM_MAP_PUT_PAGES, PAGE_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, scratch); + kunit_add_action_or_reset(test, iov_kunit_unmap, scratch); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over BVEC:\n"); + for (i = 0; i < 16; i++) { + size = 256 * 1024 * 1024; + a = ktime_get_real(); + do { + size_t part = min(size, npages * PAGE_SIZE); + + bvec = kunit_kmalloc_array(test, npages, sizeof(bvec[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, bvec); + for (j = 0; j < npages; j++) + bvec_set_page(&bvec[j], page, PAGE_SIZE, 0); + + iov_iter_bvec(&iter, ITER_DEST, bvec, npages, part); + copied = copy_to_iter(scratch, part, &iter); + KUNIT_EXPECT_EQ(test, copied, part); + size -= part; + } while (size > 0); + b = ktime_get_real(); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_XARRAY. + */ +static void __init iov_kunit_benchmark_xarray(struct kunit *test) +{ + struct iov_iter iter; + struct xarray *xarray; + struct page *page, **pages; + unsigned int samples[16]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE; + void *scratch; + int i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + xarray = iov_kunit_create_xarray(test); + + for (i = 0; i < npages; i++) { + void *x = xa_store(xarray, i, page, GFP_KERNEL); + + KUNIT_ASSERT_FALSE(test, xa_is_err(x)); + } + + /* Create a single large buffer to copy to/from. */ + pages = kunit_kmalloc_array(test, npages, sizeof(pages[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pages); + for (i = 0; i < npages; i++) + pages[i] = page; + + scratch = vmap(pages, npages, VM_MAP | VM_MAP_PUT_PAGES, PAGE_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, scratch); + kunit_add_action_or_reset(test, iov_kunit_unmap, scratch); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over XARRAY:\n"); + for (i = 0; i < 16; i++) { + iov_iter_xarray(&iter, ITER_DEST, xarray, 0, size); + a = ktime_get_real(); + copied = copy_to_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + static struct kunit_case __refdata iov_kunit_cases[] = { KUNIT_CASE(iov_kunit_copy_to_kvec), KUNIT_CASE(iov_kunit_copy_from_kvec), @@ -766,6 +944,9 @@ static struct kunit_case __refdata iov_kunit_cases[] = { KUNIT_CASE(iov_kunit_extract_pages_kvec), KUNIT_CASE(iov_kunit_extract_pages_bvec), KUNIT_CASE(iov_kunit_extract_pages_xarray), + KUNIT_CASE(iov_kunit_benchmark_bvec), + KUNIT_CASE(iov_kunit_benchmark_bvec_split), + KUNIT_CASE(iov_kunit_benchmark_xarray), {} }; From patchwork Wed Sep 13 16:56:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383566 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D94FDEE01E9 for ; Wed, 13 Sep 2023 16:57:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B4066B01F3; Wed, 13 Sep 2023 12:57:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 63DE96B01F4; Wed, 13 Sep 2023 12:57:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 41B2B6B01F5; Wed, 13 Sep 2023 12:57:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 280F06B01F3 for ; Wed, 13 Sep 2023 12:57:06 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D42C9120454 for ; Wed, 13 Sep 2023 16:57:05 +0000 (UTC) X-FDA: 81232179210.05.33D7BA3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 1D489160014 for ; Wed, 13 Sep 2023 16:57:02 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="FEfh9ti/"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624223; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=437kiFAHLHJfEBaWwZrq8WSevs6s9dyVAFCFY8Y8ee4=; b=RzlZscqB7RRifwMp4w0YbVdjBYV60JxoIVmJ3kGnIsi2C9duSyDSUE/3xhbcpPv5U8pRwX 9lHfg6lUCNqxA63NETQQHwIHQJRd0mIQ5Nwhv9YDDKXilPxFBka7Ipsu+wXQeDxlgwsX+K PE3X895ETsm9E40pL81pT72diugapcw= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="FEfh9ti/"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624223; a=rsa-sha256; cv=none; b=J+pwHbH4n1B+ivtJh48dM75zTAHwjVQdnu70wP1vRqV2X1CuZIph/ObseV0i3ta9JmU9Ys L6V6ZMG0OTCOuXst0Qh3S0H5iIsZiCoqmp/E0XYWF61/zH/SBIi12H9DnR6Osxqy3IIdZr 6t/Dy9TX5t0QzItaSJspgh2Ab2jEdwc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624222; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=437kiFAHLHJfEBaWwZrq8WSevs6s9dyVAFCFY8Y8ee4=; b=FEfh9ti/pqOOq8LBKgl+gA5GbuDsrGst/WHxhY4+UBdT9xvGKZA+zv+VF9KnP/UrLfAKfX C/IgwF6v2B2/AJjfKZr5G+5GjkQjvc3Z0jJw5Sp88K6z/N/HmFd/a5Sqm3h6Pf6o4Mo8gX LDH7TYKaSpjc7lS4yOtARmbk01S341o= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-313-ofpFxPGzPwO-sSg4BhMp7A-1; Wed, 13 Sep 2023 12:56:59 -0400 X-MC-Unique: ofpFxPGzPwO-sSg4BhMp7A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5F9CA8032FE; Wed, 13 Sep 2023 16:56:58 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id A7EE42026D4B; Wed, 13 Sep 2023 16:56:56 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 02/13] iov_iter: Renumber ITER_* constants Date: Wed, 13 Sep 2023 17:56:37 +0100 Message-ID: <20230913165648.2570623-3-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: 1D489160014 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: j5hinnwa6r93jdjqh1xjooxux91nx86w X-HE-Tag: 1694624222-99314 X-HE-Meta: U2FsdGVkX19d4YWGzgcZrGYKNCc57bnrYLtGf/ycGC4+BYTpqmQWNVtObkXxhT9BoeK+XYbkfFBnAS14OSuNv7sk9sMEPUlcppOFPdlOdcAKtjEPqlXTSUmSouuI9MDNAAbHUFlT9acQiUCF8bJ2RKt2BJMTDdBIgxXZ/lIuw1Q7J+cE3LTDlYni8HA7K3GyIxUuFfl6RdCia5RTJBvA/J+J6GPJTdyHijwFmIf2+nPi0vzZIp39pAoSLz+UoqgDXtWop/rjdftqXqaPuqM9uuO9ndW+2WbCLqhCRcyvM77WemJrOcx/Bt0UqPvrTVucAXgWwwjcuttdaq+fqnaNYXNKB9Joh6k2U+Yl2KB8Yy0lgbuMuDrHgnxViWvv3lfzMLOpm/itMfGGGM1P2XZEjZXj1jFpfeqevB08+9V7zws4EeIZ3udzOrM4Kd75QrLA3jEniya8fVXy5wiuNGBPX3Hex0YRxeFUjOLLFMwODgl1LzPclAlxGlPAWHa0k5TDocPmixOyVuzxIq6u1z2drYgrYnGssTbPZvmDS2Suw+N1/zRAcK81DEa9b9e/btZaxwAHVMX9+HOLH1ti0sctZyoRzvC/37ReLZQGt/yBtkZVzmixUZVXqCPHTvS2q7PPhjRfx0SOei22iZ2qYhpY8+RjEfvp5ZnAJsUwDOBokE8pgQBdKJRdwCjtu6ENJFlU5NKQ+D/3NZuAjzZRsrSbRrVAqaF3Htpv8pUujlPIfPHCc1bt34I9iJa6x6pEBIHsfhQMBTEYIq935WHPXfMz5Fb0Uw8hRh2TJigBXYAimfz2kuZRFMBEWqxvbgq7v7LU7UsC+UOUq810PL6k6OWaEiJKB8Vz4kZd5fj7uqD3HNdumOkz3baX3ZTXDedTZDCu5F8axOvvtd/OXyhqnO2q4vLghYrbElaBbV0mXeZ4hfuWbaR69bSC/SrxlVGYRKe3485mCkQtl3HTmD5a16v HJvGTaUu EgymcgfMe01aLiRoWEif/JT5b1JvSoUR/+MB0TuAlUk+jW/Qnpmv4LkKDxWSURb3BlSL584E56L/ReGh4qarZhZw7i9hfIvxPYCMFSM64NdKKP+rxF2YxmjAakCuWcD0XPv4vwi3rlZ9rrgBHXfeTWGyB44+1qWqzOCohx7TkikuSAQyQdqqZZw02qra9cKrx5g8oeMEPQom46108ST1Kaqy7X1xQCfpshsHlMF+UocI1Z3gkDiimxHVVrEDgWmZQZW06Jvlm6NpeVTVgHilBZObYpw2aCzH+POIvTJWlQ1JgWaDxJpXzRGad9XzmEoDZWBJLVFUO/8heuwobAEcYtgpPI4GdC0DaseAs6INoIMDulUAmHgYtZE1gjFcHgmWgbBg6o1WnsoM3VF6QYhRCFczkEIaKy6zH8cotq8xoZ35lC8xeM05cFPNKWa/K+99w7CHjqTSZ17SzaPBp57BZtnmu67sFmhviwfyFFkP+JSCE81jjJJtg86Xlup3K8m+O7S3CPnJF8ddx+FlSzRi5qtk066BGC5sEtkQVGlG+zc+KsIXSv+nO9XrQrAP2FLqt11qRL/WPKTgLXDk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Renumber the ITER_* iterator-type constants to put things in the same order as in the iteration functions and to group user-backed iterators at the bottom. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- include/linux/uio.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index 42bce38a8e87..d1801c46e89e 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -21,12 +21,12 @@ struct kvec { enum iter_type { /* iter types */ + ITER_UBUF, ITER_IOVEC, - ITER_KVEC, ITER_BVEC, + ITER_KVEC, ITER_XARRAY, ITER_DISCARD, - ITER_UBUF, }; #define ITER_SOURCE 1 // == WRITE From patchwork Wed Sep 13 16:56:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4AA1EE01E0 for ; Wed, 13 Sep 2023 16:57:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9EC976B01F4; Wed, 13 Sep 2023 12:57:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9998C6B01F5; Wed, 13 Sep 2023 12:57:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7EBF16B01F6; Wed, 13 Sep 2023 12:57:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6D9C16B01F4 for ; Wed, 13 Sep 2023 12:57:07 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 49460B41F6 for ; Wed, 13 Sep 2023 16:57:07 +0000 (UTC) X-FDA: 81232179294.19.E79EB7B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 8603B1A0005 for ; Wed, 13 Sep 2023 16:57:05 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OZGolSBv; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624225; a=rsa-sha256; cv=none; b=RsMgm0w76JnTtJO42sKC2B2s69e8sIBp6oBLBxZA73WvKYWkyXCQuYg3Vlf4oLnq6MUt7z uV55iuTU5YMPKBk/98buxl5f1hy+PXXbDNq+nTWdMdn735/kYEzZLqDAQV9PlBJdbqZiJ4 o+pi2xPNmu6DTgBz1jTvl5t8sFZv+/8= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OZGolSBv; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624225; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z7dUGMUCdPzyoXpN55evy5eiv8axvEjBjfUl9Q9dpbg=; b=bBQD4eSkxi3TRblCSyVl2cQJJ+ItcbDoTzC+Ryrc8CDcovwwnuOvCdgKs/OPMvn1jzbEIQ K+ee7AFV7KUnXzVLhQcwo0MC+rskmvfboCFlvTq26p2gGbSIKEcpv3zIZLjU84uMVfRro3 9tW/zsDAOu4+AyrXkHbcOGWgo8sVLYE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624224; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z7dUGMUCdPzyoXpN55evy5eiv8axvEjBjfUl9Q9dpbg=; b=OZGolSBvzk3B/YVM/aDJSgTCXUA8wJzeN5NCyALftfAaK1dkgjfO2Jae3hfwOkL1k79HsT XcOZ0GiNFP7HcM2ZHRtGDJASFlymEfAeFgVwOwiivOctz6m7umJVEhU7ETRKhv7UMW+D9Z nSVBsx6O5x5Sv0bNpI92g10vV+k4pnY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-354-B53T9xcDN2imp2fxxNS_HQ-1; Wed, 13 Sep 2023 12:57:01 -0400 X-MC-Unique: B53T9xcDN2imp2fxxNS_HQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A0E9C185A79C; Wed, 13 Sep 2023 16:57:00 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 07E2263F9C; Wed, 13 Sep 2023 16:56:58 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 03/13] iov_iter: Derive user-backedness from the iterator type Date: Wed, 13 Sep 2023 17:56:38 +0100 Message-ID: <20230913165648.2570623-4-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 8603B1A0005 X-Stat-Signature: yc3nnijsuub31pmbr413bss7fmzh1315 X-HE-Tag: 1694624225-566872 X-HE-Meta: U2FsdGVkX19F58oMklNOhdW5OziQuycRkv5++tGYsgXutO+Jl/Ytw/2lYQeilWnMt65YVXwoOGZlvJUmwVOmcX0uOo2gbqpZufMJqCKvvZL+Qnpy1Ky81YGm3QedWkEf67u9pcLhbGgoLUEfqEQXcADVFj0wmFCYAYzv0bbgAfup1aq1pJjWjPEcd6aJVq5HYYkz6XC1Te0v/tTdeOWX4j60Wu75lXO1Q44J1M1TPIz3S3YGaPZ0M3DjLHYoApf/K+qEEdwygblnjIkWkvM/12Gmf8FH50wedtnO/vvi+a7frBIn+2Smu445+H3+gstVFVKl8V+SlsrPPygwOuVZKTubKhAS9FhK3KVpinYNW4sJAeIhuz4wGNJplIP8N3dVyvRgtWP53ge7I+fPz9fS+52z8Iey/RtCXTNgNEoDBxpOGnmsAdUK2XGZuA+jUKyRMOZPCc7/Ek9qtKluoGSDxt9k9X+10LfwaTLx0+mOw4F7jh1uIVQriCH3LcdmnIbV40OVa7rvomQihRKNweu/0PJZnoZOQGhNsyD9dHZ/PFwfpXicgRJafaqSxAIaDqCKB1Rnm4wLAqglb0fkLnADim82PDFqgDdUwV3RNHYLdS7jq1o6updj/xV6roRw2UDk6XyANPV9vMHRk9wogB+lt6eN9X4J5UGhLSEEK/+3YXcYhwpI14tMGLe6C17zCC2oH9+/RfCeyIvuxnTXMJkq76fYDeYoUSb0PIFxcDKyRGIq9OVmLRqD5zUY6NjwieDGdM03PHith9LulnRVmF4TwghlAzIAyGtoRcYUQUXpjltNXcQJulIilwaQiSv8CEc8jtgQ123J5+VWeo7iodX+Ige2sejymF36c8Hv8KliQf0/5jBeXHANcc1qAKCt2slcGymPliiwDwOvPBZHXWq6tEYRlhm8rSlV4yFy17x9RRgTJVDR56ayTv4/gQ+3kZlBn+QhbqxUZbSk/bqvUAj C2BK0Hsn Sb7NuR3zDUv/DmDMBStHH2U+cDqSiCHz11omOXYCVu98ME0VBOI92kbAh33G72rZSa6z2svpZqTXl9VEbk5bVt0dUomxgjI0Ux5IGUGmSmz4hzU7HO7PqRse8DV5Vqx20Sbs5EgcNYxKzbAqhNubQofr5ZyuDgYXyCLD9Sdae1cwwVSHquAUVZcgyLRHdw4NVsbWY1iiTo+xK8yunYoPJm+aqQv7eviUwOtd3RmxA+4xdIkcGhRht9ZlFEpbICIl3p/S0y/vUTZo/kOpDpYZQADpBbtqMMbLMy+gIwh5nzDebrbX0w2p0nUrbvx5QPrqrupcjfueQsCibgFbf5n4LYOC+D5aYl32BVHsXY6xynkCUrqeL6N3Pd5vKSJOJvbRY7/oMjeaqN4X8q8qeJVApi0JdoR4q7LDz6Xcdq2AwdnwiQ52pJyTJS8TpGxHPYAFR2fMy2pKujo3baPUqExMYHXg8XaH8eS/Kn4hKdifzXuOA+fCi6Qyst01Wc8xpK4S34j8DJck1pY88Yv80pcdAk/PxRc6fF2WCkCLlnMtcy5kRJF+8UaaaE2Q73hFMUpFKdPCdesmbU6ulXUU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the iterator type to determine whether an iterator is user-backed or not rather than using a special flag for it. Now that ITER_UBUF and ITER_IOVEC are 0 and 1, they can be checked with a single comparison. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- include/linux/uio.h | 4 +--- lib/iov_iter.c | 1 - 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index d1801c46e89e..e2a248dad80b 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -43,7 +43,6 @@ struct iov_iter { bool copy_mc; bool nofault; bool data_source; - bool user_backed; union { size_t iov_offset; int last_offset; @@ -143,7 +142,7 @@ static inline unsigned char iov_iter_rw(const struct iov_iter *i) static inline bool user_backed_iter(const struct iov_iter *i) { - return i->user_backed; + return iter_is_ubuf(i) || iter_is_iovec(i); } /* @@ -383,7 +382,6 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, *i = (struct iov_iter) { .iter_type = ITER_UBUF, .copy_mc = false, - .user_backed = true, .data_source = direction, .ubuf = buf, .count = count, diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 27234a820eeb..227c9f536b94 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -290,7 +290,6 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, .iter_type = ITER_IOVEC, .copy_mc = false, .nofault = false, - .user_backed = true, .data_source = direction, .__iov = iov, .nr_segs = nr_segs, From patchwork Wed Sep 13 16:56:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DDEEEE01EA for ; Wed, 13 Sep 2023 16:57:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B23006B01F6; Wed, 13 Sep 2023 12:57:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AD12D6B01F9; Wed, 13 Sep 2023 12:57:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92F056B01F8; Wed, 13 Sep 2023 12:57:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7B6BE6B01F6 for ; Wed, 13 Sep 2023 12:57:10 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 42A0A80E5E for ; Wed, 13 Sep 2023 16:57:10 +0000 (UTC) X-FDA: 81232179420.23.3CFABBF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 49C7E40006 for ; Wed, 13 Sep 2023 16:57:08 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eT82Z5w6; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624228; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D4sDntfB05lYx2oXLtvEA75x1n6eFZDOWAb9msr/5sA=; b=zYQ0fL0EypjV4KCKOQAP0EY1PETQK68ofturPpiTnHzhnzS2Zc2XdXaOpzj3y8iGoLyPpE 4exR4hF8mcNO91jH/q7MCfEtXrrivCwVJm2XSzzNEkALQvqyOX+81fLbn8UVFDxCAcLnIz Bb+8g3QCANWAZvIxvPX7V6F52o/2eiU= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eT82Z5w6; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf01.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624228; a=rsa-sha256; cv=none; b=ENJRQ0klZ71UhEfKnHflmJLjXPjLhEbamUT5s4HDYonJu3aiULtpK+tQwmPnMMfmkbT4UO zCLyYjLc9CGNQPteqi/Hi7+MDX0/7clb2u9722/YVUSvJTCItEXp61e7e7fAiH3wrN8yzW glgJFnpb4MRPCcir3/OcqW+EcXov1Ks= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624227; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D4sDntfB05lYx2oXLtvEA75x1n6eFZDOWAb9msr/5sA=; b=eT82Z5w6s0hvbRezjwcsEEACUMmUyuNFXtxvMlfA3O+89eIcKl7+iBmSd7O9FC8UngBFDU 8846+3+QZvgnMSw+PTN/tvX5IYnzF2VOh23RaD8SjTVfpB+H0SGDS9DDDWK7Bxp/U7BuzO rcAlNuP9al6FGvzBrPsZIhyF5YTw4XE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-611-ULwguQmgMDW4TifQeFpDzQ-1; Wed, 13 Sep 2023 12:57:04 -0400 X-MC-Unique: ULwguQmgMDW4TifQeFpDzQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1FD8A101B456; Wed, 13 Sep 2023 16:57:03 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 47CA92026D4B; Wed, 13 Sep 2023 16:57:01 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 04/13] iov_iter: Convert iterate*() to inline funcs Date: Wed, 13 Sep 2023 17:56:39 +0100 Message-ID: <20230913165648.2570623-5-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: 49C7E40006 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: z8nwapke95uaqp3nfzr3xgwwiya3wb5k X-HE-Tag: 1694624228-310881 X-HE-Meta: U2FsdGVkX1/PAmx+fcqA6/LYl1M9yh59jx/AzrxKGhvOn8wggpklrXiD9YEViHQ5AhiF5mlA7eo+6twF+oD2uS8mDfm8dUaVix29y2rse8ck3NGbw0XTg6hY63fNU5c0Msw1FMgAwUj55N866UG3aTTV5gYioRVmy8XgCY7Qd+AZrl5nYO6F2ZEJCQzm8lTO/yLyJZV693+AMysYNAXr78PGu9cR2u9abvNannFwaNmsUh4ThtQhV49AMHuLo5bdBpPVnQ7t9hWCfhK87NU/MEEtNTB3pWfod3xhQrWVBjG9bL8M36DhY+4JvrVtVXHBc1HwWPow8kSUD5cyfhIlXzLAHIUmbYua8+Ubt7lLxI9WOI6tBU0F97XjuUSIGfvaStF4HENgqsDJ0V38LhuFQjWzwSWC871NhPklLW+OVQqGb050agExPYIEBWLIXA3mNWkEMJzoNXurduZs5uoz/40TrXEd1q/s738vgBEStmdYoi+PeXKxzeP4XbuZMGZHmtEwT0Fy6CBhKgKvOj70CQ4c03qJ0VEbeySAluoejDMabdK2ZZI2DtcR4xWPxSlyM1AMLMmT/CvwTYmQQxh9VTgS+jX9U1JHu7h+AtlkgvParinz9Nph3hYQPjUrJQD+oe5MrdOJHU8reVEIvZiNK/65DlMlC1PiSpA+QHRmjipULM9cj96UrGwChtjaBoYRj8C/xyIbFBgL4+gtil0UYydZ0lgIXwc9r/sdRIwicrx0w8M6msPQywJselPIVDO+CEM9VNqXRIWfbXok5Qve7rw4svuvex6BbNnXWQLlmyGVj4SpRLui+xZeUeEkUgQj2KD1iAEAZmZ7Mv8LBo9fLrSMB4gse58zDQpoF2P8JulHbbbhP397NxjBVnKiHqSA1rSUHZkJJFzdBry/86uRGjKfNNOT8KH0mpNoyS2K9DnfUaIr+O3HlM9vAUq7T8Zcu+BousaomN8CWF3jcjh xFMbPSX5 hZQywyGclByVuyD0xfH8IdF4KvhBPokD5p6D6VzIkaAZFWlb42tshvnEEqMt3ZuEspw+S6hRxCyevivZXCFmyPFE7g8FRrcMOFLNSGL/FYBqrw0kSN2b5LfdvoNCgv7wQ9nEMhonqZ92yblduoL4N3TLnA87tYdk0g5KO4HpFEz9Dp5knywAheCf1raE2qJwq9hMDW//lRFEXNVriF+IZs7YKC0as01efCb8brWxW9s05FybKrvyZz5N7jG5sPBU9/gXvs5nnOcM6AJPuqT2GnnWgQJiIJ6uyohYowHxR2Q5oGfuyRsIWw3QzkS+uQ24QJXlC0ZF6SSL1mjxfJaNzsmWAwvcM4S/TrvzF2k7BnsSrZ0LAXze0MbDN/u1x6C+8R0NQcy1EleN9v9lwbl0kTQEbnmD/YYKbEyavkuE2mApRatbbo9/z80Msg+iF0Ln86GbgjfC2hAIfrZfKmHvJZmb44hNJ0vqMfNE4VhdVc1OjvGLer6V5xTDgKWp6RyM6Flt7QPeUHRlAdY66yS5kBfSr2RJG+ZDFlK8P1FqTIpSJbti6G5WPThCFFQRofptS0Ed/NssVlSjoPe6F9bCJ7jI/CBpjciGeh4G1W6sECGpgs9ngdjUaq7sybL9CZ2oO1XRj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert the iov_iter iteration macros to inline functions to make the code easier to follow. The functions are marked __always_inline as we don't want to end up with indirect calls in the code. This, however, leaves dealing with ->copy_mc in an awkard situation since the step function (memcpy_from_iter_mc()) needs to test the flag in the iterator, but isn't passed the iterator. This will be dealt with in a follow-up patch. The variable names in the per-type iterator functions have been harmonised as much as possible and made clearer as to the variable purpose. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org Link: https://lore.kernel.org/r/3710261.1691764329@warthog.procyon.org.uk/ # v1 Link: https://lore.kernel.org/r/855.1692047347@warthog.procyon.org.uk/ # v2 Link: https://lore.kernel.org/r/20230816120741.534415-1-dhowells@redhat.com/ # v3 --- Notes: Changes ======= ver #4) - Fix iterate_bvec() and iterate_kvec() to update iter->bvec and iter->kvec after subtracting it to calculate iter->nr_segs. - Change iterate_xarray() to use start+progress rather than increasing start to reduce code size. ver #3) - Use min_t(size_t,) not min() to avoid a warning on Hexagon. - Inline all the step functions. ver #2) - Rebased on top of Willy's changes in linux-next. - Change the checksum argument to the iteration functions to be a general void* and use it to pass iter->copy_mc flag to memcpy_from_iter_mc() to avoid using a function pointer. - Arrange the end of the iterate_*() functions to look the same to give the optimiser the best chance. - Make iterate_and_advance() a wrapper around iterate_and_advance2(). - Adjust iterate_and_advance2() to use if-else-if-else-if-else rather than switch(), to put ITER_BVEC before KVEC and to mark UBUF and IOVEC as likely(). - Move "iter->count += progress" into iterate_and_advance2() from the iterate functions. - Mark a number of the iterator helpers with __always_inline. - Fix _copy_from_iter_flushcache() to use memcpy_from_iter_flushcache() not memcpy_from_iter(). lib/iov_iter.c | 611 ++++++++++++++++++++++++++++++------------------- 1 file changed, 370 insertions(+), 241 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 227c9f536b94..b3ce6fa5f7a5 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -14,188 +14,263 @@ #include #include -/* covers ubuf and kbuf alike */ -#define iterate_buf(i, n, base, len, off, __p, STEP) { \ - size_t __maybe_unused off = 0; \ - len = n; \ - base = __p + i->iov_offset; \ - len -= (STEP); \ - i->iov_offset += len; \ - n = len; \ -} - -/* covers iovec and kvec alike */ -#define iterate_iovec(i, n, base, len, off, __p, STEP) { \ - size_t off = 0; \ - size_t skip = i->iov_offset; \ - do { \ - len = min(n, __p->iov_len - skip); \ - if (likely(len)) { \ - base = __p->iov_base + skip; \ - len -= (STEP); \ - off += len; \ - skip += len; \ - n -= len; \ - if (skip < __p->iov_len) \ - break; \ - } \ - __p++; \ - skip = 0; \ - } while (n); \ - i->iov_offset = skip; \ - n = off; \ -} - -#define iterate_bvec(i, n, base, len, off, p, STEP) { \ - size_t off = 0; \ - unsigned skip = i->iov_offset; \ - while (n) { \ - unsigned offset = p->bv_offset + skip; \ - unsigned left; \ - void *kaddr = kmap_local_page(p->bv_page + \ - offset / PAGE_SIZE); \ - base = kaddr + offset % PAGE_SIZE; \ - len = min(min(n, (size_t)(p->bv_len - skip)), \ - (size_t)(PAGE_SIZE - offset % PAGE_SIZE)); \ - left = (STEP); \ - kunmap_local(kaddr); \ - len -= left; \ - off += len; \ - skip += len; \ - if (skip == p->bv_len) { \ - skip = 0; \ - p++; \ - } \ - n -= len; \ - if (left) \ - break; \ - } \ - i->iov_offset = skip; \ - n = off; \ -} - -#define iterate_xarray(i, n, base, len, __off, STEP) { \ - __label__ __out; \ - size_t __off = 0; \ - struct folio *folio; \ - loff_t start = i->xarray_start + i->iov_offset; \ - pgoff_t index = start / PAGE_SIZE; \ - XA_STATE(xas, i->xarray, index); \ - \ - len = PAGE_SIZE - offset_in_page(start); \ - rcu_read_lock(); \ - xas_for_each(&xas, folio, ULONG_MAX) { \ - unsigned left; \ - size_t offset; \ - if (xas_retry(&xas, folio)) \ - continue; \ - if (WARN_ON(xa_is_value(folio))) \ - break; \ - if (WARN_ON(folio_test_hugetlb(folio))) \ - break; \ - offset = offset_in_folio(folio, start + __off); \ - while (offset < folio_size(folio)) { \ - base = kmap_local_folio(folio, offset); \ - len = min(n, len); \ - left = (STEP); \ - kunmap_local(base); \ - len -= left; \ - __off += len; \ - n -= len; \ - if (left || n == 0) \ - goto __out; \ - offset += len; \ - len = PAGE_SIZE; \ - } \ - } \ -__out: \ - rcu_read_unlock(); \ - i->iov_offset += __off; \ - n = __off; \ -} - -#define __iterate_and_advance(i, n, base, len, off, I, K) { \ - if (unlikely(i->count < n)) \ - n = i->count; \ - if (likely(n)) { \ - if (likely(iter_is_ubuf(i))) { \ - void __user *base; \ - size_t len; \ - iterate_buf(i, n, base, len, off, \ - i->ubuf, (I)) \ - } else if (likely(iter_is_iovec(i))) { \ - const struct iovec *iov = iter_iov(i); \ - void __user *base; \ - size_t len; \ - iterate_iovec(i, n, base, len, off, \ - iov, (I)) \ - i->nr_segs -= iov - iter_iov(i); \ - i->__iov = iov; \ - } else if (iov_iter_is_bvec(i)) { \ - const struct bio_vec *bvec = i->bvec; \ - void *base; \ - size_t len; \ - iterate_bvec(i, n, base, len, off, \ - bvec, (K)) \ - i->nr_segs -= bvec - i->bvec; \ - i->bvec = bvec; \ - } else if (iov_iter_is_kvec(i)) { \ - const struct kvec *kvec = i->kvec; \ - void *base; \ - size_t len; \ - iterate_iovec(i, n, base, len, off, \ - kvec, (K)) \ - i->nr_segs -= kvec - i->kvec; \ - i->kvec = kvec; \ - } else if (iov_iter_is_xarray(i)) { \ - void *base; \ - size_t len; \ - iterate_xarray(i, n, base, len, off, \ - (K)) \ - } \ - i->count -= n; \ - } \ -} -#define iterate_and_advance(i, n, base, len, off, I, K) \ - __iterate_and_advance(i, n, base, len, off, I, ((void)(K),0)) - -static int copyout(void __user *to, const void *from, size_t n) +typedef size_t (*iov_step_f)(void *iter_base, size_t progress, size_t len, + void *priv, void *priv2); +typedef size_t (*iov_ustep_f)(void __user *iter_base, size_t progress, size_t len, + void *priv, void *priv2); + +static __always_inline +size_t iterate_ubuf(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_ustep_f step) { - if (should_fail_usercopy()) - return n; - if (access_ok(to, n)) { - instrument_copy_to_user(to, from, n); - n = raw_copy_to_user(to, from, n); + void __user *base = iter->ubuf; + size_t progress = 0, remain; + + remain = step(base + iter->iov_offset, 0, len, priv, priv2); + progress = len - remain; + iter->iov_offset += progress; + return progress; +} + +static __always_inline +size_t iterate_iovec(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_ustep_f step) +{ + const struct iovec *p = iter->__iov; + size_t progress = 0, skip = iter->iov_offset; + + do { + size_t remain, consumed; + size_t part = min(len, p->iov_len - skip); + + if (likely(part)) { + remain = step(p->iov_base + skip, progress, part, priv, priv2); + consumed = part - remain; + progress += consumed; + skip += consumed; + len -= consumed; + if (skip < p->iov_len) + break; + } + p++; + skip = 0; + } while (len); + + iter->__iov = p; + iter->nr_segs -= p - iter->__iov; + iter->iov_offset = skip; + return progress; +} + +static __always_inline +size_t iterate_kvec(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_step_f step) +{ + const struct kvec *p = iter->kvec; + size_t progress = 0, skip = iter->iov_offset; + + do { + size_t remain, consumed; + size_t part = min(len, p->iov_len - skip); + + if (likely(part)) { + remain = step(p->iov_base + skip, progress, part, priv, priv2); + consumed = part - remain; + progress += consumed; + skip += consumed; + len -= consumed; + if (skip < p->iov_len) + break; + } + p++; + skip = 0; + } while (len); + + iter->nr_segs -= p - iter->kvec; + iter->kvec = p; + iter->iov_offset = skip; + return progress; +} + +static __always_inline +size_t iterate_bvec(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_step_f step) +{ + const struct bio_vec *p = iter->bvec; + size_t progress = 0, skip = iter->iov_offset; + + do { + size_t remain, consumed; + size_t offset = p->bv_offset + skip, part; + void *kaddr = kmap_local_page(p->bv_page + offset / PAGE_SIZE); + + part = min3(len, + (size_t)(p->bv_len - skip), + (size_t)(PAGE_SIZE - offset % PAGE_SIZE)); + remain = step(kaddr + offset % PAGE_SIZE, progress, part, priv, priv2); + kunmap_local(kaddr); + consumed = part - remain; + len -= consumed; + progress += consumed; + skip += consumed; + if (skip >= p->bv_len) { + skip = 0; + p++; + } + if (remain) + break; + } while (len); + + iter->nr_segs -= p - iter->bvec; + iter->bvec = p; + iter->iov_offset = skip; + return progress; +} + +static __always_inline +size_t iterate_xarray(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_step_f step) +{ + struct folio *folio; + size_t progress = 0; + loff_t start = iter->xarray_start + iter->iov_offset; + pgoff_t index = start / PAGE_SIZE; + XA_STATE(xas, iter->xarray, index); + + rcu_read_lock(); + xas_for_each(&xas, folio, ULONG_MAX) { + size_t remain, consumed, offset, part, flen; + + if (xas_retry(&xas, folio)) + continue; + if (WARN_ON(xa_is_value(folio))) + break; + if (WARN_ON(folio_test_hugetlb(folio))) + break; + + offset = offset_in_folio(folio, start + progress); + flen = min(folio_size(folio) - offset, len); + + while (flen) { + void *base = kmap_local_folio(folio, offset); + + part = min_t(size_t, flen, + PAGE_SIZE - offset_in_page(offset)); + remain = step(base, progress, part, priv, priv2); + kunmap_local(base); + + consumed = part - remain; + progress += consumed; + len -= consumed; + + if (remain || len == 0) + goto out; + flen -= consumed; + offset += consumed; + } } - return n; + +out: + rcu_read_unlock(); + iter->iov_offset += progress; + return progress; } -static int copyout_nofault(void __user *to, const void *from, size_t n) +static __always_inline +size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv, + void *priv2, iov_ustep_f ustep, iov_step_f step) { - long res; + size_t progress; + if (unlikely(iter->count < len)) + len = iter->count; + if (unlikely(!len)) + return 0; + + if (likely(iter_is_ubuf(iter))) + progress = iterate_ubuf(iter, len, priv, priv2, ustep); + else if (likely(iter_is_iovec(iter))) + progress = iterate_iovec(iter, len, priv, priv2, ustep); + else if (iov_iter_is_bvec(iter)) + progress = iterate_bvec(iter, len, priv, priv2, step); + else if (iov_iter_is_kvec(iter)) + progress = iterate_kvec(iter, len, priv, priv2, step); + else if (iov_iter_is_xarray(iter)) + progress = iterate_xarray(iter, len, priv, priv2, step); + else + progress = len; + iter->count -= progress; + return progress; +} + +static __always_inline +size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv, + iov_ustep_f ustep, iov_step_f step) +{ + return iterate_and_advance2(iter, len, priv, NULL, ustep, step); +} + +static __always_inline +size_t copy_to_user_iter(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ if (should_fail_usercopy()) - return n; + return len; + if (access_ok(iter_to, len)) { + from += progress; + instrument_copy_to_user(iter_to, from, len); + len = raw_copy_to_user(iter_to, from, len); + } + return len; +} - res = copy_to_user_nofault(to, from, n); +static __always_inline +size_t copy_to_user_iter_nofault(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + ssize_t res; - return res < 0 ? n : res; + if (should_fail_usercopy()) + return len; + + from += progress; + res = copy_to_user_nofault(iter_to, from, len); + return res < 0 ? len : res; } -static int copyin(void *to, const void __user *from, size_t n) +static __always_inline +size_t copy_from_user_iter(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) { - size_t res = n; + size_t res = len; if (should_fail_usercopy()) - return n; - if (access_ok(from, n)) { - instrument_copy_from_user_before(to, from, n); - res = raw_copy_from_user(to, from, n); - instrument_copy_from_user_after(to, from, n, res); + return len; + if (access_ok(iter_from, len)) { + to += progress; + instrument_copy_from_user_before(to, iter_from, len); + res = raw_copy_from_user(to, iter_from, len); + instrument_copy_from_user_after(to, iter_from, len, res); } return res; } +static __always_inline +size_t memcpy_to_iter(void *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + memcpy(iter_to, from + progress, len); + return 0; +} + +static __always_inline +size_t memcpy_from_iter(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + memcpy(to + progress, iter_from, len); + return 0; +} + /* * fault_in_iov_iter_readable - fault in iov iterator for reading * @i: iterator @@ -312,23 +387,29 @@ size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) return 0; if (user_backed_iter(i)) might_fault(); - iterate_and_advance(i, bytes, base, len, off, - copyout(base, addr + off, len), - memcpy(base, addr + off, len) - ) - - return bytes; + return iterate_and_advance(i, bytes, (void *)addr, + copy_to_user_iter, memcpy_to_iter); } EXPORT_SYMBOL(_copy_to_iter); #ifdef CONFIG_ARCH_HAS_COPY_MC -static int copyout_mc(void __user *to, const void *from, size_t n) -{ - if (access_ok(to, n)) { - instrument_copy_to_user(to, from, n); - n = copy_mc_to_user((__force void *) to, from, n); +static __always_inline +size_t copy_to_user_iter_mc(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + if (access_ok(iter_to, len)) { + from += progress; + instrument_copy_to_user(iter_to, from, len); + len = copy_mc_to_user(iter_to, from, len); } - return n; + return len; +} + +static __always_inline +size_t memcpy_to_iter_mc(void *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + return copy_mc_to_kernel(iter_to, from + progress, len); } /** @@ -361,22 +442,20 @@ size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) return 0; if (user_backed_iter(i)) might_fault(); - __iterate_and_advance(i, bytes, base, len, off, - copyout_mc(base, addr + off, len), - copy_mc_to_kernel(base, addr + off, len) - ) - - return bytes; + return iterate_and_advance(i, bytes, (void *)addr, + copy_to_user_iter_mc, memcpy_to_iter_mc); } EXPORT_SYMBOL_GPL(_copy_mc_to_iter); #endif /* CONFIG_ARCH_HAS_COPY_MC */ -static void *memcpy_from_iter(struct iov_iter *i, void *to, const void *from, - size_t size) +static size_t memcpy_from_iter_mc(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) { - if (iov_iter_is_copy_mc(i)) - return (void *)copy_mc_to_kernel(to, from, size); - return memcpy(to, from, size); + struct iov_iter *iter = priv2; + + if (iov_iter_is_copy_mc(iter)) + return copy_mc_to_kernel(to + progress, iter_from, len); + return memcpy_from_iter(iter_from, progress, len, to, priv2); } size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) @@ -386,30 +465,46 @@ size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) if (user_backed_iter(i)) might_fault(); - iterate_and_advance(i, bytes, base, len, off, - copyin(addr + off, base, len), - memcpy_from_iter(i, addr + off, base, len) - ) - - return bytes; + return iterate_and_advance2(i, bytes, addr, i, + copy_from_user_iter, + memcpy_from_iter_mc); } EXPORT_SYMBOL(_copy_from_iter); +static __always_inline +size_t copy_from_user_iter_nocache(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + return __copy_from_user_inatomic_nocache(to + progress, iter_from, len); +} + size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) { if (WARN_ON_ONCE(!i->data_source)) return 0; - iterate_and_advance(i, bytes, base, len, off, - __copy_from_user_inatomic_nocache(addr + off, base, len), - memcpy(addr + off, base, len) - ) - - return bytes; + return iterate_and_advance(i, bytes, addr, + copy_from_user_iter_nocache, + memcpy_from_iter); } EXPORT_SYMBOL(_copy_from_iter_nocache); #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE +static __always_inline +size_t copy_from_user_iter_flushcache(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + return __copy_from_user_flushcache(to + progress, iter_from, len); +} + +static __always_inline +size_t memcpy_from_iter_flushcache(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + memcpy_flushcache(to + progress, iter_from, len); + return 0; +} + /** * _copy_from_iter_flushcache - write destination through cpu cache * @addr: destination kernel address @@ -431,12 +526,9 @@ size_t _copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) if (WARN_ON_ONCE(!i->data_source)) return 0; - iterate_and_advance(i, bytes, base, len, off, - __copy_from_user_flushcache(addr + off, base, len), - memcpy_flushcache(addr + off, base, len) - ) - - return bytes; + return iterate_and_advance(i, bytes, addr, + copy_from_user_iter_flushcache, + memcpy_from_iter_flushcache); } EXPORT_SYMBOL_GPL(_copy_from_iter_flushcache); #endif @@ -508,10 +600,9 @@ size_t copy_page_to_iter_nofault(struct page *page, unsigned offset, size_t byte void *kaddr = kmap_local_page(page); size_t n = min(bytes, (size_t)PAGE_SIZE - offset); - iterate_and_advance(i, n, base, len, off, - copyout_nofault(base, kaddr + offset + off, len), - memcpy(base, kaddr + offset + off, len) - ) + n = iterate_and_advance(i, bytes, kaddr, + copy_to_user_iter_nofault, + memcpy_to_iter); kunmap_local(kaddr); res += n; bytes -= n; @@ -554,14 +645,25 @@ size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, } EXPORT_SYMBOL(copy_page_from_iter); -size_t iov_iter_zero(size_t bytes, struct iov_iter *i) +static __always_inline +size_t zero_to_user_iter(void __user *iter_to, size_t progress, + size_t len, void *priv, void *priv2) { - iterate_and_advance(i, bytes, base, len, count, - clear_user(base, len), - memset(base, 0, len) - ) + return clear_user(iter_to, len); +} - return bytes; +static __always_inline +size_t zero_to_iter(void *iter_to, size_t progress, + size_t len, void *priv, void *priv2) +{ + memset(iter_to, 0, len); + return 0; +} + +size_t iov_iter_zero(size_t bytes, struct iov_iter *i) +{ + return iterate_and_advance(i, bytes, NULL, + zero_to_user_iter, zero_to_iter); } EXPORT_SYMBOL(iov_iter_zero); @@ -586,10 +688,9 @@ size_t copy_page_from_iter_atomic(struct page *page, size_t offset, } p = kmap_atomic(page) + offset; - iterate_and_advance(i, n, base, len, off, - copyin(p + off, base, len), - memcpy_from_iter(i, p + off, base, len) - ) + n = iterate_and_advance2(i, n, p, i, + copy_from_user_iter, + memcpy_from_iter_mc); kunmap_atomic(p); copied += n; offset += n; @@ -1180,32 +1281,64 @@ ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, } EXPORT_SYMBOL(iov_iter_get_pages_alloc2); +static __always_inline +size_t copy_from_user_iter_csum(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + __wsum next, *csum = priv2; + + next = csum_and_copy_from_user(iter_from, to + progress, len); + *csum = csum_block_add(*csum, next, progress); + return next ? 0 : len; +} + +static __always_inline +size_t memcpy_from_iter_csum(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + __wsum *csum = priv2; + + *csum = csum_and_memcpy(to + progress, iter_from, len, *csum, progress); + return 0; +} + size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) { - __wsum sum, next; - sum = *csum; if (WARN_ON_ONCE(!i->data_source)) return 0; - - iterate_and_advance(i, bytes, base, len, off, ({ - next = csum_and_copy_from_user(base, addr + off, len); - sum = csum_block_add(sum, next, off); - next ? 0 : len; - }), ({ - sum = csum_and_memcpy(addr + off, base, len, sum, off); - }) - ) - *csum = sum; - return bytes; + return iterate_and_advance2(i, bytes, addr, csum, + copy_from_user_iter_csum, + memcpy_from_iter_csum); } EXPORT_SYMBOL(csum_and_copy_from_iter); +static __always_inline +size_t copy_to_user_iter_csum(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + __wsum next, *csum = priv2; + + next = csum_and_copy_to_user(from + progress, iter_to, len); + *csum = csum_block_add(*csum, next, progress); + return next ? 0 : len; +} + +static __always_inline +size_t memcpy_to_iter_csum(void *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + __wsum *csum = priv2; + + *csum = csum_and_memcpy(iter_to, from + progress, len, *csum, progress); + return 0; +} + size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, struct iov_iter *i) { struct csum_state *csstate = _csstate; - __wsum sum, next; + __wsum sum; if (WARN_ON_ONCE(i->data_source)) return 0; @@ -1219,14 +1352,10 @@ size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, } sum = csum_shift(csstate->csum, csstate->off); - iterate_and_advance(i, bytes, base, len, off, ({ - next = csum_and_copy_to_user(addr + off, base, len); - sum = csum_block_add(sum, next, off); - next ? 0 : len; - }), ({ - sum = csum_and_memcpy(base, addr + off, len, sum, off); - }) - ) + + bytes = iterate_and_advance2(i, bytes, (void *)addr, &sum, + copy_to_user_iter_csum, + memcpy_to_iter_csum); csstate->csum = csum_shift(sum, csstate->off); csstate->off += bytes; return bytes; From patchwork Wed Sep 13 16:56:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383570 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65EEAEE01EC for ; Wed, 13 Sep 2023 16:57:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2227D6B01FA; Wed, 13 Sep 2023 12:57:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 136E06B01FB; Wed, 13 Sep 2023 12:57:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA66F6B01FC; Wed, 13 Sep 2023 12:57:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D4F616B01FA for ; Wed, 13 Sep 2023 12:57:13 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 944CB407A7 for ; Wed, 13 Sep 2023 16:57:13 +0000 (UTC) X-FDA: 81232179546.16.D1008E2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id ADBDAC0024 for ; Wed, 13 Sep 2023 16:57:11 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fc0dX0nl; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624231; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7f30E5rfqXOL8OIlALoKZyRz07q/M/xUz+BJRcAuap4=; b=gkb/PnQH/gLa//wc7fZBPfRm6A4jFh/1W+Uu5UCNk0S1FItR4E37YoUW4f9hsKR7PEwwa0 dShdAvOU1IzUtxSyDh7J3XbSlXWQcbIx/eRyQYwfXE+7rn6YLbXSnE3Xw0OH5hewh9aull UbXQi79Ax0DD3/1nZSWbyEBAfXNohXo= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fc0dX0nl; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624231; a=rsa-sha256; cv=none; b=tEPxRH5RwiRufX4ObP35SXxuIROXUfo0yAWgS5cwyUC+2oZVtwtt9zQ3ZmynSwBnREdwHU aYG4KZyldsgOAh5rQeFernRLvh2q/ZdsLMeFu9wfZmq1KG1OLyp//QZy8LkLnTxGXM5o8B 1LKyttlPHfq1vcNHN8Y1PLjROFIwSaQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624231; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7f30E5rfqXOL8OIlALoKZyRz07q/M/xUz+BJRcAuap4=; b=fc0dX0nl53IPYBwdjLtvkp6eXV8qPooF3WpwIvglLhUvF8qIUH3FIh2eO7c5xgA0R2ToCe 09phGjuZFNreshpusisz5Y/iNq5xX0XZ2SCQD5DTzbZ7MHUwjp4ZNXWD8jsNtB+hDg3SfT PEqHVQEjo2kJ0OMxDJh5etgFYi8BQvk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-510-cMka_6aIPNiC8mOgNkW5Mw-1; Wed, 13 Sep 2023 12:57:06 -0400 X-MC-Unique: cMka_6aIPNiC8mOgNkW5Mw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9603B185A797; Wed, 13 Sep 2023 16:57:05 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id D5E5F40C6EBF; Wed, 13 Sep 2023 16:57:03 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 05/13] iov: Move iterator functions to a header file Date: Wed, 13 Sep 2023 17:56:40 +0100 Message-ID: <20230913165648.2570623-6-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: ADBDAC0024 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: rwry1djyhp5trw9d4ads63z5te6u14pb X-HE-Tag: 1694624231-49033 X-HE-Meta: U2FsdGVkX19mXVyKE4S6hsgx+uuQlkYtLIEe/488OT+/jhaxqQjSEzxTEHCSEcxg3PboMwtU/F8TKOLuPRvELvJTYHMgJ8Jm/twC50nR8DBh7G3BO0MjhvEZ4izNU8kakbdajLTVvyJ7/wrSbZYRtVuK2RNnDrbh8+/zG+Crpxti3cBJELhSIz7LdJC2BAx/oefmbyRLAKS2i01Tf4xmxJBaWHhu5cj+wL1g8OvoNDBxlPaJ4gpe2x3U5qJiS7Gh7yh1DC1hkd8PZWtWdTInJ3rqbxEaGBaDFBYANR/Z+uSVXO7lP+3Y26VN3M68biayK+iry2+w3b84ZTQWcWZOiYW2Js3ZhDyHdmOwN6LLI3n1dK5iEjVj+3lbxb+PplmCUlSY5Hkpfd5RkeOKA1hYsgkf6J+dmfZf6nHUjUYT+GD3spVxUfVDa8fkPeaVfFqv9lqwDYa7U4czTcjsCs0buIXY7x+JP/M8MqI8VABhjx1+KRvvN/CSSzrvKG+guZ0EExYFSGKky3c10r1yY9be/hbQbkSAGfdQ4+YH2paNpX9xcawWzo4TZlH8s5pKqTy387miwItRJIzqUYtkkrbyIf6EIMZmlwmOQwO5wh2YdexkbIkHS3AxXLjh0Ay2oCXJegPNzYeYd5xYj9cBYmICOPLd8ClPAULzbJvS1GnAFur+VJFxtszDV1XDaCIv2fhoxbx1OpuQ4nXAAfbizFa0uqZw/gOiU6q3fMSbiRbuI1yrCSfDUuoNbh3wcY3WH8srQgBRmUrtS+7tau3+YfHqppOcmdyRZGtBeB7UDXbVk8OM90rjkxZ/oQ7ycNZx+l4eIlxip4t8NTzg3bPY9Lj2z1BMX6gQj8Ud1pv0UUy7i9LrCZxMnQ1mDpjeJhjQthQ46yc4FNEtAPEXv5TBHlddhaFGNRXdC0rcnENxdgmaAF4ZQd3mPX4iyL0BffQB+ABooiBs88BSRGAYuhPNzmE hKqDt78O BHYQChOTqv8XbbyeAv7HqKQf2d6c+GXrx2ixmwPr0y4QsxyebmATL2VL1C7+rX1dRdTJI5jJb6xs9UfmUGAk/I97DEVP9KVzNXQwFIxjOdEaIrY0G0OGStrk0oqK3cQ8V6GdQ+57wykWjcf8jYQ7drqnXPHKvLxGegCTDOav0wU3gh+qKrjOgJokfuo5yQT8WviEliIPNxPpAR6pDnOqDAqN2vHzOO+9nCv8S2br9S61p6uRVpxk46N7ej8O6f+BQ/QQ3Vs+TwbbdGzJ19P7IIs1wLUxnwVOj/sPUTFiOVpw/OLPkzIsK0lrexd2L2iJ9QkV7QMemjZu+LMc3ihwQceOwBHFUGALBPEK7s4OWjkfSSaTzI1DutQDUEHHjlKYYoR6r3asdk6uUbyzHCKy1IEsBNi2MRugas1NIva9anqGjpYziWp6RwQa6eT9uaqtFXiC2Dz1EIA7Fc5OKcWdigMcuENukVQMpiCNcjG9uD57nY6woHVl/x70iqaottvCBww7Med9skQ09KVny6E605Jydx1blA9WfE4Vtn8UDRgPMjoeqWvP1t7ADJA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move the iterator functions to a header file so that other operations that need to scan over an iterator can be added. For instance, the rbd driver could use this to scan a buffer to see if it is all zeros and libceph could use this to generate a crc. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- include/linux/iov_iter.h | 261 +++++++++++++++++++++++++++++++++++++++ lib/iov_iter.c | 197 +---------------------------- 2 files changed, 262 insertions(+), 196 deletions(-) create mode 100644 include/linux/iov_iter.h diff --git a/include/linux/iov_iter.h b/include/linux/iov_iter.h new file mode 100644 index 000000000000..836854847cdf --- /dev/null +++ b/include/linux/iov_iter.h @@ -0,0 +1,261 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* I/O iterator iteration building functions. + * + * Copyright (C) 2023 Red Hat, Inc. All Rights Reserved. + * Written by David Howells (dhowells@redhat.com) + */ + +#ifndef _LINUX_IOV_ITER_H +#define _LINUX_IOV_ITER_H + +#include +#include + +typedef size_t (*iov_step_f)(void *iter_base, size_t progress, size_t len, + void *priv, void *priv2); +typedef size_t (*iov_ustep_f)(void __user *iter_base, size_t progress, size_t len, + void *priv, void *priv2); + +/* + * Handle ITER_UBUF. + */ +static __always_inline +size_t iterate_ubuf(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_ustep_f step) +{ + void __user *base = iter->ubuf; + size_t progress = 0, remain; + + remain = step(base + iter->iov_offset, 0, len, priv, priv2); + progress = len - remain; + iter->iov_offset += progress; + return progress; +} + +/* + * Handle ITER_IOVEC. + */ +static __always_inline +size_t iterate_iovec(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_ustep_f step) +{ + const struct iovec *p = iter->__iov; + size_t progress = 0, skip = iter->iov_offset; + + do { + size_t remain, consumed; + size_t part = min(len, p->iov_len - skip); + + if (likely(part)) { + remain = step(p->iov_base + skip, progress, part, priv, priv2); + consumed = part - remain; + progress += consumed; + skip += consumed; + len -= consumed; + if (skip < p->iov_len) + break; + } + p++; + skip = 0; + } while (len); + + iter->__iov = p; + iter->nr_segs -= p - iter->__iov; + iter->iov_offset = skip; + return progress; +} + +/* + * Handle ITER_KVEC. + */ +static __always_inline +size_t iterate_kvec(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_step_f step) +{ + const struct kvec *p = iter->kvec; + size_t progress = 0, skip = iter->iov_offset; + + do { + size_t remain, consumed; + size_t part = min(len, p->iov_len - skip); + + if (likely(part)) { + remain = step(p->iov_base + skip, progress, part, priv, priv2); + consumed = part - remain; + progress += consumed; + skip += consumed; + len -= consumed; + if (skip < p->iov_len) + break; + } + p++; + skip = 0; + } while (len); + + iter->nr_segs -= p - iter->kvec; + iter->kvec = p; + iter->iov_offset = skip; + return progress; +} + +/* + * Handle ITER_BVEC. + */ +static __always_inline +size_t iterate_bvec(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_step_f step) +{ + const struct bio_vec *p = iter->bvec; + size_t progress = 0, skip = iter->iov_offset; + + do { + size_t remain, consumed; + size_t offset = p->bv_offset + skip, part; + void *kaddr = kmap_local_page(p->bv_page + offset / PAGE_SIZE); + + part = min3(len, + (size_t)(p->bv_len - skip), + (size_t)(PAGE_SIZE - offset % PAGE_SIZE)); + remain = step(kaddr + offset % PAGE_SIZE, progress, part, priv, priv2); + kunmap_local(kaddr); + consumed = part - remain; + len -= consumed; + progress += consumed; + skip += consumed; + if (skip >= p->bv_len) { + skip = 0; + p++; + } + if (remain) + break; + } while (len); + + iter->nr_segs -= p - iter->bvec; + iter->bvec = p; + iter->iov_offset = skip; + return progress; +} + +/* + * Handle ITER_XARRAY. + */ +static __always_inline +size_t iterate_xarray(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_step_f step) +{ + struct folio *folio; + size_t progress = 0; + loff_t start = iter->xarray_start + iter->iov_offset; + pgoff_t index = start / PAGE_SIZE; + XA_STATE(xas, iter->xarray, index); + + rcu_read_lock(); + xas_for_each(&xas, folio, ULONG_MAX) { + size_t remain, consumed, offset, part, flen; + + if (xas_retry(&xas, folio)) + continue; + if (WARN_ON(xa_is_value(folio))) + break; + if (WARN_ON(folio_test_hugetlb(folio))) + break; + + offset = offset_in_folio(folio, start + progress); + flen = min(folio_size(folio) - offset, len); + + while (flen) { + void *base = kmap_local_folio(folio, offset); + + part = min_t(size_t, flen, + PAGE_SIZE - offset_in_page(offset)); + remain = step(base, progress, part, priv, priv2); + kunmap_local(base); + + consumed = part - remain; + progress += consumed; + len -= consumed; + + if (remain || len == 0) + goto out; + flen -= consumed; + offset += consumed; + } + } + +out: + rcu_read_unlock(); + iter->iov_offset += progress; + return progress; +} + +/** + * iterate_and_advance2 - Iterate over an iterator + * @iter: The iterator to iterate over. + * @len: The amount to iterate over. + * @priv: Data for the step functions. + * @priv2: More data for the step functions. + * @ustep: Function for UBUF/IOVEC iterators; given __user addresses. + * @step: Function for other iterators; given kernel addresses. + * + * Iterate over the next part of an iterator, up to the specified length. The + * buffer is presented in segments, which for kernel iteration are broken up by + * physical pages and mapped, with the mapped address being presented. + * + * Two step functions, @step and @ustep, must be provided, one for handling + * mapped kernel addresses and the other is given user addresses which have the + * potential to fault since no pinning is performed. + * + * The step functions are passed the address and length of the segment, @priv, + * @priv2 and the amount of data so far iterated over (which can, for example, + * be added to @priv to point to the right part of a second buffer). The step + * functions should return the amount of the segment they didn't process (ie. 0 + * indicates complete processsing). + * + * This function returns the amount of data processed (ie. 0 means nothing was + * processed and the value of @len means processes to completion). + */ +static __always_inline +size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv, + void *priv2, iov_ustep_f ustep, iov_step_f step) +{ + size_t progress; + + if (unlikely(iter->count < len)) + len = iter->count; + if (unlikely(!len)) + return 0; + + if (likely(iter_is_ubuf(iter))) + progress = iterate_ubuf(iter, len, priv, priv2, ustep); + else if (likely(iter_is_iovec(iter))) + progress = iterate_iovec(iter, len, priv, priv2, ustep); + else if (iov_iter_is_bvec(iter)) + progress = iterate_bvec(iter, len, priv, priv2, step); + else if (iov_iter_is_kvec(iter)) + progress = iterate_kvec(iter, len, priv, priv2, step); + else if (iov_iter_is_xarray(iter)) + progress = iterate_xarray(iter, len, priv, priv2, step); + else + progress = len; + iter->count -= progress; + return progress; +} + +/** + * iterate_and_advance - Iterate over an iterator + * @iter: The iterator to iterate over. + * @len: The amount to iterate over. + * @priv: Data for the step functions. + * @ustep: Function for UBUF/IOVEC iterators; given __user addresses. + * @step: Function for other iterators; given kernel addresses. + * + * As iterate_and_advance2(), but priv2 is always NULL. + */ +static __always_inline +size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv, + iov_ustep_f ustep, iov_step_f step) +{ + return iterate_and_advance2(iter, len, priv, NULL, ustep, step); +} + +#endif /* _LINUX_IOV_ITER_H */ diff --git a/lib/iov_iter.c b/lib/iov_iter.c index b3ce6fa5f7a5..65374ee91ecd 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -13,202 +13,7 @@ #include #include #include - -typedef size_t (*iov_step_f)(void *iter_base, size_t progress, size_t len, - void *priv, void *priv2); -typedef size_t (*iov_ustep_f)(void __user *iter_base, size_t progress, size_t len, - void *priv, void *priv2); - -static __always_inline -size_t iterate_ubuf(struct iov_iter *iter, size_t len, void *priv, void *priv2, - iov_ustep_f step) -{ - void __user *base = iter->ubuf; - size_t progress = 0, remain; - - remain = step(base + iter->iov_offset, 0, len, priv, priv2); - progress = len - remain; - iter->iov_offset += progress; - return progress; -} - -static __always_inline -size_t iterate_iovec(struct iov_iter *iter, size_t len, void *priv, void *priv2, - iov_ustep_f step) -{ - const struct iovec *p = iter->__iov; - size_t progress = 0, skip = iter->iov_offset; - - do { - size_t remain, consumed; - size_t part = min(len, p->iov_len - skip); - - if (likely(part)) { - remain = step(p->iov_base + skip, progress, part, priv, priv2); - consumed = part - remain; - progress += consumed; - skip += consumed; - len -= consumed; - if (skip < p->iov_len) - break; - } - p++; - skip = 0; - } while (len); - - iter->__iov = p; - iter->nr_segs -= p - iter->__iov; - iter->iov_offset = skip; - return progress; -} - -static __always_inline -size_t iterate_kvec(struct iov_iter *iter, size_t len, void *priv, void *priv2, - iov_step_f step) -{ - const struct kvec *p = iter->kvec; - size_t progress = 0, skip = iter->iov_offset; - - do { - size_t remain, consumed; - size_t part = min(len, p->iov_len - skip); - - if (likely(part)) { - remain = step(p->iov_base + skip, progress, part, priv, priv2); - consumed = part - remain; - progress += consumed; - skip += consumed; - len -= consumed; - if (skip < p->iov_len) - break; - } - p++; - skip = 0; - } while (len); - - iter->nr_segs -= p - iter->kvec; - iter->kvec = p; - iter->iov_offset = skip; - return progress; -} - -static __always_inline -size_t iterate_bvec(struct iov_iter *iter, size_t len, void *priv, void *priv2, - iov_step_f step) -{ - const struct bio_vec *p = iter->bvec; - size_t progress = 0, skip = iter->iov_offset; - - do { - size_t remain, consumed; - size_t offset = p->bv_offset + skip, part; - void *kaddr = kmap_local_page(p->bv_page + offset / PAGE_SIZE); - - part = min3(len, - (size_t)(p->bv_len - skip), - (size_t)(PAGE_SIZE - offset % PAGE_SIZE)); - remain = step(kaddr + offset % PAGE_SIZE, progress, part, priv, priv2); - kunmap_local(kaddr); - consumed = part - remain; - len -= consumed; - progress += consumed; - skip += consumed; - if (skip >= p->bv_len) { - skip = 0; - p++; - } - if (remain) - break; - } while (len); - - iter->nr_segs -= p - iter->bvec; - iter->bvec = p; - iter->iov_offset = skip; - return progress; -} - -static __always_inline -size_t iterate_xarray(struct iov_iter *iter, size_t len, void *priv, void *priv2, - iov_step_f step) -{ - struct folio *folio; - size_t progress = 0; - loff_t start = iter->xarray_start + iter->iov_offset; - pgoff_t index = start / PAGE_SIZE; - XA_STATE(xas, iter->xarray, index); - - rcu_read_lock(); - xas_for_each(&xas, folio, ULONG_MAX) { - size_t remain, consumed, offset, part, flen; - - if (xas_retry(&xas, folio)) - continue; - if (WARN_ON(xa_is_value(folio))) - break; - if (WARN_ON(folio_test_hugetlb(folio))) - break; - - offset = offset_in_folio(folio, start + progress); - flen = min(folio_size(folio) - offset, len); - - while (flen) { - void *base = kmap_local_folio(folio, offset); - - part = min_t(size_t, flen, - PAGE_SIZE - offset_in_page(offset)); - remain = step(base, progress, part, priv, priv2); - kunmap_local(base); - - consumed = part - remain; - progress += consumed; - len -= consumed; - - if (remain || len == 0) - goto out; - flen -= consumed; - offset += consumed; - } - } - -out: - rcu_read_unlock(); - iter->iov_offset += progress; - return progress; -} - -static __always_inline -size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv, - void *priv2, iov_ustep_f ustep, iov_step_f step) -{ - size_t progress; - - if (unlikely(iter->count < len)) - len = iter->count; - if (unlikely(!len)) - return 0; - - if (likely(iter_is_ubuf(iter))) - progress = iterate_ubuf(iter, len, priv, priv2, ustep); - else if (likely(iter_is_iovec(iter))) - progress = iterate_iovec(iter, len, priv, priv2, ustep); - else if (iov_iter_is_bvec(iter)) - progress = iterate_bvec(iter, len, priv, priv2, step); - else if (iov_iter_is_kvec(iter)) - progress = iterate_kvec(iter, len, priv, priv2, step); - else if (iov_iter_is_xarray(iter)) - progress = iterate_xarray(iter, len, priv, priv2, step); - else - progress = len; - iter->count -= progress; - return progress; -} - -static __always_inline -size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv, - iov_ustep_f ustep, iov_step_f step) -{ - return iterate_and_advance2(iter, len, priv, NULL, ustep, step); -} +#include static __always_inline size_t copy_to_user_iter(void __user *iter_to, size_t progress, From patchwork Wed Sep 13 16:56:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFBC9EE01E9 for ; Wed, 13 Sep 2023 16:57:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D1D246B01F8; Wed, 13 Sep 2023 12:57:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCE3C6B01F9; Wed, 13 Sep 2023 12:57:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B48426B01FA; Wed, 13 Sep 2023 12:57:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9B9846B01F8 for ; Wed, 13 Sep 2023 12:57:12 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7611BB39B0 for ; Wed, 13 Sep 2023 16:57:12 +0000 (UTC) X-FDA: 81232179504.28.110FACB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id B80C0140017 for ; Wed, 13 Sep 2023 16:57:10 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IyPDjQjg; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624230; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rhHz2rkHFFU/pzH+hQA/LimHSXRi7J1g2T3941mk2Gc=; b=deS0yULua5WwnfFEUKgZBLZ0IECUZUxiYqLH9HMVkDZILNtCxXb2cinV2rkIpAVyNZE0hk o3DLzIIg0GB7lddbjXki4FJQURfBZUEXFB828fY5nIX5vDtmdOCPSo3xxtM2OkrbaBQxUm K+hPfW4fmqMkW4LPIvBLWCjPohBSU8E= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IyPDjQjg; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624230; a=rsa-sha256; cv=none; b=VtiXVz908A47wV3oC43+UliLaVbFohf9AXeNYkeHjvtiC0+zHoiQAYDIEvCRx9NC+T+KWI i8/+Op3sHy363e0dQthlZYyA9R0Suf7t6n7PWOJdRmGbzha0cBnsfI8eoFqDace/dpHRJH sLf6lI8Hs4nmkUQLqfTpsIBiv5y3Xks= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624230; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rhHz2rkHFFU/pzH+hQA/LimHSXRi7J1g2T3941mk2Gc=; b=IyPDjQjgaw2ztZeGQCRT9mlirGb+KBEKluH4X6WRwQwQYCRClFHFqquMpVKqIsnv3ykIDw 28byJwR8j5j6Awf3bclKbdl2kmqQFD9EjKughsmvQtg6rnU8vKB3ne3wUOjcLplZ1BRnpT Ey/Cihw9KAwxNmvJIAkMQMlcD2vnO3E= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-670-2tcJ0K_0Mga54Zz4FQYBNQ-1; Wed, 13 Sep 2023 12:57:08 -0400 X-MC-Unique: 2tcJ0K_0Mga54Zz4FQYBNQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D5F09101A529; Wed, 13 Sep 2023 16:57:07 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3B4AF1054FC1; Wed, 13 Sep 2023 16:57:06 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 06/13] iov_iter: Add a kernel-type iterator-only iteration function Date: Wed, 13 Sep 2023 17:56:41 +0100 Message-ID: <20230913165648.2570623-7-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: B80C0140017 X-Stat-Signature: u9ytsc6f8bbpaem9gs8dsjemikhpec58 X-HE-Tag: 1694624230-646471 X-HE-Meta: U2FsdGVkX1+poH7evL1hw24zGFXMdXAlSfkWdc+of7mY1hIA1t8nOXa5/18ZoUpZR6v/OiBY/u/DZ3LryAjj3v79nWA0cBctzpoHhY5e2mhd7SoTZPmuBmMw1dE5jBVZLUas7AAKie9W7v+7bpunyz0BjCVp1ul6afFdxFhRTFWJrtHMx7s9UX9G6xJmNqUkGIPEnqbpJJIFwL709Si/vEpi6TNydkJleg3K/uxKnDQM4k3JnImC41GtmKl+53M3A1u9tA/+l/t7QVM7YruXXNHNX43Xpj9+Z+8KQvvA7vLczcstGy3O1BbhYgKKAzJYBMs0Rr83yPE372mKlKUCFmzEe23vpTE5QyoGOD+I45uoGHnYpkgarBvNZYwWjaAfkex7dR7LpIM7qZLyVW7VuF3gL+N2wShOgWCjvGZ4cq2jRDp38zcWHjnd3dL/NcevjFWRU2MTdgU/qJRGzODGFdvZue/DTm7juTEZUF6LZE2isN7Te7MYnFCd4coFYl1VdeInSkNEruWWLTTPLREFJAJkqJW0hQtd0I8FCGX8VkokOkFKLNsDjRwbszWEX0deiJX6FnvrKFqd9P5zqibT93EHfCU/UElK/BMWM/UlrbUU6JHxaQaqWyxQ/dwyY/D/WWh5OC/GK0rg2jKefSEsmsSoninkqr9JaRqU5JuLzYhfhNgHlhtbbMuvD7oQjIUbhGq8yOHwdhmfj6eAm+1yIGv+h6t4log4JEaeIcEycsU2IhC2zBszSgTom3SKc+aCKopC+TLJY9zTIVZAYEwyMXc4NefHRH4KmWGD934bnqllWH92gY5ih17VidXMy5hyIobSsjoIF9Br8m0DrAsWESkQdM4bPkiDwu7gcFGskPYzyoNZOtLJn21v93u5r7korqamuuvleZHNSL+59XQo7/8QANhvqe/7EiOcPdYis58aUkDacxJ3Z/h0tBX8PbhzTbka8PDrZjMAUCplVtD S545Pg/1 Npmkw3vyYGwlAmtLd+xs4Xf7pSrqmG97FJRYruJfCv9zlphZ/znvnnQV2yvJn3ifpcWDB1c+sOCFdWjkcXG0y9FyZw5pp8vNMixeKLHkZ+Ih75xEPcpyjFiSMsudp8jF+TIv/jWYVeLB5szgDK/wDoSnJ9CVal7Cx7p67jCkimjB20wgQx6y4gI4dTVRCAC+ZYw8UCYiQJ/+xr1Kt/ELCn6EaJKKI8/ofRGEds/JWEJF0Xfgx3svNWab7ere/NGL2BXRaf7s/h3vKAq/wPAW08FKpD36bovBLoP50DkhU0xuLcRnl+KR1qta1tJNUedv3XbQc/DnZ3oDU3/SHKVcSR4Jua8hRQA+PyoeitBZoTZJVHk/nRBsqNAWbBHomLR/tS6sXbOTi5VFk+xwPl7GWjgIqhWpDN9z9tKPC84M+ill6s7A1R6HvyhSooSe1tYn9p+fItBqi1l8d3ureoULwSDA5TI90zAzZaNxBXuM2qu7zOwJKFPcy1AHgh+wLLYAaRPxoSRmo9PE8cZ2lTIsU8XHyoM96dZ1/SdN7Gu1yydYOACoBUCWUPS0f4YNCSEo6SDQ459TAKQKSX00= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add an iteration function that can only iterate over kernel internal-type iterators (ie. BVEC, KVEC, XARRAY) and not user-backed iterators (ie. UBUF and IOVEC). This allows for smaller iterators to be built when it is known the caller won't have a user-backed iterator. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- include/linux/iov_iter.h | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/include/linux/iov_iter.h b/include/linux/iov_iter.h index 836854847cdf..46e628640934 100644 --- a/include/linux/iov_iter.h +++ b/include/linux/iov_iter.h @@ -258,4 +258,39 @@ size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv, return iterate_and_advance2(iter, len, priv, NULL, ustep, step); } +/** + * iterate_and_advance_kernel - Iterate over a kernel iterator + * @iter: The iterator to iterate over. + * @len: The amount to iterate over. + * @priv: Data for the step functions. + * @step: Processing function; given kernel addresses. + * + * Like iterate_and_advance2(), but rejected UBUF and IOVEC iterators and does + * not take a user-step function. + */ +static __always_inline +size_t iterate_and_advance_kernel(struct iov_iter *iter, size_t len, void *priv, + void *priv2, iov_step_f step) +{ + size_t progress; + + if (unlikely(iter->count < len)) + len = iter->count; + if (unlikely(!len)) + return 0; + + if (iov_iter_is_bvec(iter)) + progress = iterate_bvec(iter, len, priv, priv2, step); + else if (iov_iter_is_kvec(iter)) + progress = iterate_kvec(iter, len, priv, priv2, step); + else if (iov_iter_is_xarray(iter)) + progress = iterate_xarray(iter, len, priv, priv2, step); + else if (WARN_ON_ONCE(unlikely(iter_is_ubuf(iter) || iter_is_iovec(iter)))) + progress = 0; + else + progress = len; + iter->count -= progress; + return progress; +} + #endif /* _LINUX_IOV_ITER_H */ From patchwork Wed Sep 13 16:56:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8035DEE01E9 for ; Wed, 13 Sep 2023 16:57:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D4136B01FE; Wed, 13 Sep 2023 12:57:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1832D6B01FF; Wed, 13 Sep 2023 12:57:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 024336B0200; Wed, 13 Sep 2023 12:57:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E043B6B01FE for ; Wed, 13 Sep 2023 12:57:20 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B523180482 for ; Wed, 13 Sep 2023 16:57:20 +0000 (UTC) X-FDA: 81232179840.10.438B25A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 0300040021 for ; Wed, 13 Sep 2023 16:57:18 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HfOn669x; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624239; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xJaTbGB+togQTw9SIdIdixTkcz5/QJ2b3c5fZ9o3sSg=; b=raTY/HTUlabuURJoSTYrm8aE+2c0HFM/aGEBcK6JJpDB2Fpr2vwnc/qhz7cUfxAKlioZrz ErxaLMzYCpo3yPwIkUrAks67N3kTEYYTjTOdbHffgLKsutd/ABY4c3OzrdTgIq2pL5X4xe uPVM8twJ90ZCwQUZhitDsA79w0iKEf8= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HfOn669x; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624239; a=rsa-sha256; cv=none; b=l1AllzG/p0WDvtAdMHghh5ZgUQXH+CJo760uZmXL9FSFiNR9Xqi6x/p2cNSTo2Uey8ddlC jruGBcovvPdYJ7kIgz6IoUOBH7yJvoA5RsuY+6CNUG4DgKQpHPVxaSHHdObnJf110T4vW0 AhiDNpncLADDostU2GUI3Oc2fc5tRcA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624238; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xJaTbGB+togQTw9SIdIdixTkcz5/QJ2b3c5fZ9o3sSg=; b=HfOn669xlwTfoWZq0V70ffw3GETThHlu+aO8dBRezYafMfssbN+RwDifn0aO005+JaE4nr FmYeWfvJOugIKOCdfyp4GFa5qgJtJI+mIEXb5UsjS+McOIYfdjXYAXgeJw3XD+6dcpzNns oWtsQhDP7lM8xhzh+UhYkBpSGbKURsA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-694-6buX1QvqNt-weeixz82rJw-1; Wed, 13 Sep 2023 12:57:12 -0400 X-MC-Unique: 6buX1QvqNt-weeixz82rJw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 222FF18312C9; Wed, 13 Sep 2023 16:57:10 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7B42840C6EBF; Wed, 13 Sep 2023 16:57:08 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 07/13] iov_iter: Make copy_from_iter() always handle MCE Date: Wed, 13 Sep 2023 17:56:42 +0100 Message-ID: <20230913165648.2570623-8-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: 0300040021 X-Rspam-User: X-Stat-Signature: s519ebb4sq3tpjq7i66ambd9epsx3pt8 X-Rspamd-Server: rspam01 X-HE-Tag: 1694624238-213512 X-HE-Meta: U2FsdGVkX19B/EeYkyK7gw1NNYhdl2BAb5We0CFyrncM+u8K9oORdIh6/CJlh1bf4IEiPwGstPV1DEA+G253MqRlsXtQJR4CLoCKIKvyybEDqNj3TO1gYStdaDnYM3nrhJVd9xVNoUwDhY2E1RdXQS9ywPFjFLZugKyWssyezGvfVoKtpLbeiY3ZUeGkaslVGc5PtGRM21l5itqq55WS3VYQNGnzRptSnxakyjfUPsn7fS5gi4ZVdTGrrRbhSULtFN1mc+HoUazhC3NsNuiqfS07me/zpgdTV0KY0sP94TDdTt+cJ6BP4b/uMNN5ZIU0CxouYV09GBwDBkHNN4fIb8+fPpEr2fuhUgTCQMei8976KBHTw7pT/fTul58wAILvOgj+Jm1dHY/07hLPJXRDejENB7lFz2ca3jgyz44sMmnQH7z8d6Bq/UKuNSHuW7Y1osC2nGkuPag3b3Ko3kfsc3u1IKd/g0RdbIQgWVc0s0xYD1fb8Kl3PIDmx0QzNlCbGuRX+2AU5R1WjLI37rdU12P/L9O0gQN80gVR5QujkCjHfO49WF0/i+LreeGPLLnA+maMc8DJhCjscYdj1e8BgDgPIp/MC1q2UODvBtnqXj/09snmRkPd3be970Wn3R/yffQ4gu3L85ep+O0tSDXnrtE9Rl7u+FbO04Oy7jmXsbJDHv58bUyFw4221ThyNe4leFc74buFXxzZoSTSWwu3UPOlFHJ2c4VK0NOSTZPyXPRYE5sj2lp9au5ZVez/GYwsxGhMLwwOPqVgNd4e4zJpLTe4viF/S1eamt7CCLxDOOw5oYb8VRb7zZ4qEVrY1sHe5rl02Dn7v+iX33J3Orf1/9Q/OzAVlr6KHlIV/0UISsNYnI495WS9KbmDlk1bhS/DLz97Yc6T4t/VUGiYv4S1rxuyTkx3PYIr9Fs2RPNsPxSd+hNbXYLLYuJEtnoISUOJGzCZ48NZNErCIFvz+Ke TCF3d317 tiGQyUGciDZxLNtaIEArG0/SYmwxgo/RE1vx5TudzPUN6KJrsl9hQ47tWiARh1JlwJIzsAId8uvtm7FyO52INnauWDFX7K+aWtd00RdgihrqLPi2vo24gGVHhvWnI9+KeE7gtJNAu7Pd91aQXevf6cUSdDgM1KFHTI0CWZwpfyETaNUvyEkVkiOeQDJU+2r9FvC27+JmshGxs6zOeYK4kJ+yV50RxSc6sD2/aXqbbT3DSWLcVnvVOKXa5UKaUqtg3Q41WGV1oJoMjC8dccOJfc1v9QQ/apXxXv7DZVxHthMoh7z0HV0taXyvquFSvEdj06Bziyi7drfjHT1IdULFlEWVIqKObL4JItB8QqcGTmkBwu/czfH0zsWNXS/i9Up43evjpfr69mgcC3yr7AdQjYTut+ERumIcZciUCKc60rt7Zydn6cyS6vG/xvnJ5HSvR7bC7gFdiu6JnEnynRya0YncliBkG8kWVev9ycisHLJft8/HdkZTVUwcQvyVmo+gpsr8m49ghSkebG27BNjjXkgIy9mjbgobpCUbP2apsNRtboSZq9c7LxKpUfY0g4lBhetKG/hnFcUnF0E0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make copy_from_iter() always catch an MCE and return a short copy and make the coredump code rely on that. This requires arch support in the form of a memcpy_mc() function that returns the length copied. [?] Is it better to kill the thread in the event of an MCE occurring? Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- arch/x86/include/asm/mce.h | 23 +++++++++++++++++++++++ fs/coredump.c | 1 - lib/iov_iter.c | 12 +++++------- 3 files changed, 28 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/mce.h b/arch/x86/include/asm/mce.h index 180b1cbfcc4e..77ce2044536c 100644 --- a/arch/x86/include/asm/mce.h +++ b/arch/x86/include/asm/mce.h @@ -353,4 +353,27 @@ static inline void mce_hygon_feature_init(struct cpuinfo_x86 *c) { return mce_am unsigned long copy_mc_fragile_handle_tail(char *to, char *from, unsigned len); +static __always_inline __must_check +size_t memcpy_mc(void *to, const void *from, size_t len) +{ +#ifdef CONFIG_ARCH_HAS_COPY_MC + /* + * If CPU has FSRM feature, use 'rep movs'. + * Otherwise, use rep_movs_alternative. + */ + asm volatile( + "1:\n\t" + ALTERNATIVE("rep movsb", + "call rep_movs_alternative", ALT_NOT(X86_FEATURE_FSRM)) + "2:\n" + _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_DEFAULT_MCE_SAFE) + :"+c" (len), "+D" (to), "+S" (from), ASM_CALL_CONSTRAINT + : : "memory", "rax", "r8", "r9", "r10", "r11"); +#else + memcpy(to, from, len); + return 0; +#endif + return len; +} + #endif /* _ASM_X86_MCE_H */ diff --git a/fs/coredump.c b/fs/coredump.c index 9d235fa14ab9..ad54102a5e14 100644 --- a/fs/coredump.c +++ b/fs/coredump.c @@ -884,7 +884,6 @@ static int dump_emit_page(struct coredump_params *cprm, struct page *page) pos = file->f_pos; bvec_set_page(&bvec, page, PAGE_SIZE, 0); iov_iter_bvec(&iter, ITER_SOURCE, &bvec, 1, PAGE_SIZE); - iov_iter_set_copy_mc(&iter); n = __kernel_write_iter(cprm->file, &iter, &pos); if (n != PAGE_SIZE) return 0; diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 65374ee91ecd..b574601783bc 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -14,6 +14,7 @@ #include #include #include +#include static __always_inline size_t copy_to_user_iter(void __user *iter_to, size_t progress, @@ -253,14 +254,11 @@ size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) EXPORT_SYMBOL_GPL(_copy_mc_to_iter); #endif /* CONFIG_ARCH_HAS_COPY_MC */ -static size_t memcpy_from_iter_mc(void *iter_from, size_t progress, - size_t len, void *to, void *priv2) +static __always_inline +size_t memcpy_from_iter_mc(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) { - struct iov_iter *iter = priv2; - - if (iov_iter_is_copy_mc(iter)) - return copy_mc_to_kernel(to + progress, iter_from, len); - return memcpy_from_iter(iter_from, progress, len, to, priv2); + return memcpy_mc(to + progress, iter_from, len); } size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) From patchwork Wed Sep 13 16:56:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383572 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4DFCEE01E0 for ; Wed, 13 Sep 2023 16:57:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 901296B01FF; Wed, 13 Sep 2023 12:57:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8AE7C6B0200; Wed, 13 Sep 2023 12:57:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7292E6B0201; Wed, 13 Sep 2023 12:57:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5F5E66B01FF for ; Wed, 13 Sep 2023 12:57:21 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4514A80351 for ; Wed, 13 Sep 2023 16:57:21 +0000 (UTC) X-FDA: 81232179882.05.3D632FD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 87E3AA0037 for ; Wed, 13 Sep 2023 16:57:19 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FCn0jFFg; spf=pass (imf15.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624239; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p+t8ChtS3C0mCAtd18JzNSIJcCnKtYlpMvVgh9PlkwE=; b=dFeA0ei4Y1pvkHbZjVojdC+Z5wxQEJxy3399AgZRz14Ph8ivuo10Mf+SMfonhJ+vg2HShA QnsHJ0EK5YYu63AQhtmXqecCFGHrm9tiLiP5pS6MJi/i/VVCaJkORph4SNu4zwg0iiWNUN vw24fU0H0oCsMmedqt+ldAUbvO9jXvo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624239; a=rsa-sha256; cv=none; b=ZiuP4jJQELx9P9r1nYvcBKhUo4L19bPTWbi9o+J7/XQe8stxrhSD3GnbUUemiEToamEYlf uwGpzg4KH2Oi9nfyYHIruZFeO3/7+TnKIu1AjbJpuHA+7aKwqYAufUzLXBoWvDyjFWfH1+ 9EbuIcWMHecznw7Fmt3uNqR1zS0qgXY= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FCn0jFFg; spf=pass (imf15.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624238; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p+t8ChtS3C0mCAtd18JzNSIJcCnKtYlpMvVgh9PlkwE=; b=FCn0jFFgFML3ZQz2s8axxaM05t/MuIAlVnMPJ+LnbpNxYlkBT2Eel+MfaobEQtF9NOvMmI 2UqojYup1cI4MQBAhxR+KSmhnUIQM/EVrHlk9n4gOU03GBCZWmI9z1K7Lef/XU9N0lhnw0 ZdygDC04K2SJ5uZfNuDBdRxxlB/hEFk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-542-5xC_0HubPGGLzilTHkSTMQ-1; Wed, 13 Sep 2023 12:57:13 -0400 X-MC-Unique: 5xC_0HubPGGLzilTHkSTMQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 783FB85531D; Wed, 13 Sep 2023 16:57:12 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id D43F240C6EA8; Wed, 13 Sep 2023 16:57:10 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 08/13] iov_iter: Remove the copy_mc flag and associated functions Date: Wed, 13 Sep 2023 17:56:43 +0100 Message-ID: <20230913165648.2570623-9-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: 87E3AA0037 X-Rspam-User: X-Stat-Signature: 9xcbqt99yhqqc36dzgywbyn5qk1koqnf X-Rspamd-Server: rspam03 X-HE-Tag: 1694624239-619319 X-HE-Meta: U2FsdGVkX1/HULi65kd2ij/TFaAF36AnVFj+Zgxgj9ITtiJd/tBYTqHiERgJLtwFjmtyKxSM8oj+sxM6xDf8cZ1pbd86WInIMZyDZH2npHMocjhmxM3LlBLvgt0vLmWSIgMV6rcQf1h+mjuoWmZ4qLLbLni0IqrlnVMkckRVEdWSF4cadz/Oi8bx+R0+Pu+ExPqjBkQfVh6FEFGMeLBAUWfIz8he+vpd48CyheAetPMHMPqZ2dKNsFdTDxdPjnFx8+3ppPZbP2HxQPaAl1kpwD/DXBTRlTdp7GZmXFdD/ZkW0/jauYoZJygi3mdNXL6JAhYF7h6L+jGGvxxr7HmvGaqOnwy2PE/lE3YnNCqPI+GspDZT+URr6xpX34KH+9ueyj4ZfzPRvnt+bnek879wjbv6DFE2w4xbKFFPY+V3NnoGtHIk+88WfVa5RHfEyWlDeMHELftIzqsRf5HqoJnAB4yJ0/KJilmpiSV60KGIyRhSkf2qwGB0CDnnoSbZVSU6trC8KlDTpaX2iHgikvZ6IhNxtVFp/dSpe+glPRYQEVz8clV6IITlKPYRmcSvbEofmaj+98Z27oZ3LZ7isyDK92VDqOeK3fIe6oMPpGFrHSoREAxraQEtn02lIR8Rvtgbn9FZRzz++1voIr3egLTcx7/C1Kdvp0ZYvdTTCRai6h3DN/q7jebiGlVv7zOTbS9gVIK9P8CXSvYBzoCbBc+1VigZuiynvqMP7c9u7Pw5YVBWHKkOmq0+kH+/REntQWEd0LI4afLPR7dtRymgX5m77fGfLvPWIDojQ83GEJ2PT3r0yRaRaCD1lRhVhNo7pqT1FwGiyHVWkUfN2JZQMTzhe7xOaBhWBrZ5nN09rrkw+25d8sKzNT/sCLbKxiEZIj2ueEeLmMabYtFbFyhhTRYmQRpk0ot7ugIa5TniwJWIejHnx1YM4IZ8ObyhZ6M2P8F9bjoxWsR9zj8OlOFHcgR k4kX31Ws f7dNtOFumcMsLySnsHEis+iN+sQKvFc/28bXaQXLddrCIDSGpzkZ7htX9P7BABYsEfcqum8aOEjt1VIlsmurQ0Vdq2E5InVZEKZ+D6R78TEoOUMaMj698DsHvSzxgD+sOGqRAL8p2IZRWNflKbOuaIxBj+5KosFKgsOnAGP5qY0G/bs6yW0ECI/D4ZgH5rEs9ErzT2abdQpK4LbiCI9+rFlCN0CZ58gzRrrE3M4Nz9Jau7auifXJYMYU78USJXpKIsFzqFB2F9UbqOdrffawB7hyIXFFk2XhIZwQDE/NpJMKnQoGkcPQHEClJLOHXzdyXK6+KC3wAuFlbK0FmSo8RabrEAGE3mZlmb1mVBFrPafBMvWNWJB8NZj4dT2j9O1gpaV6FY4xx1xyLWnL4ABLf85qd1xgoH2w9quedXLoRbO+zG+NeeDybzJHLiTVNeOMTcODGnlYgcwyW8nVoklM0m/Nr4XxmzXB9p1oWPWVrkQ7xhaZ6XUE/mwffyCXdxXrYusHFH07aD3Spjn60P2HpQdtrURllCiHR5RdIF8cuGeEE2RlhRMy50gQ6x0viDtQtnMQ9i0w/ea8x6Ak= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: copy_to/from_iter() now always return a short copy on MCE, so we don't need to have special handling. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- include/linux/uio.h | 16 ---------------- lib/iov_iter.c | 5 ----- 2 files changed, 21 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index e2a248dad80b..cc250892872e 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -40,7 +40,6 @@ struct iov_iter_state { struct iov_iter { u8 iter_type; - bool copy_mc; bool nofault; bool data_source; union { @@ -251,22 +250,8 @@ size_t _copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i); #ifdef CONFIG_ARCH_HAS_COPY_MC size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i); -static inline void iov_iter_set_copy_mc(struct iov_iter *i) -{ - i->copy_mc = true; -} - -static inline bool iov_iter_is_copy_mc(const struct iov_iter *i) -{ - return i->copy_mc; -} #else #define _copy_mc_to_iter _copy_to_iter -static inline void iov_iter_set_copy_mc(struct iov_iter *i) { } -static inline bool iov_iter_is_copy_mc(const struct iov_iter *i) -{ - return false; -} #endif size_t iov_iter_zero(size_t bytes, struct iov_iter *); @@ -381,7 +366,6 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, WARN_ON(direction & ~(READ | WRITE)); *i = (struct iov_iter) { .iter_type = ITER_UBUF, - .copy_mc = false, .data_source = direction, .ubuf = buf, .count = count, diff --git a/lib/iov_iter.c b/lib/iov_iter.c index b574601783bc..66d2a16774fe 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -169,7 +169,6 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, WARN_ON(direction & ~(READ | WRITE)); *i = (struct iov_iter) { .iter_type = ITER_IOVEC, - .copy_mc = false, .nofault = false, .data_source = direction, .__iov = iov, @@ -629,7 +628,6 @@ void iov_iter_kvec(struct iov_iter *i, unsigned int direction, WARN_ON(direction & ~(READ | WRITE)); *i = (struct iov_iter){ .iter_type = ITER_KVEC, - .copy_mc = false, .data_source = direction, .kvec = kvec, .nr_segs = nr_segs, @@ -646,7 +644,6 @@ void iov_iter_bvec(struct iov_iter *i, unsigned int direction, WARN_ON(direction & ~(READ | WRITE)); *i = (struct iov_iter){ .iter_type = ITER_BVEC, - .copy_mc = false, .data_source = direction, .bvec = bvec, .nr_segs = nr_segs, @@ -675,7 +672,6 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction, BUG_ON(direction & ~1); *i = (struct iov_iter) { .iter_type = ITER_XARRAY, - .copy_mc = false, .data_source = direction, .xarray = xarray, .xarray_start = start, @@ -699,7 +695,6 @@ void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count) BUG_ON(direction != READ); *i = (struct iov_iter){ .iter_type = ITER_DISCARD, - .copy_mc = false, .data_source = false, .count = count, .iov_offset = 0 From patchwork Wed Sep 13 16:56:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82382EE01E9 for ; Wed, 13 Sep 2023 16:57:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 298486B0200; Wed, 13 Sep 2023 12:57:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2460C6B0201; Wed, 13 Sep 2023 12:57:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0987B6B0202; Wed, 13 Sep 2023 12:57:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E95996B0200 for ; Wed, 13 Sep 2023 12:57:21 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BCC7F80E38 for ; Wed, 13 Sep 2023 16:57:21 +0000 (UTC) X-FDA: 81232179882.27.0AAA7CE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 1813D40018 for ; Wed, 13 Sep 2023 16:57:19 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MEzJL8x2; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624240; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9uMtbW5W1n5t6FTdNdAKrkGJdOcWoyjxyU/ZljquCxs=; b=7qMn9GPwhxCsLqBSrrXuWMPkQQLAT8610OOd2HhgOv0ZFkVZZUb8VE7002b169fI3X2lwq R6O+G03uPBh65wAD2RlPZAe94otDqZlOlqYrglbDO24KXuFg8CBLQ1pndFbEkKK1gutEO7 +DNLeTI1jhRcO2OKr4M/lXB7QEmD4a8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624240; a=rsa-sha256; cv=none; b=iXB1GpizKd7huVYNg4AFnIp150yzB4tkTXoGM8/c4aGWW9mAJ4cc00pHgvpzD3S15F1BTH D5R49ah4WBPzfARGjqbsqwGDZ+PbkD0ak+jVGugdHBe6dttT4LzrPqjRMIuQMyRYbOwf6K RGYRlU0d4giiQyA1re9MPsLc67kJX90= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MEzJL8x2; spf=pass (imf17.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624239; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9uMtbW5W1n5t6FTdNdAKrkGJdOcWoyjxyU/ZljquCxs=; b=MEzJL8x2v4yrpugmXF5YHB4kgJEcu5MdsYXicGFpnARWDcct1GzpO10Gmr/vC7lA2JI2w7 wN1U8GNbEcUNEFTqbkei1vw8hsJKhm7QaD5T9F6CrP1kxOQfVuN1wAxSk9kVef+PvKXkBv MQ2+3gRPikXikY69i8mEieLOWtO4PDE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-453-8IErGMcgOqWpOUPhN4ftHA-1; Wed, 13 Sep 2023 12:57:15 -0400 X-MC-Unique: 8IErGMcgOqWpOUPhN4ftHA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B771D8219BE; Wed, 13 Sep 2023 16:57:14 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 139332026D4B; Wed, 13 Sep 2023 16:57:12 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 09/13] iov_iter, net: Move csum_and_copy_to/from_iter() to net/ Date: Wed, 13 Sep 2023 17:56:44 +0100 Message-ID: <20230913165648.2570623-10-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: 1813D40018 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: abtup3nt83omqmmi7rzd6dg41wjrcoko X-HE-Tag: 1694624239-49249 X-HE-Meta: U2FsdGVkX18FY1VHjkdbeeOPL7ppxcNOz7zp22PoxeqYnQ6pWja4kL5oEX8Ex1nXeAkvi7N677mJ6oMC7sH3soI76DeF0R+0OvluEtJqZASW2enGQjkL5aJSnwP+1MEF1PM4yUC5FbwO26hq4wS2zdWmFLCe6SUE/Q1mouj5ngof/dCofpmpuxoozST6xL+4hD0SuKiSHK16fbcxwBkX0IBWpOzpHDkCnjcr/b4wqcL8BAsovL2JN50dKJSsJKlWoBfo7J1/tEm6ggPNI+6o+tz3gfiZweETAf1nbi6KSuUlSiG5kNQM+4DRG1KzqmLpe/wI3RPJzgwyr7BthrBKerHPpxLj5vvbbIBiuHqR8tmAXADdPwfFNBHKivlSLNjtSuIiCdOgP8vM7LVwqt0Y7lnYmVSCEq7JLuLASoTm7wCvmTOMQfnfadrcfWDgkyiB0nw6ecbvxOwcdNXZei8FEMinM6Bya3+NXtvId3ZFgKYJRu3qCk39FH+jXxqSoiBabHbeFU83OJCQmQiOUXqFan4frfacj2JvJetLdPLOnuIjAfJeP5mYIHfkFCmL5BQnsYRClv9nqfRuFb9EkYn01cJmrrDGTZwJK7yeAld4+/7rqJSVAbwg/pYobryaoaMoDJWtPex3FV95o6rgjJJ4Sab9g3W/e9qjdfeS/kvnHDi9ly+2DScYWzz9TPbWJVX15zOkTLOM3+6TsHmkZ9SB5f5K5Km/r51bjyj4HJ5hPxUL5t4vy7B/eBle5gRSgaOVlb7q3vJPweVUmTVHKxJY+TNwzTgrJUWPafUwLpcDlbfdSpuPl4abP4KaZToUPXCsMPF6RCm9/HaE/DGLWb89372weiRo3LfDCkGbKTxbCIPqXHjKtv0JK0cqTSz45mRkVkl3JTWEZlwq1kybqDTKBJlDbUBER7hzwXE2xF4GWHWhlXsQ2ZENI2KhJTGbmhhPzC8vcRMDE6Dakf7dYWK B3bemk6r 0zEKUtFzcNOBq12pV6xFmOTBYut19EQAWndLRYotVJ8Wpj1kgSRtk9aWl6LzSUKHZBEyaJZecJUODqHl9X16V5sb0xdLDJzUk5B9130qjofhss2nDK3pkGWaWsHFdseOKRgIPNaQQw4HX8rKkgHH7dI9I7p6C0r0hccTEl6atKVkbGggiAmV5raJra9oioYctdgiQze9h6PGEbNKHebAzQXYPGXonEr+7M9x1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move csum_and_copy_to/from_iter() to net code now that the iteration framework can be #included. Signed-off-by: David Howells --- include/linux/skbuff.h | 25 ++++++++++++ include/linux/uio.h | 18 --------- lib/iov_iter.c | 89 ------------------------------------------ net/core/datagram.c | 50 +++++++++++++++++++++++- net/core/skbuff.c | 33 ++++++++++++++++ 5 files changed, 107 insertions(+), 108 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 4174c4b82d13..d0656cc11c16 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3679,6 +3679,31 @@ static inline int __must_check skb_put_padto(struct sk_buff *skb, unsigned int l return __skb_put_padto(skb, len, true); } +static inline __wsum csum_and_memcpy(void *to, const void *from, size_t len, + __wsum sum, size_t off) +{ + __wsum next = csum_partial_copy_nocheck(from, to, len); + return csum_block_add(sum, next, off); +} + +struct csum_state { + __wsum csum; + size_t off; +}; + +size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i); + +static __always_inline __must_check +bool csum_and_copy_from_iter_full(void *addr, size_t bytes, + __wsum *csum, struct iov_iter *i) +{ + size_t copied = csum_and_copy_from_iter(addr, bytes, csum, i); + if (likely(copied == bytes)) + return true; + iov_iter_revert(i, copied); + return false; +} + static inline int skb_add_data(struct sk_buff *skb, struct iov_iter *from, int copy) { diff --git a/include/linux/uio.h b/include/linux/uio.h index cc250892872e..c335b95626af 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -326,24 +326,6 @@ iov_iter_npages_cap(struct iov_iter *i, int maxpages, size_t max_bytes) return npages; } -struct csum_state { - __wsum csum; - size_t off; -}; - -size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *csstate, struct iov_iter *i); -size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i); - -static __always_inline __must_check -bool csum_and_copy_from_iter_full(void *addr, size_t bytes, - __wsum *csum, struct iov_iter *i) -{ - size_t copied = csum_and_copy_from_iter(addr, bytes, csum, i); - if (likely(copied == bytes)) - return true; - iov_iter_revert(i, copied); - return false; -} size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, struct iov_iter *i); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 66d2a16774fe..5b2d053f057f 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -10,7 +10,6 @@ #include #include #include -#include #include #include #include @@ -179,13 +178,6 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, } EXPORT_SYMBOL(iov_iter_init); -static __wsum csum_and_memcpy(void *to, const void *from, size_t len, - __wsum sum, size_t off) -{ - __wsum next = csum_partial_copy_nocheck(from, to, len); - return csum_block_add(sum, next, off); -} - size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { if (WARN_ON_ONCE(i->data_source)) @@ -1079,87 +1071,6 @@ ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, } EXPORT_SYMBOL(iov_iter_get_pages_alloc2); -static __always_inline -size_t copy_from_user_iter_csum(void __user *iter_from, size_t progress, - size_t len, void *to, void *priv2) -{ - __wsum next, *csum = priv2; - - next = csum_and_copy_from_user(iter_from, to + progress, len); - *csum = csum_block_add(*csum, next, progress); - return next ? 0 : len; -} - -static __always_inline -size_t memcpy_from_iter_csum(void *iter_from, size_t progress, - size_t len, void *to, void *priv2) -{ - __wsum *csum = priv2; - - *csum = csum_and_memcpy(to + progress, iter_from, len, *csum, progress); - return 0; -} - -size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, - struct iov_iter *i) -{ - if (WARN_ON_ONCE(!i->data_source)) - return 0; - return iterate_and_advance2(i, bytes, addr, csum, - copy_from_user_iter_csum, - memcpy_from_iter_csum); -} -EXPORT_SYMBOL(csum_and_copy_from_iter); - -static __always_inline -size_t copy_to_user_iter_csum(void __user *iter_to, size_t progress, - size_t len, void *from, void *priv2) -{ - __wsum next, *csum = priv2; - - next = csum_and_copy_to_user(from + progress, iter_to, len); - *csum = csum_block_add(*csum, next, progress); - return next ? 0 : len; -} - -static __always_inline -size_t memcpy_to_iter_csum(void *iter_to, size_t progress, - size_t len, void *from, void *priv2) -{ - __wsum *csum = priv2; - - *csum = csum_and_memcpy(iter_to, from + progress, len, *csum, progress); - return 0; -} - -size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, - struct iov_iter *i) -{ - struct csum_state *csstate = _csstate; - __wsum sum; - - if (WARN_ON_ONCE(i->data_source)) - return 0; - if (unlikely(iov_iter_is_discard(i))) { - // can't use csum_memcpy() for that one - data is not copied - csstate->csum = csum_block_add(csstate->csum, - csum_partial(addr, bytes, 0), - csstate->off); - csstate->off += bytes; - return bytes; - } - - sum = csum_shift(csstate->csum, csstate->off); - - bytes = iterate_and_advance2(i, bytes, (void *)addr, &sum, - copy_to_user_iter_csum, - memcpy_to_iter_csum); - csstate->csum = csum_shift(sum, csstate->off); - csstate->off += bytes; - return bytes; -} -EXPORT_SYMBOL(csum_and_copy_to_iter); - size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, struct iov_iter *i) { diff --git a/net/core/datagram.c b/net/core/datagram.c index 176eb5834746..37c89d0933b7 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -50,7 +50,7 @@ #include #include #include -#include +#include #include #include @@ -716,6 +716,54 @@ int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *from) } EXPORT_SYMBOL(zerocopy_sg_from_iter); +static __always_inline +size_t copy_to_user_iter_csum(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + __wsum next, *csum = priv2; + + next = csum_and_copy_to_user(from + progress, iter_to, len); + *csum = csum_block_add(*csum, next, progress); + return next ? 0 : len; +} + +static __always_inline +size_t memcpy_to_iter_csum(void *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + __wsum *csum = priv2; + + *csum = csum_and_memcpy(iter_to, from + progress, len, *csum, progress); + return 0; +} + +static size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, + struct iov_iter *i) +{ + struct csum_state *csstate = _csstate; + __wsum sum; + + if (WARN_ON_ONCE(i->data_source)) + return 0; + if (unlikely(iov_iter_is_discard(i))) { + // can't use csum_memcpy() for that one - data is not copied + csstate->csum = csum_block_add(csstate->csum, + csum_partial(addr, bytes, 0), + csstate->off); + csstate->off += bytes; + return bytes; + } + + sum = csum_shift(csstate->csum, csstate->off); + + bytes = iterate_and_advance2(i, bytes, (void *)addr, &sum, + copy_to_user_iter_csum, + memcpy_to_iter_csum); + csstate->csum = csum_shift(sum, csstate->off); + csstate->off += bytes; + return bytes; +} + /** * skb_copy_and_csum_datagram - Copy datagram to an iovec iterator * and update a checksum. diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 4eaf7ed0d1f4..5dbdfce2d05f 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -62,6 +62,7 @@ #include #include #include +#include #include #include @@ -6931,3 +6932,35 @@ ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, return spliced ?: ret; } EXPORT_SYMBOL(skb_splice_from_iter); + +static __always_inline +size_t memcpy_from_iter_csum(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + __wsum *csum = priv2; + + *csum = csum_and_memcpy(to + progress, iter_from, len, *csum, progress); + return 0; +} + +static __always_inline +size_t copy_from_user_iter_csum(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + __wsum next, *csum = priv2; + + next = csum_and_copy_from_user(iter_from, to + progress, len); + *csum = csum_block_add(*csum, next, progress); + return next ? 0 : len; +} + +size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, + struct iov_iter *i) +{ + if (WARN_ON_ONCE(!i->data_source)) + return 0; + return iterate_and_advance2(i, bytes, addr, csum, + copy_from_user_iter_csum, + memcpy_from_iter_csum); +} +EXPORT_SYMBOL(csum_and_copy_from_iter); From patchwork Wed Sep 13 16:56:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0C67EE01EA for ; Wed, 13 Sep 2023 16:57:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 83F866B0205; Wed, 13 Sep 2023 12:57:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7EF0C6B0206; Wed, 13 Sep 2023 12:57:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 667316B0207; Wed, 13 Sep 2023 12:57:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 53EA06B0205 for ; Wed, 13 Sep 2023 12:57:28 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0B3DF1404E3 for ; Wed, 13 Sep 2023 16:57:28 +0000 (UTC) X-FDA: 81232180176.30.7C4356C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 4B71E20029 for ; Wed, 13 Sep 2023 16:57:25 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JQP9vyD7; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624245; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i+29vAFZrXF9naWng0ipN/2z31gEil+IMifE0VymLF4=; b=aq4AM/DGD5TwGifzjAwlds+Q+LnaqGijz6UTHBmYwW6dtsFM5NP9gBMpdX4DTrUySZr7Mo 9LQu+YbjeZCDXY2zHbGOI3zOD10Ax7B0RJi8uYClGBwxiz+F4vcuE3WNdqiFLC9DrZqax6 koA8FyWi66IDsRoSTD6Wx7+f554seZA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624245; a=rsa-sha256; cv=none; b=i9VfDZ9jABPhDP7LZS/LOoVDVS6eTbCsZaPumrt5GapV3rnQviUmJEdpfsVGzBw6ukdmHW CHid1ycMhzmyqEXLpuwoDVehcDVUxPoPBf3rGOt1XzVBnvOTzuqkLU7gN62XBhMx8ZZ53b 3hiDacVyw39kD0d0yi1KOb99jXDX/CM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JQP9vyD7; spf=pass (imf13.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624244; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i+29vAFZrXF9naWng0ipN/2z31gEil+IMifE0VymLF4=; b=JQP9vyD7fQWsuYrP2ettkddyKXIQvwkl4xMokCSzknDunTUr/YNo95k6h15qs3acYs14+G 7TWrykYzNP5v1IXQhXYOfYpAzVP67PdnpcWT8PeWiUpPNGtYUqr4QCZ61J6/5ItqWu2n9m h9mNRNgNTZr9WTpOaqffCpnibTlOtwk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-536-VozdxTMENjmI9CZVmuDJyA-1; Wed, 13 Sep 2023 12:57:18 -0400 X-MC-Unique: VozdxTMENjmI9CZVmuDJyA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 728E5857A9E; Wed, 13 Sep 2023 16:57:17 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5DF682026D4B; Wed, 13 Sep 2023 16:57:15 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Subject: [PATCH v4 10/13] iov_iter, net: Fold in csum_and_memcpy() Date: Wed, 13 Sep 2023 17:56:45 +0100 Message-ID: <20230913165648.2570623-11-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Queue-Id: 4B71E20029 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: zjmjdbrr3xd8y15f7yy8xn1tcnjfjfrq X-HE-Tag: 1694624245-469701 X-HE-Meta: U2FsdGVkX191uyML3th7qCY5PawWPpdllYuWTZq+CMRbK9BHQCwpJBBVavzxrBAtqLGfaRh0QMVIi6Js2h8GkeYK6tD+qBRb3YaKhHZYS5rfrzcEQELMhudQ3lv2RA8Ppm19gfs1+ydAmYguFiEXvx/0fNojlCe60b6Vn5nbZ/1Y370lHpgFVTkgp9g2IPQG5+QCO6AORMWnAYPHPErYc8JR2MdXy6ZFWKDu54OHF8zj/avP4LPNXOAkiiQANCHwKvwDYmG6tWPSr4EHPQZgaICVEJuv/obqnTAAhM4kMnF1tVwMktDhaGvGOS0DlGkZzm20TxonxeyG0hekkPp/DgnrYb/B+VjITHF0szN175P+0nIB4UC0Xzm3VJgyOqydyjTkK/F+9jrrrHScB26PDZWFDoSgAk4wbzv5sEvekQA8rsBeSmQUVWk+63NzjDiAbqV/Zja2m+lQyHv0GAHjLOHTFnCpg/R02EYvlMnYWa3DuOaSWmTpKvsT6WSqAvKwLBdsIqkOr+5ylMDY+94GZUEqFmTedUS2Q9NnsMH/t5DnRyVCAfaar3JeFrk1Gpdoiirt50DE10friylM5/HvWSgJbsihJD6GzMx9Fyzi5KgGhjSmEpUlx/bxKqPY3/xIeK3M4rOuRvl6aKHE6bg814H794tmQj9c7MIJ0t7nX09IuxOWTJCm4ImWEgXbJ6+JH3xQwc9Rh15qTMsHZ/m84HfSwDgxeLynj74dcmLXPXxIIHC4GSPZa6qK1Jsv1F4e6ox6seh2DqUmShyJkGC7YpGJNOYphL8+8/XN/QjPIs0KHpB2Q+YoW2euehLxScuS1M6Wsp9XnXiRquTDXe4AjRjSGpjO/NRbdKJG2dGmgLgGD5WcMEDa6qoI7Dey6nAshXoaWMfbJyPXd2X0sZ8foNP4EJGZJ8IWNg3R+RmHiuMkZrbOLTcSO9mRMR67MfCNkc4OjJ7aTx0/4I6EQN7 AuQ8prtr 09IawYj8C9LQVvB8SqsM9DuAFdSM/5AAkme3LaRT9y0j02bkXvlTYfD/bT5/DxjyMvnNULt9LiTy73vPSWIKieR/TubmmLN2a/EI4IxSBAVZwRg9wF6eCWhgpIhnBQGf4zoNu+gZjCQpguwy5RzjXLPGvIQide9tOr/IkRPV1FlrVfXKMZPWtz1oN4tZEY/w3uK7tH0lfE0hg0WXo2LziIi78llGT8Yo+ajbPpbMKrU3WUuF6LG0/VCX/dncQxx0UXHVTaytkXp0mU9iLT69kcR5GAwk57FI0ItgQzW1yFxiuZFxK+CE91lfhlPQgu9S9LdtckLjS2tJe1sORgGqJgRsh15fdoH8XbYYyTyaV9bZ1MXflTIimvCgC+w49nTo0L0MP8Gbiak2cthEFtEPa1MuJJc7RMTfxT7ve+noN96mV+XnAMNE2v6QWZTwEgo6F1AZWHLG7iteiwpmE7BDeVZ4eXUhc4xHugdYDG3i8GushfZ3qbjsyPgKc2pwV5UkStmcvTc6EnrLGbaAcDU9E3rXlt7IIcMQxHADb3AcSOLFOrHACABn1gah7++1kkAnwm38pFOOQJGvPfhsJbYc6ZaiCVXxK7mcjPQeDmgXg50L+CJrIFl36Gn337K4FgfCDTWOoyJ2QCiyw2sGKZUNJmkTG0HOYEU2VHTH+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Fold csum_and_memcpy() in to its callers. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- include/linux/skbuff.h | 7 ------- net/core/datagram.c | 3 ++- net/core/skbuff.c | 3 ++- 3 files changed, 4 insertions(+), 9 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index d0656cc11c16..c81ef5d76953 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3679,13 +3679,6 @@ static inline int __must_check skb_put_padto(struct sk_buff *skb, unsigned int l return __skb_put_padto(skb, len, true); } -static inline __wsum csum_and_memcpy(void *to, const void *from, size_t len, - __wsum sum, size_t off) -{ - __wsum next = csum_partial_copy_nocheck(from, to, len); - return csum_block_add(sum, next, off); -} - struct csum_state { __wsum csum; size_t off; diff --git a/net/core/datagram.c b/net/core/datagram.c index 37c89d0933b7..452620dd41e4 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -732,8 +732,9 @@ size_t memcpy_to_iter_csum(void *iter_to, size_t progress, size_t len, void *from, void *priv2) { __wsum *csum = priv2; + __wsum next = csum_partial_copy_nocheck(from, iter_to, len); - *csum = csum_and_memcpy(iter_to, from + progress, len, *csum, progress); + *csum = csum_block_add(*csum, next, progress); return 0; } diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 5dbdfce2d05f..3efed86321db 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6938,8 +6938,9 @@ size_t memcpy_from_iter_csum(void *iter_from, size_t progress, size_t len, void *to, void *priv2) { __wsum *csum = priv2; + __wsum next = csum_partial_copy_nocheck(iter_from, to + progress, len); - *csum = csum_and_memcpy(to + progress, iter_from, len, *csum, progress); + *csum = csum_block_add(*csum, next, progress); return 0; } From patchwork Wed Sep 13 16:56:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383574 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 356DAEE01E0 for ; Wed, 13 Sep 2023 16:57:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 869726B0203; Wed, 13 Sep 2023 12:57:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 816A46B0204; Wed, 13 Sep 2023 12:57:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6652D6B0205; Wed, 13 Sep 2023 12:57:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4E3046B0203 for ; Wed, 13 Sep 2023 12:57:27 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2F61A80BA8 for ; Wed, 13 Sep 2023 16:57:27 +0000 (UTC) X-FDA: 81232180134.03.615A1A3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 742A018002B for ; Wed, 13 Sep 2023 16:57:25 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KAOSma6p; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624245; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OEnm3txfdKWMLGG8rrdFEcQBV2WWgkp8Kq4P7DhBB1I=; b=BrSgt+wgkjJ1FtWT9vuKKBW4xgmNtnQDDI55tAQ8/R7hgxh8p3lP6YAnQDWGfkvWCdHXZu ChdKKNX3Xlr2ahCL+s3YMKf8z9dVn70er0/R2lLtBE2nzTGdkQVf7Ta2TaP8cRv0y0e1lC N73vFh3eTjZLQPy/MWvJ91JbK9BIvZY= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KAOSma6p; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624245; a=rsa-sha256; cv=none; b=BEAFNAl/+nXPKB5J3PMVwzfZjFWQqs+mpmHhAGgBgkKa8fMRX7o392Hr8yv8M7p3AFYIGx nHrHR6RLRw/rYXqR2DBbTpvvUIzxD7EZisEDCLTJtguspGw7nSDU5ci6TPCFEUoZSTR/C3 q07vUdgUnde4svOae0XfjJ8/hgVjgwI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624244; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OEnm3txfdKWMLGG8rrdFEcQBV2WWgkp8Kq4P7DhBB1I=; b=KAOSma6pM5lmy2GZIiOA5owQ26thITzEZfnHcNb/4i97I4hymihXQtNu/E/wF0HRTzjd9p VVp8E8x7ZdLxERI9jRo3K82fLEHxiEyUG+7e19JuX9HH+a7nWbRSxHvfM9HSjhL6YN1zl9 eIYHzyFEKNO7Nv+6UwDmvWeC3LnHrec= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-60-yGDwbzOOMhetGn8REiVGKQ-1; Wed, 13 Sep 2023 12:57:21 -0400 X-MC-Unique: yGDwbzOOMhetGn8REiVGKQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 32F07185A79B; Wed, 13 Sep 2023 16:57:20 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2D8792026D4B; Wed, 13 Sep 2023 16:57:18 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Subject: [PATCH v4 11/13] iov_iter, net: Merge csum_and_copy_from_iter{,_full}() together Date: Wed, 13 Sep 2023 17:56:46 +0100 Message-ID: <20230913165648.2570623-12-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 742A018002B X-Stat-Signature: wrz5qyt3hfociiwykaxx9dcsqw7k5nqq X-Rspam-User: X-HE-Tag: 1694624245-724467 X-HE-Meta: U2FsdGVkX1/ew9Duu1MG8C1iLkgTQbDR0GKVFAIo4cl3eG82yFCV2oa9k8HZK9L/RDyxWHE6EJeCw3SOexDHsChtbkusy6TeRsfH2RZrM1RDTzllCXel4L9UFMdVTvCWeE0uDIPlNHFd2pPe+UQ5MZDXBwQTFQiOn1wgnOt2dRNiEP1RZP1S5+NVT5OC/AdA/O/f1TR7C528xSxv6gCmgwq4MwJc56T1HCtVp2q9kwogHHKr/GRy46h3wdJxx9Ak82xFO2xuR2U4r6pE/h5mU1+xtoGj7pq5hQ9ed1C+lsG9Lw9zzlrQ5jY0+PQ1JtUeaOOnYvE0XN/Sou+jUTjYQ/IXMth/Jj3S58z/gDG+aL2ys77qOFwFBVUaF1x94WTA7CPwoO3Y+a5BEB2rapDuQmvnnsA4d9JRr8EI1RQ26pLCzFcu1P4EYA8bmF8l7dErnyIlM8/xMzI79Jr4z8+pAnDtlM/MfDqxtNxrKFYtmgHQMwCNe5RSF1JnFQcGpMjoHp2js9b5IlAIHZIhBypmHN77jNF0/8m45/wL0rB2Ug8qBcjyuJjy+Xcf0S9ZXrg8z+Gu7BFB0INEDIEOm/xA++mEvukMVTdRJhAagMRbtHmNVquVueRmGsf46oRt7y2cJ0jwGv6AZJOTO60vvaMC3tNUWv4qNaSKzVAs1VjRM2fXPVCdpi1YPXNXZ/HMGjIF+B/Wpm1yuUh+o1lJYdTR+F90Fjfvj8RGwZ5wMhjgoWAVyL8APGI64fPh1BfeEBjMNBDfr4cgqORTi2SNRf3YGgG1TBF1YokVhoL5nlA7jLmISuej+lrRSjCk2eNrEuJ+PZAlMqiyvsMm7/VibOue9mWgDjOtyQ72SmaasTm0uCE8jC3sXK4ElVBwWDnUiH/UOHGJLVldGns2arwpDqR4mwEgMDtm+j3d9cacAkvNM9ZKOA8Y0E/O2oElYKKBQxXwrHOgyvQLaw41KcL9fpc Iv+DsP7R U1N7515pLP9urxTbWe2RHSlBtxRrAEX+/e7NLIpRKNVf9xzjmtapURLHAUhr3b2kTCRYkEl1KrJvYLPP6mVPMuVGTk3Zl7SCwvdRC2+A+sRek0sFjqlGonKrh7fNA8GivmF1Od0gUGRSgsptKARwgydlcP1Jq9uYe44s1ykM3Qzxi5SEOfMm+YVSvaJUKOsvfLhiqpCen/ZQrtYdRtK7Qd7iyJp5mYUpitZCdAu74Bwl4dxqvVxvM6r6nkWSOJuZg+EN0eEHh7cdWL61/+Fasj68PLYY1CCvJrHxRDu4CUa6Qq+fGzojkrelvHBtpeCTg/sgZRWuZS6WFkyGju45ySz8oufnwpyfVES2AECgVF//BRS42UYYTvZzI+G+hIGhhun+S9Vny2Yv5VH/Gu8oXUo1oQT6HGHbwmidsxbfzr0mkyi6/FboZn2y7UpSDPD80nQIYxB8qvQ5lHL1SpXwruaV3B/D3pjN7xiSFvYFp7i6k5nAdKQKfr7mBSK4JR7co4QVPDAP34XtV0hzQi87E6mj0ziVztLMgk6WPEUsBc85hMKKNetXRkLRbqrigDNZuKFJpexEURTRqdg6WGo5IZdJTK5rIl4dwdUoATL3v5BRmtxe2YexHTdowAP6VezLpuRH4876hfa60kYOWjT1vteys7Uu97dsrGI/C X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move csum_and_copy_from_iter_full() out of line and then merge csum_and_copy_from_iter() into its only caller. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- include/linux/skbuff.h | 19 ++----------------- net/core/datagram.c | 5 +++++ net/core/skbuff.c | 20 +++++++++++++------- 3 files changed, 20 insertions(+), 24 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index c81ef5d76953..be402f55f6d6 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3679,23 +3679,8 @@ static inline int __must_check skb_put_padto(struct sk_buff *skb, unsigned int l return __skb_put_padto(skb, len, true); } -struct csum_state { - __wsum csum; - size_t off; -}; - -size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i); - -static __always_inline __must_check -bool csum_and_copy_from_iter_full(void *addr, size_t bytes, - __wsum *csum, struct iov_iter *i) -{ - size_t copied = csum_and_copy_from_iter(addr, bytes, csum, i); - if (likely(copied == bytes)) - return true; - iov_iter_revert(i, copied); - return false; -} +bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) + __must_check; static inline int skb_add_data(struct sk_buff *skb, struct iov_iter *from, int copy) diff --git a/net/core/datagram.c b/net/core/datagram.c index 452620dd41e4..722311eeee18 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -738,6 +738,11 @@ size_t memcpy_to_iter_csum(void *iter_to, size_t progress, return 0; } +struct csum_state { + __wsum csum; + size_t off; +}; + static size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, struct iov_iter *i) { diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 3efed86321db..2bfa6a7ba244 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6955,13 +6955,19 @@ size_t copy_from_user_iter_csum(void __user *iter_from, size_t progress, return next ? 0 : len; } -size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, - struct iov_iter *i) +bool csum_and_copy_from_iter_full(void *addr, size_t bytes, + __wsum *csum, struct iov_iter *i) { + size_t copied; + if (WARN_ON_ONCE(!i->data_source)) - return 0; - return iterate_and_advance2(i, bytes, addr, csum, - copy_from_user_iter_csum, - memcpy_from_iter_csum); + return false; + copied = iterate_and_advance2(i, bytes, addr, csum, + copy_from_user_iter_csum, + memcpy_from_iter_csum); + if (likely(copied == bytes)) + return true; + iov_iter_revert(i, copied); + return false; } -EXPORT_SYMBOL(csum_and_copy_from_iter); +EXPORT_SYMBOL(csum_and_copy_from_iter_full); From patchwork Wed Sep 13 16:56:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A46DEE01EA for ; Wed, 13 Sep 2023 16:57:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 900676B0208; Wed, 13 Sep 2023 12:57:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8AE046B0209; Wed, 13 Sep 2023 12:57:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7281D6B020A; Wed, 13 Sep 2023 12:57:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5EAF86B0209 for ; Wed, 13 Sep 2023 12:57:33 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3F0551404B7 for ; Wed, 13 Sep 2023 16:57:33 +0000 (UTC) X-FDA: 81232180386.11.04FDA77 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 93584A0035 for ; Wed, 13 Sep 2023 16:57:31 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RFEDV2uE; spf=pass (imf15.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624251; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=c2st9mzpODW5EfnFYy73QjLkmO48eOaJEcOSoL9qczM=; b=jnHKpc/NLcFFOdHTojeM4LtUJRVyekT+YRC3ztCi9ahlaPvHUgmPokOsov2eUPqNk+acLT /J++MSaX5cNr5MStuXZJo2FRM83LTvjC/FAMlh/ClcJdAZlfGYmvyfq3gcmc5Rs9XceE4j lwjxTz82SztuZVefliMOqFrnDZ8VdgM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624251; a=rsa-sha256; cv=none; b=bz3R9IiQsvxLZrXQ0hDqzzBe3hOKDQbAqYY8DfCS9FRl7jFnQN0dUwgmtbLGtEhCNRuDkN k+HFf75YEZRxZYtlFTEZywedFOkaUazAUQtwR64N+Z12HzTM9kFfWELtqTvrjHwiRREjHm tMoUhN2ucKYkUYKYiqWeBEMJ1+HNbIs= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RFEDV2uE; spf=pass (imf15.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624251; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c2st9mzpODW5EfnFYy73QjLkmO48eOaJEcOSoL9qczM=; b=RFEDV2uE5/kKcKcdHW1SLKkp901HMexROSAMJldGFjba2YTZE6zPcePrnnoiq32PBnEACV c5CGoy67P9aBhWzXgsGDKxbUV5Utn5yV4znDhsrzcH2AJdAa4mx/+omM/cfrGrwLM78b5f LF6/WseT8hFwdcG9bTdIP9w4HyU1ZsU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-628-Z0pMlrrIOVKNFRYw1td4xA-1; Wed, 13 Sep 2023 12:57:23 -0400 X-MC-Unique: Z0pMlrrIOVKNFRYw1td4xA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D46F2101FAA0; Wed, 13 Sep 2023 16:57:22 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id CAFFF40C6EA8; Wed, 13 Sep 2023 16:57:20 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Subject: [PATCH v4 12/13] iov_iter, net: Move hash_and_copy_to_iter() to net/ Date: Wed, 13 Sep 2023 17:56:47 +0100 Message-ID: <20230913165648.2570623-13-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: 93584A0035 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 6r8zn5ikgi3yhmrap1kkrjn1fgdm9p66 X-HE-Tag: 1694624251-61474 X-HE-Meta: U2FsdGVkX18fJ5fLy7dFiaQx1dPBnAl7gw/2PkciYJUsbeT/iQRGHLa1YGIO0+eS+yDvlCm/zCy0av8Nz2tAEGdhah+CyZKV045tg+ZJBfTE0176Tz2khg6dgIzkU/Q+TXeLHz3UEczqlwqGV/y9V4LvYUEFqi+zgGOwCMZUo3kaz/Ei/ccVvJJtjen6rBvWpR+MArqsWQ+Cjjfn7YomvF5dQN5A3fFnWIFmh552l+a2rCKHcT/WD3C21J4ilTi55VcjNEDfqrIe8e7TVgmI195o+U6/u1OM9qDCJ+Qw/IKK4HnASy7n/c7PZXU2VbmTqNB1sbOfRc7ZpWaqpAXsEWfsuMOtcHkKvIzoMfN6FZt0LRI3BAo17G59AiqVQ79C5xYb7mCDpnHa3aVMB1wHqBNNq9iU1dnCs2VOCBtn45Y4qk/DZUm09ya32i0tqA2SiOp/Y9n40B+M4d0lFskWkd6IieR27L9ZPYRQytADSmoCFlX+T6jDKQ7Rqcjmsc0SorzrpCjVNOKUVitWWex7f2EHX1QogTzXkewwAXsy2hU3dDldeVKhjJIRV6mbsSdRqKac5Wyiyu2or5ZhNdFkUDxEG0LPFw0g868yLt5PlQdIOTrClPq/5PSq3Jm83XCA0Z8k9U6+ZOkpuzBgKd9S6PX9a3FqroqK24AN47XfFTmObMJfK+qQCHsE6TeFmRLrBSdIAh5tZxNmvhQoURRxka+TMjEBTni//JD0ZF0puiUPQMAUlTqMv0s7rXfBZzUxbCpnoig4I2fytegnhJcLzsE3Pc/wRCC6BnP2U/7SF3ZBQ1ysV9pppiQWX1BCoTyCD9mqY2hm9VlTLi+2uEqoMILwC0HACrPN/FPwTPUE9GMa3oj1RcCa7GGaBLuHF/YQ1SJRSsMilfYbU2AKrusHZaJbIvkU7Akv7KLFfCgQUCFzWlFdlJgvJYnfUToU1N56OpdzpDIGw2rxaKP8PgN i05oFj3Z h1DN+/xCgBrlfmzL2gm59h7ULQ08SbkAiSh0gX/zhZRl95R1W3EpatiwiitfZ831dCpA6qQ19LlDAWMhKEV0ui/q7TMkGtM4IfxsJYD0zovrr3lR4bnhi2cNtg707Pp/6klhHROazhVdSmYLnS3xYLW0hmgrXBSipagpKOvUvlplLjfCzg2sOdCHXj4rcu/oYcDQYHD1hIpZkkqRXouElcgDzKVOhnA7qKz0UpSrr59i1UkbdY7Vk98LG+0xYoynBdOu33hL8NIM0pnX7Ej0KRGYGQux1FEJLe/en1YsfIwPJe7UzwPcz3xJeQ84CBDnQ7NrVjbPlqNELRA+dp3qEPm4YWnvKm79nbZ+xQ/6e2E/v6xqLR1txIQ3hXxs5HX4pBzEzzLkH3NhPCdpAgTcOPQjLLDHH9MUq5vb+QpnILPsCNZGn/ZsRU6VNKHKS8TAqFIkHc/1O3YarqSFsGPz+ODrXMOC/J052zipeK21eGTIfMSxsQV8TrlXCiec7G96AHuUMi4iLh2jMyY87MvGDqK788IpjDAYOpr7GYr88YxNNSpGp+Q6hG2ECp8fK2aGe7L4J0hi3a16H4no4P3PzU8IH/hHMRNQf5eYCtTBHcDzzbwjTFSzDhldAT0UVXPADILP8IKLhi+1BpzUFkJWPewuEIJmEx862NDMo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move hash_and_copy_to_iter() to be with its only caller in networking code. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- include/linux/uio.h | 3 --- lib/iov_iter.c | 20 -------------------- net/core/datagram.c | 19 +++++++++++++++++++ 3 files changed, 19 insertions(+), 23 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index c335b95626af..6bb00645fe46 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -326,9 +326,6 @@ iov_iter_npages_cap(struct iov_iter *i, int maxpages, size_t max_bytes) return npages; } -size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, - struct iov_iter *i); - struct iovec *iovec_from_user(const struct iovec __user *uvector, unsigned long nr_segs, unsigned long fast_segs, struct iovec *fast_iov, bool compat); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 5b2d053f057f..77520834ebe4 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1,5 +1,4 @@ // SPDX-License-Identifier: GPL-2.0-only -#include #include #include #include @@ -1071,25 +1070,6 @@ ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, } EXPORT_SYMBOL(iov_iter_get_pages_alloc2); -size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, - struct iov_iter *i) -{ -#ifdef CONFIG_CRYPTO_HASH - struct ahash_request *hash = hashp; - struct scatterlist sg; - size_t copied; - - copied = copy_to_iter(addr, bytes, i); - sg_init_one(&sg, addr, copied); - ahash_request_set_crypt(hash, &sg, NULL, copied); - crypto_ahash_update(hash); - return copied; -#else - return 0; -#endif -} -EXPORT_SYMBOL(hash_and_copy_to_iter); - static int iov_npages(const struct iov_iter *i, int maxpages) { size_t skip = i->iov_offset, size = i->count; diff --git a/net/core/datagram.c b/net/core/datagram.c index 722311eeee18..103d46fa0eeb 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -61,6 +61,7 @@ #include #include #include +#include /* * Is a socket 'connection oriented' ? @@ -489,6 +490,24 @@ static int __skb_datagram_iter(const struct sk_buff *skb, int offset, return 0; } +static size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, + struct iov_iter *i) +{ +#ifdef CONFIG_CRYPTO_HASH + struct ahash_request *hash = hashp; + struct scatterlist sg; + size_t copied; + + copied = copy_to_iter(addr, bytes, i); + sg_init_one(&sg, addr, copied); + ahash_request_set_crypt(hash, &sg, NULL, copied); + crypto_ahash_update(hash); + return copied; +#else + return 0; +#endif +} + /** * skb_copy_and_hash_datagram_iter - Copy datagram to an iovec iterator * and update a hash. From patchwork Wed Sep 13 16:56:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13383576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92BA1EE01E9 for ; Wed, 13 Sep 2023 16:57:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A89E6B0207; Wed, 13 Sep 2023 12:57:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3317E6B0208; Wed, 13 Sep 2023 12:57:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1831B6B0209; Wed, 13 Sep 2023 12:57:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 044856B0207 for ; Wed, 13 Sep 2023 12:57:33 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C45411404B7 for ; Wed, 13 Sep 2023 16:57:32 +0000 (UTC) X-FDA: 81232180344.13.6495DDE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 0EDE5C000E for ; Wed, 13 Sep 2023 16:57:30 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HyYyDGQ5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694624251; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jTW4hj6R6SH2WgBE9Xhmo5Z05cFVRaLxhGn+V3gLo/A=; b=LXsqI9ogvGC76iwcWqQ+67nsqHkRh7qjGkbpXCVbmA29xy0+3/ke8Q66RSV231l/qly1yf IgQOfXODUrekmtcgTyYR8ylCOC4+8XAbQP3dJEk+Z2r08R02o3SJHlzRbZy1VgRQvAKkSw FwgeGlMzGC/Ag3qVVhF30vQ3XJ7Sk7g= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HyYyDGQ5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of dhowells@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694624251; a=rsa-sha256; cv=none; b=leWlUJlySVmg3OohT7lzkXyxF+SPuodeSWX5sFYsgmftgpybGwLFo9+RHqfkLm0apLwCRS Hk3cVl9n2onLv9F6NDFHdjBwNmWXh7/Mwlwam8hW5ILE7F1w3RPphxoLyyXgVoJHzVXeci LWAkXgMKR5LdGipS3t4pFPVJzUKbb5w= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624250; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jTW4hj6R6SH2WgBE9Xhmo5Z05cFVRaLxhGn+V3gLo/A=; b=HyYyDGQ5+crGYtT2e4BA+No/lpFhxIcPzQ57idYmn6PZxoZx3CZBjMaC+nl3vTWz4wXxiz 4dwxQeN+tVtJvHhBiFplrnh21kWeHat6QoJDIgGQNevcMkrTlwR/NKBznW4RLDyQNM5bus JZCH/y3oowQa5deYcKHGtssGnu07wws= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-583-kw6qakgRMmKfBnOo7mji8A-1; Wed, 13 Sep 2023 12:57:27 -0400 X-MC-Unique: kw6qakgRMmKfBnOo7mji8A-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0D8F6855321; Wed, 13 Sep 2023 16:57:26 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id C823740C6EA8; Wed, 13 Sep 2023 16:57:23 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Subject: [PATCH v4 13/13] iov_iter: Create a fake device to allow iov_iter testing/benchmarking Date: Wed, 13 Sep 2023 17:56:48 +0100 Message-ID: <20230913165648.2570623-14-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Rspamd-Queue-Id: 0EDE5C000E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: m4ud48etarwqse5tendfd5dje9woyo7u X-HE-Tag: 1694624250-445530 X-HE-Meta: U2FsdGVkX18kQKHuv4+tHElED/dgDGaPEdcE8VkNcx9vmu1Epzye4ptRE/voBWrbl1OolV0Vn7KcgpCjeSEfK7iSptgtQmHy0oIs7+oZWsXqjhpoxYMfqxUoJ/1M7UMUwUSxRidBfIJvHcXdi6SLyrGgENK/A2J+sWw/6ZurdTDWoojX0oVr1SC5Jzc8TktY9BIwCKGo+oYQDt/Z0LGxdlwwaIJ5OKNy9SGDj0CY7iN87S3LYKWf3bPmnXY6O/B0E4Oz62QXTeMH0D9Pu4SSnEWPMgFLBOHfkkxqLudhGoP3OBCWRa35SoPs67kY0e06j/+8tXKCldIqy7f4rMKXw25GggXR9pP9cqsQ5xlv8s8ykzm2imDL13+uIlKfL0Oo1Rd/yz0zfjg3+hEESX90gR+uTAessnZfl0MFgOsGn/IeVXr/9Q2MpepvBWRW5YOHlrMx35k6nCJ6fHY2np05PE2MqtHZrlk5UhTHQdpF5E4VTdsAaL1a2z7OaYoN6ChZiLMwb9fs9/M6hfUpjNozRuY+H/dxD8GGu+/Iz+IxopG5DUvExMbyIW7L0aIr7n/cnb0/ZFtFXyn5/eYliKgxlvzhjfIQmGdU6fwvTRiTsLE5ZpccGSGRLoqQA6wCPjtfrpWLE+LN25LWZb6lZ2tzbbohzLytnRqK1o946/uNBiXi8xWQBbufuI9Uxb9bCvDYe22A3atjcbVNRJTFubALaFyn3x3MMcHBDsn+tL/uCsWJ1Tmhb0eGIBfdSnWYJoXb2dYbfhZx1tQ/OeB9lH8+ag3xT8+hVd6ogWAwuSi1hL+BJ/wOI+YmnGE6asJuVdIBsikTp2rg1T7QMZvYT4qYNX605QI1gzynDzCaKbw/JdM7OHYy8bTta2V6PRT9N9782CebxJXeXbnPfxX4Zr1azc9J+0Y7nuUW9TXKscqSB1t+KSB8QYOnW58g9G8+CHT1RQxGYl7sHt2k/ShpGBP aKADcoZ5 5g2/jhy4ykefT+fOBK3O3GzsL9XnxEzlSkohcILLD8ekWu5Ma7v5RnNRAV3gODBCsYHTpvJuJq6B9FjYZCIi6fbT3FhFc/BAy1CDKmIGXhDQoq8kgPaIQmVVC+lDl51I6AYP4oBBCpJXotWILVJOCSiJLILGRIF23hw/Ia38lm8mV7Cob3Bt2wQXynCv0vfpWcSUgXscw5QrIMp8LpfFdUOAmvLfsUUNef+bWINSSLltDkxezceWbJ5I7Y6RQwSfVtZUkDaorL6fXP2q77Vbhq0+UeKEhaJAOga7Wxkl+kXV3oztmjLSAtV0oYJ768ntjBnYjBiLklVF3OES33GL60i7v/Gp3sPx17S54XPTvhlk8KdttbRAZKHqiuigeU3KtEW2/W1myFkB/O1yLrMwh8SxHZNJhqkL7J15AA/zMRpQhcOhSteHdQBM2LxPRgMyTtCKPdD4OWxYiZL+niwJJqwMMdGkpob7msEXJwFZVtxoEn5PdYY9yfMPTkFmC2EaFPoDVnJVgtsmOsoFCv90V1njAvkbLst/SYDHzwi0ia4itTy3ndTqMQWHby48ag4aXWaJxtmMdjSsLLYyQuY8SdNm6PFixj/k2vEO+b+KSMuFkp8vUY8bB9jxbZIvIL03K3BDuXmX8SqU56XpKBiSjnxWFqHqnnv0e/Q9Ejm1EspbQ6BAyulPXsGQlTA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Create a fake device to allow testing and benchmarking of UBUF and IOVEC iterators. /dev/iov-test is created and can be driven with pwritev() in which case it copies everything to a sink page and it can be written with preadv() in which case it copies repeatedly from a patterned page. The time taken is logged with tracepoints. This can be driven by something like: echo 1 >/sys/kernel/debug/tracing/events/iov_test/enable cmd="r -b 1M -V 256 0 256M"; xfs_io -c "open /dev/iov-test" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" cmd="w -b 1M -V 256 0 256M"; xfs_io -c "open /dev/iov-test" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" showing something like: ...: iov_test_read: size=10000000 done=10000000 ty=1 nr=256 dur=27653 ...: iov_test_write: size=10000000 done=10000000 ty=1 nr=256 dur=31792 in the trace log. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- lib/Kconfig.debug | 8 +++ lib/Makefile | 1 + lib/test_iov_iter.c | 134 ++++++++++++++++++++++++++++++++++++++ lib/test_iov_iter_trace.h | 80 +++++++++++++++++++++++ 4 files changed, 223 insertions(+) create mode 100644 lib/test_iov_iter.c create mode 100644 lib/test_iov_iter_trace.h diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index fa307f93fa2e..cf8392c51344 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2524,6 +2524,14 @@ config TEST_SYSCTL If unsure, say N. +config TEST_IOV_ITER + tristate "iov_iter test driver" + help + This creates a misc device that can be used as a way to test various + I/O iterator functions through the use of readv/writev and ioctl. + + If unsure, say N. + config BITFIELD_KUNIT tristate "KUnit test bitfield functions at runtime" if !KUNIT_ALL_TESTS depends on KUNIT diff --git a/lib/Makefile b/lib/Makefile index 740109b6e2c8..f6419544a749 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -65,6 +65,7 @@ CFLAGS_test_bitops.o += -Werror obj-$(CONFIG_CPUMASK_KUNIT_TEST) += cpumask_kunit.o obj-$(CONFIG_TEST_SYSCTL) += test_sysctl.o obj-$(CONFIG_TEST_IOV_ITER) += kunit_iov_iter.o +obj-$(CONFIG_TEST_IOV_ITER) += test_iov_iter.o obj-$(CONFIG_HASH_KUNIT_TEST) += test_hash.o obj-$(CONFIG_TEST_IDA) += test_ida.o obj-$(CONFIG_TEST_UBSAN) += test_ubsan.o diff --git a/lib/test_iov_iter.c b/lib/test_iov_iter.c new file mode 100644 index 000000000000..afa70647dbde --- /dev/null +++ b/lib/test_iov_iter.c @@ -0,0 +1,134 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* I/O iterator testing device. + * + * Copyright (C) 2023 Red Hat, Inc. All Rights Reserved. + * Written by David Howells (dhowells@redhat.com) + */ + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include +#include +#include +#define CREATE_TRACE_POINTS +#include "test_iov_iter_trace.h" + +MODULE_DESCRIPTION("iov_iter testing"); +MODULE_AUTHOR("David Howells "); +MODULE_LICENSE("GPL"); + +static ssize_t iov_test_read_iter(struct kiocb *iocb, struct iov_iter *iter) +{ + struct folio *folio = iocb->ki_filp->private_data; + unsigned int nr_segs = iter->nr_segs; + size_t size = iov_iter_count(iter), fsize = folio_size(folio); + size_t copied = 0, offset = 0, i; + ktime_t a, b; + u8 *p; + + /* Pattern the buffer */ + p = kmap_local_folio(folio, 0); + for (i = 0; i < folio_size(folio); i++) + p[i] = i & 0xff; + kunmap_local(p); + + a = ktime_get_real(); + while (iov_iter_count(iter)) { + size_t done, part = min(iov_iter_count(iter), fsize - offset); + + done = copy_folio_to_iter(folio, offset, part, iter); + if (done == 0) + break; + copied += done; + offset = (offset + done) & (fsize - 1); + } + + b = ktime_get_real(); + trace_iov_test_read(size, copied, iov_iter_type(iter), nr_segs, + ktime_to_us(ktime_sub(b, a))); + return copied; +} + +static ssize_t iov_test_write_iter(struct kiocb *iocb, struct iov_iter *iter) +{ + struct folio *folio = iocb->ki_filp->private_data; + unsigned int nr_segs = iter->nr_segs; + size_t size = iov_iter_count(iter), fsize = folio_size(folio); + size_t copied = 0, offset = 0; + ktime_t a = ktime_get_real(), b; + + while (iov_iter_count(iter)) { + size_t done, part = min(iov_iter_count(iter), fsize - offset); + + done = copy_page_from_iter(folio_page(folio, 0), offset, part, iter); + if (done == 0) + break; + copied += done; + offset = (offset + done) & (fsize - 1); + } + + b = ktime_get_real(); + trace_iov_test_write(size, copied, iov_iter_type(iter), nr_segs, + ktime_to_us(ktime_sub(b, a))); + return copied; +} + +static int iov_test_open(struct inode *inode, struct file *file) +{ + struct folio *folio; + + if (!capable(CAP_SYS_ADMIN)) + return -EPERM; + + folio = folio_alloc(GFP_KERNEL, 0); + if (!folio) + return -ENOMEM; + file->private_data = folio; + return 0; +} + +static int iov_test_release(struct inode *inode, struct file *file) +{ + struct folio *folio = file->private_data; + + folio_put(folio); + return 0; +} + +static const struct file_operations iov_test_fops = { + .owner = THIS_MODULE, + .open = iov_test_open, + .release = iov_test_release, + .read_iter = iov_test_read_iter, + .write_iter = iov_test_write_iter, + .splice_read = copy_splice_read, + .llseek = noop_llseek, +}; + +static struct miscdevice iov_test_dev = { + .minor = MISC_DYNAMIC_MINOR, + .name = "iov-test", + .fops = &iov_test_fops, +}; + +static int __init iov_test_init(void) +{ + int ret; + + ret = misc_register(&iov_test_dev); + if (ret < 0) + return ret; + pr_info("Loaded\n"); + return 0; +} +module_init(iov_test_init); + +static void __exit iov_test_exit(void) +{ + pr_info("Unloading\n"); + + misc_deregister(&iov_test_dev); +} +module_exit(iov_test_exit); diff --git a/lib/test_iov_iter_trace.h b/lib/test_iov_iter_trace.h new file mode 100644 index 000000000000..b99cade5d004 --- /dev/null +++ b/lib/test_iov_iter_trace.h @@ -0,0 +1,80 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* I/O iterator testing device. + * + * Copyright (C) 2023 Red Hat, Inc. All Rights Reserved. + * Written by David Howells (dhowells@redhat.com) + */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM iov_test + +#if !defined(_IOV_TEST_TRACE_H) || defined(TRACE_HEADER_MULTI_READ) +#define _IOV_TEST_TRACE_H + +#include + +TRACE_EVENT(iov_test_read, + TP_PROTO(size_t size, size_t done, enum iter_type type, + unsigned int nr_segs, u64 duration), + + TP_ARGS(size, done, type, nr_segs, duration), + + TP_STRUCT__entry( + __field(size_t, size) + __field(size_t, done) + __field(enum iter_type, type) + __field(unsigned int, nr_segs) + __field(u64, duration) + ), + + TP_fast_assign( + __entry->size = size; + __entry->done = done; + __entry->type = type; + __entry->nr_segs = nr_segs; + __entry->duration = duration; + ), + + TP_printk("size=%zx done=%zx ty=%u nr=%u dur=%llu", + __entry->size, + __entry->done, + __entry->type, + __entry->nr_segs, + __entry->duration) + ); + +TRACE_EVENT(iov_test_write, + TP_PROTO(size_t size, size_t done, enum iter_type type, + unsigned int nr_segs, u64 duration), + + TP_ARGS(size, done, type, nr_segs, duration), + + TP_STRUCT__entry( + __field(size_t, size) + __field(size_t, done) + __field(enum iter_type, type) + __field(unsigned int, nr_segs) + __field(u64, duration) + ), + + TP_fast_assign( + __entry->size = size; + __entry->done = done; + __entry->type = type; + __entry->nr_segs = nr_segs; + __entry->duration = duration; + ), + + TP_printk("size=%zx done=%zx ty=%u nr=%u dur=%llu", + __entry->size, + __entry->done, + __entry->type, + __entry->nr_segs, + __entry->duration) + ); + +#endif /* _IOV_TEST_TRACE_H */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH . +#define TRACE_INCLUDE_FILE test_iov_iter_trace +#include