From patchwork Wed Sep 20 13:03:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 13392747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35200CE79D1 for ; Wed, 20 Sep 2023 13:05:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B78EE6B0162; Wed, 20 Sep 2023 09:05:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AFDBE6B0163; Wed, 20 Sep 2023 09:05:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 928F96B0164; Wed, 20 Sep 2023 09:05:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7B3726B0162 for ; Wed, 20 Sep 2023 09:05:22 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 30FF31A080D for ; Wed, 20 Sep 2023 13:05:22 +0000 (UTC) X-FDA: 81256996884.04.34736F3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id EFECD180007 for ; Wed, 20 Sep 2023 13:05:19 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=crim0Bd7; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695215120; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YJCqPShPuxX0OSXQWhEKRturENKmTo+NpvJVS0wCU18=; b=pm7WapzQPzOcTCYAvt7w7Bo/+aV+/ut8fx8LoY3SD0UX9tmflzG+5DzJ7WO/nDQ92JW4B0 46Srm31erVXLLyFX/7575lswFMit98Pj+4eVyzfJrFOh17aLu6Zma3vPekGg8/qibzsFP2 IaSGoAI3hd0C10pYYu2/XYEEWbd8lRY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695215120; a=rsa-sha256; cv=none; b=5j4LuYPVbS6/+qItwD4G9TaIL6W+e3XlcbkbjL7+bLaThxBQrwIZOECXIWRj2qQX5Bl+h7 yFfnZh1ZRemB/eSrTsYdaw6QuVIXc06ZPbSGDvyNGmaztcIF4C4MkklisynOfoHTRG3oLJ 4zEN0GyPl3AqceDhgosSGVa+0Y9+r6U= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=crim0Bd7; spf=pass (imf06.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695215119; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YJCqPShPuxX0OSXQWhEKRturENKmTo+NpvJVS0wCU18=; b=crim0Bd7q8XcIih6T6ycIY4Fz2F3YwluOwN0NWyjxSfJLOZjKCCczc/yN+X7IDxaXetyAk /TZIuVtQkyLHr4UJNgI16CRYjG70bg2gT/f314SyQxvNrya08PxvW8gCRyf/RQWLNHftal OhCNKs9Ee4maBcpEIgptc/BQpAMpVR0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-688-aNGlfMQdOTe2VK1zYWIPzQ-1; Wed, 20 Sep 2023 09:05:14 -0400 X-MC-Unique: aNGlfMQdOTe2VK1zYWIPzQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C3D67858F19; Wed, 20 Sep 2023 13:05:13 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7BAA02156701; Wed, 20 Sep 2023 13:05:11 +0000 (UTC) From: David Howells To: Jens Axboe Cc: David Howells , Al Viro , Linus Torvalds , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Brendan Higgins , David Gow , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, Christian Brauner , David Hildenbrand , John Hubbard Subject: [RFC PATCH v2 8/9] iov_iter: Add benchmarking kunit tests Date: Wed, 20 Sep 2023 14:03:59 +0100 Message-ID: <20230920130400.203330-9-dhowells@redhat.com> In-Reply-To: <20230920130400.203330-1-dhowells@redhat.com> References: <20230920130400.203330-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 X-Stat-Signature: 95gk8be9m4ksjrsom8i9pgt3ua66uqh4 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: EFECD180007 X-Rspam-User: X-HE-Tag: 1695215119-527137 X-HE-Meta: U2FsdGVkX18NBoG/iMJ0NBVrHFUHDL0XBESWDgaDjvOLPu2nrl8uP+bsCtUsuahxJMS43UGuTq3MlsaoTYQZOVBN9qk2BcKOmDl4hTTCDJNFi67kH1moPjAT8mN9Lwky9Hmt2EcKhUYlDl+PzNJfDoxwjvuuo24snKG2XFJbGJpNvkdHrV+LnthiXu1J1usWbd3rWWbio0o+VtTIehFW1FMlhwA6Bo4tWhII3513hnqpXWYBk/UnfZo4fz5Y4JfxAEf9FSXPm8yS80iF5clP0i7nBQhU4PgjX3iRa12u+J4fQJHUukIfPrgwOr4V0KB1DAHZ4VQWlUZVARBpCdXpSuU9lkMTGEBU/75VLf6tq8hnOLGTNiHh0p1sbhszd9rr1sxytqy9br12cGc4QXITn7HeMH8VrmacNPo1QqmfN+4eLfnTR2TGHpG0T0RJCJe4Yfd4yY4A1SDyheFdjdSwuhhck0CbcYNri1Gd7TxQ1b6QsTJFB2SbM0fTSNHPL706frDHd+MkR9t3Np7AGi3pqJUtiLtrdv7WxpmYGcBR3hT/OJF5X3TxKakz67Xo1587muvLsCB1noscw3ZiyTtpUW1i3CvoW5gZrIYxj7ILd4vLEMiWooXBLahY7Klp/hQM0rlPRveHmBsoS1qTpvYs5dN/sNUsas0jWw0iBuvHdKA5snEls+bRVDrAU4T7nB/nIFOp98Qr8O8j4LgFhQs/y7Y00UpdE3P1qlp4YaEcK/w7+q/5fyROv2ZaGRNRVwV3cB/Y+F/6RwyMvo+rPf2MYMdsWR9VXzbpiLCITZCyROhW0r5iIMpxhwRVvR+uHP3ylhWDy1YIr9dC6RxxoI8xPUrGKUUE/ORlvYoz5lQuDHCAVSrLw4buMemwTTiBytEat57p6x9TYcfC6GWJqkpo1Q2H+iud5sB220Y16m4aodXU11JNc96/mNnBxUDJEYUHZHCsxxCGPVqJQpOV+/W 3BAymV0m N7juhBh2Kf/I6JoHdnkRrZdSHod6nY8CeuyLBrP12gzZJdV97i4O7Er2+o9XN9OiQ4xALnZc0mMR1VSw6T2gUXleXDLXz2sxvPqnQJ+lkoAGPISwPp45lIo3wSdeahy5K/Pl17Z6ppVCJ4Khf9VLfsx4/tS37K1Or4WGtt4n2MCkbc/q3ZSmGZSc0YTIVeH6rO93dACu+a/d3B1MmYgUnCVmADE1XepMKpXLNAKqySwraqqz2+wXUdx9DHiBFUmOc6REymhf9xMZ78uM/KU7nqyAOJNWujOvgTV1QABu9eZfj/wuyFdap4B2MRhqfiaw3ugazcoPZu10YCkvC2SLTLsXVGbQfUqyeRowUoYYky/nBGZz0XNcaRUydcyi/jKGzrmI5b3egbC9F/+GQ3SVYDM248x65cURAO97mJaaUx+xdCbAmpRXgeeUoyazqKtwB1xwXPx4Dnj/zxD71mCM2y5y7lLpT4waoKej+FqIBXS68//7fP8cXKzHTgbNOfITDDfAl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add kunit tests to benchmark 256MiB copies to a KVEC iterator, a BVEC iterator, an XARRAY iterator and to a loop that allocates 256-page BVECs and fills them in (similar to a maximal bio struct being set up). Signed-off-by: David Howells cc: Christoph Hellwig cc: Christian Brauner cc: Jens Axboe cc: Al Viro cc: David Hildenbrand cc: John Hubbard cc: Brendan Higgins cc: David Gow cc: linux-kselftest@vger.kernel.org cc: kunit-dev@googlegroups.com cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- lib/kunit_iov_iter.c | 251 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 251 insertions(+) diff --git a/lib/kunit_iov_iter.c b/lib/kunit_iov_iter.c index 2994c3f348ab..17f85f24b239 100644 --- a/lib/kunit_iov_iter.c +++ b/lib/kunit_iov_iter.c @@ -1261,6 +1261,253 @@ static void __init iov_kunit_extract_pages_xarray(struct kunit *test) KUNIT_SUCCEED(); } +static void iov_kunit_free_page(void *data) +{ + __free_page(data); +} + +#define IOV_KUNIT_NR_SAMPLES 16 +static void __init iov_kunit_benchmark_print_stats(struct kunit *test, + unsigned int *samples) +{ + unsigned long long sumsq = 0; + unsigned long total = 0, mean, stddev; + unsigned int n = IOV_KUNIT_NR_SAMPLES; + int i; + + //for (i = 0; i < n; i++) + // kunit_info(test, "run %x: %u uS\n", i, samples[i]); + + /* Ignore the 0th sample as that may include extra overhead such as + * setting up PTEs. + */ + samples++; + n--; + for (i = 0; i < n; i++) + total += samples[i]; + mean = total / n; + + for (i = 0; i < n; i++) { + long s = samples[i] - mean; + + sumsq += s * s; + } + stddev = int_sqrt64(sumsq); + + kunit_info(test, "avg %lu uS, stddev %lu uS\n", mean, stddev); +} + +/* + * Create a source buffer for benchmarking. + */ +static void *__init iov_kunit_create_source(struct kunit *test, size_t npages) +{ + struct page *page, **pages; + void *scratch; + size_t i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + pages = kunit_kmalloc_array(test, npages, sizeof(pages[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pages); + for (i = 0; i < npages; i++) { + pages[i] = page; + get_page(page); + } + + scratch = vmap(pages, npages, VM_MAP | VM_MAP_PUT_PAGES, PAGE_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, scratch); + kunit_add_action_or_reset(test, iov_kunit_unmap, scratch); + return scratch; +} + +/* + * Time copying 256MiB through an ITER_KVEC. + */ +static void __init iov_kunit_benchmark_kvec(struct kunit *test) +{ + struct iov_iter iter; + struct kvec kvec[8]; + unsigned int samples[IOV_KUNIT_NR_SAMPLES]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE, part; + void *scratch, *buffer; + int i; + + /* Allocate a huge buffer and populate it with pages. */ + buffer = iov_kunit_create_source(test, npages); + + /* Create a single large buffer to copy to/from. */ + scratch = iov_kunit_create_source(test, npages); + + /* Split the target over a number of kvecs */ + copied = 0; + for (i = 0; i < ARRAY_SIZE(kvec); i++) { + part = size / ARRAY_SIZE(kvec); + kvec[i].iov_base = buffer + copied; + kvec[i].iov_len = part; + copied += part; + } + kvec[i - 1].iov_len += size - part; + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over KVEC:\n"); + for (i = 0; i < IOV_KUNIT_NR_SAMPLES; i++) { + iov_iter_kvec(&iter, ITER_SOURCE, kvec, ARRAY_SIZE(kvec), size); + + a = ktime_get_real(); + copied = copy_from_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_BVEC. + */ +static void __init iov_kunit_benchmark_bvec(struct kunit *test) +{ + struct iov_iter iter; + struct bio_vec *bvec; + struct page *page; + unsigned int samples[IOV_KUNIT_NR_SAMPLES]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE; + void *scratch; + int i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + bvec = kunit_kmalloc_array(test, npages, sizeof(bvec[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, bvec); + for (i = 0; i < npages; i++) + bvec_set_page(&bvec[i], page, PAGE_SIZE, 0); + + /* Create a single large buffer to copy to/from. */ + scratch = iov_kunit_create_source(test, npages); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over BVEC:\n"); + for (i = 0; i < IOV_KUNIT_NR_SAMPLES; i++) { + iov_iter_bvec(&iter, ITER_SOURCE, bvec, npages, size); + a = ktime_get_real(); + copied = copy_from_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_BVEC in 256 page chunks. + */ +static void __init iov_kunit_benchmark_bvec_split(struct kunit *test) +{ + struct iov_iter iter; + struct bio_vec *bvec; + struct page *page; + unsigned int samples[IOV_KUNIT_NR_SAMPLES]; + ktime_t a, b; + ssize_t copied; + size_t size, npages = 64; + void *scratch; + int i, j; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + /* Create a single large buffer to copy to/from. */ + scratch = iov_kunit_create_source(test, npages); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over BVEC:\n"); + for (i = 0; i < IOV_KUNIT_NR_SAMPLES; i++) { + size = 256 * 1024 * 1024; + a = ktime_get_real(); + do { + size_t part = min_t(size_t, size, npages * PAGE_SIZE); + + bvec = kunit_kmalloc_array(test, npages, sizeof(bvec[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, bvec); + for (j = 0; j < npages; j++) + bvec_set_page(&bvec[j], page, PAGE_SIZE, 0); + + iov_iter_bvec(&iter, ITER_SOURCE, bvec, npages, part); + copied = copy_from_iter(scratch, part, &iter); + KUNIT_EXPECT_EQ(test, copied, part); + size -= part; + } while (size > 0); + b = ktime_get_real(); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_XARRAY. + */ +static void __init iov_kunit_benchmark_xarray(struct kunit *test) +{ + struct iov_iter iter; + struct xarray *xarray; + struct page *page; + unsigned int samples[IOV_KUNIT_NR_SAMPLES]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE; + void *scratch; + int i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + xarray = iov_kunit_create_xarray(test); + + for (i = 0; i < npages; i++) { + void *x = xa_store(xarray, i, page, GFP_KERNEL); + + KUNIT_ASSERT_FALSE(test, xa_is_err(x)); + } + + /* Create a single large buffer to copy to/from. */ + scratch = iov_kunit_create_source(test, npages); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over XARRAY:\n"); + for (i = 0; i < IOV_KUNIT_NR_SAMPLES; i++) { + iov_iter_xarray(&iter, ITER_SOURCE, xarray, 0, size); + a = ktime_get_real(); + copied = copy_from_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + static struct kunit_case __refdata iov_kunit_cases[] = { KUNIT_CASE(iov_kunit_copy_to_ubuf), KUNIT_CASE(iov_kunit_copy_from_ubuf), @@ -1277,6 +1524,10 @@ static struct kunit_case __refdata iov_kunit_cases[] = { KUNIT_CASE(iov_kunit_extract_pages_kvec), KUNIT_CASE(iov_kunit_extract_pages_bvec), KUNIT_CASE(iov_kunit_extract_pages_xarray), + KUNIT_CASE(iov_kunit_benchmark_kvec), + KUNIT_CASE(iov_kunit_benchmark_bvec), + KUNIT_CASE(iov_kunit_benchmark_bvec_split), + KUNIT_CASE(iov_kunit_benchmark_xarray), {} };