From patchwork Wed Oct 5 18:03:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 13000009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7F7ACC4332F for ; Thu, 6 Oct 2022 07:13:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665040419; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=bHWr/xzgpDmMRYVu+iR1IHBtK0ArPzYzTxToWTgiufk=; b=Pv9FVo+KgdU3w5rntHfO3QbSp+oS3dg02hXysn9LulhKpZ36DqKVYe8ogwyZE3zXCGGOdG d4tY+hUp4h+8ajLHa1Wg+b9PpjO7ds+FQaHyjdTWUZ0dOBlVPQ1oKDKI9Z/bhdMKJAX2sf lJMZ0iEsR5MEdh1Td9sMzKERG9LOP6k= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-581-HU1vnP3PNM6M_kwybU2Sew-1; Thu, 06 Oct 2022 03:13:36 -0400 X-MC-Unique: HU1vnP3PNM6M_kwybU2Sew-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 31A8229324A6; Thu, 6 Oct 2022 07:13:34 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (unknown [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 18CCD2024CC5; Thu, 6 Oct 2022 07:13:34 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 2E3601947BA9; Thu, 6 Oct 2022 07:13:33 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id CB6071946A4E for ; Wed, 5 Oct 2022 18:04:07 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id B13AF4B3FED; Wed, 5 Oct 2022 18:04:07 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast02.extmail.prod.ext.rdu2.redhat.com [10.11.55.18]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AA8934B3FE0 for ; Wed, 5 Oct 2022 18:04:07 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E76DE918875 for ; Wed, 5 Oct 2022 18:04:01 +0000 (UTC) Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-287-vZZ9wZGhNRuWs464zVmikQ-1; Wed, 05 Oct 2022 14:03:53 -0400 X-MC-Unique: vZZ9wZGhNRuWs464zVmikQ-1 Received: by mail-pj1-f43.google.com with SMTP id n18-20020a17090ade9200b0020b0012097cso278099pjv.0; Wed, 05 Oct 2022 11:03:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IKwMLpWApD5aFrIIIf8CGQFhqbcknhmLeA7p0rloKo4=; b=fCk+LASNd2t8EK1EOK8Dx4MZ+/H2pNDEfYq5zOExLCPmcWAHNaCCb4d1nrglzGleQy QJtjp43A9pVnM2M7zLhXTo/hPkWiww0LzHDaa6sN9PBoZNGgL2Zl+jk73DxRQ5Ee280r y07QDVrGVlgs90tMpdwgL/UHB/ZMXgf8VkrGCNeKcRQaJulO28I6xUJADYqvV/NphJEB nHfJgxA5a25aSrV22USPQBGJGU64BXwO43LC2eBwIumsyDB8GA8lHm4X9PyuFXUl0UY2 hJA8t3dABPvodoKJrq6uZPOWB44MV9mAo3uq+dQBxrvUjDMl92aM/aLHxOPDHGW4SjrH qTFg== X-Gm-Message-State: ACrzQf0TrdVCgdz04PBwGKgXhzbKwDOzh9zXFYli4c0Qq60PwVodOJDK FNxo7NRf1dGYzNCazyr1LrQow8oxbyk= X-Google-Smtp-Source: AMsMyM53gfS50h/TX2/GWR7BIW/H3B+J8W4P92wIc0AYeapZJ8wWdOf6715h39n5ZlkC6HAcQe53Tg== X-Received: by 2002:a17:90b:188b:b0:20a:8fc8:60b8 with SMTP id mn11-20020a17090b188b00b0020a8fc860b8mr6326296pjb.37.1664993032035; Wed, 05 Oct 2022 11:03:52 -0700 (PDT) Received: from localhost.localdomain (c-67-174-241-145.hsd1.ca.comcast.net. [67.174.241.145]) by smtp.gmail.com with ESMTPSA id y17-20020a170903011100b001788494b764sm10674639plc.231.2022.10.05.11.03.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Oct 2022 11:03:51 -0700 (PDT) From: Yang Shi To: mgorman@techsingularity.net, agk@redhat.com, snitzer@kernel.org, dm-devel@redhat.com, akpm@linux-foundation.org Date: Wed, 5 Oct 2022 11:03:37 -0700 Message-Id: <20221005180341.1738796-1-shy828301@gmail.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Mailman-Approved-At: Thu, 06 Oct 2022 07:13:31 +0000 Subject: [dm-devel] [RFC PATCH 0/4] Introduce mempool pages bulk allocator the use it in dm-crypt X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-block@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Errors-To: dm-devel-bounces@redhat.com Sender: "dm-devel" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com We have full disk encryption enabled, profiling shows page allocations may incur a noticeable overhead when writing. The dm-crypt creates an "out" bio for writing. And fill the "out" bio with the same amount of pages as "in" bio. But the driver allocates one page at a time in a loop. For 1M bio it means the driver has to call page allocator 256 times. It seems not that efficient. Since v5.13 we have page bulk allocator supported, so dm-crypt could use it to do page allocations more efficiently. I could just call the page bulk allocator in dm-crypt driver before the mempool allocator, but it seems ad-hoc and the quick search shows some others do the similar thing, for example, f2fs compress, block bounce, g2fs, ufs, etc. So it seems more neat to implement a bulk allocation API for mempool. So introduce the mempool page bulk allocator. The below APIs are introduced: - mempool_init_pages_bulk() - mempool_create_pages_bulk() They initialize the mempool for page bulk allocator. The pool is filled by alloc_page() in a loop. - mempool_alloc_pages_bulk_list() - mempool_alloc_pages_bulk_array() They do bulk allocation from mempool. They do the below conceptually: 1. Call bulk page allocator 2. If the allocation is fulfilled then return otherwise try to allocate the remaining pages from the mempool 3. If it is fulfilled then return otherwise retry from #1 with sleepable gfp 4. If it is still failed, sleep for a while to wait for the mempool is refilled, then retry from #1 The populated pages will stay on the list or array until the callers consume them or free them. Since mempool allocator is guaranteed to success in the sleepable context, so the two APIs return true for success or false for fail. It is the caller's responsibility to handle failure case (partial allocation), just like the page bulk allocator. The mempool typically is an object agnostic allocator, but bulk allocation is only supported by pages, so the mempool bulk allocator is for page allocation only as well. With the mempool bulk allocator the IOPS of dm-crypt with 1M I/O would get improved by approxiamately 6%. The test is done on a VM with 80 vCPU and 64GB memory with an encrypted ram device (the impact from storage hardware could be minimized so that we could benchmark the dm-crypt layer more accurately). Before the patch: Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=402MiB/s][r=0,w=402 IOPS][eta 00m:00s] crypt: (groupid=0, jobs=1): err= 0: pid=233950: Thu Sep 15 16:23:10 2022 write: IOPS=402, BW=403MiB/s (423MB/s)(23.6GiB/60002msec) slat (usec): min=2425, max=3819, avg=2480.84, stdev=34.00 clat (usec): min=7, max=165751, avg=156398.72, stdev=4691.03 lat (msec): min=2, max=168, avg=158.88, stdev= 4.69 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 165], 99.95th=[ 165], | 99.99th=[ 167] bw ( KiB/s): min=405504, max=413696, per=99.71%, avg=411845.53, stdev=1155.04, samples=120 iops : min= 396, max= 404, avg=402.17, stdev= 1.15, samples=120 lat (usec) : 10=0.01% lat (msec) : 4=0.01%, 10=0.01%, 20=0.02%, 50=0.05%, 100=0.08% lat (msec) : 250=100.09% cpu : usr=3.74%, sys=95.66%, ctx=27, majf=0, minf=4 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=0,24138,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=403MiB/s (423MB/s), 403MiB/s-403MiB/s (423MB/s-423MB/s), io=23.6GiB (25.4GB), run=60002-60002msec After the patch: Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=430MiB/s][r=0,w=430 IOPS][eta 00m:00s] crypt: (groupid=0, jobs=1): err= 0: pid=288730: Thu Sep 15 16:25:39 2022 write: IOPS=430, BW=431MiB/s (452MB/s)(25.3GiB/60002msec) slat (usec): min=2253, max=3213, avg=2319.49, stdev=34.29 clat (usec): min=6, max=149337, avg=146257.68, stdev=4239.52 lat (msec): min=2, max=151, avg=148.58, stdev= 4.24 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 146], 10.00th=[ 146], 20.00th=[ 146], | 30.00th=[ 146], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 146], 80.00th=[ 146], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min=438272, max=442368, per=99.73%, avg=440463.57, stdev=1305.60, samples=120 iops : min= 428, max= 432, avg=430.12, stdev= 1.28, samples=120 lat (usec) : 10=0.01% lat (msec) : 4=0.01%, 10=0.01%, 20=0.02%, 50=0.05%, 100=0.09% lat (msec) : 250=100.07% cpu : usr=3.78%, sys=95.37%, ctx=12778, majf=0, minf=4 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=0,25814,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=431MiB/s (452MB/s), 431MiB/s-431MiB/s (452MB/s-452MB/s), io=25.3GiB (27.1GB), run=60002-60002msec The function tracing also shows the time consumed by page allocations is reduced significantly. The test allocated 1M (256 pages) bio in the same environment. Before the patch: It took approximately 600us by excluding the bio_add_page() calls. 2720.630754 | 56) xfs_io-38859 | 2.571 us | mempool_alloc(); 2720.630757 | 56) xfs_io-38859 | 0.937 us | bio_add_page(); 2720.630758 | 56) xfs_io-38859 | 1.772 us | mempool_alloc(); 2720.630760 | 56) xfs_io-38859 | 0.852 us | bio_add_page(); …. 2720.631559 | 56) xfs_io-38859 | 2.058 us | mempool_alloc(); 2720.631561 | 56) xfs_io-38859 | 0.717 us | bio_add_page(); 2720.631562 | 56) xfs_io-38859 | 2.014 us | mempool_alloc(); 2720.631564 | 56) xfs_io-38859 | 0.620 us | bio_add_page(); After the patch: It took approxiamately 30us. 11564.266385 | 22) xfs_io-136183 | + 30.551 us | __alloc_pages_bulk(); Page allocations overhead is around 6% (600us/9853us) in dm-crypt layer shown by function trace. The data also matches the IOPS data shown by fio. And the benchmark with 4K size I/O doesn't show measurable regression. Yang Shi (4): mm: mempool: extract common initialization code mm: mempool: introduce page bulk allocator md: dm-crypt: move crypt_free_buffer_pages ahead md: dm-crypt: use mempool page bulk allocator drivers/md/dm-crypt.c | 92 ++++++++++++++++------------- include/linux/mempool.h | 19 ++++++ mm/mempool.c | 227 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------- 3 files changed, 276 insertions(+), 62 deletions(-) --- dm-devel mailing list dm-devel@redhat.com https://listman.redhat.com/mailman/listinfo/dm-devel