From patchwork Mon Jul 19 02:17:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 12384587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FAB3C636C8 for ; Mon, 19 Jul 2021 02:18:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BF15561168 for ; Mon, 19 Jul 2021 02:18:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF15561168 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 639E06B012D; Sun, 18 Jul 2021 22:18:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6103C8D00FA; Sun, 18 Jul 2021 22:18:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 416BB6B012F; Sun, 18 Jul 2021 22:18:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0238.hostedemail.com [216.40.44.238]) by kanga.kvack.org (Postfix) with ESMTP id 1AFC36B012D for ; Sun, 18 Jul 2021 22:18:10 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id B62B3180301CC for ; Mon, 19 Jul 2021 02:18:08 +0000 (UTC) X-FDA: 78377727456.04.49FE838 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) by imf17.hostedemail.com (Postfix) with ESMTP id 77BE0F0003B8 for ; Mon, 19 Jul 2021 02:18:08 +0000 (UTC) Received: by mail-qk1-f171.google.com with SMTP id a80so3951020qkg.11 for ; Sun, 18 Jul 2021 19:18:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8jO4nyGRdRSlBrG+WtYYiYryUcVMfftUEaMWXVUBDHo=; b=SKGJyt71wEAPu9gRSETzpS6ZnOKx6a1bVyzPQQZ7wf8q9e73XNqg20Rep9KCsgrJy+ 6Hc3QqOXLFgPBymAhgv3OdB98lsEtfYwD4ait9F9Gu9xVNNl1pArXFG2hWMtL3OO6agz SxH8WuxRAMY8Vfw1gSSYi+51LcEQc9Oi+X4vTKka5FVLiWWv51br36NbOaUPqU1D51w9 SjDbXTB+VSgkS8I43DHq5z+4+zymugAbk3ZEwYK0fZpqFfb3P53OivY0S/RgGvti2oPv 4ZuW/H1CYbkP//CQIV/63MPo7glEXQQl+QC6U1KHSvlP4kzxCfosoWd9BYTo07k0Y6y4 l+SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8jO4nyGRdRSlBrG+WtYYiYryUcVMfftUEaMWXVUBDHo=; b=kiTYXuq+MwNRkvZXJcNHoOSsNs9lI9Y4Lkw3W/AfAJxppiRjn3YBQh8V7fxB9SYxqe NqtDqxFRm39UunvCnUraXGfSCG79xrguPTR4qDv2U5bbd0/IhKT1ZSJXgvyc5ODaiEdi yAaH/yfwtOrgoBTzYUsFkX1W4GRwRGv7DMS/jTwfcCsOHtPOnJCo/1+VxLdEECtcWVlv mORzK/lgqsnXjhBIjh0DNcfQi6BUsa2Bc8rzeuNjTYGgAvu0SUyLG3FpHvD75QUEgyjk abiFB5UcpqLOK4j7TUrOljgP4v/Uep9cxfzDTtEGHd0nuZSsPUDTxlpH7nQY3fm7MOk5 F6Og== X-Gm-Message-State: AOAM5309vmFYaJIXclVK0fa4eencxoRjGvXbHJDVM5rcq6sgQk37jHsA Ku/ej9W8XBrJ0tlMoR8Nfmk= X-Google-Smtp-Source: ABdhPJzYvqeefWrAr6+pDZNfg1zE1hHMkxSmnfBnpZkpOG8dZNMF6D/di1+589JFQPt7g2NDmTm3GA== X-Received: by 2002:a05:620a:f91:: with SMTP id b17mr3591105qkn.107.1626661087768; Sun, 18 Jul 2021 19:18:07 -0700 (PDT) Received: from localhost ([207.98.216.60]) by smtp.gmail.com with ESMTPSA id o27sm3351949qkk.124.2021.07.18.19.18.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Jul 2021 19:18:07 -0700 (PDT) From: Yury Norov To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-mmc@vger.kernel.org Cc: Yury Norov , Andy Shevchenko , Rasmus Villemoes , Petr Mladek , Sergey Senozhatsky , Tejun Heo , Steven Rostedt , Wolfram Sang , Ulf Hansson , Dennis Zhou , Christoph Lameter , Andrew Morton Subject: [PATCH 1/4] mm/percpu: micro-optimize pcpu_is_populated() Date: Sun, 18 Jul 2021 19:17:52 -0700 Message-Id: <20210719021755.883182-2-yury.norov@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210719021755.883182-1-yury.norov@gmail.com> References: <20210719021755.883182-1-yury.norov@gmail.com> MIME-Version: 1.0 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=SKGJyt71; spf=pass (imf17.hostedemail.com: domain of yurynorov@gmail.com designates 209.85.222.171 as permitted sender) smtp.mailfrom=yurynorov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam05 X-Stat-Signature: aqrq7swyu4kctehxkhexiaoetg9jb4fp X-Rspamd-Queue-Id: 77BE0F0003B8 X-HE-Tag: 1626661088-369698 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: bitmap_next_clear_region() calls find_next_zero_bit() and find_next_bit() sequentially to find a range of clear bits. In case of pcpu_is_populated() there's a chance to return earlier if bitmap has all bits set. Signed-off-by: Yury Norov Acked-by: Dennis Zhou --- mm/percpu.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index 7f2e0151c4e2..25461571dcc5 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1071,17 +1071,18 @@ static void pcpu_block_update_hint_free(struct pcpu_chunk *chunk, int bit_off, static bool pcpu_is_populated(struct pcpu_chunk *chunk, int bit_off, int bits, int *next_off) { - unsigned int page_start, page_end, rs, re; + unsigned int start, end; - page_start = PFN_DOWN(bit_off * PCPU_MIN_ALLOC_SIZE); - page_end = PFN_UP((bit_off + bits) * PCPU_MIN_ALLOC_SIZE); + start = PFN_DOWN(bit_off * PCPU_MIN_ALLOC_SIZE); + end = PFN_UP((bit_off + bits) * PCPU_MIN_ALLOC_SIZE); - rs = page_start; - bitmap_next_clear_region(chunk->populated, &rs, &re, page_end); - if (rs >= page_end) + start = find_next_zero_bit(chunk->populated, end, start); + if (start >= end) return true; - *next_off = re * PAGE_SIZE / PCPU_MIN_ALLOC_SIZE; + end = find_next_bit(chunk->populated, end, start + 1); + + *next_off = end * PAGE_SIZE / PCPU_MIN_ALLOC_SIZE; return false; } From patchwork Mon Jul 19 02:17:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 12384589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C71FC636CA for ; Mon, 19 Jul 2021 02:18:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AED886100C for ; Mon, 19 Jul 2021 02:18:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AED886100C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E9EEB6B012E; Sun, 18 Jul 2021 22:18:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E289C8D00FA; Sun, 18 Jul 2021 22:18:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB7706B0130; Sun, 18 Jul 2021 22:18:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0185.hostedemail.com [216.40.44.185]) by kanga.kvack.org (Postfix) with ESMTP id 93E2F6B012E for ; Sun, 18 Jul 2021 22:18:11 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 369B8230EC for ; Mon, 19 Jul 2021 02:18:10 +0000 (UTC) X-FDA: 78377727540.17.C6636F6 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) by imf09.hostedemail.com (Postfix) with ESMTP id D2C5B3000105 for ; Mon, 19 Jul 2021 02:18:09 +0000 (UTC) Received: by mail-qk1-f179.google.com with SMTP id p202so15290412qka.12 for ; Sun, 18 Jul 2021 19:18:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AepViv9+lraIg5lovfET8HkOedqURaUO6BN5RLwUkTw=; b=tfC4qqRgbEoIWLY7t2wU1RcF8kvmFBTKu6551JF8Esk4RHcfSsk5gwL4iph5hVN4+F NE4n7adPW6keivUUrXoI2UiorZnIMJS7JcdeH9NcA+IV/r4gJaZd49dbYd8JHxGgudX6 YZ7Eb5RzfVAoZoKuIJ1UMPHkaH0neF/LavKag0KA62jqOIMLuBvjE2v5yy10p5ZEQKRw WXzCy2LeD7xpodNUrENgNCgt6cZZL3YS11MwolmjbSZRo8GAXeyXm9oavXGrWFaNyd1a 91EZ+f8cCqg3S6aUKXrQ758ScAeXytwWjIu7FX6HPV99aho4K52NfC6EEbk7780FrqKh TSmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AepViv9+lraIg5lovfET8HkOedqURaUO6BN5RLwUkTw=; b=cPxBNZJOfSp7GgQrZ0Ks8TodrG0pXjkMW8Q4xAyq3cvOaZLAItY4sh4uku4E36IbMn TCMWIkURTeviZAhemGEe6uvl23U4uVUui6jdacz/UmS2U2rI3LZhmhIHmGERYzPODJpH sGg6Q+ANY2bAR8L5qHCZ6XrvYEfedeP9ZvzdSGjf9X/DffqtnEi84sDfFPQGbP0Aw1oN MwqBBBYpMN7LPAJylb+kxjVYovhjNDPLK0GkMpQ0dN6+47jYVp0SvwLyoMkp2bgNImd2 4u3/GZrWH7mC9KhPZZkF2adKqtso+27zFgOg0cQoJz4jK1Y/iz89hz1kW8+mospJ3L48 2z1A== X-Gm-Message-State: AOAM532Ty9pEao8X7uD1a7KS7wcuAB8nDYkafXbx+3ShIJRlkDAJkWFM q0467blotfLgyQqHXbD90zZmcZrIOXg= X-Google-Smtp-Source: ABdhPJzPh0/EQ+L49RR9Xpzz54/IIi6eS56/TvEmCax+ScGC6hUV/rfTHPcmlCnEGf7J5tzR5KqO0Q== X-Received: by 2002:a37:a252:: with SMTP id l79mr22051587qke.128.1626661089165; Sun, 18 Jul 2021 19:18:09 -0700 (PDT) Received: from localhost ([207.98.216.60]) by smtp.gmail.com with ESMTPSA id c22sm5885849qtd.42.2021.07.18.19.18.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Jul 2021 19:18:08 -0700 (PDT) From: Yury Norov To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-mmc@vger.kernel.org Cc: Yury Norov , Andy Shevchenko , Rasmus Villemoes , Petr Mladek , Sergey Senozhatsky , Tejun Heo , Steven Rostedt , Wolfram Sang , Ulf Hansson , Dennis Zhou , Christoph Lameter , Andrew Morton Subject: [PATCH 2/4] bitmap: unify find_bit operations Date: Sun, 18 Jul 2021 19:17:53 -0700 Message-Id: <20210719021755.883182-3-yury.norov@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210719021755.883182-1-yury.norov@gmail.com> References: <20210719021755.883182-1-yury.norov@gmail.com> MIME-Version: 1.0 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=tfC4qqRg; spf=pass (imf09.hostedemail.com: domain of yurynorov@gmail.com designates 209.85.222.179 as permitted sender) smtp.mailfrom=yurynorov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam02 X-Stat-Signature: 5pbgmy9ibgebpz6wtrnosd7idw4fywpq X-Rspamd-Queue-Id: D2C5B3000105 X-HE-Tag: 1626661089-325207 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: bitmap_for_each_{set,clear}_region() are similar to for_each_bit() macros in include/linux/find.h, but interface and implementation of them are different. This patch adds for_each_bitrange() macros and drops unused bitmap_*_region() API in sake of unification. Signed-off-by: Yury Norov Acked-by: Dennis Zhou Acked-by: Ulf Hansson # For MMC Tested-by: Wolfram Sang --- drivers/mmc/host/renesas_sdhi_core.c | 2 +- include/linux/bitmap.h | 41 -------------------- include/linux/find.h | 56 ++++++++++++++++++++++++++++ mm/percpu.c | 20 ++++------ 4 files changed, 65 insertions(+), 54 deletions(-) diff --git a/drivers/mmc/host/renesas_sdhi_core.c b/drivers/mmc/host/renesas_sdhi_core.c index e49ca0f7fe9a..efd33b1fc467 100644 --- a/drivers/mmc/host/renesas_sdhi_core.c +++ b/drivers/mmc/host/renesas_sdhi_core.c @@ -647,7 +647,7 @@ static int renesas_sdhi_select_tuning(struct tmio_mmc_host *host) * is at least SH_MOBILE_SDHI_MIN_TAP_ROW probes long then use the * center index as the tap, otherwise bail out. */ - bitmap_for_each_set_region(bitmap, rs, re, 0, taps_size) { + for_each_set_bitrange(rs, re, bitmap, taps_size) { if (re - rs > tap_cnt) { tap_end = re; tap_start = rs; diff --git a/include/linux/bitmap.h b/include/linux/bitmap.h index 3f7c6731b203..96670abf49bd 100644 --- a/include/linux/bitmap.h +++ b/include/linux/bitmap.h @@ -55,12 +55,6 @@ struct device; * bitmap_clear(dst, pos, nbits) Clear specified bit area * bitmap_find_next_zero_area(buf, len, pos, n, mask) Find bit free area * bitmap_find_next_zero_area_off(buf, len, pos, n, mask, mask_off) as above - * bitmap_next_clear_region(map, &start, &end, nbits) Find next clear region - * bitmap_next_set_region(map, &start, &end, nbits) Find next set region - * bitmap_for_each_clear_region(map, rs, re, start, end) - * Iterate over all clear regions - * bitmap_for_each_set_region(map, rs, re, start, end) - * Iterate over all set regions * bitmap_shift_right(dst, src, n, nbits) *dst = *src >> n * bitmap_shift_left(dst, src, n, nbits) *dst = *src << n * bitmap_cut(dst, src, first, n, nbits) Cut n bits from first, copy rest @@ -459,41 +453,6 @@ static inline void bitmap_replace(unsigned long *dst, __bitmap_replace(dst, old, new, mask, nbits); } -static inline void bitmap_next_clear_region(unsigned long *bitmap, - unsigned int *rs, unsigned int *re, - unsigned int end) -{ - *rs = find_next_zero_bit(bitmap, end, *rs); - *re = find_next_bit(bitmap, end, *rs + 1); -} - -static inline void bitmap_next_set_region(unsigned long *bitmap, - unsigned int *rs, unsigned int *re, - unsigned int end) -{ - *rs = find_next_bit(bitmap, end, *rs); - *re = find_next_zero_bit(bitmap, end, *rs + 1); -} - -/* - * Bitmap region iterators. Iterates over the bitmap between [@start, @end). - * @rs and @re should be integer variables and will be set to start and end - * index of the current clear or set region. - */ -#define bitmap_for_each_clear_region(bitmap, rs, re, start, end) \ - for ((rs) = (start), \ - bitmap_next_clear_region((bitmap), &(rs), &(re), (end)); \ - (rs) < (re); \ - (rs) = (re) + 1, \ - bitmap_next_clear_region((bitmap), &(rs), &(re), (end))) - -#define bitmap_for_each_set_region(bitmap, rs, re, start, end) \ - for ((rs) = (start), \ - bitmap_next_set_region((bitmap), &(rs), &(re), (end)); \ - (rs) < (re); \ - (rs) = (re) + 1, \ - bitmap_next_set_region((bitmap), &(rs), &(re), (end))) - /** * BITMAP_FROM_U64() - Represent u64 value in the format suitable for bitmap. * @n: u64 value diff --git a/include/linux/find.h b/include/linux/find.h index ae9ed52b52b8..5bb6db213bcb 100644 --- a/include/linux/find.h +++ b/include/linux/find.h @@ -301,6 +301,62 @@ unsigned long find_next_bit_le(const void *addr, unsigned (bit) < (size); \ (bit) = find_next_zero_bit((addr), (size), (bit) + 1)) +/** + * for_each_set_bitrange - iterate over all set bit ranges [b; e) + * @b: bit offset of start of current bitrange (first set bit) + * @e: bit offset of end of current bitrange (first unset bit) + * @addr: bitmap address to base the search on + * @size: bitmap size in number of bits + */ +#define for_each_set_bitrange(b, e, addr, size) \ + for ((b) = find_next_bit((addr), (size), 0), \ + (e) = find_next_zero_bit((addr), (size), (b) + 1); \ + (b) < (size); \ + (b) = find_next_bit((addr), (size), (e) + 1), \ + (e) = find_next_zero_bit((addr), (size), (b) + 1)) + +/** + * for_each_set_bitrange_from - iterate over set bit ranges [b; e) + * @b: bit offset of start of current bitrange (first set bit); must be initialized + * @e: bit offset of end of current bitrange (first unset bit) + * @addr: bitmap address to base the search on + * @size: bitmap size in number of bits + */ +#define for_each_set_bitrange_from(b, e, addr, size) \ + for ((b) = find_next_bit((addr), (size), (b)), \ + (e) = find_next_zero_bit((addr), (size), (b) + 1); \ + (b) < (size); \ + (b) = find_next_bit((addr), (size), (e) + 1), \ + (e) = find_next_zero_bit((addr), (size), (b) + 1)) + +/** + * for_each_clear_bitrange - iterate over all unset bit ranges [b; e) + * @b: bit offset of start of current bitrange (first unset bit) + * @e: bit offset of end of current bitrange (first set bit) + * @addr: bitmap address to base the search on + * @size: bitmap size in number of bits + */ +#define for_each_clear_bitrange(b, e, addr, size) \ + for ((b) = find_next_zero_bit((addr), (size), 0), \ + (e) = find_next_bit((addr), (size), (b) + 1); \ + (b) < (size); \ + (b) = find_next_zero_bit((addr), (size), (e) + 1), \ + (e) = find_next_bit((addr), (size), (b) + 1)) + +/** + * for_each_clear_bitrange_from - iterate over unset bit ranges [b; e) + * @b: bit offset of start of current bitrange (first unset bit); must be initialized + * @e: bit offset of end of current bitrange (first set bit) + * @addr: bitmap address to base the search on + * @size: bitmap size in number of bits + */ +#define for_each_clear_bitrange_from(b, e, addr, size) \ + for ((b) = find_next_zero_bit((addr), (size), (b)), \ + (e) = find_next_bit((addr), (size), (b) + 1); \ + (b) < (size); \ + (b) = find_next_zero_bit((addr), (size), (e) + 1), \ + (e) = find_next_bit((addr), (size), (b) + 1)) + /** * for_each_set_clump8 - iterate over bitmap for each 8-bit clump with set bits * @start: bit offset to start search and to store the current iteration offset diff --git a/mm/percpu.c b/mm/percpu.c index 25461571dcc5..6d518e822983 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -780,7 +780,7 @@ static void pcpu_block_refresh_hint(struct pcpu_chunk *chunk, int index) { struct pcpu_block_md *block = chunk->md_blocks + index; unsigned long *alloc_map = pcpu_index_alloc_map(chunk, index); - unsigned int rs, re, start; /* region start, region end */ + unsigned int start, end; /* region start, region end */ /* promote scan_hint to contig_hint */ if (block->scan_hint) { @@ -796,9 +796,8 @@ static void pcpu_block_refresh_hint(struct pcpu_chunk *chunk, int index) block->right_free = 0; /* iterate over free areas and update the contig hints */ - bitmap_for_each_clear_region(alloc_map, rs, re, start, - PCPU_BITMAP_BLOCK_BITS) - pcpu_block_update(block, rs, re); + for_each_clear_bitrange_from(start, end, alloc_map, PCPU_BITMAP_BLOCK_BITS) + pcpu_block_update(block, start, end); } /** @@ -1856,13 +1855,12 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, /* populate if not all pages are already there */ if (!is_atomic) { - unsigned int page_start, page_end, rs, re; + unsigned int page_end, rs, re; - page_start = PFN_DOWN(off); + rs = PFN_DOWN(off); page_end = PFN_UP(off + size); - bitmap_for_each_clear_region(chunk->populated, rs, re, - page_start, page_end) { + for_each_clear_bitrange_from(rs, re, chunk->populated, page_end) { WARN_ON(chunk->immutable); ret = pcpu_populate_chunk(chunk, rs, re, pcpu_gfp); @@ -2018,8 +2016,7 @@ static void pcpu_balance_free(bool empty_only) list_for_each_entry_safe(chunk, next, &to_free, list) { unsigned int rs, re; - bitmap_for_each_set_region(chunk->populated, rs, re, 0, - chunk->nr_pages) { + for_each_set_bitrange(rs, re, chunk->populated, chunk->nr_pages) { pcpu_depopulate_chunk(chunk, rs, re); spin_lock_irq(&pcpu_lock); pcpu_chunk_depopulated(chunk, rs, re); @@ -2089,8 +2086,7 @@ static void pcpu_balance_populated(void) continue; /* @chunk can't go away while pcpu_alloc_mutex is held */ - bitmap_for_each_clear_region(chunk->populated, rs, re, 0, - chunk->nr_pages) { + for_each_clear_bitrange(rs, re, chunk->populated, chunk->nr_pages) { int nr = min_t(int, re - rs, nr_to_pop); spin_unlock_irq(&pcpu_lock); From patchwork Mon Jul 19 02:17:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 12384591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6CE8C636CE for ; Mon, 19 Jul 2021 02:18:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8A98361168 for ; Mon, 19 Jul 2021 02:18:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8A98361168 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 32C1B6B012F; Sun, 18 Jul 2021 22:18:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BC1B6B0130; Sun, 18 Jul 2021 22:18:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F16C36B0131; Sun, 18 Jul 2021 22:18:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0063.hostedemail.com [216.40.44.63]) by kanga.kvack.org (Postfix) with ESMTP id CFA006B012F for ; Sun, 18 Jul 2021 22:18:12 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 7E10E180882AF for ; Mon, 19 Jul 2021 02:18:11 +0000 (UTC) X-FDA: 78377727582.26.33667DE Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) by imf22.hostedemail.com (Postfix) with ESMTP id 2F41B1924 for ; Mon, 19 Jul 2021 02:18:11 +0000 (UTC) Received: by mail-qv1-f45.google.com with SMTP id w2so7629799qvh.13 for ; Sun, 18 Jul 2021 19:18:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=F05MjF1EjPo9ALm7Ep9kFMbBo/dzAk/z9NhYdhgbLQU=; b=c2JnXWJDh86iqNrQ6XAfurPn8Ie54j98BTn8TjxSUeD7s2VhqxnTAPDTOaWsh/Jc8N OFH55XkPpYC2ZkF3rR0Mue2csD7bqk5bdHV/IqKmyPtq2iHFKOcRe5MysqBwR6ekRbm4 skimyqWhoJWEVo0FU1E8ofgjIghmwyuDm+aF9W5TN4Vg8izknB8DLlPJ9WVAl308AGgB XskN6a3h24i5tPll4sDkRqkESVOEgyJxW1+IqjqTfLgn2X5ljk7QTxnAisLlnwD9clyN /Lmd8rWgVe8ypGcdM4ubs6cuEYrkAC9rIJTL4U7zJVKPCH6LYe5OsRbL/BJ+X4evGqSU lHng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=F05MjF1EjPo9ALm7Ep9kFMbBo/dzAk/z9NhYdhgbLQU=; b=Nb4zrUHb/Ss2H3akgn2L+UFPXZ20LZ3CLgYN61EE/kcEzx9MjEwKMMXCUBgb2H2uNq GDh0wm+g21jsD2BL9vAABgAm22KHfnqPs4D0mzjnoZSYHHdcAzMXbI+56h+oFiFSZ9LM UIVA3/QxBYSCPvVsvVUp044W9Hmlk8hC7afVnTTzZeWvMX/UHtL0/FoXSB9A9Xj0ldVA U5cGMymaSRJAG7jg1F+0so3lwbKEs8C8AxuhwvYU+FCSwm4SP2Ryqq43zTtp+Qr0H+cU flC+6N0jfF57RPkV8CLGBB4+XFezZx8uqRK4oa2fk2MoCaZ6AoDRTlf+jRRBJXwlkhnB O3Tw== X-Gm-Message-State: AOAM530/W6acOhVDBJ3JxDLeq4jjDHZplZKdeA4M75hDiVBy+im7OWC7 wQNyktckLye1/n9OlLcYEWM= X-Google-Smtp-Source: ABdhPJw5IaXLvY4GY3gixUsEQH1zgiCmVLiM4jB+jFuk8uQo1DBCTKt27AdmcwKgHMmo4ADCH4O/UQ== X-Received: by 2002:ad4:5c49:: with SMTP id a9mr13679048qva.27.1626661090504; Sun, 18 Jul 2021 19:18:10 -0700 (PDT) Received: from localhost ([207.98.216.60]) by smtp.gmail.com with ESMTPSA id k10sm54870qtp.76.2021.07.18.19.18.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Jul 2021 19:18:10 -0700 (PDT) From: Yury Norov To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-mmc@vger.kernel.org Cc: Yury Norov , Andy Shevchenko , Rasmus Villemoes , Petr Mladek , Sergey Senozhatsky , Tejun Heo , Steven Rostedt , Wolfram Sang , Ulf Hansson , Dennis Zhou , Christoph Lameter , Andrew Morton Subject: [PATCH 3/4] lib: bitmap: add performance test for bitmap_print_to_pagebuf Date: Sun, 18 Jul 2021 19:17:54 -0700 Message-Id: <20210719021755.883182-4-yury.norov@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210719021755.883182-1-yury.norov@gmail.com> References: <20210719021755.883182-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2F41B1924 X-Stat-Signature: ntixbsnogdye7d8js64e57nczua3m3ft Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=c2JnXWJD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of yurynorov@gmail.com designates 209.85.219.45 as permitted sender) smtp.mailfrom=yurynorov@gmail.com X-HE-Tag: 1626661091-428538 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Functional tests for bitmap_print_to_pagebuf() are provided in lib/test_printf.c. This patch adds performance test for a case of fully set bitmap. Signed-off-by: Yury Norov --- lib/test_bitmap.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/lib/test_bitmap.c b/lib/test_bitmap.c index 4ea73f5aed41..452d525007da 100644 --- a/lib/test_bitmap.c +++ b/lib/test_bitmap.c @@ -430,6 +430,42 @@ static void __init test_bitmap_parselist(void) } } +static void __init test_bitmap_printlist(void) +{ + unsigned long *bmap = kmalloc(PAGE_SIZE, GFP_KERNEL); + char *buf = kmalloc(PAGE_SIZE, GFP_KERNEL); + char expected[256]; + int ret, slen; + ktime_t time; + + if (!buf || !bmap) + goto out; + + memset(bmap, -1, PAGE_SIZE); + slen = snprintf(expected, 256, "0-%ld", PAGE_SIZE * 8 - 1); + if (slen < 0) + goto out; + + time = ktime_get(); + ret = bitmap_print_to_pagebuf(true, buf, bmap, PAGE_SIZE * 8); + time = ktime_get() - time; + + if (ret != slen + 1) { + pr_err("bitmap_print_to_pagebuf: result is %d, expected %d\n", ret, slen); + goto out; + } + + if (strncmp(buf, expected, slen)) { + pr_err("bitmap_print_to_pagebuf: result is %s, expected %s\n", buf, expected); + goto out; + } + + pr_err("bitmap_print_to_pagebuf: input is '%s', Time: %llu\n", buf, time); +out: + kfree(buf); + kfree(bmap); +} + static const unsigned long parse_test[] __initconst = { BITMAP_FROM_U64(0), BITMAP_FROM_U64(1), @@ -669,6 +705,7 @@ static void __init selftest(void) test_bitmap_arr32(); test_bitmap_parse(); test_bitmap_parselist(); + test_bitmap_printlist(); test_mem_optimisations(); test_for_each_set_clump8(); test_bitmap_cut(); From patchwork Mon Jul 19 02:17:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yury Norov X-Patchwork-Id: 12384593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6471C636C8 for ; Mon, 19 Jul 2021 02:18:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 73F4B610A2 for ; Mon, 19 Jul 2021 02:18:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 73F4B610A2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 59D786B0130; Sun, 18 Jul 2021 22:18:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 52A966B0131; Sun, 18 Jul 2021 22:18:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 351836B0132; Sun, 18 Jul 2021 22:18:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id 14B5D6B0130 for ; Sun, 18 Jul 2021 22:18:14 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B1E1C20BF0 for ; Mon, 19 Jul 2021 02:18:12 +0000 (UTC) X-FDA: 78377727624.08.CDA97D4 Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) by imf01.hostedemail.com (Postfix) with ESMTP id 754F55011AA9 for ; Mon, 19 Jul 2021 02:18:12 +0000 (UTC) Received: by mail-qt1-f180.google.com with SMTP id g8so12014296qtj.1 for ; Sun, 18 Jul 2021 19:18:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=v7LGF7A24MqvtOPGVDdt9eL48iMH5UQbKnUazAYVpTs=; b=f3DJt2H0o9omDVvuQN5v1QSb/r6QSjoBV97l95h4zLyzLthlOdnOCwvg+HUNREWxyz o/swR0MA67DbNo0RhPjIpzAPMoH7INbTUYgfZkVYDlK5fogvieMixk1GssQ61ex62bag IOTixqiI8hrPRhtEhM/c4rJhLUk33yqtmA/fo/PdO9abiEkJNhFr1AXT3jb+/Ln8iUiw sVy0bhEIwuMbsdwyHfndj0Qgf34nneiEFhS+L6RIkVr5lS4RNmj2QSjW1yanNQ2XHpNs Im0v0WcyKkSx4kYtbzSpurvl+EqpE+ucHC+fOSLZNu6GCEiYByV+YFFI9UZQMfz21Opu RxRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=v7LGF7A24MqvtOPGVDdt9eL48iMH5UQbKnUazAYVpTs=; b=IS63Min0W7A9jZ2rW6MPjuUpzh2vgOGggZ/gsZHPuQhzrn4y3rn2xuUyRr5hbE+w0D el1riXYyFfBHoVQyoK229/K4P5HgzaAT4Ku2pYzsQTKYrFyg6igaXbX6t5X074p+UsE1 95KEF/tmSybJNCB8ezwpubFZn0Y4Il62itb9rn6KXAxQrZ3vI2IC/h179awUG7CcO0R+ 3fdj2FaDsuoUygPkVNuT7uLAvV2yGfSprQUuSlng3dbWwCEoe03Ts9fAFGxcM1747fLJ bp+wLnqmuUVQh1dyMeFisFjPbRmFBvOHHWmmwilur67J/xKvwDk5EPdCG1oBkcamRP0a 1/lg== X-Gm-Message-State: AOAM530kR7qhUEdQt9JEiRScCAlVg3c6FE6b8abPpzdGir6wcSZme5Bh ePVrDhPGQ4N+BUg9Yn0RH+Q= X-Google-Smtp-Source: ABdhPJxops0MED8R2XXX4dM9TVbDCyoBT8virK5Xmff0q2qDiCbX/HZRqNJl0c2CgnpKQuMoFQcJaQ== X-Received: by 2002:ac8:7115:: with SMTP id z21mr14032886qto.85.1626661091797; Sun, 18 Jul 2021 19:18:11 -0700 (PDT) Received: from localhost ([207.98.216.60]) by smtp.gmail.com with ESMTPSA id d7sm6085796qth.70.2021.07.18.19.18.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Jul 2021 19:18:11 -0700 (PDT) From: Yury Norov To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-mmc@vger.kernel.org Cc: Yury Norov , Andy Shevchenko , Rasmus Villemoes , Petr Mladek , Sergey Senozhatsky , Tejun Heo , Steven Rostedt , Wolfram Sang , Ulf Hansson , Dennis Zhou , Christoph Lameter , Andrew Morton Subject: [PATCH 4/4] vsprintf: rework bitmap_list_string Date: Sun, 18 Jul 2021 19:17:55 -0700 Message-Id: <20210719021755.883182-5-yury.norov@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210719021755.883182-1-yury.norov@gmail.com> References: <20210719021755.883182-1-yury.norov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 754F55011AA9 X-Stat-Signature: 1sggba5xc71tiaf46fo7degdzppygs4j Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=f3DJt2H0; spf=pass (imf01.hostedemail.com: domain of yurynorov@gmail.com designates 209.85.160.180 as permitted sender) smtp.mailfrom=yurynorov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1626661092-243033 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: bitmap_list_string() is very ineffective when printing bitmaps with long ranges of set bits because it calls find_next_bit() for each bit in the bitmap. We can do better by detecting ranges of set bits. In my environment, before/after is 943008/31008 ns. Signed-off-by: Yury Norov --- lib/vsprintf.c | 24 +++++++----------------- 1 file changed, 7 insertions(+), 17 deletions(-) diff --git a/lib/vsprintf.c b/lib/vsprintf.c index 3b8b3f20051a..361799075706 100644 --- a/lib/vsprintf.c +++ b/lib/vsprintf.c @@ -1241,20 +1241,13 @@ char *bitmap_list_string(char *buf, char *end, unsigned long *bitmap, struct printf_spec spec, const char *fmt) { int nr_bits = max_t(int, spec.field_width, 0); - /* current bit is 'cur', most recently seen range is [rbot, rtop] */ - int cur, rbot, rtop; bool first = true; + int rbot, rtop; if (check_pointer(&buf, end, bitmap, spec)) return buf; - rbot = cur = find_first_bit(bitmap, nr_bits); - while (cur < nr_bits) { - rtop = cur; - cur = find_next_bit(bitmap, nr_bits, cur + 1); - if (cur < nr_bits && cur <= rtop + 1) - continue; - + for_each_set_bitrange(rbot, rtop, bitmap, nr_bits) { if (!first) { if (buf < end) *buf = ','; @@ -1263,15 +1256,12 @@ char *bitmap_list_string(char *buf, char *end, unsigned long *bitmap, first = false; buf = number(buf, end, rbot, default_dec_spec); - if (rbot < rtop) { - if (buf < end) - *buf = '-'; - buf++; - - buf = number(buf, end, rtop, default_dec_spec); - } + if (rtop == rbot + 1) + continue; - rbot = cur; + if (buf < end) + *buf = '-'; + buf = number(++buf, end, rtop - 1, default_dec_spec); } return buf; }