From patchwork Thu Jul 20 09:45:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13320236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9ECBC04FE0 for ; Thu, 20 Jul 2023 09:48:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229553AbjGTJsd (ORCPT ); Thu, 20 Jul 2023 05:48:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231389AbjGTJsK (ORCPT ); Thu, 20 Jul 2023 05:48:10 -0400 Received: from out-54.mta0.migadu.com (out-54.mta0.migadu.com [IPv6:2001:41d0:1004:224b::36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E9092701 for ; Thu, 20 Jul 2023 02:47:16 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1689846435; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dQrUdY1vRsj8E+dCm96975wE2JF9r7J+GLmQavXqC6U=; b=ppP0xq24ptj/OlMY3SPOuHL6Ff4dQYJc12lo4pwlHFUru/0kT+yUO+nGANDCOXMijpY0zh 0xRV/YP9illwyEGWdRzcnR+C2a2oqd4/o3nAXfqCCcJ3QCVAM4J4lj3lf7TdGBwhgX25Kc aqsmNNgQe9Mnd4K/gYhs9jp73e9/KfE= From: chengming.zhou@linux.dev To: axboe@kernel.dk, osandov@fb.com, ming.lei@redhat.com, kbusch@kernel.org, krisman@suse.de Cc: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, zhouchengming@bytedance.com Subject: [PATCH 1/6] sbitmap: fix hint wrap in the failure case Date: Thu, 20 Jul 2023 17:45:50 +0800 Message-ID: <20230720094555.1397621-2-chengming.zhou@linux.dev> In-Reply-To: <20230720094555.1397621-1-chengming.zhou@linux.dev> References: <20230720094555.1397621-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Chengming Zhou ``` hint = nr + 1; if (hint >= depth - 1) hint = 0; ``` Now we wrap the hint to 0 in the failure case, but: 1. hint == depth - 1, is actually an available offset hint, which we shouldn't wrap hint to 0. 2. In the strict round_robin non-wrap case, we shouldn't wrap at all. ``` wrap = wrap && hint; ``` We only need to check wrap based on the original hint ( > 0), don't need to recheck the new hint which maybe updated in the failure case. Also delete the mismatched comments by the way. Signed-off-by: Chengming Zhou --- lib/sbitmap.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/lib/sbitmap.c b/lib/sbitmap.c index eff4e42c425a..5ed6c2adf58e 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -144,12 +144,7 @@ static int __sbitmap_get_word(unsigned long *word, unsigned long depth, while (1) { nr = find_next_zero_bit(word, depth, hint); if (unlikely(nr >= depth)) { - /* - * We started with an offset, and we didn't reset the - * offset to 0 in a failure case, so start from 0 to - * exhaust the map. - */ - if (hint && wrap) { + if (wrap) { hint = 0; continue; } @@ -160,8 +155,13 @@ static int __sbitmap_get_word(unsigned long *word, unsigned long depth, break; hint = nr + 1; - if (hint >= depth - 1) - hint = 0; + if (hint >= depth) { + if (wrap) { + hint = 0; + continue; + } + return -1; + } } return nr; From patchwork Thu Jul 20 09:45:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13320237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FE2EEB64DA for ; Thu, 20 Jul 2023 09:48:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229563AbjGTJsg (ORCPT ); Thu, 20 Jul 2023 05:48:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231400AbjGTJsL (ORCPT ); Thu, 20 Jul 2023 05:48:11 -0400 Received: from out-24.mta0.migadu.com (out-24.mta0.migadu.com [IPv6:2001:41d0:1004:224b::18]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9732E2706 for ; Thu, 20 Jul 2023 02:47:21 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1689846440; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HexLk5YdZ9UMqQVRKs36MeqJ3bE8mocAttG9oHtUF6A=; b=T3g0daIqsxUtbedfWKPmCc61Qc12FtWKResFJpeO2mr/aRXdYhFQp9Pq27ndDZNyrYc5Kt 7uDJsxRP4E45zPxPxh7vmCqiZoRPQfuQmeTGRkOu8Atn66cRWwHReZ+iWGrbu+ify0SdMJ eCiiqpd8AYC3WMEkSJGYMsndmZ5Jq5E= From: chengming.zhou@linux.dev To: axboe@kernel.dk, osandov@fb.com, ming.lei@redhat.com, kbusch@kernel.org, krisman@suse.de Cc: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, zhouchengming@bytedance.com Subject: [PATCH 2/6] sbitmap: fix round-robin non-wrap find with hint > 0 Date: Thu, 20 Jul 2023 17:45:51 +0800 Message-ID: <20230720094555.1397621-3-chengming.zhou@linux.dev> In-Reply-To: <20230720094555.1397621-1-chengming.zhou@linux.dev> References: <20230720094555.1397621-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Chengming Zhou If we have alloc_hint > 0 and don't wrap, we need to recheck sb->map[index] with hint == 0 to exhaust the map. Signed-off-by: Chengming Zhou --- lib/sbitmap.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/lib/sbitmap.c b/lib/sbitmap.c index 5ed6c2adf58e..ccb96d1f92ba 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -192,10 +192,18 @@ static int sbitmap_find_bit(struct sbitmap *sb, unsigned int alloc_hint, bool wrap) { + unsigned int map_nr = sb->map_nr; unsigned int i; int nr = -1; - for (i = 0; i < sb->map_nr; i++) { + /* + * If we have alloc_hint > 0 and don't wrap, we need to + * recheck sb->map[index] with hint == 0 to exhaust the map. + */ + if (alloc_hint && !wrap) + map_nr += 1; + + for (i = 0; i < map_nr; i++) { nr = sbitmap_find_bit_in_word(&sb->map[index], min_t(unsigned int, __map_depth(sb, index), From patchwork Thu Jul 20 09:45:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13320239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55243EB64DC for ; Thu, 20 Jul 2023 09:49:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230470AbjGTJtS (ORCPT ); Thu, 20 Jul 2023 05:49:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231480AbjGTJsT (ORCPT ); Thu, 20 Jul 2023 05:48:19 -0400 Received: from out-31.mta0.migadu.com (out-31.mta0.migadu.com [IPv6:2001:41d0:1004:224b::1f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E7A6935B8 for ; Thu, 20 Jul 2023 02:47:27 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1689846446; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TQ3vuI5C6djnpQbCi1PW59Gj+FCebISZRVuxTvWKvqo=; b=w4MmY0D51B5mbGeKCwtzsoKhqXXNdLV/Jh+Z8f4ocFu4r2lgNM2FMx5joP6MZg61eMRcPe SWTBP0I8vkvNLPEK871eybgV69GeQS2Ltb1S/JfKkNAgDSRsA3dseXHhNnlHdXo/ewd8zK JY+lUtfuC2q3a1WZ9PWsuyHaW+deVy0= From: chengming.zhou@linux.dev To: axboe@kernel.dk, osandov@fb.com, ming.lei@redhat.com, kbusch@kernel.org, krisman@suse.de Cc: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, zhouchengming@bytedance.com Subject: [PATCH 3/6] sbitmap: don't loop twice in find_next_zero_bit() Date: Thu, 20 Jul 2023 17:45:52 +0800 Message-ID: <20230720094555.1397621-4-chengming.zhou@linux.dev> In-Reply-To: <20230720094555.1397621-1-chengming.zhou@linux.dev> References: <20230720094555.1397621-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Chengming Zhou __sbitmap_get_shallow() try to allocate a free bit with a limited depth, which always wrap when finding a free bit in the word, even in the round-robin case. So it seems we don't need strict round-robin here. This way will loop twice in find_next_zero_bit() in the word, no point in looping twice in find_next_zero_bit() if we don't want strict round-robin for this case. Signed-off-by: Chengming Zhou --- lib/sbitmap.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/sbitmap.c b/lib/sbitmap.c index ccb96d1f92ba..0f3943bd3940 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -268,7 +268,9 @@ static int __sbitmap_get_shallow(struct sbitmap *sb, unsigned int index; index = SB_NR_TO_INDEX(sb, alloc_hint); - alloc_hint = SB_NR_TO_BIT(sb, alloc_hint); + + /* No point in looping twice in find_next_zero_bit() for this case. */ + alloc_hint = 0; return sbitmap_find_bit(sb, shallow_depth, index, alloc_hint, true); } From patchwork Thu Jul 20 09:45:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13320240 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1F9FEB64DA for ; Thu, 20 Jul 2023 09:49:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231270AbjGTJtb (ORCPT ); Thu, 20 Jul 2023 05:49:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229672AbjGTJs2 (ORCPT ); Thu, 20 Jul 2023 05:48:28 -0400 Received: from out-23.mta0.migadu.com (out-23.mta0.migadu.com [IPv6:2001:41d0:1004:224b::17]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4EE826A9 for ; Thu, 20 Jul 2023 02:47:32 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1689846450; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CHTcss6l/Dml/Uch+o0A/n+25yQoOIqj7dE4TlbyHH4=; b=Ai++OZMUjHmw2983yCNBjVkPFxFeR1arX5zXXbaGjA90hfi5mvTPK+xbJURrXi/u+0CWWN RkZhb6KGm+bKwEOt2ud4UlHdP4VWFqqslwKhjxgk4nP6y8kd6taike6pQ5cLKPz4EMMIpE 7N58wZ4pmp7tHfkumvcPMa3P5e/aYD4= From: chengming.zhou@linux.dev To: axboe@kernel.dk, osandov@fb.com, ming.lei@redhat.com, kbusch@kernel.org, krisman@suse.de Cc: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, zhouchengming@bytedance.com Subject: [PATCH 4/6] sbitmap: remove offset wrap logic when finding bit in word Date: Thu, 20 Jul 2023 17:45:53 +0800 Message-ID: <20230720094555.1397621-5-chengming.zhou@linux.dev> In-Reply-To: <20230720094555.1397621-1-chengming.zhou@linux.dev> References: <20230720094555.1397621-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Chengming Zhou We actually don't need the offset hint wrap logic in word: 1. Strict round-robin mode: just search from that offset hint, we will recheck that word at last if offset hint > 0. 2. No round-robin mode: we use offset hint == 0, so don't need to wrap offset hint too. We remove offset wrap logic and simplify the code. Signed-off-by: Chengming Zhou --- lib/sbitmap.c | 36 ++++++++++-------------------------- 1 file changed, 10 insertions(+), 26 deletions(-) diff --git a/lib/sbitmap.c b/lib/sbitmap.c index 0f3943bd3940..50bdf3a31947 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -134,34 +134,21 @@ void sbitmap_resize(struct sbitmap *sb, unsigned int depth) EXPORT_SYMBOL_GPL(sbitmap_resize); static int __sbitmap_get_word(unsigned long *word, unsigned long depth, - unsigned int hint, bool wrap) + unsigned int hint) { int nr; - /* don't wrap if starting from 0 */ - wrap = wrap && hint; - while (1) { nr = find_next_zero_bit(word, depth, hint); - if (unlikely(nr >= depth)) { - if (wrap) { - hint = 0; - continue; - } + if (unlikely(nr >= depth)) return -1; - } if (!test_and_set_bit_lock(nr, word)) break; hint = nr + 1; - if (hint >= depth) { - if (wrap) { - hint = 0; - continue; - } + if (hint >= depth) return -1; - } } return nr; @@ -169,14 +156,13 @@ static int __sbitmap_get_word(unsigned long *word, unsigned long depth, static int sbitmap_find_bit_in_word(struct sbitmap_word *map, unsigned int depth, - unsigned int alloc_hint, - bool wrap) + unsigned int alloc_hint) { int nr; do { nr = __sbitmap_get_word(&map->word, depth, - alloc_hint, wrap); + alloc_hint); if (nr != -1) break; if (!sbitmap_deferred_clear(map)) @@ -189,8 +175,7 @@ static int sbitmap_find_bit_in_word(struct sbitmap_word *map, static int sbitmap_find_bit(struct sbitmap *sb, unsigned int depth, unsigned int index, - unsigned int alloc_hint, - bool wrap) + unsigned int alloc_hint) { unsigned int map_nr = sb->map_nr; unsigned int i; @@ -200,7 +185,7 @@ static int sbitmap_find_bit(struct sbitmap *sb, * If we have alloc_hint > 0 and don't wrap, we need to * recheck sb->map[index] with hint == 0 to exhaust the map. */ - if (alloc_hint && !wrap) + if (alloc_hint) map_nr += 1; for (i = 0; i < map_nr; i++) { @@ -208,7 +193,7 @@ static int sbitmap_find_bit(struct sbitmap *sb, min_t(unsigned int, __map_depth(sb, index), depth), - alloc_hint, wrap); + alloc_hint); if (nr != -1) { nr += index << sb->shift; @@ -240,8 +225,7 @@ static int __sbitmap_get(struct sbitmap *sb, unsigned int alloc_hint) else alloc_hint = 0; - return sbitmap_find_bit(sb, UINT_MAX, index, alloc_hint, - !sb->round_robin); + return sbitmap_find_bit(sb, UINT_MAX, index, alloc_hint); } int sbitmap_get(struct sbitmap *sb) @@ -272,7 +256,7 @@ static int __sbitmap_get_shallow(struct sbitmap *sb, /* No point in looping twice in find_next_zero_bit() for this case. */ alloc_hint = 0; - return sbitmap_find_bit(sb, shallow_depth, index, alloc_hint, true); + return sbitmap_find_bit(sb, shallow_depth, index, alloc_hint); } int sbitmap_get_shallow(struct sbitmap *sb, unsigned long shallow_depth) From patchwork Thu Jul 20 09:45:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13320241 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41005EB64DA for ; Thu, 20 Jul 2023 09:49:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230371AbjGTJtl (ORCPT ); Thu, 20 Jul 2023 05:49:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230400AbjGTJsp (ORCPT ); Thu, 20 Jul 2023 05:48:45 -0400 Received: from out-51.mta0.migadu.com (out-51.mta0.migadu.com [91.218.175.51]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CFC683AAD for ; Thu, 20 Jul 2023 02:47:36 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1689846454; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p34J6xnRb86eKyDcpXgSq81rjV9gO+WCulIJcSmybAc=; b=FO6tJlnKG0PcTyIP5O724vhJaUPzs5PqLkd247BAIxAFGdYM4v7WurTFDGaN1YoOreq4c4 b/h8BL7Yc08WxbDl3yG4Uw0l/8qSIJEuH5EsaTErtlXjMS1BEQPE+uV6BtlyiezOPVg7ob IdnA189BksTiUPxZouFvEL2EiJX56tI= From: chengming.zhou@linux.dev To: axboe@kernel.dk, osandov@fb.com, ming.lei@redhat.com, kbusch@kernel.org, krisman@suse.de Cc: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, zhouchengming@bytedance.com Subject: [PATCH 5/6] sbitmap: wake_index doesn't need to be atomic_t Date: Thu, 20 Jul 2023 17:45:54 +0800 Message-ID: <20230720094555.1397621-6-chengming.zhou@linux.dev> In-Reply-To: <20230720094555.1397621-1-chengming.zhou@linux.dev> References: <20230720094555.1397621-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Chengming Zhou We use wake_index to remember from which to wake up next time, which doesn't need to be atomic_t since we only read it once before wakeups, and write it once after wakeups. Signed-off-by: Chengming Zhou --- include/linux/sbitmap.h | 2 +- lib/sbitmap.c | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/include/linux/sbitmap.h b/include/linux/sbitmap.h index d662cf136021..bdbe478ba4dc 100644 --- a/include/linux/sbitmap.h +++ b/include/linux/sbitmap.h @@ -116,7 +116,7 @@ struct sbitmap_queue { /** * @wake_index: Next wait queue in @ws to wake up. */ - atomic_t wake_index; + unsigned int wake_index; /** * @ws: Wait queues. diff --git a/lib/sbitmap.c b/lib/sbitmap.c index 50bdf3a31947..6778ab3fc6a5 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -419,7 +419,7 @@ int sbitmap_queue_init_node(struct sbitmap_queue *sbq, unsigned int depth, sbq->min_shallow_depth = UINT_MAX; sbq->wake_batch = sbq_calc_wake_batch(sbq, depth); - atomic_set(&sbq->wake_index, 0); + sbq->wake_index = 0; atomic_set(&sbq->ws_active, 0); atomic_set(&sbq->completion_cnt, 0); atomic_set(&sbq->wakeup_cnt, 0); @@ -549,7 +549,7 @@ static void __sbitmap_queue_wake_up(struct sbitmap_queue *sbq, int nr) if (!atomic_read(&sbq->ws_active)) return; - wake_index = atomic_read(&sbq->wake_index); + wake_index = READ_ONCE(sbq->wake_index); for (i = 0; i < SBQ_WAIT_QUEUES; i++) { struct sbq_wait_state *ws = &sbq->ws[wake_index]; @@ -570,8 +570,8 @@ static void __sbitmap_queue_wake_up(struct sbitmap_queue *sbq, int nr) break; } - if (wake_index != atomic_read(&sbq->wake_index)) - atomic_set(&sbq->wake_index, wake_index); + if (wake_index != READ_ONCE(sbq->wake_index)) + WRITE_ONCE(sbq->wake_index, wake_index); } void sbitmap_queue_wake_up(struct sbitmap_queue *sbq, int nr) @@ -672,7 +672,7 @@ void sbitmap_queue_wake_all(struct sbitmap_queue *sbq) * sbitmap_queue_wake_up(). */ smp_mb(); - wake_index = atomic_read(&sbq->wake_index); + wake_index = READ_ONCE(sbq->wake_index); for (i = 0; i < SBQ_WAIT_QUEUES; i++) { struct sbq_wait_state *ws = &sbq->ws[wake_index]; @@ -702,7 +702,7 @@ void sbitmap_queue_show(struct sbitmap_queue *sbq, struct seq_file *m) seq_puts(m, "}\n"); seq_printf(m, "wake_batch=%u\n", sbq->wake_batch); - seq_printf(m, "wake_index=%d\n", atomic_read(&sbq->wake_index)); + seq_printf(m, "wake_index=%d\n", READ_ONCE(sbq->wake_index)); seq_printf(m, "ws_active=%d\n", atomic_read(&sbq->ws_active)); seq_puts(m, "ws={\n"); From patchwork Thu Jul 20 09:45:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13320242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCD99EB64DA for ; Thu, 20 Jul 2023 09:49:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231251AbjGTJtw (ORCPT ); Thu, 20 Jul 2023 05:49:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230158AbjGTJt1 (ORCPT ); Thu, 20 Jul 2023 05:49:27 -0400 Received: from out-6.mta0.migadu.com (out-6.mta0.migadu.com [91.218.175.6]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D008F197 for ; Thu, 20 Jul 2023 02:47:42 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1689846460; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iYyPqspkOuXVMavHO7B8mxFBAqzpe8Xduz/Ay3699EE=; b=pw5sQb18kA//kymlNb+qvXIDo6jCV6+mmT0zU6VFQozPJNN8PoBdLacoMJrd/1FqdpJvcW VwuVolXq85zUUtFfX40ZQSiEGTsPAgEbohducF49YxBq9Q/Wf59Odv9Bl0BlUS6IqohIJQ /iml5vko4ls+yXcePH2pF7w/oW2Of3U= From: chengming.zhou@linux.dev To: axboe@kernel.dk, osandov@fb.com, ming.lei@redhat.com, kbusch@kernel.org, krisman@suse.de Cc: linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, zhouchengming@bytedance.com Subject: [PATCH 6/6] sbitmap: check ws_active before check waitqueues Date: Thu, 20 Jul 2023 17:45:55 +0800 Message-ID: <20230720094555.1397621-7-chengming.zhou@linux.dev> In-Reply-To: <20230720094555.1397621-1-chengming.zhou@linux.dev> References: <20230720094555.1397621-1-chengming.zhou@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org From: Chengming Zhou When !ws_active, we don't need to check waitqueues at all. So add this check in sbitmap_queue_wake_all(), like we do in sbitmap_queue_wake_up(). Signed-off-by: Chengming Zhou --- lib/sbitmap.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/sbitmap.c b/lib/sbitmap.c index 6778ab3fc6a5..38c265e4ef9d 100644 --- a/lib/sbitmap.c +++ b/lib/sbitmap.c @@ -672,6 +672,10 @@ void sbitmap_queue_wake_all(struct sbitmap_queue *sbq) * sbitmap_queue_wake_up(). */ smp_mb(); + + if (!atomic_read(&sbq->ws_active)) + return; + wake_index = READ_ONCE(sbq->wake_index); for (i = 0; i < SBQ_WAIT_QUEUES; i++) { struct sbq_wait_state *ws = &sbq->ws[wake_index];