From patchwork Thu May 26 16:35:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862538 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 557CAC433FE for ; Thu, 26 May 2022 16:36:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348259AbiEZQg6 (ORCPT ); Thu, 26 May 2022 12:36:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348153AbiEZQgY (ORCPT ); Thu, 26 May 2022 12:36:24 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8E8C4C403; Thu, 26 May 2022 09:36:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=IWEOZJwft2vuBCToQwiXtUQwH05rs0AR5scLKqJJLCk=; b=YqQ7aAvSMpq5Xh/9MSHmojdO5v 9Z/vu0Kuu6vaYv1TiAUyxO8pkxmmXZq3FEueXi/crC4oyBu59XgENqunIz9igI6SO8anh+H7qmT8O AmPVW2Wq1F11rg6NOU8WETjDcKrq20zdd3ff4gz8DyN1FLsWHtXtO5qRZl2RsS808qIIGWipba7gw 0FYOm82WMyLemFo13vvW7bK4aQmGqumiAhmLXF4djwQidhxKUyHrOkE3qrSHWhm7bd/ZIyn8HlWBA 5cQ/34gLmikwocQ+0YEGK743B3Tn3tdrXldXwrVUKIGC+BS24CJgBdGNglVQF8GZ8/PdYAHcLDLIX sBnP9Kew==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTD-008A7S-8b; Thu, 26 May 2022 10:36:21 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGT7-0008X6-VI; Thu, 26 May 2022 10:36:14 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe , Christoph Hellwig Date: Thu, 26 May 2022 10:35:48 -0600 Message-Id: <20220526163604.32736-2-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com, hch@lst.de X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 01/17] md/raid5-log: Drop extern decorators for function prototypes X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org extern is not necessary and recommended against when defining prototype functions in headers. checkpatch.pl complains about these. So remove them. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5-log.h | 75 ++++++++++++++++++++---------------------- 1 file changed, 36 insertions(+), 39 deletions(-) diff --git a/drivers/md/raid5-log.h b/drivers/md/raid5-log.h index 43c714a8798c..270ced4f770f 100644 --- a/drivers/md/raid5-log.h +++ b/drivers/md/raid5-log.h @@ -2,49 +2,46 @@ #ifndef _RAID5_LOG_H #define _RAID5_LOG_H -extern int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev); -extern void r5l_exit_log(struct r5conf *conf); -extern int r5l_write_stripe(struct r5l_log *log, struct stripe_head *head_sh); -extern void r5l_write_stripe_run(struct r5l_log *log); -extern void r5l_flush_stripe_to_raid(struct r5l_log *log); -extern void r5l_stripe_write_finished(struct stripe_head *sh); -extern int r5l_handle_flush_request(struct r5l_log *log, struct bio *bio); -extern void r5l_quiesce(struct r5l_log *log, int quiesce); -extern bool r5l_log_disk_error(struct r5conf *conf); -extern bool r5c_is_writeback(struct r5l_log *log); -extern int -r5c_try_caching_write(struct r5conf *conf, struct stripe_head *sh, - struct stripe_head_state *s, int disks); -extern void -r5c_finish_stripe_write_out(struct r5conf *conf, struct stripe_head *sh, - struct stripe_head_state *s); -extern void r5c_release_extra_page(struct stripe_head *sh); -extern void r5c_use_extra_page(struct stripe_head *sh); -extern void r5l_wake_reclaim(struct r5l_log *log, sector_t space); -extern void r5c_handle_cached_data_endio(struct r5conf *conf, - struct stripe_head *sh, int disks); -extern int r5c_cache_data(struct r5l_log *log, struct stripe_head *sh); -extern void r5c_make_stripe_write_out(struct stripe_head *sh); -extern void r5c_flush_cache(struct r5conf *conf, int num); -extern void r5c_check_stripe_cache_usage(struct r5conf *conf); -extern void r5c_check_cached_full_stripe(struct r5conf *conf); +int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev); +void r5l_exit_log(struct r5conf *conf); +int r5l_write_stripe(struct r5l_log *log, struct stripe_head *head_sh); +void r5l_write_stripe_run(struct r5l_log *log); +void r5l_flush_stripe_to_raid(struct r5l_log *log); +void r5l_stripe_write_finished(struct stripe_head *sh); +int r5l_handle_flush_request(struct r5l_log *log, struct bio *bio); +void r5l_quiesce(struct r5l_log *log, int quiesce); +bool r5l_log_disk_error(struct r5conf *conf); +bool r5c_is_writeback(struct r5l_log *log); +int r5c_try_caching_write(struct r5conf *conf, struct stripe_head *sh, + struct stripe_head_state *s, int disks); +void r5c_finish_stripe_write_out(struct r5conf *conf, struct stripe_head *sh, + struct stripe_head_state *s); +void r5c_release_extra_page(struct stripe_head *sh); +void r5c_use_extra_page(struct stripe_head *sh); +void r5l_wake_reclaim(struct r5l_log *log, sector_t space); +void r5c_handle_cached_data_endio(struct r5conf *conf, + struct stripe_head *sh, int disks); +int r5c_cache_data(struct r5l_log *log, struct stripe_head *sh); +void r5c_make_stripe_write_out(struct stripe_head *sh); +void r5c_flush_cache(struct r5conf *conf, int num); +void r5c_check_stripe_cache_usage(struct r5conf *conf); +void r5c_check_cached_full_stripe(struct r5conf *conf); extern struct md_sysfs_entry r5c_journal_mode; -extern void r5c_update_on_rdev_error(struct mddev *mddev, - struct md_rdev *rdev); -extern bool r5c_big_stripe_cached(struct r5conf *conf, sector_t sect); -extern int r5l_start(struct r5l_log *log); +void r5c_update_on_rdev_error(struct mddev *mddev, struct md_rdev *rdev); +bool r5c_big_stripe_cached(struct r5conf *conf, sector_t sect); +int r5l_start(struct r5l_log *log); -extern struct dma_async_tx_descriptor * +struct dma_async_tx_descriptor * ops_run_partial_parity(struct stripe_head *sh, struct raid5_percpu *percpu, struct dma_async_tx_descriptor *tx); -extern int ppl_init_log(struct r5conf *conf); -extern void ppl_exit_log(struct r5conf *conf); -extern int ppl_write_stripe(struct r5conf *conf, struct stripe_head *sh); -extern void ppl_write_stripe_run(struct r5conf *conf); -extern void ppl_stripe_write_finished(struct stripe_head *sh); -extern int ppl_modify_log(struct r5conf *conf, struct md_rdev *rdev, bool add); -extern void ppl_quiesce(struct r5conf *conf, int quiesce); -extern int ppl_handle_flush_request(struct r5l_log *log, struct bio *bio); +int ppl_init_log(struct r5conf *conf); +void ppl_exit_log(struct r5conf *conf); +int ppl_write_stripe(struct r5conf *conf, struct stripe_head *sh); +void ppl_write_stripe_run(struct r5conf *conf); +void ppl_stripe_write_finished(struct stripe_head *sh); +int ppl_modify_log(struct r5conf *conf, struct md_rdev *rdev, bool add); +void ppl_quiesce(struct r5conf *conf, int quiesce); +int ppl_handle_flush_request(struct r5l_log *log, struct bio *bio); extern struct md_sysfs_entry ppl_write_hint; static inline bool raid5_has_log(struct r5conf *conf) From patchwork Thu May 26 16:35:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CBEEC433EF for ; Thu, 26 May 2022 16:36:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242520AbiEZQgW (ORCPT ); Thu, 26 May 2022 12:36:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230417AbiEZQgW (ORCPT ); Thu, 26 May 2022 12:36:22 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75C3936170; Thu, 26 May 2022 09:36:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=EBCha7Kz5pRLaLgWVut+C7W0HPOknhHzDfjqwmF0hVg=; b=nWOvRUfpgZWzzr6O2i6z5jk6J3 OIpWkl8Owfysr2pq2UEzbJbuKCE9ZRImgb58ZbRSu62NZ2ehLIEJ4tco4VyuecJt1fxFe4r6vOSbS heYyKGzpqznuXi2UaDUAkUCk9C78+5dgNIE7/OPCST1cLyZF2gTZ88P4shJu6Nm7cHvHp8/jusVeQ wRLEtAQPqqqQMh3AgD6qihhvVj+5PjBDy8HrHwSpafVUvuLFuX1PlRpcC6h3E9v2x+A0k8D7txyuF GN4bDQvMfZYeU/LzJH7X/n2P/6K5ZX8rWiZcdGojiaiMLXlxBpaINDpFmRyPJ3GVXdMsR1P1IDxLY AUb5GfMQ==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTC-008A7T-NE; Thu, 26 May 2022 10:36:19 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGT8-0008X9-4G; Thu, 26 May 2022 10:36:14 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe Date: Thu, 26 May 2022 10:35:49 -0600 Message-Id: <20220526163604.32736-3-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 02/17] md/raid5-cache: Add r5c_conf_is_writeback() helper X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org Most calls to r5c_is_writeback() have to dereference conf to get the log. The log pointer should be protected by RCU, but isn't. Push the dereference of conf->log into a new r5c_conf_is_writeback() helper to reduce the number of sites that dereference conf->log so this can be improved. No functional changes intended. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5-cache.c | 7 ++++++- drivers/md/raid5-log.h | 2 +- drivers/md/raid5.c | 16 ++++++++-------- 3 files changed, 15 insertions(+), 10 deletions(-) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 83c184eddbda..96f51ce9b6c1 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -247,12 +247,17 @@ enum r5l_io_unit_state { IO_UNIT_STRIPE_END = 3, /* stripes data finished writing to raid */ }; -bool r5c_is_writeback(struct r5l_log *log) +static bool r5c_is_writeback(struct r5l_log *log) { return (log != NULL && log->r5c_journal_mode == R5C_JOURNAL_MODE_WRITE_BACK); } +bool r5c_conf_is_writeback(struct r5conf *conf) +{ + return r5c_is_writeback(conf->log); +} + static sector_t r5l_ring_add(struct r5l_log *log, sector_t start, sector_t inc) { start += inc; diff --git a/drivers/md/raid5-log.h b/drivers/md/raid5-log.h index 270ced4f770f..720e164e4d4d 100644 --- a/drivers/md/raid5-log.h +++ b/drivers/md/raid5-log.h @@ -11,7 +11,7 @@ void r5l_stripe_write_finished(struct stripe_head *sh); int r5l_handle_flush_request(struct r5l_log *log, struct bio *bio); void r5l_quiesce(struct r5l_log *log, int quiesce); bool r5l_log_disk_error(struct r5conf *conf); -bool r5c_is_writeback(struct r5l_log *log); +bool r5c_conf_is_writeback(struct r5conf *conf); int r5c_try_caching_write(struct r5conf *conf, struct stripe_head *sh, struct stripe_head_state *s, int disks); void r5c_finish_stripe_write_out(struct r5conf *conf, struct stripe_head *sh, diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index 5d09256d7f81..458d88faf2e9 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -224,7 +224,7 @@ static void do_release_stripe(struct r5conf *conf, struct stripe_head *sh, BUG_ON(!list_empty(&sh->lru)); BUG_ON(atomic_read(&conf->active_stripes)==0); - if (r5c_is_writeback(conf->log)) + if (r5c_conf_is_writeback(conf)) for (i = sh->disks; i--; ) if (test_bit(R5_InJournal, &sh->dev[i].flags)) injournal++; @@ -236,7 +236,7 @@ static void do_release_stripe(struct r5conf *conf, struct stripe_head *sh, * 2. when resync is requested fot the stripe. */ if (test_bit(STRIPE_SYNC_REQUESTED, &sh->state) || - (conf->quiesce && r5c_is_writeback(conf->log) && + (conf->quiesce && r5c_conf_is_writeback(conf) && !test_bit(STRIPE_HANDLE, &sh->state) && injournal != 0)) { if (test_bit(STRIPE_R5C_CACHING, &sh->state)) r5c_make_stripe_write_out(sh); @@ -274,7 +274,7 @@ static void do_release_stripe(struct r5conf *conf, struct stripe_head *sh, md_wakeup_thread(conf->mddev->thread); atomic_dec(&conf->active_stripes); if (!test_bit(STRIPE_EXPANDING, &sh->state)) { - if (!r5c_is_writeback(conf->log)) + if (!r5c_conf_is_writeback(conf)) list_add_tail(&sh->lru, temp_inactive_list); else { WARN_ON(test_bit(R5_InJournal, &sh->dev[sh->pd_idx].flags)); @@ -1786,7 +1786,7 @@ static void ops_complete_prexor(void *stripe_head_ref) pr_debug("%s: stripe %llu\n", __func__, (unsigned long long)sh->sector); - if (r5c_is_writeback(sh->raid_conf->log)) + if (r5c_conf_is_writeback(sh->raid_conf)) /* * raid5-cache write back uses orig_page during prexor. * After prexor, it is time to free orig_page @@ -1905,9 +1905,9 @@ ops_run_biodrain(struct stripe_head *sh, struct dma_async_tx_descriptor *tx) tx = async_copy_data(1, wbi, &dev->page, dev->offset, dev->sector, tx, sh, - r5c_is_writeback(conf->log)); + r5c_conf_is_writeback(conf)); if (dev->page != dev->orig_page && - !r5c_is_writeback(conf->log)) { + !r5c_conf_is_writeback(conf)) { set_bit(R5_SkipCopy, &dev->flags); clear_bit(R5_UPTODATE, &dev->flags); clear_bit(R5_OVERWRITE, &dev->flags); @@ -5085,7 +5085,7 @@ static void handle_stripe(struct stripe_head *sh) */ if (!sh->reconstruct_state && !sh->check_state && !sh->log_io) { - if (!r5c_is_writeback(conf->log)) { + if (!r5c_conf_is_writeback(conf)) { if (s.to_write) handle_stripe_dirtying(conf, sh, &s, disks); } else { /* write back cache */ @@ -5533,7 +5533,7 @@ static struct stripe_head *__get_priority_stripe(struct r5conf *conf, int group) struct stripe_head *sh, *tmp; struct list_head *handle_list = NULL; struct r5worker_group *wg; - bool second_try = !r5c_is_writeback(conf->log) && + bool second_try = !r5c_conf_is_writeback(conf) && !r5l_log_disk_error(conf); bool try_loprio = test_bit(R5C_LOG_TIGHT, &conf->cache_state) || r5l_log_disk_error(conf); From patchwork Thu May 26 16:35:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3013C433F5 for ; Thu, 26 May 2022 16:36:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348159AbiEZQgZ (ORCPT ); Thu, 26 May 2022 12:36:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42104 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344564AbiEZQgX (ORCPT ); Thu, 26 May 2022 12:36:23 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 585AD3DA53; Thu, 26 May 2022 09:36:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=jqXxJ/n39vbHiCMTJ9YpoTIdD2FJ0weE01mlKeUY3bo=; b=VwMBpSUHA60HDH4RqVNZKCreBb mGJ6qJy9649E97/4Zd+G62ZzVQ1plgOMmls7c4zbh4Z7KWxyKs3L3t4p/D8yfy1MPvOs70SYAa5eZ RpG9x/DpqmdkrA/+b6rNarVo7k92BHXV/MeDwaaFuLrzVS0zDPISHEgtCo1cv7XRFoC1Wbxwg5mA7 vmNw+x11e5sVK6/NBChsECx+KrHqMYdzixby28oY4QezPeg1buqoKUJe7A539uU14B5tj6AxR91VN kf996VZ4mQ5PYh+qJGuvpLWHo6v/eZDcfTuHyvwrE8Apvmovvz6JCPQWrnaWUBQgSccd1MK7UzhYN vJsTk1Jw==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTD-008A7V-8b; Thu, 26 May 2022 10:36:20 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGT8-0008XC-9G; Thu, 26 May 2022 10:36:14 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe , Christoph Hellwig Date: Thu, 26 May 2022 10:35:50 -0600 Message-Id: <20220526163604.32736-4-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com, hch@lst.de X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 03/17] md/raid5-cache: Refactor r5l_start() to take a struct r5conf X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org All calls to r5l_start() have to dereference conf to get the log. The log pointer should be protected by RCU, but isn't. Push the dereference of conf->log into r5l_start() to reduce the number of sites that dereference conf->log so this can be improved. No functional changes intended. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5-cache.c | 9 +++------ drivers/md/raid5-log.h | 2 +- drivers/md/raid5.c | 4 ++-- 3 files changed, 6 insertions(+), 9 deletions(-) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 96f51ce9b6c1..7c5dc45b1ea8 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -3034,20 +3034,17 @@ static int r5l_load_log(struct r5l_log *log) return ret; } -int r5l_start(struct r5l_log *log) +int r5l_start(struct r5conf *conf) { + struct r5l_log *log = conf->log; int ret; if (!log) return 0; ret = r5l_load_log(log); - if (ret) { - struct mddev *mddev = log->rdev->mddev; - struct r5conf *conf = mddev->private; - + if (ret) r5l_exit_log(conf); - } return ret; } diff --git a/drivers/md/raid5-log.h b/drivers/md/raid5-log.h index 720e164e4d4d..db4c1262b380 100644 --- a/drivers/md/raid5-log.h +++ b/drivers/md/raid5-log.h @@ -29,7 +29,7 @@ void r5c_check_cached_full_stripe(struct r5conf *conf); extern struct md_sysfs_entry r5c_journal_mode; void r5c_update_on_rdev_error(struct mddev *mddev, struct md_rdev *rdev); bool r5c_big_stripe_cached(struct r5conf *conf, sector_t sect); -int r5l_start(struct r5l_log *log); +int r5l_start(struct r5conf *conf); struct dma_async_tx_descriptor * ops_run_partial_parity(struct stripe_head *sh, struct raid5_percpu *percpu, diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index 458d88faf2e9..83911affcd88 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -8040,7 +8040,7 @@ static int raid5_add_disk(struct mddev *mddev, struct md_rdev *rdev) if (ret) return ret; - ret = r5l_start(conf->log); + ret = r5l_start(conf); if (ret) return ret; @@ -8743,7 +8743,7 @@ static int raid5_start(struct mddev *mddev) { struct r5conf *conf = mddev->private; - return r5l_start(conf->log); + return r5l_start(conf); } static struct md_personality raid6_personality = From patchwork Thu May 26 16:35:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862526 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7815C43219 for ; Thu, 26 May 2022 16:36:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348191AbiEZQgh (ORCPT ); Thu, 26 May 2022 12:36:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348148AbiEZQgX (ORCPT ); Thu, 26 May 2022 12:36:23 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B44624CD40; Thu, 26 May 2022 09:36:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=ikRjs5UJZGQWBAATZMqTcVeSb4A/Fwku2+gm6UjqWh0=; b=mYjtZSuQTpE/WrxjnvOl4Ht9Dh ZMTk//H6ZUgBXcsCoVQiTVjwJlDPg+ewhYZpNGNXWPl9//kyjPgwVMus/GpnbIlD6pm/7MfliKCw5 U2oALwSH4pb4269W8BJUWPftGFfWtB+Educpt0ANyCOGz4z8JoySIxgz8GadaPTd+0gfCkriEdgwf Eaz7tVg1J9Qce4ZbOHOnL4FG5V8eKZ6p2BJ/TXwtO5xFL7KruG79S9euc/O3AiTT4ajIpRPES0ff9 xlgZxFORuvg548UjE/qALVv2UAs5geTOKxC/dWpUR99PoMRYb6PzVrastUlbEJN0wAA/b0hVmm8mv RqHBHRqg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTD-008A7W-8b; Thu, 26 May 2022 10:36:21 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGT8-0008XF-EQ; Thu, 26 May 2022 10:36:14 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe , Christoph Hellwig Date: Thu, 26 May 2022 10:35:51 -0600 Message-Id: <20220526163604.32736-5-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com, hch@lst.de X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 04/17] md/raid5-cache: Refactor r5l_flush_stripe_to_raid() to take a struct r5conf X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org All calls to r5l_flush_stripe_to_raid() have to dereference conf to get the log. The log pointer should be protected by RCU, but isn't. Push the dereference of conf->log into r5l_flush_stripe_to_raid() to reduce the number of sites that dereference conf->log so this can be improved. No functional changes intended. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5-cache.c | 3 ++- drivers/md/raid5-log.h | 4 ++-- drivers/md/raid5.c | 6 +++--- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 7c5dc45b1ea8..edff4e8d07dc 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -1289,8 +1289,9 @@ static void r5l_log_flush_endio(struct bio *bio) * only write stripes of an io_unit to raid disks till the io_unit is the first * one whose data/parity is in log. */ -void r5l_flush_stripe_to_raid(struct r5l_log *log) +void r5l_flush_stripe_to_raid(struct r5conf *conf) { + struct r5l_log *log = conf->log; bool do_flush; if (!log || !log->need_cache_flush) diff --git a/drivers/md/raid5-log.h b/drivers/md/raid5-log.h index db4c1262b380..5ace25d11ea4 100644 --- a/drivers/md/raid5-log.h +++ b/drivers/md/raid5-log.h @@ -6,7 +6,7 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev); void r5l_exit_log(struct r5conf *conf); int r5l_write_stripe(struct r5l_log *log, struct stripe_head *head_sh); void r5l_write_stripe_run(struct r5l_log *log); -void r5l_flush_stripe_to_raid(struct r5l_log *log); +void r5l_flush_stripe_to_raid(struct r5conf *conf); void r5l_stripe_write_finished(struct stripe_head *sh); int r5l_handle_flush_request(struct r5l_log *log, struct bio *bio); void r5l_quiesce(struct r5l_log *log, int quiesce); @@ -96,7 +96,7 @@ static inline void log_write_stripe_run(struct r5conf *conf) static inline void log_flush_stripe_to_raid(struct r5conf *conf) { if (conf->log) - r5l_flush_stripe_to_raid(conf->log); + r5l_flush_stripe_to_raid(conf); else if (raid5_has_ppl(conf)) ppl_write_stripe_run(conf); } diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index 83911affcd88..f1b55495de53 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -6427,7 +6427,7 @@ static int handle_active_stripes(struct r5conf *conf, int group, release_inactive_stripe_list(conf, temp_inactive_list, NR_STRIPE_HASH_LOCKS); - r5l_flush_stripe_to_raid(conf->log); + r5l_flush_stripe_to_raid(conf); if (release_inactive) { spin_lock_irq(&conf->device_lock); return 0; @@ -6483,7 +6483,7 @@ static void raid5_do_work(struct work_struct *work) flush_deferred_bios(conf); - r5l_flush_stripe_to_raid(conf->log); + r5l_flush_stripe_to_raid(conf); async_tx_issue_pending_all(); blk_finish_plug(&plug); @@ -6570,7 +6570,7 @@ static void raid5d(struct md_thread *thread) flush_deferred_bios(conf); - r5l_flush_stripe_to_raid(conf->log); + r5l_flush_stripe_to_raid(conf); async_tx_issue_pending_all(); blk_finish_plug(&plug); From patchwork Thu May 26 16:35:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862532 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04C03C433EF for ; Thu, 26 May 2022 16:36:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235458AbiEZQgq (ORCPT ); Thu, 26 May 2022 12:36:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42422 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348193AbiEZQgh (ORCPT ); Thu, 26 May 2022 12:36:37 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D35484FC7B; Thu, 26 May 2022 09:36:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=pefqnAQAPzwF0QGui2ZmdTDHAKSVQx30Ff/PluyW8NI=; b=s/6M5aFEhwfoxrUb5VaYNZMBX2 tS3ngVspS1/pZYA+3WaBs6E87Ivcs/UD+w/Ik1vEZRp4YqbORoSySkftV0/0hKOnU6G6KrQ7zqH1m BckEJB7hgiyJ/wg8Q+xpL2JWyN44/QuayvOCy+Zr7YlEEcJeD6smY2qfB0dZVG7lpvcPSenFXsgCm qEaf8ewHB7l53HxAVr3eGHhwoQcUntfpr7IY4eD1bTamoxQor0oj1yO3T6wIuE8Uiq/G7F+6BxKFb 4xYlhCIFFycbUl56/JglWMy3Fbqvt5+Xs+rbzPpkQ6z2jud8C/ZKnmKoI2BABAxbKB3EYNZlLFqt6 0xkk90IA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTH-008A7V-0y; Thu, 26 May 2022 10:36:23 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGT8-0008XI-Jf; Thu, 26 May 2022 10:36:14 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe , Christoph Hellwig Date: Thu, 26 May 2022 10:35:52 -0600 Message-Id: <20220526163604.32736-6-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com, hch@lst.de X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 05/17] md/raid5-cache: Refactor r5l_wake_reclaim() to take a struct r5conf X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org All calls to r5l_wake_reclaim() have to dereference conf to get the log. The log pointer should be protected by RCU, but isn't. Push the dereference of conf->log into r5l_wake_reclaim() to reduce the number of sites that dereference conf->log so this can be improved. No functional changes intended. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5-cache.c | 41 +++++++++++++++++++++------------------- drivers/md/raid5-log.h | 2 +- drivers/md/raid5.c | 2 +- 3 files changed, 24 insertions(+), 21 deletions(-) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index edff4e8d07dc..3427f95cb358 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -326,7 +326,20 @@ void r5c_handle_cached_data_endio(struct r5conf *conf, } } -void r5l_wake_reclaim(struct r5l_log *log, sector_t space); +static void __r5l_wake_reclaim(struct r5l_log *log, sector_t space) +{ + unsigned long target; + unsigned long new = (unsigned long)space; /* overflow in theory */ + + if (!log) + return; + do { + target = log->reclaim_target; + if (new < target) + return; + } while (cmpxchg(&log->reclaim_target, target, new) != target); + md_wakeup_thread(log->reclaim_thread); +} /* Check whether we should flush some stripes to free up stripe cache */ void r5c_check_stripe_cache_usage(struct r5conf *conf) @@ -349,7 +362,7 @@ void r5c_check_stripe_cache_usage(struct r5conf *conf) */ if (total_cached > conf->min_nr_stripes * 1 / 2 || atomic_read(&conf->empty_inactive_list_nr) > 0) - r5l_wake_reclaim(conf->log, 0); + __r5l_wake_reclaim(conf->log, 0); } /* @@ -368,7 +381,7 @@ void r5c_check_cached_full_stripe(struct r5conf *conf) if (atomic_read(&conf->r5c_cached_full_stripes) >= min(R5C_FULL_STRIPE_FLUSH_BATCH(conf), conf->chunk_sectors >> RAID5_STRIPE_SHIFT(conf))) - r5l_wake_reclaim(conf->log, 0); + __r5l_wake_reclaim(conf->log, 0); } /* @@ -444,7 +457,7 @@ static inline void r5c_update_log_state(struct r5l_log *log) clear_bit(R5C_LOG_TIGHT, &conf->cache_state); if (wake_reclaim) - r5l_wake_reclaim(log, 0); + __r5l_wake_reclaim(log, 0); } /* @@ -1087,7 +1100,7 @@ int r5l_write_stripe(struct r5l_log *log, struct stripe_head *sh) mutex_unlock(&log->io_mutex); if (wake_reclaim) - r5l_wake_reclaim(log, reserve); + __r5l_wake_reclaim(log, reserve); return 0; } @@ -1239,7 +1252,7 @@ static void __r5l_stripe_write_finished(struct r5l_io_unit *io) if (r5l_reclaimable_space(log) > log->max_free_space || test_bit(R5C_LOG_TIGHT, &conf->cache_state)) - r5l_wake_reclaim(log, 0); + __r5l_wake_reclaim(log, 0); spin_unlock_irqrestore(&log->io_list_lock, flags); wake_up(&log->iounit_wait); @@ -1564,19 +1577,9 @@ static void r5l_reclaim_thread(struct md_thread *thread) r5l_do_reclaim(log); } -void r5l_wake_reclaim(struct r5l_log *log, sector_t space) +void r5l_wake_reclaim(struct r5conf *conf, sector_t space) { - unsigned long target; - unsigned long new = (unsigned long)space; /* overflow in theory */ - - if (!log) - return; - do { - target = log->reclaim_target; - if (new < target) - return; - } while (cmpxchg(&log->reclaim_target, target, new) != target); - md_wakeup_thread(log->reclaim_thread); + __r5l_wake_reclaim(conf->log, space); } void r5l_quiesce(struct r5l_log *log, int quiesce) @@ -1588,7 +1591,7 @@ void r5l_quiesce(struct r5l_log *log, int quiesce) mddev = log->rdev->mddev; wake_up(&mddev->sb_wait); kthread_park(log->reclaim_thread->tsk); - r5l_wake_reclaim(log, MaxSector); + __r5l_wake_reclaim(log, MaxSector); r5l_do_reclaim(log); } else kthread_unpark(log->reclaim_thread->tsk); diff --git a/drivers/md/raid5-log.h b/drivers/md/raid5-log.h index 5ace25d11ea4..7594d2d80520 100644 --- a/drivers/md/raid5-log.h +++ b/drivers/md/raid5-log.h @@ -18,7 +18,7 @@ void r5c_finish_stripe_write_out(struct r5conf *conf, struct stripe_head *sh, struct stripe_head_state *s); void r5c_release_extra_page(struct stripe_head *sh); void r5c_use_extra_page(struct stripe_head *sh); -void r5l_wake_reclaim(struct r5l_log *log, sector_t space); +void r5l_wake_reclaim(struct r5conf *conf, sector_t space); void r5c_handle_cached_data_endio(struct r5conf *conf, struct stripe_head *sh, int disks); int r5c_cache_data(struct r5l_log *log, struct stripe_head *sh); diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index f1b55495de53..0d486f8aaf87 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -742,7 +742,7 @@ raid5_get_active_stripe(struct r5conf *conf, sector_t sector, if (!sh) { set_bit(R5_INACTIVE_BLOCKED, &conf->cache_state); - r5l_wake_reclaim(conf->log, 0); + r5l_wake_reclaim(conf, 0); wait_event_lock_irq( conf->wait_for_stripe, !list_empty(conf->inactive_list + hash) && From patchwork Thu May 26 16:35:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 359D9C433EF for ; Thu, 26 May 2022 16:36:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348211AbiEZQgv (ORCPT ); Thu, 26 May 2022 12:36:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348192AbiEZQgh (ORCPT ); Thu, 26 May 2022 12:36:37 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CDA694FC5D; Thu, 26 May 2022 09:36:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=rT5G8qxq1d10C0jx/Qw/01z/duQzbK3ZzkhAFxNw4j4=; b=j7sD+dEOmb6nZOW+8VnttqLZAl OcE3FwlRX6FC1LFGbzUX4Xh43gqGyzcMio1PBXhC5T3ITchUJM1JUfQBnIuPXZkB2funUTavAQqnv 206hHNGPz7AH9YGAz3Pc3vP3+JDxZVGmBXfTOZJqey6shjSFnbw3PBwC4eAm2en5WWkhcSMwTyEfC V3XfQx0+VvtlwXSOCNp1UwAEaf0v4rOBbfMgSpUHfjl8Rywc1o3jEJ5hAt3ShaMRFzCnrVdCqNXkO BgVBz4BL7TwdDQRCsEgMN9odrDI1YaEe88jBoVAc5l2W/MsxGNOiyBsWnBv+oKBWR+E/70O28GUl1 w8gTv+Hw==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTG-008A7T-Sf; Thu, 26 May 2022 10:36:23 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGT8-0008XL-P6; Thu, 26 May 2022 10:36:14 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe , Christoph Hellwig Date: Thu, 26 May 2022 10:35:53 -0600 Message-Id: <20220526163604.32736-7-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com, hch@lst.de X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 06/17] md/raid5-cache: Refactor remaining functions to take a r5conf X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org Many of the interface functions dereference conf->log in inline helpers in raid5-log.h. Refactor all off these to dereference conf->log inside the function instead. This will help to fix the rcu locking when accessing conf->log. No functional changes intended. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5-cache.c | 18 +++++++++++------- drivers/md/raid5-log.h | 20 ++++++++++---------- 2 files changed, 21 insertions(+), 17 deletions(-) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 3427f95cb358..24110b687055 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -1006,9 +1006,9 @@ static inline void r5l_add_no_space_stripe(struct r5l_log *log, * running in raid5d, where reclaim could wait for raid5d too (when it flushes * data from log to raid disks), so we shouldn't wait for reclaim here */ -int r5l_write_stripe(struct r5l_log *log, struct stripe_head *sh) +int r5l_write_stripe(struct r5conf *conf, struct stripe_head *sh) { - struct r5conf *conf = sh->raid_conf; + struct r5l_log *log = conf->log; int write_disks = 0; int data_pages, parity_pages; int reserve; @@ -1104,8 +1104,9 @@ int r5l_write_stripe(struct r5l_log *log, struct stripe_head *sh) return 0; } -void r5l_write_stripe_run(struct r5l_log *log) +void r5l_write_stripe_run(struct r5conf *conf) { + struct r5l_log *log = conf->log; if (!log) return; mutex_lock(&log->io_mutex); @@ -1113,8 +1114,10 @@ void r5l_write_stripe_run(struct r5l_log *log) mutex_unlock(&log->io_mutex); } -int r5l_handle_flush_request(struct r5l_log *log, struct bio *bio) +int r5l_handle_flush_request(struct r5conf *conf, struct bio *bio) { + struct r5l_log *log = conf->log; + if (log->r5c_journal_mode == R5C_JOURNAL_MODE_WRITE_THROUGH) { /* * in write through (journal only) @@ -1582,8 +1585,9 @@ void r5l_wake_reclaim(struct r5conf *conf, sector_t space) __r5l_wake_reclaim(conf->log, space); } -void r5l_quiesce(struct r5l_log *log, int quiesce) +void r5l_quiesce(struct r5conf *conf, int quiesce) { + struct r5l_log *log = conf->log; struct mddev *mddev; if (quiesce) { @@ -2892,9 +2896,9 @@ void r5c_finish_stripe_write_out(struct r5conf *conf, set_bit(STRIPE_HANDLE, &sh->state); } -int r5c_cache_data(struct r5l_log *log, struct stripe_head *sh) +int r5c_cache_data(struct r5conf *conf, struct stripe_head *sh) { - struct r5conf *conf = sh->raid_conf; + struct r5l_log *log = conf->log; int pages = 0; int reserve; int i; diff --git a/drivers/md/raid5-log.h b/drivers/md/raid5-log.h index 7594d2d80520..e39db84c5de0 100644 --- a/drivers/md/raid5-log.h +++ b/drivers/md/raid5-log.h @@ -4,12 +4,12 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev); void r5l_exit_log(struct r5conf *conf); -int r5l_write_stripe(struct r5l_log *log, struct stripe_head *head_sh); -void r5l_write_stripe_run(struct r5l_log *log); +int r5l_write_stripe(struct r5conf *conf, struct stripe_head *head_sh); +void r5l_write_stripe_run(struct r5conf *conf); void r5l_flush_stripe_to_raid(struct r5conf *conf); void r5l_stripe_write_finished(struct stripe_head *sh); -int r5l_handle_flush_request(struct r5l_log *log, struct bio *bio); -void r5l_quiesce(struct r5l_log *log, int quiesce); +int r5l_handle_flush_request(struct r5conf *conf, struct bio *bio); +void r5l_quiesce(struct r5conf *conf, int quiesce); bool r5l_log_disk_error(struct r5conf *conf); bool r5c_conf_is_writeback(struct r5conf *conf); int r5c_try_caching_write(struct r5conf *conf, struct stripe_head *sh, @@ -21,7 +21,7 @@ void r5c_use_extra_page(struct stripe_head *sh); void r5l_wake_reclaim(struct r5conf *conf, sector_t space); void r5c_handle_cached_data_endio(struct r5conf *conf, struct stripe_head *sh, int disks); -int r5c_cache_data(struct r5l_log *log, struct stripe_head *sh); +int r5c_cache_data(struct r5conf *conf, struct stripe_head *sh); void r5c_make_stripe_write_out(struct stripe_head *sh); void r5c_flush_cache(struct r5conf *conf, int num); void r5c_check_stripe_cache_usage(struct r5conf *conf); @@ -63,10 +63,10 @@ static inline int log_stripe(struct stripe_head *sh, struct stripe_head_state *s /* writing out phase */ if (s->waiting_extra_page) return 0; - return r5l_write_stripe(conf->log, sh); + return r5l_write_stripe(conf, sh); } else if (test_bit(STRIPE_LOG_TRAPPED, &sh->state)) { /* caching phase */ - return r5c_cache_data(conf->log, sh); + return r5c_cache_data(conf, sh); } } else if (raid5_has_ppl(conf)) { return ppl_write_stripe(conf, sh); @@ -88,7 +88,7 @@ static inline void log_stripe_write_finished(struct stripe_head *sh) static inline void log_write_stripe_run(struct r5conf *conf) { if (conf->log) - r5l_write_stripe_run(conf->log); + r5l_write_stripe_run(conf); else if (raid5_has_ppl(conf)) ppl_write_stripe_run(conf); } @@ -106,7 +106,7 @@ static inline int log_handle_flush_request(struct r5conf *conf, struct bio *bio) int ret = -ENODEV; if (conf->log) - ret = r5l_handle_flush_request(conf->log, bio); + ret = r5l_handle_flush_request(conf, bio); else if (raid5_has_ppl(conf)) ret = ppl_handle_flush_request(conf->log, bio); @@ -116,7 +116,7 @@ static inline int log_handle_flush_request(struct r5conf *conf, struct bio *bio) static inline void log_quiesce(struct r5conf *conf, int quiesce) { if (conf->log) - r5l_quiesce(conf->log, quiesce); + r5l_quiesce(conf, quiesce); else if (raid5_has_ppl(conf)) ppl_quiesce(conf, quiesce); } From patchwork Thu May 26 16:35:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61A17C433F5 for ; Thu, 26 May 2022 16:36:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345155AbiEZQgn (ORCPT ); Thu, 26 May 2022 12:36:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348185AbiEZQgh (ORCPT ); Thu, 26 May 2022 12:36:37 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3E344F471; Thu, 26 May 2022 09:36:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=Vtrw1h6V0YuxVxDvHliU2zCW8YmoFTF3FBe7fyBj6JM=; b=NTsJ0mYQB0JcH8HNTkanYMfwYP hJyPSo4rfmQQUN+DBDPoBysTmJiMT5LvYJaUYl3GzUS3bVDyRRb9IcloHOe9G+6xFONsxMQb66cRB LWJRB8oKr1/o7rUWEAUIZpAfZW2KIpiCi05dGjSa5qE4BOksd9LGInH1liVaGWZf+toVdblV6YEDU Tuk8HAhYeNM6utTM81CW2/2mIsgas5vXyuIUjosm3ahdY6zvPzXXD9A0uluFXZMA47U/cqNTtP6zJ uKsENqw6ZeDlxr78mJlsijdZnSrk6ybq5xKuFFrP64xmdCIj2nU2+pN5PbyDlDigSSlDMI2baqEUT QFnOsdQA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTG-008A7W-T5; Thu, 26 May 2022 10:36:23 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGT8-0008XO-Ul; Thu, 26 May 2022 10:36:15 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe , Christoph Hellwig Date: Thu, 26 May 2022 10:35:54 -0600 Message-Id: <20220526163604.32736-8-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com, hch@lst.de X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 07/17] md/raid5-ppl: Drop unused argument from ppl_handle_flush_request() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org ppl_handle_flush_request() takes an struct r5log argument but doesn't use it. It has no buisiness taking this argument as it is only used by raid5-cache and has no way to derference it anyway. Remove the argument. No functional changes intended. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5-log.h | 4 ++-- drivers/md/raid5-ppl.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/md/raid5-log.h b/drivers/md/raid5-log.h index e39db84c5de0..1c184fe20939 100644 --- a/drivers/md/raid5-log.h +++ b/drivers/md/raid5-log.h @@ -41,7 +41,7 @@ void ppl_write_stripe_run(struct r5conf *conf); void ppl_stripe_write_finished(struct stripe_head *sh); int ppl_modify_log(struct r5conf *conf, struct md_rdev *rdev, bool add); void ppl_quiesce(struct r5conf *conf, int quiesce); -int ppl_handle_flush_request(struct r5l_log *log, struct bio *bio); +int ppl_handle_flush_request(struct bio *bio); extern struct md_sysfs_entry ppl_write_hint; static inline bool raid5_has_log(struct r5conf *conf) @@ -108,7 +108,7 @@ static inline int log_handle_flush_request(struct r5conf *conf, struct bio *bio) if (conf->log) ret = r5l_handle_flush_request(conf, bio); else if (raid5_has_ppl(conf)) - ret = ppl_handle_flush_request(conf->log, bio); + ret = ppl_handle_flush_request(bio); return ret; } diff --git a/drivers/md/raid5-ppl.c b/drivers/md/raid5-ppl.c index 973e2e06f19c..4f5bdb4cad2b 100644 --- a/drivers/md/raid5-ppl.c +++ b/drivers/md/raid5-ppl.c @@ -679,7 +679,7 @@ void ppl_quiesce(struct r5conf *conf, int quiesce) } } -int ppl_handle_flush_request(struct r5l_log *log, struct bio *bio) +int ppl_handle_flush_request(struct bio *bio) { if (bio->bi_iter.bi_size == 0) { bio_endio(bio); From patchwork Thu May 26 16:35:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80CC7C433EF for ; Thu, 26 May 2022 16:36:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348207AbiEZQgo (ORCPT ); Thu, 26 May 2022 12:36:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346042AbiEZQge (ORCPT ); Thu, 26 May 2022 12:36:34 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0C884EDDF; Thu, 26 May 2022 09:36:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=EJp+9sZ34Y9fqOBjq+Gi6IuCQ2335JDwkpGO6bYI9Ak=; b=mn47hcEhuA7bfg8fQQdfCen6LP 2RY8KUU6xyftTpn8x/YD+1VI832dC8ieoNAOk4oOqYF50OwBaGMBxQDvBpcJxzpVzSuulK1wQWLuo 0I6K8zQtl6I2SzfGlPK090pwUO4ZoK8ky3WD93/sLM2/DzEKPBJfJNi7T0XyaFwU7ZbX3O+QsxQ7H JZJYYsD9OXB/kmejfwat+D0sH0KqojULRWSdwL/LAHBAvCprw8GFhVHYZquXZPJ3chXgRwdVJe2mc oMxeYQzePKmXHfk7U309Ujw9cwatN0Hemv0aUbLl0cxdC5LTYDiGi2PvCFaGPQyIltfXtjddlxfpl lVDEgkDQ==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTG-008A7R-NR; Thu, 26 May 2022 10:36:23 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGT9-0008XS-4f; Thu, 26 May 2022 10:36:15 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe , Christoph Hellwig Date: Thu, 26 May 2022 10:35:55 -0600 Message-Id: <20220526163604.32736-9-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com, hch@lst.de X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 08/17] md/raid5-cache: Pass the log through to r5c_finish_cache_stripe() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org r5c_finish_cache_stripe() dereferences conf->log, which will need an rcu_read_lock(). But that is not necessary here as the log is already available in the call sites through other means. No functional changes intended. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5-cache.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 24110b687055..8284ce3e5cf6 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -507,10 +507,9 @@ static void r5c_handle_parity_cached(struct stripe_head *sh) * Setting proper flags after writing (or flushing) data and/or parity to the * log device. This is called from r5l_log_endio() or r5l_log_flush_endio(). */ -static void r5c_finish_cache_stripe(struct stripe_head *sh) +static void r5c_finish_cache_stripe(struct r5l_log *log, + struct stripe_head *sh) { - struct r5l_log *log = sh->raid_conf->log; - if (log->r5c_journal_mode == R5C_JOURNAL_MODE_WRITE_THROUGH) { BUG_ON(test_bit(STRIPE_R5C_CACHING, &sh->state)); /* @@ -528,14 +527,14 @@ static void r5c_finish_cache_stripe(struct stripe_head *sh) } } -static void r5l_io_run_stripes(struct r5l_io_unit *io) +static void r5l_io_run_stripes(struct r5l_log *log, struct r5l_io_unit *io) { struct stripe_head *sh, *next; list_for_each_entry_safe(sh, next, &io->stripe_list, log_list) { list_del_init(&sh->log_list); - r5c_finish_cache_stripe(sh); + r5c_finish_cache_stripe(log, sh); set_bit(STRIPE_HANDLE, &sh->state); raid5_release_stripe(sh); @@ -554,7 +553,7 @@ static void r5l_log_run_stripes(struct r5l_log *log) break; list_move_tail(&io->log_sibling, &log->finished_ios); - r5l_io_run_stripes(io); + r5l_io_run_stripes(log, io); } } @@ -1284,7 +1283,7 @@ static void r5l_log_flush_endio(struct bio *bio) spin_lock_irqsave(&log->io_list_lock, flags); list_for_each_entry(io, &log->flushing_ios, log_sibling) - r5l_io_run_stripes(io); + r5l_io_run_stripes(log, io); list_splice_tail_init(&log->flushing_ios, &log->finished_ios); spin_unlock_irqrestore(&log->io_list_lock, flags); From patchwork Thu May 26 16:35:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 020BEC433EF for ; Thu, 26 May 2022 16:36:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348252AbiEZQgy (ORCPT ); Thu, 26 May 2022 12:36:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348175AbiEZQgd (ORCPT ); Thu, 26 May 2022 12:36:33 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 938763DA53; Thu, 26 May 2022 09:36:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=+Dyt/77+mWlqTxiTO6FVET0axWZRroBByVkNU8RIwCQ=; b=fXeq9jvSJ9kFbrzaqV1w8Lksmt dA89QGcjrCaHzPYQj+8TTmlLALxSmi5ZM/4QYuEFSQwNvy2xn/dtBE9w2MJysnfUhZZBGedJuQI51 I6dV7qAqXwHhHXOQL6PHlgGe1KijeTGp0n9/NCUtM+CaJ45C+pPM3ahFsf9pbhC2OWW46gsNxctZe RrM0B4k+CpDrw0UoGTY5VBAZ3Y8vD7xtouwRXGKAZos3ZCMsIAMTdqP6sBbh/B+QsWOZ+QV0a2mbE 0+GWU4tIvTIsT9JtdguINhAFq+vwN3e3ruQOLZZTq19C3Y1NjhX8Ng/Il1/9WkYlw8LrNKN1h9leV kG/ixbow==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTF-008A7W-Vh; Thu, 26 May 2022 10:36:22 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGT9-0008XX-GJ; Thu, 26 May 2022 10:36:15 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe , Christoph Hellwig Date: Thu, 26 May 2022 10:35:56 -0600 Message-Id: <20220526163604.32736-10-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com, hch@lst.de X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 09/17] md/raid5-cache: Don't pass conf to r5c_calculate_new_cp() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org r5c_calculate_new_cp() only uses conf to dereference the log which both callers already have a pointer to and no longer need to obtain a conf through a complicated dereference chain for this use. No functional changes intended. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5-cache.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 8284ce3e5cf6..9a8f2a988b03 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -1168,10 +1168,9 @@ static void r5l_run_no_space_stripes(struct r5l_log *log) * for write through mode, returns log->next_checkpoint * for write back, returns log_start of first sh in stripe_in_journal_list */ -static sector_t r5c_calculate_new_cp(struct r5conf *conf) +static sector_t r5c_calculate_new_cp(struct r5l_log *log) { struct stripe_head *sh; - struct r5l_log *log = conf->log; sector_t new_cp; unsigned long flags; @@ -1179,12 +1178,12 @@ static sector_t r5c_calculate_new_cp(struct r5conf *conf) return log->next_checkpoint; spin_lock_irqsave(&log->stripe_in_journal_lock, flags); - if (list_empty(&conf->log->stripe_in_journal_list)) { + if (list_empty(&log->stripe_in_journal_list)) { /* all stripes flushed */ spin_unlock_irqrestore(&log->stripe_in_journal_lock, flags); return log->next_checkpoint; } - sh = list_first_entry(&conf->log->stripe_in_journal_list, + sh = list_first_entry(&log->stripe_in_journal_list, struct stripe_head, r5c); new_cp = sh->log_start; spin_unlock_irqrestore(&log->stripe_in_journal_lock, flags); @@ -1193,10 +1192,8 @@ static sector_t r5c_calculate_new_cp(struct r5conf *conf) static sector_t r5l_reclaimable_space(struct r5l_log *log) { - struct r5conf *conf = log->rdev->mddev->private; - return r5l_ring_distance(log, log->last_checkpoint, - r5c_calculate_new_cp(conf)); + r5c_calculate_new_cp(log)); } static void r5l_run_no_mem_stripe(struct r5l_log *log) @@ -1517,7 +1514,6 @@ static void r5c_do_reclaim(struct r5conf *conf) static void r5l_do_reclaim(struct r5l_log *log) { - struct r5conf *conf = log->rdev->mddev->private; sector_t reclaim_target = xchg(&log->reclaim_target, 0); sector_t reclaimable; sector_t next_checkpoint; @@ -1546,7 +1542,7 @@ static void r5l_do_reclaim(struct r5l_log *log) log->io_list_lock); } - next_checkpoint = r5c_calculate_new_cp(conf); + next_checkpoint = r5c_calculate_new_cp(log); spin_unlock_irq(&log->io_list_lock); if (reclaimable == 0 || !write_super) From patchwork Thu May 26 16:35:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA213C433F5 for ; Thu, 26 May 2022 16:36:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346481AbiEZQgt (ORCPT ); Thu, 26 May 2022 12:36:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348194AbiEZQgh (ORCPT ); Thu, 26 May 2022 12:36:37 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E9644EA36; Thu, 26 May 2022 09:36:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=TA8S/q/SfyBaqNlhSvQ2KjlZ3mUYd8WeiUhUSFWVr1s=; b=HUzG+l2VVLsTRICWLJ1QCKZfX6 uKiXwxPzdJWLObHDwtqzfjsIuoUGdibe82tJPnfmwdGficPPfeHFjP2qBFgWFLzm2yK+UDAZQDes4 J3lzsVJyNwI3D4xXZsg1R7L9dM60Vi0OKrw61tvgZGjRFpq/FpLSOX2yuGaDiXrqKq/oxhuWQAOal ESWF61LspMkrqKVFfXkH+1v5BHy7x13gb3ZLENFj36vlQNxNFQVb8EOnhrhkKkmmgr3YkVpgvSlQ7 Y0YNNXeNezF+FD44Owxv7gBMIIkG4EgbenIQuXE+2Rd8MObmvwMhYXzFvBvWom2wiROdLEkNWqtPD bv0EAQog==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTF-008A7S-Vj; Thu, 26 May 2022 10:36:23 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGT9-0008Xb-ON; Thu, 26 May 2022 10:36:15 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe , Christoph Hellwig Date: Thu, 26 May 2022 10:35:57 -0600 Message-Id: <20220526163604.32736-11-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com, hch@lst.de X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 10/17] md/raid5-cache: Take struct r5l_log in r5c_log_required_to_flush_cache() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org The only call site of r5c_log_required_to_flush_cache() already has a log pointer and has already checked that it is writeback. So the dereference and writeback check is redundant. Just pass the log pointer. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5-cache.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 9a8f2a988b03..a3c4d43d6deb 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -412,13 +412,9 @@ void r5c_check_cached_full_stripe(struct r5conf *conf) * (stripe_in_journal_count) * (max_degraded + 1) + * (group_cnt + 1) * (raid_disks - max_degraded) */ -static sector_t r5c_log_required_to_flush_cache(struct r5conf *conf) +static sector_t r5c_log_required_to_flush_cache(struct r5conf *conf, + struct r5l_log *log) { - struct r5l_log *log = conf->log; - - if (!r5c_is_writeback(log)) - return 0; - return BLOCK_SECTORS * ((conf->max_degraded + 1) * atomic_read(&log->stripe_in_journal_count) + (conf->raid_disks - conf->max_degraded) * (conf->group_cnt + 1)); @@ -443,7 +439,7 @@ static inline void r5c_update_log_state(struct r5l_log *log) free_space = r5l_ring_distance(log, log->log_start, log->last_checkpoint); - reclaim_space = r5c_log_required_to_flush_cache(conf); + reclaim_space = r5c_log_required_to_flush_cache(conf, log); if (free_space < 2 * reclaim_space) set_bit(R5C_LOG_CRITICAL, &conf->cache_state); else { From patchwork Thu May 26 16:35:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06C98C433EF for ; Thu, 26 May 2022 16:36:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348255AbiEZQg4 (ORCPT ); Thu, 26 May 2022 12:36:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348154AbiEZQgY (ORCPT ); Thu, 26 May 2022 12:36:24 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5FC1E3B3C0; Thu, 26 May 2022 09:36:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=QyHrnH5ZuXktkhsga6G9I5lk9psWpw8ft+8WQu8yre4=; b=SGF2gA9OrU3pLiVsW/FZQRJago JPn8NFyCFlLAhhLSKCOPdFrg+2ZaD62MRS8+Mf9euTicQJDdWT9fumN5CZKBSF8RLATO+lO2E6J/V V5cTyn7QfpB4D7WjTB6N0oqR6POltT3STWuv6HzCW9EULMGkw+d2pyTiWrH3bKl0ENN9qtykN0NM8 QRab/EX5hzeTDzY8NKGBzzEalOOVBOjjXjLUbxt78ClUplEnU6Hyshrzi29hdd0uI74ibJID2iy2p eyj7CRN9xEXhznyhD/oYRd92gXQ4pfXUKk9dGKab4aCPp8RXp1ZD2pAl9kQeFSeHglqx8fonrU94h AduRxbpA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTF-008A7R-Rs; Thu, 26 May 2022 10:36:22 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGT9-0008Xf-VQ; Thu, 26 May 2022 10:36:16 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe , Christoph Hellwig Date: Thu, 26 May 2022 10:35:58 -0600 Message-Id: <20220526163604.32736-12-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com, hch@lst.de X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 11/17] md/raid5: Ensure array is suspended for calls to log_exit() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org The raid5-cache code relies on there being no IO in flight when log_exit() is called. There are two places where this is not guaranteed so add mddev_suspend() and mddev_resume() calls to these sites. The site in raid5_remove_disk() has a comment saying that it is called in raid5d and thus cannot wait for pending writes; however that does not appear to be correct anymore (if it ever was) as raid5_remove_disk() is called from hot_remove_disk() which only appears to be called in the md_ioctl(). Thus, the comment is removed, as well as the racy check and replaced with calls to suspend/resume. The site in raid5_change_consistency_policy() is in the error path, and another similar call site already has suspend/resume calls just below it; so it should be equally safe to make that change here. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index 0d486f8aaf87..4b25a0262cb5 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -7938,18 +7938,9 @@ static int raid5_remove_disk(struct mddev *mddev, struct md_rdev *rdev) print_raid5_conf(conf); if (test_bit(Journal, &rdev->flags) && conf->log) { - /* - * we can't wait pending write here, as this is called in - * raid5d, wait will deadlock. - * neilb: there is no locking about new writes here, - * so this cannot be safe. - */ - if (atomic_read(&conf->active_stripes) || - atomic_read(&conf->r5c_cached_full_stripes) || - atomic_read(&conf->r5c_cached_partial_stripes)) { - return -EBUSY; - } + mddev_suspend(mddev); log_exit(conf); + mddev_resume(mddev); return 0; } if (rdev == rcu_access_pointer(p->rdev)) @@ -8697,8 +8688,11 @@ static int raid5_change_consistency_policy(struct mddev *mddev, const char *buf) err = log_init(conf, NULL, true); if (!err) { err = resize_stripes(conf, conf->pool_size); - if (err) + if (err) { + mddev_suspend(mddev); log_exit(conf); + mddev_resume(mddev); + } } } else err = -EINVAL; From patchwork Thu May 26 16:35:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C962C4332F for ; Thu, 26 May 2022 16:36:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348240AbiEZQgp (ORCPT ); Thu, 26 May 2022 12:36:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344564AbiEZQge (ORCPT ); Thu, 26 May 2022 12:36:34 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C231948E61; Thu, 26 May 2022 09:36:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=XV1DXWrKKUyaYd2rc55uj5hiPcVYzJEV2A7rBcUZk4Q=; b=MAm+qCnVxM/C65FlKFwRk9eaTh sHUhQOA8U1pN6j1RcbR64Gi1JDtEar1gF/WD8Il1BKzm/JyEOXD++tsl+9zJkCbCh2ttIx73vjZai 2M/nf7Xw+NOwPQZPtcnU2gsqcmDJdvdO5SjYWqHwEV28TMpGDWbaGyytQP7W8Qt6RJ0KJkejcvi7S Woy3kIHq73SNuRSW8SllyaZfDQhlTh3xRGZGgDFFiSUPMydlYdXjZ7a17c3iW50NeSxV4OKT2E1Qy jV8OoPeKKzibrwASWzwOX9g4bB9VS8GV03qUolDYhJVGl8E9Qo3n7DjavVILftIUwx85jLowSWSMq vkbqAHcw==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTF-008A7V-RK; Thu, 26 May 2022 10:36:22 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGTA-0008Xj-6S; Thu, 26 May 2022 10:36:16 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe Date: Thu, 26 May 2022 10:35:59 -0600 Message-Id: <20220526163604.32736-13-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 12/17] md/raid5-cache: Move struct r5l_log definition to raid5-log.h X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org Move struct r5l_log definition to raid5-log.h. While this reduces encapsulation, it is necessary for the definition of r5l_log to be public so that rcu_access_pointer() can be used on conf-log in the next patch. rcu_access_pointer(p) doesn't technically dereference the log pointer however, it does use typeof(*p) and some older GCC versions (anything earlier than gcc-10) will wrongly try to dereference the structure: include/linux/rcupdate.h:384:9: error: dereferencing pointer to incomplete type ‘struct r5l_log’ typeof(*p) *local = (typeof(*p) *__force)READ_ONCE(p); \ ^ include/linux/rcupdate.h:495:31: note: in expansion of macro ‘__rcu_access_pointer’ #define rcu_access_pointer(p) __rcu_access_pointer((p), __UNIQUE_ID(rcu), __rcu) To prevent this, simply provide the definition where rcu_access_pointer() may be used. Reported-by: Donald Buczek Signed-off-by: Logan Gunthorpe --- drivers/md/raid5-cache.c | 75 ---------------------------------------- drivers/md/raid5-log.h | 75 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+), 75 deletions(-) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index a3c4d43d6deb..6349cfaae2c8 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -79,81 +79,6 @@ static char *r5c_journal_mode_str[] = {"write-through", * - return IO for pending writes */ -struct r5l_log { - struct md_rdev *rdev; - - u32 uuid_checksum; - - sector_t device_size; /* log device size, round to - * BLOCK_SECTORS */ - sector_t max_free_space; /* reclaim run if free space is at - * this size */ - - sector_t last_checkpoint; /* log tail. where recovery scan - * starts from */ - u64 last_cp_seq; /* log tail sequence */ - - sector_t log_start; /* log head. where new data appends */ - u64 seq; /* log head sequence */ - - sector_t next_checkpoint; - - struct mutex io_mutex; - struct r5l_io_unit *current_io; /* current io_unit accepting new data */ - - spinlock_t io_list_lock; - struct list_head running_ios; /* io_units which are still running, - * and have not yet been completely - * written to the log */ - struct list_head io_end_ios; /* io_units which have been completely - * written to the log but not yet written - * to the RAID */ - struct list_head flushing_ios; /* io_units which are waiting for log - * cache flush */ - struct list_head finished_ios; /* io_units which settle down in log disk */ - struct bio flush_bio; - - struct list_head no_mem_stripes; /* pending stripes, -ENOMEM */ - - struct kmem_cache *io_kc; - mempool_t io_pool; - struct bio_set bs; - mempool_t meta_pool; - - struct md_thread *reclaim_thread; - unsigned long reclaim_target; /* number of space that need to be - * reclaimed. if it's 0, reclaim spaces - * used by io_units which are in - * IO_UNIT_STRIPE_END state (eg, reclaim - * dones't wait for specific io_unit - * switching to IO_UNIT_STRIPE_END - * state) */ - wait_queue_head_t iounit_wait; - - struct list_head no_space_stripes; /* pending stripes, log has no space */ - spinlock_t no_space_stripes_lock; - - bool need_cache_flush; - - /* for r5c_cache */ - enum r5c_journal_mode r5c_journal_mode; - - /* all stripes in r5cache, in the order of seq at sh->log_start */ - struct list_head stripe_in_journal_list; - - spinlock_t stripe_in_journal_lock; - atomic_t stripe_in_journal_count; - - /* to submit async io_units, to fulfill ordering of flush */ - struct work_struct deferred_io_work; - /* to disable write back during in degraded mode */ - struct work_struct disable_writeback_work; - - /* to for chunk_aligned_read in writeback mode, details below */ - spinlock_t tree_lock; - struct radix_tree_root big_stripe_tree; -}; - /* * Enable chunk_aligned_read() with write back cache. * diff --git a/drivers/md/raid5-log.h b/drivers/md/raid5-log.h index 1c184fe20939..5948c41f1f2e 100644 --- a/drivers/md/raid5-log.h +++ b/drivers/md/raid5-log.h @@ -2,6 +2,81 @@ #ifndef _RAID5_LOG_H #define _RAID5_LOG_H +struct r5l_log { + struct md_rdev *rdev; + + u32 uuid_checksum; + + sector_t device_size; /* log device size, round to + * BLOCK_SECTORS */ + sector_t max_free_space; /* reclaim run if free space is at + * this size */ + + sector_t last_checkpoint; /* log tail. where recovery scan + * starts from */ + u64 last_cp_seq; /* log tail sequence */ + + sector_t log_start; /* log head. where new data appends */ + u64 seq; /* log head sequence */ + + sector_t next_checkpoint; + + struct mutex io_mutex; + struct r5l_io_unit *current_io; /* current io_unit accepting new data */ + + spinlock_t io_list_lock; + struct list_head running_ios; /* io_units which are still running, + * and have not yet been completely + * written to the log */ + struct list_head io_end_ios; /* io_units which have been completely + * written to the log but not yet written + * to the RAID */ + struct list_head flushing_ios; /* io_units which are waiting for log + * cache flush */ + struct list_head finished_ios; /* io_units which settle down in log disk */ + struct bio flush_bio; + + struct list_head no_mem_stripes; /* pending stripes, -ENOMEM */ + + struct kmem_cache *io_kc; + mempool_t io_pool; + struct bio_set bs; + mempool_t meta_pool; + + struct md_thread *reclaim_thread; + unsigned long reclaim_target; /* number of space that need to be + * reclaimed. if it's 0, reclaim spaces + * used by io_units which are in + * IO_UNIT_STRIPE_END state (eg, reclaim + * dones't wait for specific io_unit + * switching to IO_UNIT_STRIPE_END + * state) */ + wait_queue_head_t iounit_wait; + + struct list_head no_space_stripes; /* pending stripes, log has no space */ + spinlock_t no_space_stripes_lock; + + bool need_cache_flush; + + /* for r5c_cache */ + enum r5c_journal_mode r5c_journal_mode; + + /* all stripes in r5cache, in the order of seq at sh->log_start */ + struct list_head stripe_in_journal_list; + + spinlock_t stripe_in_journal_lock; + atomic_t stripe_in_journal_count; + + /* to submit async io_units, to fulfill ordering of flush */ + struct work_struct deferred_io_work; + /* to disable write back during in degraded mode */ + struct work_struct disable_writeback_work; + + /* to for chunk_aligned_read in writeback mode, details below */ + spinlock_t tree_lock; + struct radix_tree_root big_stripe_tree; +}; + int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev); void r5l_exit_log(struct r5conf *conf); int r5l_write_stripe(struct r5conf *conf, struct stripe_head *head_sh); From patchwork Thu May 26 16:36:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8A1FC433FE for ; Thu, 26 May 2022 16:36:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348242AbiEZQgw (ORCPT ); Thu, 26 May 2022 12:36:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42356 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348177AbiEZQgd (ORCPT ); Thu, 26 May 2022 12:36:33 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9395F3ED04; Thu, 26 May 2022 09:36:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=LVPA3vmvSA7P6Ad1y1+3/RRw2c624cA8sePqr/TnEdw=; b=f9jflO1yh9JgjMhbxeu7BjEpsG /V9H282fl47jauXwBDloKk90psGADmHwCxu28KuTSfnjQPl6I/lDSgp5WVkkL3EOcsi/lfIjA4HR7 6fY7Jc8IWziqY1uMaIb2pWNBStgerHsH7fU+0o3ZQz24NthR/e/vCuhzwiy2S8Dhurx0TNFnY8V0u LhPIO+a1E2CWdJ8z2aXRXxjGTqmY65gzOvFoF6A0sJMxI3QdHqKTlZHHIHCLn6RR86G1th3X4uIIX MtQBe4Wf5OpWVXK/XWkgoyIjr4Kk44JlNPwZjC9k66FMENeneJsUh+8rBPzmtjeT23oN5sVxAK6op PQJk/FWA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTF-008A7T-C3; Thu, 26 May 2022 10:36:22 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGTA-0008Xn-E1; Thu, 26 May 2022 10:36:16 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe Date: Thu, 26 May 2022 10:36:00 -0600 Message-Id: <20220526163604.32736-14-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 13/17] md/raid5-cache: Add RCU protection to conf->log accesses X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org The mdadm test 21raid5cache randomly fails with NULL pointer accesses of conf->log when run repeatedly. conf->log was sort of protected with RCU, but most dereferences were not done with the correct functions. Add rcu_read_locks(), rcu_dereference_protected() and rcu_access_pointers() calls to the appropriate places and mark the pointer with __rcu. Signed-off-by: Logan Gunthorpe --- drivers/md/raid5-cache.c | 132 +++++++++++++++++++++++++++------------ drivers/md/raid5-log.h | 14 ++--- drivers/md/raid5.c | 4 +- drivers/md/raid5.h | 2 +- 4 files changed, 102 insertions(+), 50 deletions(-) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 6349cfaae2c8..7c56ee977c3a 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -180,7 +180,12 @@ static bool r5c_is_writeback(struct r5l_log *log) bool r5c_conf_is_writeback(struct r5conf *conf) { - return r5c_is_writeback(conf->log); + bool ret; + + rcu_read_lock(); + ret = r5c_is_writeback(rcu_dereference(conf->log)); + rcu_read_unlock(); + return ret; } static sector_t r5l_ring_add(struct r5l_log *log, sector_t start, sector_t inc) @@ -266,12 +271,23 @@ static void __r5l_wake_reclaim(struct r5l_log *log, sector_t space) md_wakeup_thread(log->reclaim_thread); } +static struct r5l_log *get_log_for_io(struct r5conf *conf) +{ + /* + * rcu_dereference_protected is safe because the array will be + * quiesced before log_exit() so it can't be called while + * an IO is in progress. + */ + return rcu_dereference_protected(conf->log, true); +} + /* Check whether we should flush some stripes to free up stripe cache */ void r5c_check_stripe_cache_usage(struct r5conf *conf) { + struct r5l_log *log = get_log_for_io(conf); int total_cached; - if (!r5c_is_writeback(conf->log)) + if (!r5c_is_writeback(log)) return; total_cached = atomic_read(&conf->r5c_cached_partial_stripes) + @@ -287,7 +303,7 @@ void r5c_check_stripe_cache_usage(struct r5conf *conf) */ if (total_cached > conf->min_nr_stripes * 1 / 2 || atomic_read(&conf->empty_inactive_list_nr) > 0) - __r5l_wake_reclaim(conf->log, 0); + __r5l_wake_reclaim(log, 0); } /* @@ -296,7 +312,9 @@ void r5c_check_stripe_cache_usage(struct r5conf *conf) */ void r5c_check_cached_full_stripe(struct r5conf *conf) { - if (!r5c_is_writeback(conf->log)) + struct r5l_log *log = get_log_for_io(conf); + + if (!r5c_is_writeback(log)) return; /* @@ -306,7 +324,7 @@ void r5c_check_cached_full_stripe(struct r5conf *conf) if (atomic_read(&conf->r5c_cached_full_stripes) >= min(R5C_FULL_STRIPE_FLUSH_BATCH(conf), conf->chunk_sectors >> RAID5_STRIPE_SHIFT(conf))) - __r5l_wake_reclaim(conf->log, 0); + __r5l_wake_reclaim(log, 0); } /* @@ -388,7 +406,7 @@ static inline void r5c_update_log_state(struct r5l_log *log) void r5c_make_stripe_write_out(struct stripe_head *sh) { struct r5conf *conf = sh->raid_conf; - struct r5l_log *log = conf->log; + struct r5l_log *log = get_log_for_io(conf); BUG_ON(!r5c_is_writeback(log)); @@ -630,7 +648,7 @@ static void r5c_disable_writeback_async(struct work_struct *work) /* wait superblock change before suspend */ wait_event(mddev->sb_wait, - conf->log == NULL || + !rcu_access_pointer(conf->log) || (!test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags) && (locked = mddev_trylock(mddev)))); if (locked) { @@ -928,7 +946,7 @@ static inline void r5l_add_no_space_stripe(struct r5l_log *log, */ int r5l_write_stripe(struct r5conf *conf, struct stripe_head *sh) { - struct r5l_log *log = conf->log; + struct r5l_log *log = get_log_for_io(conf); int write_disks = 0; int data_pages, parity_pages; int reserve; @@ -1026,7 +1044,8 @@ int r5l_write_stripe(struct r5conf *conf, struct stripe_head *sh) void r5l_write_stripe_run(struct r5conf *conf) { - struct r5l_log *log = conf->log; + struct r5l_log *log = get_log_for_io(conf); + if (!log) return; mutex_lock(&log->io_mutex); @@ -1036,7 +1055,7 @@ void r5l_write_stripe_run(struct r5conf *conf) int r5l_handle_flush_request(struct r5conf *conf, struct bio *bio) { - struct r5l_log *log = conf->log; + struct r5l_log *log = get_log_for_io(conf); if (log->r5c_journal_mode == R5C_JOURNAL_MODE_WRITE_THROUGH) { /* @@ -1224,7 +1243,7 @@ static void r5l_log_flush_endio(struct bio *bio) */ void r5l_flush_stripe_to_raid(struct r5conf *conf) { - struct r5l_log *log = conf->log; + struct r5l_log *log = get_log_for_io(conf); bool do_flush; if (!log || !log->need_cache_flush) @@ -1339,7 +1358,7 @@ void r5c_flush_cache(struct r5conf *conf, int num) struct stripe_head *sh, *next; lockdep_assert_held(&conf->device_lock); - if (!conf->log) + if (!rcu_access_pointer(conf->log)) return; count = 0; @@ -1358,9 +1377,8 @@ void r5c_flush_cache(struct r5conf *conf, int num) } } -static void r5c_do_reclaim(struct r5conf *conf) +static void r5c_do_reclaim(struct r5conf *conf, struct r5l_log *log) { - struct r5l_log *log = conf->log; struct stripe_head *sh; int count = 0; unsigned long flags; @@ -1488,22 +1506,28 @@ static void r5l_reclaim_thread(struct md_thread *thread) { struct mddev *mddev = thread->mddev; struct r5conf *conf = mddev->private; - struct r5l_log *log = conf->log; + struct r5l_log *log; + /* + * This is safe, because the reclaim thread will be stopped before + * the log is freed in log_exit() + */ + log = rcu_dereference_protected(conf->log, true); if (!log) return; - r5c_do_reclaim(conf); + + r5c_do_reclaim(conf, log); r5l_do_reclaim(log); } void r5l_wake_reclaim(struct r5conf *conf, sector_t space) { - __r5l_wake_reclaim(conf->log, space); + __r5l_wake_reclaim(get_log_for_io(conf), space); } void r5l_quiesce(struct r5conf *conf, int quiesce) { - struct r5l_log *log = conf->log; + struct r5l_log *log = get_log_for_io(conf); struct mddev *mddev; if (quiesce) { @@ -2461,16 +2485,20 @@ static void r5l_write_super(struct r5l_log *log, sector_t cp) static ssize_t r5c_journal_mode_show(struct mddev *mddev, char *page) { struct r5conf *conf; - int ret; + struct r5l_log *log; + int ret = 0; spin_lock(&mddev->lock); conf = mddev->private; - if (!conf || !conf->log) { - spin_unlock(&mddev->lock); - return 0; - } + if (!conf) + goto out_spin_unlock; - switch (conf->log->r5c_journal_mode) { + rcu_read_lock(); + log = rcu_dereference(conf->log); + if (!log) + goto out; + + switch (log->r5c_journal_mode) { case R5C_JOURNAL_MODE_WRITE_THROUGH: ret = snprintf( page, PAGE_SIZE, "[%s] %s\n", @@ -2486,6 +2514,10 @@ static ssize_t r5c_journal_mode_show(struct mddev *mddev, char *page) default: ret = 0; } + +out: + rcu_read_unlock(); +out_spin_unlock: spin_unlock(&mddev->lock); return ret; } @@ -2499,13 +2531,15 @@ static ssize_t r5c_journal_mode_show(struct mddev *mddev, char *page) int r5c_journal_mode_set(struct mddev *mddev, int mode) { struct r5conf *conf; + struct r5l_log *log; + int ret = 0; if (mode < R5C_JOURNAL_MODE_WRITE_THROUGH || mode > R5C_JOURNAL_MODE_WRITE_BACK) return -EINVAL; conf = mddev->private; - if (!conf || !conf->log) + if (!conf || !rcu_access_pointer(conf->log)) return -ENODEV; if (raid5_calc_degraded(conf) > 0 && @@ -2513,12 +2547,19 @@ int r5c_journal_mode_set(struct mddev *mddev, int mode) return -EINVAL; mddev_suspend(mddev); - conf->log->r5c_journal_mode = mode; + rcu_read_lock(); + log = rcu_dereference(conf->log); + if (log) { + log->r5c_journal_mode = mode; + pr_debug("md/raid:%s: setting r5c cache mode to %d: %s\n", + mdname(mddev), mode, r5c_journal_mode_str[mode]); + } else { + ret = -ENODEV; + } + rcu_read_unlock(); mddev_resume(mddev); - pr_debug("md/raid:%s: setting r5c cache mode to %d: %s\n", - mdname(mddev), mode, r5c_journal_mode_str[mode]); - return 0; + return ret; } EXPORT_SYMBOL(r5c_journal_mode_set); @@ -2564,7 +2605,7 @@ int r5c_try_caching_write(struct r5conf *conf, struct stripe_head_state *s, int disks) { - struct r5l_log *log = conf->log; + struct r5l_log *log = get_log_for_io(conf); int i; struct r5dev *dev; int to_cache = 0; @@ -2731,7 +2772,7 @@ void r5c_finish_stripe_write_out(struct r5conf *conf, struct stripe_head *sh, struct stripe_head_state *s) { - struct r5l_log *log = conf->log; + struct r5l_log *log = get_log_for_io(conf); int i; int do_wakeup = 0; sector_t tree_index; @@ -2814,7 +2855,7 @@ void r5c_finish_stripe_write_out(struct r5conf *conf, int r5c_cache_data(struct r5conf *conf, struct stripe_head *sh) { - struct r5l_log *log = conf->log; + struct r5l_log *log = get_log_for_io(conf); int pages = 0; int reserve; int i; @@ -2870,7 +2911,7 @@ int r5c_cache_data(struct r5conf *conf, struct stripe_head *sh) /* check whether this big stripe is in write back cache. */ bool r5c_big_stripe_cached(struct r5conf *conf, sector_t sect) { - struct r5l_log *log = conf->log; + struct r5l_log *log = get_log_for_io(conf); sector_t tree_index; void *slot; @@ -2880,6 +2921,7 @@ bool r5c_big_stripe_cached(struct r5conf *conf, sector_t sect) WARN_ON_ONCE(!rcu_read_lock_held()); tree_index = r5c_tree_index(conf, sect); slot = radix_tree_lookup(&log->big_stripe_tree, tree_index); + return slot != NULL; } @@ -2960,30 +3002,38 @@ static int r5l_load_log(struct r5l_log *log) int r5l_start(struct r5conf *conf) { - struct r5l_log *log = conf->log; + struct r5l_log *log; int ret; + log = rcu_dereference_protected(conf->log, + lockdep_is_held(&conf->mddev->reconfig_mutex)); if (!log) return 0; ret = r5l_load_log(log); if (ret) r5l_exit_log(conf); + return ret; } void r5c_update_on_rdev_error(struct mddev *mddev, struct md_rdev *rdev) { struct r5conf *conf = mddev->private; - struct r5l_log *log = conf->log; + struct r5l_log *log; + rcu_read_lock(); + log = rcu_dereference(conf->log); if (!log) - return; + goto out; if ((raid5_calc_degraded(conf) > 0 || test_bit(Journal, &rdev->flags)) && - conf->log->r5c_journal_mode == R5C_JOURNAL_MODE_WRITE_BACK) + log->r5c_journal_mode == R5C_JOURNAL_MODE_WRITE_BACK) schedule_work(&log->disable_writeback_work); + +out: + rcu_read_unlock(); } int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev) @@ -3091,15 +3141,17 @@ int r5l_init_log(struct r5conf *conf, struct md_rdev *rdev) void r5l_exit_log(struct r5conf *conf) { - struct r5l_log *log = conf->log; + struct r5l_log *log; - conf->log = NULL; - synchronize_rcu(); + lockdep_assert_held(&conf->mddev->reconfig_mutex); + log = rcu_replace_pointer(conf->log, NULL, 1); /* Ensure disable_writeback_work wakes up and exits */ wake_up(&conf->mddev->sb_wait); flush_work(&log->disable_writeback_work); md_unregister_thread(&log->reclaim_thread); + synchronize_rcu(); + mempool_exit(&log->meta_pool); bioset_exit(&log->bs); mempool_exit(&log->io_pool); diff --git a/drivers/md/raid5-log.h b/drivers/md/raid5-log.h index 5948c41f1f2e..5b910e2d4279 100644 --- a/drivers/md/raid5-log.h +++ b/drivers/md/raid5-log.h @@ -133,7 +133,7 @@ static inline int log_stripe(struct stripe_head *sh, struct stripe_head_state *s { struct r5conf *conf = sh->raid_conf; - if (conf->log) { + if (rcu_access_pointer(conf->log)) { if (!test_bit(STRIPE_R5C_CACHING, &sh->state)) { /* writing out phase */ if (s->waiting_extra_page) @@ -154,7 +154,7 @@ static inline void log_stripe_write_finished(struct stripe_head *sh) { struct r5conf *conf = sh->raid_conf; - if (conf->log) + if (rcu_access_pointer(conf->log)) r5l_stripe_write_finished(sh); else if (raid5_has_ppl(conf)) ppl_stripe_write_finished(sh); @@ -162,7 +162,7 @@ static inline void log_stripe_write_finished(struct stripe_head *sh) static inline void log_write_stripe_run(struct r5conf *conf) { - if (conf->log) + if (rcu_access_pointer(conf->log)) r5l_write_stripe_run(conf); else if (raid5_has_ppl(conf)) ppl_write_stripe_run(conf); @@ -170,7 +170,7 @@ static inline void log_write_stripe_run(struct r5conf *conf) static inline void log_flush_stripe_to_raid(struct r5conf *conf) { - if (conf->log) + if (rcu_access_pointer(conf->log)) r5l_flush_stripe_to_raid(conf); else if (raid5_has_ppl(conf)) ppl_write_stripe_run(conf); @@ -180,7 +180,7 @@ static inline int log_handle_flush_request(struct r5conf *conf, struct bio *bio) { int ret = -ENODEV; - if (conf->log) + if (rcu_access_pointer(conf->log)) ret = r5l_handle_flush_request(conf, bio); else if (raid5_has_ppl(conf)) ret = ppl_handle_flush_request(bio); @@ -190,7 +190,7 @@ static inline int log_handle_flush_request(struct r5conf *conf, struct bio *bio) static inline void log_quiesce(struct r5conf *conf, int quiesce) { - if (conf->log) + if (rcu_access_pointer(conf->log)) r5l_quiesce(conf, quiesce); else if (raid5_has_ppl(conf)) ppl_quiesce(conf, quiesce); @@ -198,7 +198,7 @@ static inline void log_quiesce(struct r5conf *conf, int quiesce) static inline void log_exit(struct r5conf *conf) { - if (conf->log) + if (rcu_access_pointer(conf->log)) r5l_exit_log(conf); else if (raid5_has_ppl(conf)) ppl_exit_log(conf); diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index 4b25a0262cb5..93b4b69650bb 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -7937,7 +7937,7 @@ static int raid5_remove_disk(struct mddev *mddev, struct md_rdev *rdev) struct md_rdev *tmp; print_raid5_conf(conf); - if (test_bit(Journal, &rdev->flags) && conf->log) { + if (test_bit(Journal, &rdev->flags) && rcu_access_pointer(conf->log)) { mddev_suspend(mddev); log_exit(conf); mddev_resume(mddev); @@ -8019,7 +8019,7 @@ static int raid5_add_disk(struct mddev *mddev, struct md_rdev *rdev) int last = conf->raid_disks - 1; if (test_bit(Journal, &rdev->flags)) { - if (conf->log) + if (rcu_access_pointer(conf->log)) return -EBUSY; rdev->raid_disk = 0; diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h index 638d29863503..04fe5b6f679c 100644 --- a/drivers/md/raid5.h +++ b/drivers/md/raid5.h @@ -684,7 +684,7 @@ struct r5conf { struct r5worker_group *worker_groups; int group_cnt; int worker_cnt_per_group; - struct r5l_log *log; + struct r5l_log __rcu *log; void *log_private; spinlock_t pending_bios_lock; From patchwork Thu May 26 16:36:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862528 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39420C433EF for ; Thu, 26 May 2022 16:36:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348228AbiEZQgj (ORCPT ); Thu, 26 May 2022 12:36:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348147AbiEZQgX (ORCPT ); Thu, 26 May 2022 12:36:23 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE8D34C7AA; Thu, 26 May 2022 09:36:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=/WD2ehsnnbCNIuVIODA1HdN2V1ZCmC9syKI4aXJHfpQ=; b=KapnB+8SWlmnlWl89nEjcyvn7M VI5+n1DSCnhWLcUSdtGIvXIBi6A72nhIXHMV/WkKTM5w00AvPxuY+TMaydl737990T4R/Eg0QFF6X K5Z5bHpIwqzcEq5eaZg/D1YOaElz8K+zyo0HoLqfZCtZY/9ngzjn6w4o6Cy2e9g1sApnPjcq5l0Ip fE1eNHwBEbrnnvEFM89tYIjw7y/knRzJphEo/q4bxjCHg22qCgeFBpxrdRLHdD5xiZrXOMmv1po9n mr4M+3F1CPd0Q9z2H3U3DB+VdF3J8Hww31B2/or5RfcHu3R1DDlNLRg8fYqcEz5WOFVNaocygwNe9 IWPq+hZA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTF-008A7R-1W; Thu, 26 May 2022 10:36:21 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGTA-0008Xr-Lo; Thu, 26 May 2022 10:36:16 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe , Christoph Hellwig Date: Thu, 26 May 2022 10:36:01 -0600 Message-Id: <20220526163604.32736-15-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com, hch@lst.de X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 14/17] md/raid5-cache: Annotate pslot with __rcu notation X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org radix_tree_lookup_slot() and radix_tree_replace_slot() API expect the slot returned and looked up to be marked with __rcu. Otherwise sparse warnings are generated: drivers/md/raid5-cache.c:2939:23: warning: incorrect type in assignment (different address spaces) drivers/md/raid5-cache.c:2939:23: expected void **pslot drivers/md/raid5-cache.c:2939:23: got void [noderef] __rcu ** Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5-cache.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/md/raid5-cache.c b/drivers/md/raid5-cache.c index 7c56ee977c3a..5ddfe29bc26a 100644 --- a/drivers/md/raid5-cache.c +++ b/drivers/md/raid5-cache.c @@ -2609,7 +2609,7 @@ int r5c_try_caching_write(struct r5conf *conf, int i; struct r5dev *dev; int to_cache = 0; - void **pslot; + void __rcu **pslot; sector_t tree_index; int ret; uintptr_t refcount; @@ -2776,7 +2776,7 @@ void r5c_finish_stripe_write_out(struct r5conf *conf, int i; int do_wakeup = 0; sector_t tree_index; - void **pslot; + void __rcu **pslot; uintptr_t refcount; if (!log || !test_bit(R5_InJournal, &sh->dev[sh->pd_idx].flags)) From patchwork Thu May 26 16:36:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 578DDC4332F for ; Thu, 26 May 2022 16:36:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348182AbiEZQgf (ORCPT ); Thu, 26 May 2022 12:36:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348145AbiEZQgX (ORCPT ); Thu, 26 May 2022 12:36:23 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CBA836170; Thu, 26 May 2022 09:36:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=u1yxSPEeTNho1f/GFvMeJPGUhnwcK9k9whz6X7BQdj4=; b=m79OEsmrxVkoUNX+YjIz95b9uO 9RywmDrf46ts34D5D6/5OGY8uqEVALUPMXPf+lgLB8gh3vj7enEh0HVLOT1vuK7NUdXGt+b6fVeYQ UiC9uwXGEs97mmOgJfJzsh+AA6wPVzOmlkQ6HzEuv1oPAs4hwTNglWG00no5nzXMKWvo74Re+/xYi US2kPrbmHsWvsC759G95CASYqOCh4Y1R0jNlJeCyF1PYcWbXBfm3MDP6sibxhDTEBXRzkN9wAyQ5K oUeKJBMFKiKSoAxHo/k/yF2h/0/HqS6aVPNaA9Ax7OfHH6b5W59b1+aReOUJekTCgbToXn9bypOcq +NH7erLg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTE-008A7V-DU; Thu, 26 May 2022 10:36:21 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGTA-0008Xv-Su; Thu, 26 May 2022 10:36:16 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe Date: Thu, 26 May 2022 10:36:02 -0600 Message-Id: <20220526163604.32736-16-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 15/17] md: Use enum for overloaded magic numbers used by mddev->curr_resync X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org Comments in the code document special values used for mddev->curr_resync. Make this clearer by using an enum to label these values. The only functional change is a couple places use the wrong comparison operator that implied 3 is another special value. They are all fixed to imply that 3 or greater is an active resync. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/md.c | 40 ++++++++++++++++++---------------------- drivers/md/md.h | 15 +++++++++++++++ 2 files changed, 33 insertions(+), 22 deletions(-) diff --git a/drivers/md/md.c b/drivers/md/md.c index 8273ac5eef06..0893029865eb 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -5001,7 +5001,7 @@ static ssize_t sync_speed_show(struct mddev *mddev, char *page) { unsigned long resync, dt, db; - if (mddev->curr_resync == 0) + if (mddev->curr_resync == MD_RESYNC_NONE) return sprintf(page, "none\n"); resync = mddev->curr_mark_cnt - atomic_read(&mddev->recovery_active); dt = (jiffies - mddev->resync_mark) / HZ; @@ -5020,8 +5020,8 @@ sync_completed_show(struct mddev *mddev, char *page) if (!test_bit(MD_RECOVERY_RUNNING, &mddev->recovery)) return sprintf(page, "none\n"); - if (mddev->curr_resync == 1 || - mddev->curr_resync == 2) + if (mddev->curr_resync == MD_RESYNC_YIELDED || + mddev->curr_resync == MD_RESYNC_DELAYED) return sprintf(page, "delayed\n"); if (test_bit(MD_RECOVERY_SYNC, &mddev->recovery) || @@ -8018,7 +8018,7 @@ static int status_resync(struct seq_file *seq, struct mddev *mddev) max_sectors = mddev->dev_sectors; resync = mddev->curr_resync; - if (resync <= 3) { + if (resync < MD_RESYNC_ACTIVE) { if (test_bit(MD_RECOVERY_DONE, &mddev->recovery)) /* Still cleaning up */ resync = max_sectors; @@ -8027,7 +8027,7 @@ static int status_resync(struct seq_file *seq, struct mddev *mddev) else resync -= atomic_read(&mddev->recovery_active); - if (resync == 0) { + if (resync == MD_RESYNC_NONE) { if (test_bit(MD_RESYNCING_REMOTE, &mddev->recovery)) { struct md_rdev *rdev; @@ -8051,7 +8051,7 @@ static int status_resync(struct seq_file *seq, struct mddev *mddev) } return 0; } - if (resync < 3) { + if (resync < MD_RESYNC_ACTIVE) { seq_printf(seq, "\tresync=DELAYED"); return 1; } @@ -8729,13 +8729,7 @@ void md_do_sync(struct md_thread *thread) mddev->last_sync_action = action ?: desc; - /* we overload curr_resync somewhat here. - * 0 == not engaged in resync at all - * 2 == checking that there is no conflict with another sync - * 1 == like 2, but have yielded to allow conflicting resync to - * commence - * other == active in resync - this many blocks - * + /* * Before starting a resync we must have set curr_resync to * 2, and then checked that every "conflicting" array has curr_resync * less than ours. When we find one that is the same or higher @@ -8747,7 +8741,7 @@ void md_do_sync(struct md_thread *thread) do { int mddev2_minor = -1; - mddev->curr_resync = 2; + mddev->curr_resync = MD_RESYNC_DELAYED; try_again: if (test_bit(MD_RECOVERY_INTR, &mddev->recovery)) @@ -8759,12 +8753,14 @@ void md_do_sync(struct md_thread *thread) && mddev2->curr_resync && match_mddev_units(mddev, mddev2)) { DEFINE_WAIT(wq); - if (mddev < mddev2 && mddev->curr_resync == 2) { + if (mddev < mddev2 && + mddev->curr_resync == MD_RESYNC_DELAYED) { /* arbitrarily yield */ - mddev->curr_resync = 1; + mddev->curr_resync = MD_RESYNC_YIELDED; wake_up(&resync_wait); } - if (mddev > mddev2 && mddev->curr_resync == 1) + if (mddev > mddev2 && + mddev->curr_resync == MD_RESYNC_YIELDED) /* no need to wait here, we can wait the next * time 'round when curr_resync == 2 */ @@ -8792,7 +8788,7 @@ void md_do_sync(struct md_thread *thread) finish_wait(&resync_wait, &wq); } } - } while (mddev->curr_resync < 2); + } while (mddev->curr_resync < MD_RESYNC_DELAYED); j = 0; if (test_bit(MD_RECOVERY_SYNC, &mddev->recovery)) { @@ -8876,7 +8872,7 @@ void md_do_sync(struct md_thread *thread) desc, mdname(mddev)); mddev->curr_resync = j; } else - mddev->curr_resync = 3; /* no longer delayed */ + mddev->curr_resync = MD_RESYNC_ACTIVE; /* no longer delayed */ mddev->curr_resync_completed = j; sysfs_notify_dirent_safe(mddev->sysfs_completed); md_new_event(); @@ -9011,14 +9007,14 @@ void md_do_sync(struct md_thread *thread) if (!test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery) && !test_bit(MD_RECOVERY_INTR, &mddev->recovery) && - mddev->curr_resync > 3) { + mddev->curr_resync >= MD_RESYNC_ACTIVE) { mddev->curr_resync_completed = mddev->curr_resync; sysfs_notify_dirent_safe(mddev->sysfs_completed); } mddev->pers->sync_request(mddev, max_sectors, &skipped); if (!test_bit(MD_RECOVERY_CHECK, &mddev->recovery) && - mddev->curr_resync > 3) { + mddev->curr_resync >= MD_RESYNC_ACTIVE) { if (test_bit(MD_RECOVERY_SYNC, &mddev->recovery)) { if (test_bit(MD_RECOVERY_INTR, &mddev->recovery)) { if (mddev->curr_resync >= mddev->recovery_cp) { @@ -9082,7 +9078,7 @@ void md_do_sync(struct md_thread *thread) } else if (test_bit(MD_RECOVERY_REQUESTED, &mddev->recovery)) mddev->resync_min = mddev->curr_resync_completed; set_bit(MD_RECOVERY_DONE, &mddev->recovery); - mddev->curr_resync = 0; + mddev->curr_resync = MD_RESYNC_NONE; spin_unlock(&mddev->lock); wake_up(&resync_wait); diff --git a/drivers/md/md.h b/drivers/md/md.h index 5f62c46ac2d3..2d06003a4c3f 100644 --- a/drivers/md/md.h +++ b/drivers/md/md.h @@ -288,6 +288,21 @@ struct serial_info { sector_t _subtree_last; /* highest sector in subtree of rb node */ }; +/* + * mddev->curr_resync stores the current sector of the resync but + * also has some overloaded values. + */ +enum { + /* No resync in progress */ + MD_RESYNC_NONE = 0, + /* Yielded to allow another conflicting resync to commence */ + MD_RESYNC_YIELDED = 1, + /* Delayed to check that there is no conflict with another sync */ + MD_RESYNC_DELAYED = 2, + /* Any value greater than or equal to this is in an active resync */ + MD_RESYNC_ACTIVE = 3, +}; + struct mddev { void *private; struct md_personality *pers; From patchwork Thu May 26 16:36:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862522 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7457DC433EF for ; Thu, 26 May 2022 16:36:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348152AbiEZQgX (ORCPT ); Thu, 26 May 2022 12:36:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245543AbiEZQgW (ORCPT ); Thu, 26 May 2022 12:36:22 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B528A3ED04; Thu, 26 May 2022 09:36:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=AnEIxfUMmcUWxMypKv8Fd+pUHTZRqVZQOmFGGbpQ8x4=; b=mC9Bb5u/4zA7cyUUg55DN+zbau Y5J9xLhpLpz7hmy8gM1fFUkZUDFdZw16IHtZ6CXkoDjEagDTbcY98JsqNsO49uJCtLpqVAW7x2DNB elNrIenlyMUQ1cGIlG+IP0cknHZQijYjkSUzo1y6hX7r148eX3lu9wpKWInuYFAhRTIEoh5GAdIee nvF2LhpthDkyeeddTfKfLF+WsNNmralMRHCmd5jGr8O0Jn64727w7VjvyFaHAUExDQz9tJn4uMFP8 lARUaNu/d1fENk/6ayHrR9eOjJTJRxnrnwNtz4XHObzWj4zlcGmcosBb9xhDmkEociOxEn2Hg5Jh1 LLbDemiA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTD-008A7R-Rm; Thu, 26 May 2022 10:36:20 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGTB-0008Xz-4Y; Thu, 26 May 2022 10:36:17 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe Date: Thu, 26 May 2022 10:36:03 -0600 Message-Id: <20220526163604.32736-17-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 16/17] md: Ensure resync is reported after it starts X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org The 07layouts test in mdadm fails on some systems. The failure presents itself as the backup file not being removed before the next layout is grown into: mdadm: /dev/md0: cannot create backup file /tmp/md-test-backup: File exists This is because the background mdadm process, which is responsible for cleaning up this backup file gets into an infinite loop waiting for the reshape to start. mdadm checks the mdstat file if a reshape is going and, if it is not, it waits for an event on the file or times out in 5 seconds. On faster machines, the reshape may complete before the 5 seconds times out, and thus the background mdadm process loops waiting for a reshape to start that has already occurred. mdadm reads the mdstat file to start, but mdstat does not report that the reshape has begun, even though it has indeed begun. So the mdstat_wait() call (in mdadm) which polls on the mdstat file won't ever return until timing out. The reason mdstat reports the reshape has started is due to an issue in status_resync(). recovery_active is subtracted from curr_resync which will result in a value of zero for the first chunk of reshaped data, and the resulting read will report no reshape in progress. To fix this, if "resync - recovery_active" is an overloaded value, force the value to be MD_RESYNC_ACTIVE so the code reports a resync in progress. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/md.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/drivers/md/md.c b/drivers/md/md.c index 0893029865eb..2be429874d18 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -8022,10 +8022,20 @@ static int status_resync(struct seq_file *seq, struct mddev *mddev) if (test_bit(MD_RECOVERY_DONE, &mddev->recovery)) /* Still cleaning up */ resync = max_sectors; - } else if (resync > max_sectors) + } else if (resync > max_sectors) { resync = max_sectors; - else + } else { resync -= atomic_read(&mddev->recovery_active); + if (resync < MD_RESYNC_ACTIVE) { + /* + * Resync has started, but the subtraction has + * yielded one of the special values. Force it + * to active to ensure the status reports an + * active resync. + */ + resync = MD_RESYNC_ACTIVE; + } + } if (resync == MD_RESYNC_NONE) { if (test_bit(MD_RESYNCING_REMOTE, &mddev->recovery)) { From patchwork Thu May 26 16:36:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12862524 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B207EC4332F for ; Thu, 26 May 2022 16:36:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348161AbiEZQg0 (ORCPT ); Thu, 26 May 2022 12:36:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348132AbiEZQgX (ORCPT ); Thu, 26 May 2022 12:36:23 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B79D48E61; Thu, 26 May 2022 09:36:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=tiDb5N7DtJWgf5i+viE/JdrxQoObOCh/lk7c62dUGdI=; b=H3fM8eJI/0Qi9l/+we2qC4Mb3x L1PRPZ/Eyb2qmb6zyuF5YCumWK0SqoNfzrwmsjoYUsd6AMFuyFcVgXrPf5kOWkHUUeGvjO4iNJm4X sn0Q+bxbp+wvUOAT8WrTGhmVcIPmcIRBeiMQ8gxiF7EJASkZ+zghGMhrsZuxJ58ZdED4GLwvNXRk4 gUQUxzAcorulWSm3HU/DFK//cF3Q2Uj2J5kei2+jhBN2I1cAzNXBktg+PlC7IqY/+F8Xos4ZH5M4o klHseeSuxmsyCAwnhonR/SQ3PThrtBLE17kuxG2fMNJCaWVT7cohWKym2XGoEbTeorktyiagvk90u 6YJ38UvA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nuGTD-008A7T-HN; Thu, 26 May 2022 10:36:21 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1nuGTB-0008Y3-C7; Thu, 26 May 2022 10:36:17 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Christoph Hellwig , Donald Buczek , Guoqing Jiang , Xiao Ni , Stephen Bates , Martin Oliveira , David Sloan , Logan Gunthorpe , Christoph Hellwig Date: Thu, 26 May 2022 10:36:04 -0600 Message-Id: <20220526163604.32736-18-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220526163604.32736-1-logang@deltatee.com> References: <20220526163604.32736-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, hch@infradead.org, buczek@molgen.mpg.de, guoqing.jiang@linux.dev, xni@redhat.com, sbates@raithlin.com, Martin.Oliveira@eideticom.com, David.Sloan@eideticom.com, logang@deltatee.com, hch@lst.de X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v2 17/17] md: Notify sysfs sync_completed in md_reap_sync_thread() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org The mdadm test 07layouts randomly produces a kernel hung task deadlock. The deadlock is caused by the suspend_lo/suspend_hi files being set by the mdadm background process during reshape and not being cleared because the process hangs. (Leaving aside the issue of the fragility of freezing kernel tasks by buggy userspace processes...) When the background mdadm process hangs it, is waiting (without a timeout) on a change to the sync_completed file signalling that the reshape has completed. The process is woken up a couple times when the reshape finishes but it is woken up before MD_RECOVERY_RUNNING is cleared so sync_completed_show() reports 0 instead of "none". To fix this, notify the sysfs file in md_reap_sync_thread() after MD_RECOVERY_RUNNING has been cleared. This wakes up mdadm and causes it to continue and write to suspend_lo/suspend_hi to allow IO to continue. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/md.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/md/md.c b/drivers/md/md.c index 2be429874d18..2c07c9508222 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -9476,6 +9476,7 @@ void md_reap_sync_thread(struct mddev *mddev, bool reconfig_mutex_held) wake_up(&resync_wait); /* flag recovery needed just to double check */ set_bit(MD_RECOVERY_NEEDED, &mddev->recovery); + sysfs_notify_dirent_safe(mddev->sysfs_completed); sysfs_notify_dirent_safe(mddev->sysfs_action); md_new_event(); if (mddev->event_work.func)