From patchwork Wed Jul 19 07:09:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xueshi Hu X-Patchwork-Id: 13318403 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8869EEB64DA for ; Wed, 19 Jul 2023 07:10:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229639AbjGSHKX (ORCPT ); Wed, 19 Jul 2023 03:10:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46214 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229695AbjGSHKM (ORCPT ); Wed, 19 Jul 2023 03:10:12 -0400 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF3AC1FF1 for ; Wed, 19 Jul 2023 00:10:10 -0700 (PDT) Received: by mail-pj1-x1035.google.com with SMTP id 98e67ed59e1d1-2633fe9b6c0so341552a91.1 for ; Wed, 19 Jul 2023 00:10:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20221208.gappssmtp.com; s=20221208; t=1689750610; x=1690355410; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SDGPmqXPbXy9jPXGt4LdQJJoukkPj69aQm1gZgBhB4s=; b=ZO5rJy3NY3bE04v4sM7NoIH5y3xMENwOcbNpCW7/vWTbYEg7VpsPOLgek1OH1KyUu5 rvDq3Gqqff3N45tL43/vJ2mBczz3kF99xG2bDW3U2sUBwx30amoN6lOMdM8blUwSvc9I DU7KCBecrBxugrUVLivhj0pQln78lll7BAoAfgf5+ClyW9THhAgvv79930uJ301SDscl JmD6Rut3YbafLF7vYHU3pETcbekWvcs/IXNgEetiJ1A7ESVK4t1ye8DtLNSYoV0KCzdl hJRemYZo1TQeoKgyFC/uB+sKkEvL6nUh1u9IPpYAC4gC0sRG4rrm/RBd3qJb8d5uv0CE a4TA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689750610; x=1690355410; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SDGPmqXPbXy9jPXGt4LdQJJoukkPj69aQm1gZgBhB4s=; b=e8tmBRsO1zK/Ntth+0pqZTiwdW9C676BL7msx/4CdK0mpe5B/xj/lpNdkZ1xY3Nx8k /PkXL5v1jy8hIXR0BUtYaFUcrsGFiw8Mo+fJX1dkjyNnlwyItdkji+DS+qPXMFqoSBP8 c49l9pwkbiGU19UfPuxgEFRpQDPCO7rbn4dLCLLpxx1TqKKQVv6GJNbk1Bb3XH210GxE fJSeb5j1SMT9cXKWFkEQ7miOSP/BiHFLXw88pWy1ozuCu2myBN6NAGVSQdvaU9ENHvZn PELnc3KA+qa4BKjwC1QLcV1r223717XvkJbxhNJWSc7n80RqLJPbGbNIw0Mf7o4cFaj1 D9rg== X-Gm-Message-State: ABy/qLZdVNvoQd7+dudoaA09FTnjfafIaNK4WQ5L3vmbl84n7FSVDs3b 23nKvyN2sDb/G1fvCgN0H3jL3Q== X-Google-Smtp-Source: APBJJlG//2hxKX90n2i5tHZRj1zBJ6+RGphPPYANxyzWmCdjdN582HzJp1IH+m/DGc0piYB+rOrg+w== X-Received: by 2002:a17:90a:2e13:b0:262:ec12:7c40 with SMTP id q19-20020a17090a2e1300b00262ec127c40mr1812873pjd.11.1689750609939; Wed, 19 Jul 2023 00:10:09 -0700 (PDT) Received: from nixos.tailf4e9e.ts.net ([47.75.78.161]) by smtp.googlemail.com with ESMTPSA id 11-20020a17090a194b00b002639c4f81cesm667453pjh.3.2023.07.19.00.10.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 00:10:09 -0700 (PDT) From: Xueshi Hu To: song@kernel.org Cc: linux-raid@vger.kernel.org, yukuai1@huaweicloud.com, Xueshi Hu Subject: [PATCH v3 1/3] md/raid1: freeze array more strictly when reshape Date: Wed, 19 Jul 2023 15:09:52 +0800 Message-Id: <20230719070954.3084379-2-xueshi.hu@smartx.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230719070954.3084379-1-xueshi.hu@smartx.com> References: <20230719070954.3084379-1-xueshi.hu@smartx.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org When an IO error happens, reschedule_retry() will increase r1conf::nr_queued, which makes freeze_array() unblocked. However, before all r1bio in the memory pool are released, the memory pool should not be modified. Introduce freeze_array_totally() to solve the problem. Compared to freeze_array(), it's more strict because any in-flight io needs to complete including queued io. Signed-off-by: Xueshi Hu --- drivers/md/raid1.c | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index dd25832eb045..5605c9680818 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -1072,7 +1072,7 @@ static void freeze_array(struct r1conf *conf, int extra) /* Stop sync I/O and normal I/O and wait for everything to * go quiet. * This is called in two situations: - * 1) management command handlers (reshape, remove disk, quiesce). + * 1) management command handlers (remove disk, quiesce). * 2) one normal I/O request failed. * After array_frozen is set to 1, new sync IO will be blocked at @@ -1111,6 +1111,37 @@ static void unfreeze_array(struct r1conf *conf) wake_up(&conf->wait_barrier); } +/* conf->resync_lock should be held */ +static int get_pending(struct r1conf *conf) +{ + int idx, ret; + + ret = atomic_read(&conf->nr_sync_pending); + for (idx = 0; idx < BARRIER_BUCKETS_NR; idx++) + ret += atomic_read(&conf->nr_pending[idx]); + + return ret; +} + +static void freeze_array_totally(struct r1conf *conf) +{ + /* + * freeze_array_totally() is almost the same with freeze_array() except + * it requires there's no queued io. Raid1's reshape will destroy the + * old mempool and change r1conf::raid_disks, which are necessary when + * freeing the queued io. + */ + spin_lock_irq(&conf->resync_lock); + conf->array_frozen = 1; + raid1_log(conf->mddev, "freeze totally"); + wait_event_lock_irq_cmd( + conf->wait_barrier, + get_pending(conf) == 0, + conf->resync_lock, + md_wakeup_thread(conf->mddev->thread)); + spin_unlock_irq(&conf->resync_lock); +} + static void alloc_behind_master_bio(struct r1bio *r1_bio, struct bio *bio) { @@ -3296,7 +3327,7 @@ static int raid1_reshape(struct mddev *mddev) return -ENOMEM; } - freeze_array(conf, 0); + freeze_array_totally(conf); /* ok, everything is stopped */ oldpool = conf->r1bio_pool; From patchwork Wed Jul 19 07:09:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xueshi Hu X-Patchwork-Id: 13318404 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4ED52C001DC for ; Wed, 19 Jul 2023 07:10:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229679AbjGSHKZ (ORCPT ); Wed, 19 Jul 2023 03:10:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229685AbjGSHKU (ORCPT ); Wed, 19 Jul 2023 03:10:20 -0400 Received: from mail-il1-x144.google.com (mail-il1-x144.google.com [IPv6:2607:f8b0:4864:20::144]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A72E1BF0 for ; Wed, 19 Jul 2023 00:10:13 -0700 (PDT) Received: by mail-il1-x144.google.com with SMTP id e9e14a558f8ab-345f1e0abf9so28852425ab.3 for ; Wed, 19 Jul 2023 00:10:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20221208.gappssmtp.com; s=20221208; t=1689750613; x=1690355413; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7PxoHMuRjPvVhNHPoZFIyFjoUjgrNWfrZfj8UVxX1Mo=; b=zmWkU5G1tPmeVrozmzHBBV93Io6QrdBU6v/3Geny0Xtq3qecKunlGocnBnJGNVsr20 EACMVRh2Jtbm2TMebgQy0+Y3F98o5Iw5AxvInvqxL/EFImPgx6cvoXkLRbKpMQJrsJXm 0K2HJo/oKyG3OAPdJDiW4piR88Yjqm+QdJ6zyR4rmEKydmpNr3fsVW8i3PeiaFRwQJYU ddYly5PQLyOoNppU00wCDy7P41YuSTzqrQzvkxGDtor1zzTN4eBM7R1fTQDNJ2Cz8O3e qrJfxQhKHCgtnx+Qs/vou1hP5LgU5MvNos/FaCCEvvmqqRQHaVMhmb3F9LWlQl3dzhPr UDpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689750613; x=1690355413; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7PxoHMuRjPvVhNHPoZFIyFjoUjgrNWfrZfj8UVxX1Mo=; b=c0QhoCGqu8Mniyd1Xud3EMuTdPqYgX1h8vKGyCvqQRN61YeOFzqJFzn6+ioj6DvFQr 7VwO2g5H2jTXn5w2xfTBsvkJ4YSY3aP1W4wpgfQ/5dmZ8ZXU262OMbkhaE65+G3maV6n 3Y3b9FgmADxjcH07AbQH3cSTEe8Sp6hDInTsE3jPcJ5wXW+fzYfKvtfPMgwJiasu0Fi/ 2pMqQPpA3kdPm2WebNossY7fanXPs7E00qY0Lcj4JsNnI5fgwwAU7WNaiOfveC+5YA4l RQwMsULx8F4VvfcQwwcpHj/TTNvkyz5+CnpPaK+a1fvN6TU4bChVRN+kNgT9BFYjk2Iy /bQw== X-Gm-Message-State: ABy/qLa5swcfQHQiTvKpZ0Ub3DViukSUS9OexDrge9g6k0zPZYtiASHn YNSSoB22PkuTV50kmwSEmIxDRw== X-Google-Smtp-Source: APBJJlHa+VYyYLNqlJcOAXRaecw2U5+FL4kIjwWod3bni4yQGDzonaQbSEeDsXrW0C+JQySp6Lv18g== X-Received: by 2002:a92:ca83:0:b0:348:8216:3363 with SMTP id t3-20020a92ca83000000b0034882163363mr1561145ilo.5.1689750612704; Wed, 19 Jul 2023 00:10:12 -0700 (PDT) Received: from nixos.tailf4e9e.ts.net ([47.75.78.161]) by smtp.googlemail.com with ESMTPSA id 11-20020a17090a194b00b002639c4f81cesm667453pjh.3.2023.07.19.00.10.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 00:10:12 -0700 (PDT) From: Xueshi Hu To: song@kernel.org Cc: linux-raid@vger.kernel.org, yukuai1@huaweicloud.com, Xueshi Hu Subject: [PATCH v3 2/3] md/raid1: don't allow_barrier() before r1bio got freed Date: Wed, 19 Jul 2023 15:09:53 +0800 Message-Id: <20230719070954.3084379-3-xueshi.hu@smartx.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230719070954.3084379-1-xueshi.hu@smartx.com> References: <20230719070954.3084379-1-xueshi.hu@smartx.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org allow_barrier() make reshape possible. Raid reshape changes the r1conf::raid_disks and mempool. Free the r1bio firstly and then call allow_barrier() Signed-off-by: Xueshi Hu Reviewed-by: Yu Kuai --- drivers/md/raid1.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 5605c9680818..62e86b7d1561 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -313,6 +313,7 @@ static void raid_end_bio_io(struct r1bio *r1_bio) { struct bio *bio = r1_bio->master_bio; struct r1conf *conf = r1_bio->mddev->private; + sector_t sector = r1_bio->sector; /* if nobody has done the final endio yet, do it now */ if (!test_and_set_bit(R1BIO_Returned, &r1_bio->state)) { @@ -323,13 +324,13 @@ static void raid_end_bio_io(struct r1bio *r1_bio) call_bio_endio(r1_bio); } + + free_r1bio(r1_bio); /* * Wake up any possible resync thread that waits for the device * to go idle. All I/Os, even write-behind writes, are done. */ - allow_barrier(conf, r1_bio->sector); - - free_r1bio(r1_bio); + allow_barrier(conf, sector); } /* @@ -1404,6 +1405,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio, return; } + retry_write: r1_bio = alloc_r1bio(mddev, bio); r1_bio->sectors = max_write_sectors; @@ -1419,7 +1421,6 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio, */ disks = conf->raid_disks * 2; - retry_write: blocked_rdev = NULL; rcu_read_lock(); max_sectors = r1_bio->sectors; @@ -1499,7 +1500,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio, for (j = 0; j < i; j++) if (r1_bio->bios[j]) rdev_dec_pending(conf->mirrors[j].rdev, mddev); - r1_bio->state = 0; + free_r1bio(r1_bio); allow_barrier(conf, bio->bi_iter.bi_sector); if (bio->bi_opf & REQ_NOWAIT) { @@ -2529,6 +2530,7 @@ static void handle_read_error(struct r1conf *conf, struct r1bio *r1_bio) struct mddev *mddev = conf->mddev; struct bio *bio; struct md_rdev *rdev; + sector_t sector; clear_bit(R1BIO_ReadError, &r1_bio->state); /* we got a read error. Maybe the drive is bad. Maybe just @@ -2558,12 +2560,13 @@ static void handle_read_error(struct r1conf *conf, struct r1bio *r1_bio) } rdev_dec_pending(rdev, conf->mddev); - allow_barrier(conf, r1_bio->sector); + sector = r1_bio->sector; bio = r1_bio->master_bio; /* Reuse the old r1_bio so that the IO_BLOCKED settings are preserved */ r1_bio->state = 0; raid1_read_request(mddev, bio, r1_bio->sectors, r1_bio); + allow_barrier(conf, sector); } static void raid1d(struct md_thread *thread) From patchwork Wed Jul 19 07:09:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xueshi Hu X-Patchwork-Id: 13318405 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BCAFEB64DA for ; Wed, 19 Jul 2023 07:10:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229587AbjGSHKf (ORCPT ); Wed, 19 Jul 2023 03:10:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46304 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229691AbjGSHKV (ORCPT ); Wed, 19 Jul 2023 03:10:21 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DBF481FF9 for ; Wed, 19 Jul 2023 00:10:15 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id 98e67ed59e1d1-262cc036fa4so3298336a91.3 for ; Wed, 19 Jul 2023 00:10:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smartx-com.20221208.gappssmtp.com; s=20221208; t=1689750615; x=1690355415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+GUfPaqxqaUbNGMzDV0eCmb+o/mlJufIsxGkzLSwirc=; b=D4a9wfLwOk+EeiihfhlbyTKToRAV88WCeIllIhlFdwBJG90hPB8ajT+YTto0j/09yh dVmq0F4w+7Vz/UDFJtsVrVeolhIAe0J6bOeSfOoQ6XeRADARS8mdcGR4F8QiDZim8XWZ SAttiE7YvAuha+u52ZxQ02ZrpdUyLCfhV3r90dG/ka0s0rsCFzvweNeuAGOfpN+zYrLO 4Q/XBW+5lxLvckWlOlhgcRhET7LSKQnTCqf/Nf0y/3X99zManQXy+OK5uu5tNl6WI+fY WuhN3yuV9G/aKtMVJXPdfs/EQshLypsZmEREhWV5paYGJHFrkvdt/lVLglt09XwKNF/y s/Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689750615; x=1690355415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+GUfPaqxqaUbNGMzDV0eCmb+o/mlJufIsxGkzLSwirc=; b=ZZQXE3y/S6ZixcXBbUTctOVd2o8RPp5C1xUpElueRJsiQMCPg8teZrcEXXYXk5PeeZ hxIpWdlxYj9JRoPyDruu4vwXrMf734ZfyPs1bXg7DNQ+3cfQwmoeIFYQIvxod6b9d5RQ 4waTj7zotp4RjLinFfkefZW+bOClcEn42ze06H2a7iarhzc3LIxeTkbm1NeBQ0yTGBR2 ZEAQMfj/Yf9VKRekhG9Bo+DcJg0xbKmviHx1Ej1gQfd3HotSCHwTiNaKzw4c7T+qbHug JcF7Wk+SzPK1G+9plg/tTjoqRQHsGDb/0UidJ8Oe+P22ML6RcjX75/ATZEsoDDVwiHe+ OPhA== X-Gm-Message-State: ABy/qLbpwaQ8bYbGi5cV/sTIkquS3YBAZrodbI2898o5uOmGRs2G+zt7 C0A59/rNUJhnWoyik6fokyvclQ== X-Google-Smtp-Source: APBJJlGuywjdg2Q74UiqFyaXEYhAH7hffQgOW+l/KLQj3mmV6MkSGh7JorYRCmqJYn8qhkjpPK+IUQ== X-Received: by 2002:a17:90b:354a:b0:263:e1db:7275 with SMTP id lt10-20020a17090b354a00b00263e1db7275mr12020194pjb.6.1689750615322; Wed, 19 Jul 2023 00:10:15 -0700 (PDT) Received: from nixos.tailf4e9e.ts.net ([47.75.78.161]) by smtp.googlemail.com with ESMTPSA id 11-20020a17090a194b00b002639c4f81cesm667453pjh.3.2023.07.19.00.10.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 00:10:15 -0700 (PDT) From: Xueshi Hu To: song@kernel.org Cc: linux-raid@vger.kernel.org, yukuai1@huaweicloud.com, Xueshi Hu Subject: [PATCH v3 3/3] md/raid1: check array size before reshape Date: Wed, 19 Jul 2023 15:09:54 +0800 Message-Id: <20230719070954.3084379-4-xueshi.hu@smartx.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230719070954.3084379-1-xueshi.hu@smartx.com> References: <20230719070954.3084379-1-xueshi.hu@smartx.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-raid@vger.kernel.org If array size doesn't changed, nothing need to do. Signed-off-by: Xueshi Hu --- drivers/md/raid1.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 62e86b7d1561..5840b8b0f9b7 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -3282,9 +3282,6 @@ static int raid1_reshape(struct mddev *mddev) int d, d2; int ret; - memset(&newpool, 0, sizeof(newpool)); - memset(&oldpool, 0, sizeof(oldpool)); - /* Cannot change chunk_size, layout, or level */ if (mddev->chunk_sectors != mddev->new_chunk_sectors || mddev->layout != mddev->new_layout || @@ -3295,6 +3292,12 @@ static int raid1_reshape(struct mddev *mddev) return -EINVAL; } + if (mddev->delta_disks == 0) + return 0; /* nothing to do */ + + memset(&newpool, 0, sizeof(newpool)); + memset(&oldpool, 0, sizeof(oldpool)); + if (!mddev_is_clustered(mddev)) md_allow_write(mddev);