From patchwork Thu Oct 12 16:59:01 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 10002409 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 150D760325 for ; Thu, 12 Oct 2017 16:59:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 05A3128D13 for ; Thu, 12 Oct 2017 16:59:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EE75D28D2C; Thu, 12 Oct 2017 16:59:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 544DE28D17 for ; Thu, 12 Oct 2017 16:59:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752241AbdJLQ7W (ORCPT ); Thu, 12 Oct 2017 12:59:22 -0400 Received: from esa5.hgst.iphmx.com ([216.71.153.144]:42644 "EHLO esa5.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751258AbdJLQ7T (ORCPT ); Thu, 12 Oct 2017 12:59:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1507827560; x=1539363560; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-id:content-transfer-encoding: mime-version; bh=DqTN8YFj7ry3U0EzQh9jDUpmmTx/OmcVJ7LhaNYTe0o=; b=CeiAKdadlKlgB/eKKO73W1/eqXDG4OF8y0dDeYeYw0ZJnaTuipg5BEtL ZLgOc52p8Vwob0J947PDq69iWt2odNjEF7OBfZYSxa4dxWZPcuuYUV7iq 5taYRazCdT+xR9EzbS6DzGLif7GUi0aVH2/8od1wDMXDtUD6rW1RNETxm n6k5DjXYKdTtsnk0dyXddsL/HetMH4Tv/wRQIeIXoF6dLBrrf80Re3XsQ ybNGSTs1+cuIK1WctnFCghbstRVwQQPvQkTmB8LaujsNOlYhAprq6QEpq ynCDE6e+8c1XdVX9Zvria9p2NzzQeTHEBvkvq3KDgLk35/4kfCCnW5M1b g==; X-IronPort-AV: E=Sophos;i="5.43,366,1503331200"; d="scan'208";a="57651110" Received: from mail-by2nam03lp0048.outbound.protection.outlook.com (HELO NAM03-BY2-obe.outbound.protection.outlook.com) ([216.32.180.48]) by ob1.hgst.iphmx.com with ESMTP; 13 Oct 2017 00:59:04 +0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sharedspace.onmicrosoft.com; s=selector1-wdc-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=DqTN8YFj7ry3U0EzQh9jDUpmmTx/OmcVJ7LhaNYTe0o=; b=NYlHRZD+KoG8k9vjayKLnfPqayY4M735VTe+K9BVfLtZuNTHxviOeLdQcgKihPPZYZ1iiHT/2hsDvosOgNhF0/4rR6/be43ogx/wdJLjc2lCfAVDL2BcfWUqY1L5KPv6TEC+mDKnOoHkGa+mZ5KqzKcnNcS03amJFVivZDhuEk0= Received: from CY1PR0401MB1536.namprd04.prod.outlook.com (10.163.19.154) by CY1PR0401MB1533.namprd04.prod.outlook.com (10.163.19.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.77.7; Thu, 12 Oct 2017 16:59:01 +0000 Received: from CY1PR0401MB1536.namprd04.prod.outlook.com ([10.163.19.154]) by CY1PR0401MB1536.namprd04.prod.outlook.com ([10.163.19.154]) with mapi id 15.20.0077.019; Thu, 12 Oct 2017 16:59:01 +0000 From: Bart Van Assche To: "shli@kernel.org" CC: "linux-block@vger.kernel.org" , "jthumshirn@suse.de" , "linux-raid@vger.kernel.org" , "hch@lst.de" , "martin.petersen@oracle.com" , "axboe@kernel.dk" , "ming.lei@redhat.com" , "linux-scsi@vger.kernel.org" , "oleksandr@natalenko.name" , "hare@suse.com" Subject: Re: [PATCH v8 03/10] md: Neither resync nor reshape while the system is frozen Thread-Topic: [PATCH v8 03/10] md: Neither resync nor reshape while the system is frozen Thread-Index: AQHTQgtFJ8rGtjz8I0iWk8dpEfhEkKLdqqWAgAARgQCAACXXAIABA62AgAAlj4CAAWd9AA== Date: Thu, 12 Oct 2017 16:59:01 +0000 Message-ID: <1507827540.2448.29.camel@wdc.com> References: <20171010210346.14574-1-bart.vanassche@wdc.com> <20171010210346.14574-4-bart.vanassche@wdc.com> <20171010223025.tda3lx64rdmugm54@kernel.org> <1507678384.2815.80.camel@wdc.com> <20171011014830.ps3v3yxny27wv7mc@kernel.org> <1507742275.2477.20.camel@wdc.com> <20171011193221.hkwvcysv6hahsncd@kernel.org> In-Reply-To: <20171011193221.hkwvcysv6hahsncd@kernel.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Bart.VanAssche@wdc.com; x-originating-ip: [63.163.107.100] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; CY1PR0401MB1533; 20:8vMTv1hejKp2YifYlcK+C6iATBk7ufks9uoJuw5dl6FtuEM6Z3vBXL8aM/ug+fxUkUzc8O1dCiGoQsdk/qvSmXSVAEQmm/TRJiaYVFK6sNyJ0B4mgXu3ZnNBspQapIk8/TfCkHmhr/JB94GwwOcDrSedkkj0pA3g4UJee6FIEnM= x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 49ff6a08-fde2-4efe-68a9-08d511928992 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(2017030254152)(48565401081)(2017052603199)(201703131423075)(201703031133081)(201702281549075); SRVR:CY1PR0401MB1533; x-ms-traffictypediagnostic: CY1PR0401MB1533: wdcipoutbound: EOP-TRUE x-exchange-antispam-report-test: UriScan:(9452136761055); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(10201501046)(100000703101)(100105400095)(3002001)(93006095)(93001095)(6055026)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123562025)(20161123555025)(20161123564025)(20161123560025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:CY1PR0401MB1533; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:CY1PR0401MB1533; x-forefront-prvs: 04583CED1A x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(979002)(6009001)(39860400002)(376002)(346002)(52314003)(199003)(24454002)(377424004)(189002)(66066001)(99286003)(4326008)(14454004)(81166006)(7736002)(2351001)(2906002)(7416002)(305945005)(93886005)(3846002)(25786009)(4001150100001)(53936002)(102836003)(97736004)(2501003)(33646002)(5660300001)(105586002)(8676002)(1730700003)(6512007)(8936002)(6246003)(316002)(5640700003)(6116002)(36756003)(101416001)(2900100001)(103116003)(50986999)(189998001)(3660700001)(72206003)(6506006)(6486002)(106356001)(77096006)(76176999)(68736007)(478600001)(54906003)(81156014)(3280700002)(54356999)(86362001)(229853002)(6916009)(6436002)(2950100002)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1102; SCL:1; SRVR:CY1PR0401MB1533; H:CY1PR0401MB1536.namprd04.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-ID: <8F400BE7C5D23B4E89FB4899F05A07DB@namprd04.prod.outlook.com> MIME-Version: 1.0 X-OriginatorOrg: wdc.com X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Oct 2017 16:59:01.2514 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b61c8803-16f3-4c35-9b17-6f65f441df86 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0401MB1533 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Wed, 2017-10-11 at 12:32 -0700, Shaohua Li wrote: > On Wed, Oct 11, 2017 at 05:17:56PM +0000, Bart Van Assche wrote: > > On Tue, 2017-10-10 at 18:48 -0700, Shaohua Li wrote: > > > The problem is __md_stop_writes set some bit like MD_RECOVERY_FROZEN, which > > > will prevent md_check_recovery restarting resync/reshape. I think we need > > > preserve mddev->recovery in suspend prepare and restore after resume > > > > Hello Shaohua, > > > > As far as I can see __md_stop_writes() sets MD_RECOVERY_FROZEN and can set > > MD_RECOVERY_INTR. Since md_check_recovery() clears MD_RECOVERY_INTR I think > > it should be sufficient to save and restore the state of the > > MD_RECOVERY_FROZEN flag. However, when I ran the following test: > > * echo frozen > /sys/block/md0/md/sync_action > > * cat /sys/block/md0/md/sync_action > > * systemctl hibernate > > * (power on system again) > > * cat /sys/block/md0/md/sync_action > > > > the output that appeared was as follows: > > > > frozen > > idle > > Does that mean that a hibernate or suspend followed by a resume is sufficient > > to clear the MD_RECOVERY_FROZEN flag for the md drivers and hence that the > > patch at the start of this e-mail thread does not need any further > > modifications? > > Have no idea why it shows 'idle'. From my understanding, after resume, previous > memory is stored and MD_RECOVERY_FROZEN bit should be set. Was trying to > understand what happens. Hello Shaohua, I think a user space action causes the MD_RECOVERY_FROZEN bit to be cleared. After I had added a few debug statements in the md driver this is what I found in the system log: Oct 12 09:38:39 kernel: action_store: storing action check Oct 12 09:38:39 kernel: md: data-check of RAID array md0 Oct 12 09:38:40 systemd[1]: Starting Hibernate... Oct 12 09:38:40 kernel: md: md0: data-check interrupted. [ powered on system manually ] Oct 12 09:39:05 kernel: action_store: storing action check Oct 12 09:39:05 kernel: md: data-check of RAID array md0 Oct 12 09:39:11 kernel: action_store: storing action idle Oct 12 09:39:11 kernel: md: md0: data-check interrupted. Anyway, how about the patch below? I have verified by adding a debug pr_info() statement that the newly added code really clears the MD_RECOVERY_FROZEN bit. Thanks, Bart. Subject: [PATCH] md: Neither resync nor reshape while the system is frozen Some people use the md driver on laptops and use the suspend and resume functionality. Since it is essential that submitting of new I/O requests stops before a hibernation image is created, interrupt the md resync and reshape actions if the system is being frozen. Note: the resync and reshape will restart after the system is resumed and a message similar to the following will appear in the system log: md: md0: data-check interrupted. Signed-off-by: Bart Van Assche Cc: Shaohua Li Cc: linux-raid@vger.kernel.org Cc: Ming Lei Cc: Christoph Hellwig Cc: Hannes Reinecke Cc: Johannes Thumshirn --- drivers/md/md.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++++- drivers/md/md.h | 8 ++++++++ 2 files changed, 57 insertions(+), 1 deletion(-) Reviewed-by: Shaohua Li diff --git a/drivers/md/md.c b/drivers/md/md.c index b99584e5d6b1..8b2fc750f97f 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -66,6 +66,8 @@ #include #include #include +#include +#include #include #include "md.h" @@ -7439,6 +7441,7 @@ static int md_thread(void *arg) */ allow_signal(SIGKILL); + set_freezable(); while (!kthread_should_stop()) { /* We need to wait INTERRUPTIBLE so that @@ -7449,7 +7452,7 @@ static int md_thread(void *arg) if (signal_pending(current)) flush_signals(current); - wait_event_interruptible_timeout + wait_event_freezable_timeout (thread->wqueue, test_bit(THREAD_WAKEUP, &thread->flags) || kthread_should_stop() || kthread_should_park(), @@ -8944,6 +8947,8 @@ static void md_stop_all_writes(void) int need_delay = 0; for_each_mddev(mddev, tmp) { + mddev->frozen_before_suspend = + test_bit(MD_RECOVERY_FROZEN, &mddev->recovery); if (mddev_trylock(mddev)) { if (mddev->pers) __md_stop_writes(mddev); @@ -8963,6 +8968,47 @@ static void md_stop_all_writes(void) mdelay(1000*1); } +static void md_restore_frozen_flag(void) +{ + struct list_head *tmp; + struct mddev *mddev; + + for_each_mddev(mddev, tmp) { + if (mddev->frozen_before_suspend) + set_bit(MD_RECOVERY_FROZEN, &mddev->recovery); + else + clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery); + } +} + +/* + * Ensure that neither resyncing nor reshaping occurs while the system is + * frozen. + */ +static int md_notify_pm(struct notifier_block *bl, unsigned long state, + void *unused) +{ + pr_debug("%s: state = %ld\n", __func__, state); + + switch (state) { + case PM_HIBERNATION_PREPARE: + case PM_SUSPEND_PREPARE: + case PM_RESTORE_PREPARE: + md_stop_all_writes(); + break; + case PM_POST_HIBERNATION: + case PM_POST_SUSPEND: + case PM_POST_RESTORE: + md_restore_frozen_flag(); + break; + } + return NOTIFY_DONE; +} + +static struct notifier_block md_pm_notifier = { + .notifier_call = md_notify_pm, +}; + static int md_notify_reboot(struct notifier_block *this, unsigned long code, void *x) { @@ -9009,6 +9055,7 @@ static int __init md_init(void) md_probe, NULL, NULL); register_reboot_notifier(&md_reboot_notifier); + register_pm_notifier(&md_pm_notifier); raid_table_header = register_sysctl_table(raid_root_table); md_geninit(); @@ -9248,6 +9295,7 @@ static __exit void md_exit(void) unregister_blkdev(MD_MAJOR,"md"); unregister_blkdev(mdp_major, "mdp"); + unregister_pm_notifier(&md_pm_notifier); unregister_reboot_notifier(&md_reboot_notifier); unregister_sysctl_table(raid_table_header); diff --git a/drivers/md/md.h b/drivers/md/md.h index d8287d3cd1bf..727b3aef4d26 100644 --- a/drivers/md/md.h +++ b/drivers/md/md.h @@ -356,6 +356,14 @@ struct mddev { */ int recovery_disabled; + /* Writes are stopped before hibernation, suspend and restore by + * calling md_stop_all_writes(). That function sets the + * MD_RECOVERY_FROZEN flag. The value of that flag is saved before + * writes are stopped such that it can be restored after hibernation, + * suspend or restore have finished. + */ + bool frozen_before_suspend; + int in_sync; /* know to not need resync */ /* 'open_mutex' avoids races between 'md_open' and 'do_md_stop', so * that we are never stopping an array while it is open.