From patchwork Wed Jun 19 04:17:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 11003309 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 28700924 for ; Wed, 19 Jun 2019 04:17:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 17F4D28A9D for ; Wed, 19 Jun 2019 04:17:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0C45E28AB1; Wed, 19 Jun 2019 04:17:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 60EA428A9D for ; Wed, 19 Jun 2019 04:17:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 786E16B0005; Wed, 19 Jun 2019 00:17:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 737F48E0002; Wed, 19 Jun 2019 00:17:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64FE28E0001; Wed, 19 Jun 2019 00:17:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 191816B0005 for ; Wed, 19 Jun 2019 00:17:42 -0400 (EDT) Received: by mail-ed1-f69.google.com with SMTP id o13so24285487edt.4 for ; Tue, 18 Jun 2019 21:17:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=mk7z2bflQrQ5U619V8n7hxrKZ5DzGqpWnNox93l8SO4=; b=evKFKArCpohg9iJ3tMW4zfmgyS2ZAe8C6f22kExqTXECHlU1cV3N3g83HZjS+Tagik B20bF3bu4ZV5nFNE/hSVVvz5xPsQbCIb3GG2UvWog6IJG1LNnofAeqLGh+RBl8aTrCST dqo0seV0AbSJKKUbTrrhHkpPGTt5eVqGdV+PjK8CGKKTd5ukWzlxfm6fdkCF4sWB5y5E JZW2M+mVqMgJk1p6YOiuiNWPkB0+xpcuRuQ42WVuB7cczVxe0q6q4qRv3sP9+HztYGh+ nWY++YEupZpG5tB+zkYnY55gCPKUSBe3fIkWnWZxGYtJBx961SIIaH2mKZ2yoJb4OZBU ww8A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com X-Gm-Message-State: APjAAAVGEcJBPhzh1docrYxqhdlVvvf6OPQKTvQRX5jBCjSMdHjBdABo aDeVhMT68lvkVGzWFKa0vbqbfhJf4VkWaZ4KQl1vicvth9SVQh5BhFM9ENbMJkni1kInYvjrUbH GaIUQc1CPqS33ieGp0Ufy8RnpiFNPsu/FNrpv3hyeIjM1/J7C7+B2ahKURUF8NB1AYg== X-Received: by 2002:aa7:ca41:: with SMTP id j1mr75412148edt.149.1560917861596; Tue, 18 Jun 2019 21:17:41 -0700 (PDT) X-Google-Smtp-Source: APXvYqyA3vJaOgl4sba0+ZNjzE9BfZtRqBN++AKwmxlZ/PN2cuNh1oecE4Vddv0fvUIAbtHwcB87 X-Received: by 2002:aa7:ca41:: with SMTP id j1mr75412063edt.149.1560917860185; Tue, 18 Jun 2019 21:17:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560917860; cv=none; d=google.com; s=arc-20160816; b=EgdK0lHKOJOP5W60RMscv4KVtaidJyGclx8F8/nPGxzXZOtYqFgIxvNw8wesVJKMmL Cm3wY9iLTgHFLzCAlFQ2vNp3+AKNcGSkQI2uA0ae8n9gkjRXY7WclK07G+fnYzIITYm6 NG7pOH9Jny/tavZZOU3ebymff6aLMpDr4FHbaauB9quGio8sBZVWfs4etKp1s44eew/a AzhfFpPAVrMU8v/mw1Gs5hIJzTWAdNnubY6geV1PFyZmpxskr/tIWeRmvjvo4mr5tBSv K/ByQMMbC3NaTsp7jMSeMdv4W32/SBm/rWKLE+TYT/RZlu/i/5EIsMn5V6MvjQuqqMhk 0alQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=mk7z2bflQrQ5U619V8n7hxrKZ5DzGqpWnNox93l8SO4=; b=bXOC98TVxohU2HFmjUcLmhdSPjnBX7uC9vU8enc5EME+tHeEZeTjwCimro4wgi4+3O Ty8YySAa84XGfodKu9WYdLC9QKmSwbsqI+EONc4ubT7Mrg47KGsVLhf1HiBtiwptbAGz Evr6X3SLwN7iMXMLM8t/eYvGGwbdAfpNDxAa9rAFEUyotZH/7zo1shLw4NIZb7K7rpCo xUulFyYEomF1u/IkFHh/fmHFcbhoz2eD08oukmSX5Qvc0YjnuaShr/jeWaET6INBZJGb aMcHUDTk1xkvyHOPo5jRRc85S+o5a0xS15i0glqjLNtjNMSBN+Zsa5zHvvlTXfJ2sS/y sl1A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from foss.arm.com (foss.arm.com. [217.140.110.172]) by mx.google.com with ESMTP id e28si13195245edb.265.2019.06.18.21.17.39 for ; Tue, 18 Jun 2019 21:17:40 -0700 (PDT) Received-SPF: pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) client-ip=217.140.110.172; Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3BF37360; Tue, 18 Jun 2019 21:17:39 -0700 (PDT) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.43.130]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 72C793F718; Tue, 18 Jun 2019 21:17:32 -0700 (PDT) From: Anshuman Khandual To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, akpm@linux-foundation.org, catalin.marinas@arm.com, will.deacon@arm.com Cc: mark.rutland@arm.com, mhocko@suse.com, ira.weiny@intel.com, david@redhat.com, cai@lca.pw, logang@deltatee.com, james.morse@arm.com, cpandya@codeaurora.org, arunks@codeaurora.org, dan.j.williams@intel.com, mgorman@techsingularity.net, osalvador@suse.de, ard.biesheuvel@arm.com, steve.capper@arm.com Subject: [PATCH V6 1/3] mm/hotplug: Reorder memblock_[free|remove]() calls in try_remove_memory() Date: Wed, 19 Jun 2019 09:47:38 +0530 Message-Id: <1560917860-26169-2-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1560917860-26169-1-git-send-email-anshuman.khandual@arm.com> References: <1560917860-26169-1-git-send-email-anshuman.khandual@arm.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Memory hot remove uses get_nid_for_pfn() while tearing down linked sysfs entries between memory block and node. It first checks pfn validity with pfn_valid_within() before fetching nid. With CONFIG_HOLES_IN_ZONE config (arm64 has this enabled) pfn_valid_within() calls pfn_valid(). pfn_valid() is an arch implementation on arm64 (CONFIG_HAVE_ARCH_PFN_VALID) which scans all mapped memblock regions with memblock_is_map_memory(). This creates a problem in memory hot remove path which has already removed given memory range from memory block with memblock_[remove|free] before arriving at unregister_mem_sect_under_nodes(). Hence get_nid_for_pfn() returns -1 skipping subsequent sysfs_remove_link() calls leaving node <-> memory block sysfs entries as is. Subsequent memory add operation hits BUG_ON() because of existing sysfs entries. [ 62.007176] NUMA: Unknown node for memory at 0x680000000, assuming node 0 [ 62.052517] ------------[ cut here ]------------ [ 62.053211] kernel BUG at mm/memory_hotplug.c:1143! [ 62.053868] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP [ 62.054589] Modules linked in: [ 62.054999] CPU: 19 PID: 3275 Comm: bash Not tainted 5.1.0-rc2-00004-g28cea40b2683 #41 [ 62.056274] Hardware name: linux,dummy-virt (DT) [ 62.057166] pstate: 40400005 (nZcv daif +PAN -UAO) [ 62.058083] pc : add_memory_resource+0x1cc/0x1d8 [ 62.058961] lr : add_memory_resource+0x10c/0x1d8 [ 62.059842] sp : ffff0000168b3ce0 [ 62.060477] x29: ffff0000168b3ce0 x28: ffff8005db546c00 [ 62.061501] x27: 0000000000000000 x26: 0000000000000000 [ 62.062509] x25: ffff0000111ef000 x24: ffff0000111ef5d0 [ 62.063520] x23: 0000000000000000 x22: 00000006bfffffff [ 62.064540] x21: 00000000ffffffef x20: 00000000006c0000 [ 62.065558] x19: 0000000000680000 x18: 0000000000000024 [ 62.066566] x17: 0000000000000000 x16: 0000000000000000 [ 62.067579] x15: ffffffffffffffff x14: ffff8005e412e890 [ 62.068588] x13: ffff8005d6b105d8 x12: 0000000000000000 [ 62.069610] x11: ffff8005d6b10490 x10: 0000000000000040 [ 62.070615] x9 : ffff8005e412e898 x8 : ffff8005e412e890 [ 62.071631] x7 : ffff8005d6b105d8 x6 : ffff8005db546c00 [ 62.072640] x5 : 0000000000000001 x4 : 0000000000000002 [ 62.073654] x3 : ffff8005d7049480 x2 : 0000000000000002 [ 62.074666] x1 : 0000000000000003 x0 : 00000000ffffffef [ 62.075685] Process bash (pid: 3275, stack limit = 0x00000000d754280f) [ 62.076930] Call trace: [ 62.077411] add_memory_resource+0x1cc/0x1d8 [ 62.078227] __add_memory+0x70/0xa8 [ 62.078901] probe_store+0xa4/0xc8 [ 62.079561] dev_attr_store+0x18/0x28 [ 62.080270] sysfs_kf_write+0x40/0x58 [ 62.080992] kernfs_fop_write+0xcc/0x1d8 [ 62.081744] __vfs_write+0x18/0x40 [ 62.082400] vfs_write+0xa4/0x1b0 [ 62.083037] ksys_write+0x5c/0xc0 [ 62.083681] __arm64_sys_write+0x18/0x20 [ 62.084432] el0_svc_handler+0x88/0x100 [ 62.085177] el0_svc+0x8/0xc Re-ordering memblock_[free|remove]() with arch_remove_memory() solves the problem on arm64 as pfn_valid() behaves correctly and returns positive as memblock for the address range still exists. arch_remove_memory() removes applicable memory sections from zone with __remove_pages() and tears down kernel linear mapping. Removing memblock regions afterwards is safe because there is no other memblock (bootmem) allocator user that late. So nobody is going to allocate from the removed range just to blow up later. Also nobody should be using the bootmem allocated range else we wouldn't allow to remove it. So reordering is indeed safe. Reviewed-by: David Hildenbrand Reviewed-by: Oscar Salvador Acked-by: Mark Rutland Acked-by: Michal Hocko Signed-off-by: Anshuman Khandual --- mm/memory_hotplug.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index a88c5f3..cfa5fac 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1831,13 +1831,13 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size) /* remove memmap entry */ firmware_map_remove(start, start + size, "System RAM"); - memblock_free(start, size); - memblock_remove(start, size); /* remove memory block devices before removing memory */ remove_memory_block_devices(start, size); arch_remove_memory(nid, start, size, NULL); + memblock_free(start, size); + memblock_remove(start, size); __release_memory_resource(start, size); try_offline_node(nid); From patchwork Wed Jun 19 04:17:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 11003313 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E3CB8112C for ; Wed, 19 Jun 2019 04:17:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D04A728A9D for ; Wed, 19 Jun 2019 04:17:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C372528AA7; Wed, 19 Jun 2019 04:17:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5EC0C28A9D for ; Wed, 19 Jun 2019 04:17:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D94C6B0006; Wed, 19 Jun 2019 00:17:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 58A238E0002; Wed, 19 Jun 2019 00:17:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A0108E0001; Wed, 19 Jun 2019 00:17:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 00F186B0006 for ; Wed, 19 Jun 2019 00:17:48 -0400 (EDT) Received: by mail-ed1-f71.google.com with SMTP id s5so24272198eda.10 for ; Tue, 18 Jun 2019 21:17:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=GlOBlEcIXkjl+TiR1o4jafzqgo8fD8vSAH8C3FJFjlA=; b=CGHcc9q/M58rN9uO+x47XFkf/Na4Rl2522QW5yIXQRXJjnZoOuda7GnKSSv0wsByU3 U2Lsu29AUoxKe/KwVDJgHIqwwokYNLgq7RvxhBtXekn88foqnpVcGz7+fB9G/GRSPrlZ WEeCEeIkMBGojcy44q/FqyUZB0Z17hXpslDnNzTrWLY5/FoM0Kpme+LQb6hv2FhcDueY eF9tTk3m1EsFnKSJ9w22AON0hEKyMSzPy9nKjqG5nHf6p+LO7kNrU3PTAfTJBKT2baoS IIW43pZaeXeE5zO3EuUxNQoY8WoefYszGGCQuox/k4+Mbgpk+iwUiisiUjjGzH0pncAr jSXw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com X-Gm-Message-State: APjAAAVLMZzGCZB/UqrQ9kguC8AAvd3KCOogyiou7yJOgyzVd7v38LKA mMf02bW+lzdYJj2F/1d26K5sYd+xOi/SLLukvMBWPZot5h1mLLaLrlzvLKwuhRjG793i1/mivpO AkVlF0SomjnyMvzzuH+hcJkR1qPAqf073W3P3kCfUE/FaHQqWGVwEHfGD7jSRI9tKhw== X-Received: by 2002:a17:906:69c4:: with SMTP id g4mr16601777ejs.9.1560917867503; Tue, 18 Jun 2019 21:17:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqwZAoK1bquDBAoeDeSF4zRSvg0TGFe0/X8r5dzICgfrGN/8XhJyJ6BjssScCMvSBEAhhbVh X-Received: by 2002:a17:906:69c4:: with SMTP id g4mr16601730ejs.9.1560917866624; Tue, 18 Jun 2019 21:17:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560917866; cv=none; d=google.com; s=arc-20160816; b=NDT+p8ee7xJiVjO6NK72HYxiYyDROCZQ4PNZcW1hCClb5rg+p27sF9oknIVW3PUkmO Ev8gQ/gSyg1A52O6um05XCFaUOltoioxA7w3dugJJYH3RNZ3C0F6RlwfTJAQpxmv//NQ pYy/IVuN+C8EjoOSrzJG7L02wfh51SKlMHTEu6LlpQcluGCAcXx2JGjXu65hR5+Y+UR+ Bx0JxrR8c7rMuZwrFUPsnGEyqySgeYE5xzEffZ4ebC5TqmKAYStjFviHdHU+KV3KMD42 37ZyCyrQWEBDKCvcgN2YODeGrvtA/pVjIkLdOyaPmVLphwowfoopV3VufQog93TmJ7m9 9A2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=GlOBlEcIXkjl+TiR1o4jafzqgo8fD8vSAH8C3FJFjlA=; b=k2hi54eeJRL0sXoOnqB3c8GGe/Y25RQdte7kpR2livg3TEF3l+TBW9FDg8S5GGzHsz YJhfFSujCPmETfnDuZDvMlqSXidkSi234xVPtb51Gj+0iepl6XUEGcsWkRymxdrkOAC7 PcUoFmF+AaPv4qdIzICwNmFbtAO/b91jroRqmwa76cG8z2xHNJnY/8aNyRoq9wj9Uqtg /dDSBFYd9bWQckcVz7J9CULyLTG39ZZJ8USCQvYIoNRaHmyd8f2zwfMYkg70xuY7zlOl B3boiiNr4bJkSnYeZ9o/TsxZkGfrC5bFA8kgUTK5Aw9tXPvb5jpEwiIaTW2eO/khMDib jdQA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from foss.arm.com (foss.arm.com. [217.140.110.172]) by mx.google.com with ESMTP id u12si3093775ejz.73.2019.06.18.21.17.46 for ; Tue, 18 Jun 2019 21:17:46 -0700 (PDT) Received-SPF: pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) client-ip=217.140.110.172; Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C36CA360; Tue, 18 Jun 2019 21:17:45 -0700 (PDT) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.43.130]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id BEC413F718; Tue, 18 Jun 2019 21:17:39 -0700 (PDT) From: Anshuman Khandual To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, akpm@linux-foundation.org, catalin.marinas@arm.com, will.deacon@arm.com Cc: mark.rutland@arm.com, mhocko@suse.com, ira.weiny@intel.com, david@redhat.com, cai@lca.pw, logang@deltatee.com, james.morse@arm.com, cpandya@codeaurora.org, arunks@codeaurora.org, dan.j.williams@intel.com, mgorman@techsingularity.net, osalvador@suse.de, ard.biesheuvel@arm.com, steve.capper@arm.com Subject: [PATCH V6 2/3] arm64/mm: Hold memory hotplug lock while walking for kernel page table dump Date: Wed, 19 Jun 2019 09:47:39 +0530 Message-Id: <1560917860-26169-3-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1560917860-26169-1-git-send-email-anshuman.khandual@arm.com> References: <1560917860-26169-1-git-send-email-anshuman.khandual@arm.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The arm64 page table dump code can race with concurrent modification of the kernel page tables. When a leaf entries are modified concurrently, the dump code may log stale or inconsistent information for a VA range, but this is otherwise not harmful. When intermediate levels of table are freed, the dump code will continue to use memory which has been freed and potentially reallocated for another purpose. In such cases, the dump code may dereference bogus addresses, leading to a number of potential problems. Intermediate levels of table may by freed during memory hot-remove, which will be enabled by a subsequent patch. To avoid racing with this, take the memory hotplug lock when walking the kernel page table. Acked-by: David Hildenbrand Acked-by: Mark Rutland Signed-off-by: Anshuman Khandual --- arch/arm64/mm/ptdump_debugfs.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/arm64/mm/ptdump_debugfs.c b/arch/arm64/mm/ptdump_debugfs.c index 064163f..b5eebc8 100644 --- a/arch/arm64/mm/ptdump_debugfs.c +++ b/arch/arm64/mm/ptdump_debugfs.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include #include #include @@ -7,7 +8,10 @@ static int ptdump_show(struct seq_file *m, void *v) { struct ptdump_info *info = m->private; + + get_online_mems(); ptdump_walk_pgd(m, info); + put_online_mems(); return 0; } DEFINE_SHOW_ATTRIBUTE(ptdump); From patchwork Wed Jun 19 04:17:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 11003317 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 924FA924 for ; Wed, 19 Jun 2019 04:17:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 806A828A9D for ; Wed, 19 Jun 2019 04:17:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7491628AA7; Wed, 19 Jun 2019 04:17:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7A89128A9D for ; Wed, 19 Jun 2019 04:17:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 899146B0007; Wed, 19 Jun 2019 00:17:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 849548E0002; Wed, 19 Jun 2019 00:17:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 738728E0001; Wed, 19 Jun 2019 00:17:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 242936B0007 for ; Wed, 19 Jun 2019 00:17:55 -0400 (EDT) Received: by mail-ed1-f70.google.com with SMTP id a5so24218435edx.12 for ; Tue, 18 Jun 2019 21:17:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=uWiY2WTdo/ew647hAF8F0jW57wgesm6JL1lLbG+HNBQ=; b=Jyq7/fuUPe+9odIbLisGB7lq0x2ZEaaxbONg9cGqWpe7thodKutdyBxDplqVW8sMS9 d4iQMuV5kiaYoxDI3IU5RJw1BWnZCwPI5Sdk2U1b/bfJXsIKfWwTfGPBSc60Yr5L3nm1 apQN6Tt2/p5o29QU6no5DqfltSsGmuMSZJoWX9SxcxKKyxEYMTQbB6/ti7lHqAA95zQ1 kZV9eR5RA817uefhnypH7TYHYgHAveTy4fe7bih4CnNP4rioqOAHyMQon4g+TmRZI+2F lgwRewnl9rYBakGlCkIa1GCaSas0rAMBAXeG2xte1REI2df3gPdcujsafe2vU3L8NA83 rulw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com X-Gm-Message-State: APjAAAWO0GJWp7OGJwEDB5Oy4iRmj4/L2KuAJJJ2HXQVKzZnWR8FiabR E1zxnhTqI2x+21dy5ftU2zSv+6fQpEdNxGwnnABi0BQh8DWDpRlsUDKBwz6kM42GElqjYQ6hxF5 TUkmVT7kwsxlaOJmXwhZxZ58aWXVm6eikuLKQKWOY8xdjqzZw7PP175XTn0k42qjk3g== X-Received: by 2002:a17:906:454d:: with SMTP id s13mr6815271ejq.255.1560917874661; Tue, 18 Jun 2019 21:17:54 -0700 (PDT) X-Google-Smtp-Source: APXvYqw7WYkh46L3qjMeE9utzUi73hPdgnuWBAz70dRzrNXXhWs7qLoHAMClnqH1dQdPvjNdGqeL X-Received: by 2002:a17:906:454d:: with SMTP id s13mr6815211ejq.255.1560917873451; Tue, 18 Jun 2019 21:17:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560917873; cv=none; d=google.com; s=arc-20160816; b=XQaL54jYL87/q9QTjZR3yvyUxNhAfjSaOvMiqwDN4p/oyx+jlXdauNCnGym/z5Hw9U FzBf0qEsTuTCpOMR8o2PC2GAxjs3bt3VZb6lBhB2bUIB/I8Ng42w2sCmBMD3awwhm2l6 P3YfaqACCXEVW77lqDo6QDKcH9iYcyLTX1baN3+l/0OYfoeutotDIcwS4LknAUShb32k IUMM0nXro6bFMeImEeGE0S2SqrIBt5p7AsagOzRkPK3FDYMKfnnqRNoJoIBKJtSu3HI9 O40nikP2QbAnj8ItVuJ9E3GDrG8N3x1yIQAjJjLSurxzq9rLhDHuaVF958pNinMXivJw 3sOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=uWiY2WTdo/ew647hAF8F0jW57wgesm6JL1lLbG+HNBQ=; b=QanXsToLHNLcYWyK1H1HYwRgTvr1cF3EtfPUCdLkzkP97qiGxO0gsr4XSRlC8jjb2J 6GrAf93Z2TTsCwQMZ/tbdSf2UXddKG8osQmM1exkqaMSRDzHoKwL85zd4SC8pQt9i6hE ioSK3wn+h0Hg2GC+FfgP/SXrYu7HSYL8U+S/TGjSplxEym58h+QpuQcgO0lm39fpzUS6 4LWC1xsmLpjNV9Bmon8SykjMrBM08GfV5OazmcxNMD7x6TEvGhXdpt7w1M8GNyQpfSqU FDAQuZ3UJOrULSFsDur/pS3ZHhVqB4NpgaZ+5HxdVUYS6MY+xi+6xVLG0c9oV3Dlzqid oaJA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from foss.arm.com (foss.arm.com. [217.140.110.172]) by mx.google.com with ESMTP id j42si12702039eda.80.2019.06.18.21.17.53 for ; Tue, 18 Jun 2019 21:17:53 -0700 (PDT) Received-SPF: pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) client-ip=217.140.110.172; Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6F25CCFC; Tue, 18 Jun 2019 21:17:52 -0700 (PDT) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.43.130]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 60B7F3F718; Tue, 18 Jun 2019 21:17:46 -0700 (PDT) From: Anshuman Khandual To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, akpm@linux-foundation.org, catalin.marinas@arm.com, will.deacon@arm.com Cc: mark.rutland@arm.com, mhocko@suse.com, ira.weiny@intel.com, david@redhat.com, cai@lca.pw, logang@deltatee.com, james.morse@arm.com, cpandya@codeaurora.org, arunks@codeaurora.org, dan.j.williams@intel.com, mgorman@techsingularity.net, osalvador@suse.de, ard.biesheuvel@arm.com, steve.capper@arm.com Subject: [PATCH V6 3/3] arm64/mm: Enable memory hot remove Date: Wed, 19 Jun 2019 09:47:40 +0530 Message-Id: <1560917860-26169-4-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1560917860-26169-1-git-send-email-anshuman.khandual@arm.com> References: <1560917860-26169-1-git-send-email-anshuman.khandual@arm.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The arch code for hot-remove must tear down portions of the linear map and vmemmap corresponding to memory being removed. In both cases the page tables mapping these regions must be freed, and when sparse vmemmap is in use the memory backing the vmemmap must also be freed. This patch adds a new remove_pagetable() helper which can be used to tear down either region, and calls it from vmemmap_free() and ___remove_pgd_mapping(). The sparse_vmap argument determines whether the backing memory will be freed. remove_pagetable() makes two distinct passes over the kernel page table. In the first pass it unmaps, invalidates applicable TLB cache and frees backing memory if required (vmemmap) for each mapped leaf entry. In the second pass it looks for empty page table sections whose page table page can be unmapped, TLB invalidated and freed. While freeing intermediate level page table pages bail out if any of its entries are still valid. This can happen for partially filled kernel page table either from a previously attempted failed memory hot add or while removing an address range which does not span the entire page table page range. The vmemmap region may share levels of table with the vmalloc region. There can be conflicts between hot remove freeing page table pages with a concurrent vmalloc() walking the kernel page table. This conflict can not just be solved by taking the init_mm ptl because of existing locking scheme in vmalloc(). Hence unlike linear mapping, skip freeing page table pages while tearing down vmemmap mapping. While here update arch_add_memory() to handle __add_pages() failures by just unmapping recently added kernel linear mapping. Now enable memory hot remove on arm64 platforms by default with ARCH_ENABLE_MEMORY_HOTREMOVE. This implementation is overall inspired from kernel page table tear down procedure on X86 architecture. Acked-by: David Hildenbrand Signed-off-by: Anshuman Khandual Acked-by: Steve Capper --- arch/arm64/Kconfig | 3 + arch/arm64/mm/mmu.c | 290 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 284 insertions(+), 9 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 6426f48..9375f26 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -270,6 +270,9 @@ config HAVE_GENERIC_GUP config ARCH_ENABLE_MEMORY_HOTPLUG def_bool y +config ARCH_ENABLE_MEMORY_HOTREMOVE + def_bool y + config SMP def_bool y diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 93ed0df..9e80a94 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -733,6 +733,250 @@ int kern_addr_valid(unsigned long addr) return pfn_valid(pte_pfn(pte)); } + +#ifdef CONFIG_MEMORY_HOTPLUG +static void free_hotplug_page_range(struct page *page, size_t size) +{ + WARN_ON(!page || PageReserved(page)); + free_pages((unsigned long)page_address(page), get_order(size)); +} + +static void free_hotplug_pgtable_page(struct page *page) +{ + free_hotplug_page_range(page, PAGE_SIZE); +} + +static void free_pte_table(pmd_t *pmdp, unsigned long addr) +{ + struct page *page; + pte_t *ptep; + int i; + + ptep = pte_offset_kernel(pmdp, 0UL); + for (i = 0; i < PTRS_PER_PTE; i++) { + if (!pte_none(READ_ONCE(ptep[i]))) + return; + } + + page = pmd_page(READ_ONCE(*pmdp)); + pmd_clear(pmdp); + __flush_tlb_kernel_pgtable(addr); + free_hotplug_pgtable_page(page); +} + +static void free_pmd_table(pud_t *pudp, unsigned long addr) +{ + struct page *page; + pmd_t *pmdp; + int i; + + if (CONFIG_PGTABLE_LEVELS <= 2) + return; + + pmdp = pmd_offset(pudp, 0UL); + for (i = 0; i < PTRS_PER_PMD; i++) { + if (!pmd_none(READ_ONCE(pmdp[i]))) + return; + } + + page = pud_page(READ_ONCE(*pudp)); + pud_clear(pudp); + __flush_tlb_kernel_pgtable(addr); + free_hotplug_pgtable_page(page); +} + +static void free_pud_table(pgd_t *pgdp, unsigned long addr) +{ + struct page *page; + pud_t *pudp; + int i; + + if (CONFIG_PGTABLE_LEVELS <= 3) + return; + + pudp = pud_offset(pgdp, 0UL); + for (i = 0; i < PTRS_PER_PUD; i++) { + if (!pud_none(READ_ONCE(pudp[i]))) + return; + } + + page = pgd_page(READ_ONCE(*pgdp)); + pgd_clear(pgdp); + __flush_tlb_kernel_pgtable(addr); + free_hotplug_pgtable_page(page); +} + +static void unmap_hotplug_pte_range(pmd_t *pmdp, unsigned long addr, + unsigned long end, bool sparse_vmap) +{ + struct page *page; + pte_t *ptep, pte; + + do { + ptep = pte_offset_kernel(pmdp, addr); + pte = READ_ONCE(*ptep); + if (pte_none(pte)) + continue; + + WARN_ON(!pte_present(pte)); + page = sparse_vmap ? pte_page(pte) : NULL; + pte_clear(&init_mm, addr, ptep); + flush_tlb_kernel_range(addr, addr + PAGE_SIZE); + if (sparse_vmap) + free_hotplug_page_range(page, PAGE_SIZE); + } while (addr += PAGE_SIZE, addr < end); +} + +static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr, + unsigned long end, bool sparse_vmap) +{ + unsigned long next; + struct page *page; + pmd_t *pmdp, pmd; + + do { + next = pmd_addr_end(addr, end); + pmdp = pmd_offset(pudp, addr); + pmd = READ_ONCE(*pmdp); + if (pmd_none(pmd)) + continue; + + WARN_ON(!pmd_present(pmd)); + if (pmd_sect(pmd)) { + page = sparse_vmap ? pmd_page(pmd) : NULL; + pmd_clear(pmdp); + flush_tlb_kernel_range(addr, next); + if (sparse_vmap) + free_hotplug_page_range(page, PMD_SIZE); + continue; + } + WARN_ON(!pmd_table(pmd)); + unmap_hotplug_pte_range(pmdp, addr, next, sparse_vmap); + } while (addr = next, addr < end); +} + +static void unmap_hotplug_pud_range(pgd_t *pgdp, unsigned long addr, + unsigned long end, bool sparse_vmap) +{ + unsigned long next; + struct page *page; + pud_t *pudp, pud; + + do { + next = pud_addr_end(addr, end); + pudp = pud_offset(pgdp, addr); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) + continue; + + WARN_ON(!pud_present(pud)); + if (pud_sect(pud)) { + page = sparse_vmap ? pud_page(pud) : NULL; + pud_clear(pudp); + flush_tlb_kernel_range(addr, next); + if (sparse_vmap) + free_hotplug_page_range(page, PUD_SIZE); + continue; + } + WARN_ON(!pud_table(pud)); + unmap_hotplug_pmd_range(pudp, addr, next, sparse_vmap); + } while (addr = next, addr < end); +} + +static void unmap_hotplug_range(unsigned long addr, unsigned long end, + bool sparse_vmap) +{ + unsigned long next; + pgd_t *pgdp, pgd; + + do { + next = pgd_addr_end(addr, end); + pgdp = pgd_offset_k(addr); + pgd = READ_ONCE(*pgdp); + if (pgd_none(pgd)) + continue; + + WARN_ON(!pgd_present(pgd)); + unmap_hotplug_pud_range(pgdp, addr, next, sparse_vmap); + } while (addr = next, addr < end); +} + +static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, + unsigned long end) +{ + pte_t *ptep, pte; + + do { + ptep = pte_offset_kernel(pmdp, addr); + pte = READ_ONCE(*ptep); + WARN_ON(!pte_none(pte)); + } while (addr += PAGE_SIZE, addr < end); +} + +static void free_empty_pmd_table(pud_t *pudp, unsigned long addr, + unsigned long end) +{ + unsigned long next; + pmd_t *pmdp, pmd; + + do { + next = pmd_addr_end(addr, end); + pmdp = pmd_offset(pudp, addr); + pmd = READ_ONCE(*pmdp); + if (pmd_none(pmd)) + continue; + + WARN_ON(!pmd_present(pmd) || !pmd_table(pmd) || pmd_sect(pmd)); + free_empty_pte_table(pmdp, addr, next); + free_pte_table(pmdp, addr); + } while (addr = next, addr < end); +} + +static void free_empty_pud_table(pgd_t *pgdp, unsigned long addr, + unsigned long end) +{ + unsigned long next; + pud_t *pudp, pud; + + do { + next = pud_addr_end(addr, end); + pudp = pud_offset(pgdp, addr); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) + continue; + + WARN_ON(!pud_present(pud) || !pud_table(pud) || pud_sect(pud)); + free_empty_pmd_table(pudp, addr, next); + free_pmd_table(pudp, addr); + } while (addr = next, addr < end); +} + +static void free_empty_tables(unsigned long addr, unsigned long end) +{ + unsigned long next; + pgd_t *pgdp, pgd; + + do { + next = pgd_addr_end(addr, end); + pgdp = pgd_offset_k(addr); + pgd = READ_ONCE(*pgdp); + if (pgd_none(pgd)) + continue; + + WARN_ON(!pgd_present(pgd)); + free_empty_pud_table(pgdp, addr, next); + free_pud_table(pgdp, addr); + } while (addr = next, addr < end); +} + +static void remove_pagetable(unsigned long start, unsigned long end, + bool sparse_vmap) +{ + unmap_hotplug_range(start, end, sparse_vmap); + free_empty_tables(start, end); +} +#endif + #ifdef CONFIG_SPARSEMEM_VMEMMAP #if !ARM64_SWAPPER_USES_SECTION_MAPS int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, @@ -780,6 +1024,27 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, void vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap) { +#ifdef CONFIG_MEMORY_HOTPLUG + /* + * FIXME: We should have called remove_pagetable(start, end, true). + * vmemmap and vmalloc virtual range might share intermediate kernel + * page table entries. Removing vmemmap range page table pages here + * can potentially conflict with a cuncurrent vmalloc() allocation. + * + * This is primarily because valloc() does not take init_mm ptl for + * the entire page table walk and it's modification. Instead it just + * takes the lock while allocating and installing page table pages + * via [p4d|pud|pmd|pte]_aloc(). A cuncurrently vanishing page table + * entry via memory hotremove can cause vmalloc() kernel page table + * walk pointers to be invalid on the fly which can cause corruption + * or worst, a crash. + * + * To avoid this problem, lets not free empty page table pages for + * given vmemmap range being hot-removed. Just unmap and free the + * range instead. + */ + unmap_hotplug_range(start, end, true); +#endif } #endif /* CONFIG_SPARSEMEM_VMEMMAP */ @@ -1066,10 +1331,18 @@ int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) } #ifdef CONFIG_MEMORY_HOTPLUG +static void __remove_pgd_mapping(pgd_t *pgdir, unsigned long start, u64 size) +{ + unsigned long end = start + size; + + WARN_ON(pgdir != init_mm.pgd); + remove_pagetable(start, end, false); +} + int arch_add_memory(int nid, u64 start, u64 size, struct mhp_restrictions *restrictions) { - int flags = 0; + int ret, flags = 0; if (rodata_full || debug_pagealloc_enabled()) flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; @@ -1077,9 +1350,14 @@ int arch_add_memory(int nid, u64 start, u64 size, __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), size, PAGE_KERNEL, __pgd_pgtable_alloc, flags); - return __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, + ret = __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, restrictions); + if (ret) + __remove_pgd_mapping(swapper_pg_dir, + __phys_to_virt(start), size); + return ret; } + void arch_remove_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap) { @@ -1087,14 +1365,8 @@ void arch_remove_memory(int nid, u64 start, u64 size, unsigned long nr_pages = size >> PAGE_SHIFT; struct zone *zone; - /* - * FIXME: Cleanup page tables (also in arch_add_memory() in case - * adding fails). Until then, this function should only be used - * during memory hotplug (adding memory), not for memory - * unplug. ARCH_ENABLE_MEMORY_HOTREMOVE must not be - * unlocked yet. - */ zone = page_zone(pfn_to_page(start_pfn)); __remove_pages(zone, start_pfn, nr_pages, altmap); + __remove_pgd_mapping(swapper_pg_dir, __phys_to_virt(start), size); } #endif