From patchwork Tue Jun 11 11:26:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 10986599 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 71D4F14DB for ; Tue, 11 Jun 2019 11:26:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5E8BF287CF for ; Tue, 11 Jun 2019 11:26:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 508C5287E8; Tue, 11 Jun 2019 11:26:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9EDB3287CF for ; Tue, 11 Jun 2019 11:26:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 400EC6B0005; Tue, 11 Jun 2019 07:26:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3B1796B0006; Tue, 11 Jun 2019 07:26:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A1586B0007; Tue, 11 Jun 2019 07:26:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id D28F86B0005 for ; Tue, 11 Jun 2019 07:26:02 -0400 (EDT) Received: by mail-ed1-f71.google.com with SMTP id i9so4800006edr.13 for ; Tue, 11 Jun 2019 04:26:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=AgKdtSVvchxS2bnU4W011zCEvrOcUJNL2gyRxD5fxk4=; b=sRKCsuGrKUgyz77/wr+gd+f/P2zYDHyK+jIucK+KK5fioGkLm9vorjcdyxH4X5+H/0 ngHekWMnDdynTikW32rWrypcnGuPDiUNMP3edYQ8tjaYp+jJt48GkFr7W4v76zcLtj5r De9CLfmIynuR5NDrIDqONLGZz0Nz2w+NiGlD2m46LJID1aDKYlykk0FTCwgstOtHTqc4 aW/eT+nDqSg72YCvWeU+RBt7KiE4x6AD9wD/NmCM7sK8Y3Lcf6xVqgJEvZw9QyzzKDYh yuN8cu2qqrGFxYeevVaRB1ZhR1DFp99UT8R0TUdwkvKMwpNwoYrKdAUcnVFO5JEXMeS0 R4mQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com X-Gm-Message-State: APjAAAVy0RU4PKKH/7F53aImxWQuAQ7VQp+DvvlshjDWvYXGXtBDsHAF TPA4en2rdYMyOb813K7cvB+wcki4aBD5bkltK0LqkM6lvdZGEWewvQezghall/XGNJg/KCCzRVX nIJ1EcFNQzps8oRxFdZpeQS2nTRR13UBAPvBhop181ffdd8MRulW1pVqAO7tXy3GrXA== X-Received: by 2002:a17:906:5047:: with SMTP id e7mr56094874ejk.282.1560252362163; Tue, 11 Jun 2019 04:26:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqzMIfVmnTXTkYYSKysETHWog8O+jDYif8Q2zNXf4pVoTNIVQSGy4iql5VO6KLky4XL3/VqT X-Received: by 2002:a17:906:5047:: with SMTP id e7mr56094819ejk.282.1560252361134; Tue, 11 Jun 2019 04:26:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1560252361; cv=none; d=google.com; s=arc-20160816; b=M5keNv4gBdrQe/LDpBu9STODLIusghmvD+RgpHhPYAnKxn87zgcyG/bgXNRhLWz3Qi t+jKCwd/YHwGk7iar7Skbl8Ffjio7MnD7CILCKayHOvV1w8rvjSo/iMVZq9wEgZ6eanH zxrwTHxinYGq9PciaJY+P8B3bLGEmaB+dVnp078FYT7RuFuTl8VTN29ZsmaaLcg+yRdr PRWTK2T2MSSqZ6oLPpsB66gGZy6VswcD0bXcjvUFsNCiiUTsEmXd86Aj5GoWCQBlOFE+ RaDKFo4rudK2NLjpKi2MmvmOybIBmWZW97Qx447/5lZgPBsgSPAja4TBeFkLKzSoPp85 zjSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=AgKdtSVvchxS2bnU4W011zCEvrOcUJNL2gyRxD5fxk4=; b=dzm2o76HngMoVHW/WTnD2V6XrLAnOExWjryhHgFH9OwUDyMi5CuFwTOP64TAilJ3Rc bkRojfqyKXjfoyKldtkQ4spJIc21z+TFKihKNBEaUCQBc40NlA9mOl4/RGgkF4SLyr9/ 0nuWGm6DahD6axxLhkSYd5Q4L45zAGXbpm2rAoyQws53/oRL/cj4L1VqTuf7ta6AOe9g MdYepuPNtT0tnwO3K46v71KJ6hZuSuYFcqYJr21LilaWZcki3jqVNWhn4ocGRgNAZb9Z RuhuyIYJp9HHTVo9e0Y6ZDL34jUoLhv+9KMdTdZ6rQALNrvQFxNuUi39riYMIBX6fwZH gBYQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from foss.arm.com (foss.arm.com. [217.140.110.172]) by mx.google.com with ESMTP id j5si8160339eja.306.2019.06.11.04.26.00 for ; Tue, 11 Jun 2019 04:26:01 -0700 (PDT) Received-SPF: pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) client-ip=217.140.110.172; Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 102A3344; Tue, 11 Jun 2019 04:26:00 -0700 (PDT) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.43.135]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5A08F3F557; Tue, 11 Jun 2019 04:27:39 -0700 (PDT) From: Anshuman Khandual To: akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, will.deacon@arm.com, ard.biesheuvel@arm.com Cc: osalvador@suse.de, david@redhat.com, mhocko@suse.com, mark.rutland@arm.com Subject: [PATCH V5 - Rebased] mm/hotplug: Reorder memblock_[free|remove]() calls in try_remove_memory() Date: Tue, 11 Jun 2019 16:56:13 +0530 Message-Id: <1560252373-3230-1-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <36e0126f-e2d1-239c-71f3-91125a49e019@redhat.com> References: <36e0126f-e2d1-239c-71f3-91125a49e019@redhat.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Memory hot remove uses get_nid_for_pfn() while tearing down linked sysfs entries between memory block and node. It first checks pfn validity with pfn_valid_within() before fetching nid. With CONFIG_HOLES_IN_ZONE config (arm64 has this enabled) pfn_valid_within() calls pfn_valid(). pfn_valid() is an arch implementation on arm64 (CONFIG_HAVE_ARCH_PFN_VALID) which scans all mapped memblock regions with memblock_is_map_memory(). This creates a problem in memory hot remove path which has already removed given memory range from memory block with memblock_[remove|free] before arriving at unregister_mem_sect_under_nodes(). Hence get_nid_for_pfn() returns -1 skipping subsequent sysfs_remove_link() calls leaving node <-> memory block sysfs entries as is. Subsequent memory add operation hits BUG_ON() because of existing sysfs entries. [ 62.007176] NUMA: Unknown node for memory at 0x680000000, assuming node 0 [ 62.052517] ------------[ cut here ]------------ [ 62.053211] kernel BUG at mm/memory_hotplug.c:1143! [ 62.053868] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP [ 62.054589] Modules linked in: [ 62.054999] CPU: 19 PID: 3275 Comm: bash Not tainted 5.1.0-rc2-00004-g28cea40b2683 #41 [ 62.056274] Hardware name: linux,dummy-virt (DT) [ 62.057166] pstate: 40400005 (nZcv daif +PAN -UAO) [ 62.058083] pc : add_memory_resource+0x1cc/0x1d8 [ 62.058961] lr : add_memory_resource+0x10c/0x1d8 [ 62.059842] sp : ffff0000168b3ce0 [ 62.060477] x29: ffff0000168b3ce0 x28: ffff8005db546c00 [ 62.061501] x27: 0000000000000000 x26: 0000000000000000 [ 62.062509] x25: ffff0000111ef000 x24: ffff0000111ef5d0 [ 62.063520] x23: 0000000000000000 x22: 00000006bfffffff [ 62.064540] x21: 00000000ffffffef x20: 00000000006c0000 [ 62.065558] x19: 0000000000680000 x18: 0000000000000024 [ 62.066566] x17: 0000000000000000 x16: 0000000000000000 [ 62.067579] x15: ffffffffffffffff x14: ffff8005e412e890 [ 62.068588] x13: ffff8005d6b105d8 x12: 0000000000000000 [ 62.069610] x11: ffff8005d6b10490 x10: 0000000000000040 [ 62.070615] x9 : ffff8005e412e898 x8 : ffff8005e412e890 [ 62.071631] x7 : ffff8005d6b105d8 x6 : ffff8005db546c00 [ 62.072640] x5 : 0000000000000001 x4 : 0000000000000002 [ 62.073654] x3 : ffff8005d7049480 x2 : 0000000000000002 [ 62.074666] x1 : 0000000000000003 x0 : 00000000ffffffef [ 62.075685] Process bash (pid: 3275, stack limit = 0x00000000d754280f) [ 62.076930] Call trace: [ 62.077411] add_memory_resource+0x1cc/0x1d8 [ 62.078227] __add_memory+0x70/0xa8 [ 62.078901] probe_store+0xa4/0xc8 [ 62.079561] dev_attr_store+0x18/0x28 [ 62.080270] sysfs_kf_write+0x40/0x58 [ 62.080992] kernfs_fop_write+0xcc/0x1d8 [ 62.081744] __vfs_write+0x18/0x40 [ 62.082400] vfs_write+0xa4/0x1b0 [ 62.083037] ksys_write+0x5c/0xc0 [ 62.083681] __arm64_sys_write+0x18/0x20 [ 62.084432] el0_svc_handler+0x88/0x100 [ 62.085177] el0_svc+0x8/0xc Re-ordering memblock_[free|remove]() with arch_remove_memory() solves the problem on arm64 as pfn_valid() behaves correctly and returns positive as memblock for the address range still exists. arch_remove_memory() removes applicable memory sections from zone with __remove_pages() and tears down kernel linear mapping. Removing memblock regions afterwards is safe because there is no other memblock (bootmem) allocator user that late. So nobody is going to allocate from the removed range just to blow up later. Also nobody should be using the bootmem allocated range else we wouldn't allow to remove it. So reordering is indeed safe. Reviewed-by: David Hildenbrand Reviewed-by: Oscar Salvador Acked-by: Mark Rutland Acked-by: Michal Hocko Signed-off-by: Anshuman Khandual --- Changes in PATCH V5 - Rebased (from V5) - Rebased on linux-next (next-20190611) - s/__remove_memory/try_remove_memory in the subject line - s/arch_remove_memory/memblock_[free|remove] in the subject line - A small change in the commit message as re-order happens now for memblock remove functions not arch_remove_memory() mm/memory_hotplug.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index a88c5f334e5a..cfa5facf1b38 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1831,13 +1831,13 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size) /* remove memmap entry */ firmware_map_remove(start, start + size, "System RAM"); - memblock_free(start, size); - memblock_remove(start, size); /* remove memory block devices before removing memory */ remove_memory_block_devices(start, size); arch_remove_memory(nid, start, size, NULL); + memblock_free(start, size); + memblock_remove(start, size); __release_memory_resource(start, size); try_offline_node(nid);