From patchwork Mon Jul 15 06:17:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 11043385 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 86DEB13BD for ; Mon, 15 Jul 2019 06:17:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 76E9222362 for ; Mon, 15 Jul 2019 06:17:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6858626D08; Mon, 15 Jul 2019 06:17:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B9BE722362 for ; Mon, 15 Jul 2019 06:17:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C3AE26B0006; Mon, 15 Jul 2019 02:17:33 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BC4066B0007; Mon, 15 Jul 2019 02:17:33 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A63D66B0008; Mon, 15 Jul 2019 02:17:33 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 591F46B0006 for ; Mon, 15 Jul 2019 02:17:33 -0400 (EDT) Received: by mail-ed1-f71.google.com with SMTP id i44so12911150eda.3 for ; Sun, 14 Jul 2019 23:17:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=fR0qheju1ReugcjNPqDZi/s2XyMltA//PhuA0FZEjmM=; b=nzrFLyu7x/RhMgrftkcKFMXFIahyQdQZhFR42u9ChOulp34PUzsdWvaEu+yyDzpUjM xKAE1HbVhybN45T2DapBOaWFLjTqq6b9/eSEKbPbsciWsDRJO7blSLdEd6xIdp8NBqtv 1xCDbitpRwUCp9lroStfA/xwrSCmTnymfbBXocTpy/58fwwwQNazg+42Btag8SPSAIdQ rJGKcQuIoOLgQiReVTd4ueAP5EzvPpgK5HxRvl/sGAQ25bCbcE40IlzyDrJsK+plstX+ JPhUCZBfTms1wFHeRFno1x+1+HOPHuBOTLbt8uNHHG0G3fsPUSwRYkG3FA6k3fhUAqnF RqCQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com X-Gm-Message-State: APjAAAXkUwffAYJROzz7zud4iayR3Vj2ND98rGAv9R/ICT3JSRnEOTLH PsXapEx3Fa8W+sMvEUjqouI5gGtw6+MFHL02GJTUN0RuRQj1eiTZLFO+Pw/Bi0vthoK0NL1SPTP xXOt0dLhV0CBTefzAqMVYDIfLSOq4z9W8xh0vUbp6f21TtMbpx7w96N+tR0bjrWkpxA== X-Received: by 2002:a50:8dcb:: with SMTP id s11mr21009378edh.144.1563171452929; Sun, 14 Jul 2019 23:17:32 -0700 (PDT) X-Google-Smtp-Source: APXvYqxT5Rrc9DqXIjvEJNHvVP9P8kbo9ue1AbobrbD8ftl6q7h+pXxHYExkcJ0vhfMpxmO0K+cQ X-Received: by 2002:a50:8dcb:: with SMTP id s11mr21009309edh.144.1563171451948; Sun, 14 Jul 2019 23:17:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1563171451; cv=none; d=google.com; s=arc-20160816; b=AtZsVv25pSEBldo050EIuWPmlQbuAUqFjyMc6fQkNXnhPT5ZSRx4IMOram3dolHJ14 QCsMw/fgNIuwjVHQt2LJyQjjzjERG4NoXkIdXdjQvf2lxfOUE5P1ZfR6PX13fCyViQwW hHTVbRFzQgQLhuQsamGChxJC1y1SAw3ILsxpbzRJ4PUkfH2e7WIluSMfGYLhN8Zg+0HZ beGo/bWYnWQ2u8gQ+q11khZqOqG+FHKyx01xrmCJuaC6I5YGrL51IaMW/kN2RmqVLLhG dx8ck0H5mLYvkmlorNQD9F7eXcEMIHTkw6HNmrn6jPmmB4hUI0OnEg5axpsB7epffumD 7IKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=fR0qheju1ReugcjNPqDZi/s2XyMltA//PhuA0FZEjmM=; b=Ew6kXUbPu5OYbc9HWO1WulA70y/L0K2WkdBsKjJGndS/0Rr6epvBEfWGf4K6F7ZzIM KprHLh5e+eaco4w6AESn3lUOcxsc40EX2xX9RD4SMWuISYjTks3kJk1GCewxs9f9OppU QdZ4U3o5X4Pap2u1I3jNVGZuUaTizm3WjqL/B+oCAtqI+GKx1U3XsHRkyOaTFdgralR3 bJJC8BeEv4s5au8YA0Wu6kiLbw3LTiYyqjcUVhp6sO2TfZhEqAjXhM7dTTsSiDs6qIWX vye9jN15mcm2TcRYLsLAV5lCO3Z7ufnaf22picAuTgKP6iksBn/Ehw/2ooEbC8pnrQqU yC8g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from foss.arm.com (foss.arm.com. [217.140.110.172]) by mx.google.com with ESMTP id l41si10040628eda.255.2019.07.14.23.17.31 for ; Sun, 14 Jul 2019 23:17:31 -0700 (PDT) Received-SPF: pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) client-ip=217.140.110.172; Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EF85C344; Sun, 14 Jul 2019 23:17:30 -0700 (PDT) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.40.143]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B031B3F71F; Sun, 14 Jul 2019 23:19:24 -0700 (PDT) From: Anshuman Khandual To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, akpm@linux-foundation.org, catalin.marinas@arm.com, will.deacon@arm.com Cc: mark.rutland@arm.com, mhocko@suse.com, ira.weiny@intel.com, david@redhat.com, cai@lca.pw, logang@deltatee.com, james.morse@arm.com, cpandya@codeaurora.org, arunks@codeaurora.org, dan.j.williams@intel.com, mgorman@techsingularity.net, osalvador@suse.de, ard.biesheuvel@arm.com, steve.capper@arm.com Subject: [PATCH V6 RESEND 1/3] mm/hotplug: Reorder memblock_[free|remove]() calls in try_remove_memory() Date: Mon, 15 Jul 2019 11:47:48 +0530 Message-Id: <1563171470-3117-2-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1563171470-3117-1-git-send-email-anshuman.khandual@arm.com> References: <1563171470-3117-1-git-send-email-anshuman.khandual@arm.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Memory hot remove uses get_nid_for_pfn() while tearing down linked sysfs entries between memory block and node. It first checks pfn validity with pfn_valid_within() before fetching nid. With CONFIG_HOLES_IN_ZONE config (arm64 has this enabled) pfn_valid_within() calls pfn_valid(). pfn_valid() is an arch implementation on arm64 (CONFIG_HAVE_ARCH_PFN_VALID) which scans all mapped memblock regions with memblock_is_map_memory(). This creates a problem in memory hot remove path which has already removed given memory range from memory block with memblock_[remove|free] before arriving at unregister_mem_sect_under_nodes(). Hence get_nid_for_pfn() returns -1 skipping subsequent sysfs_remove_link() calls leaving node <-> memory block sysfs entries as is. Subsequent memory add operation hits BUG_ON() because of existing sysfs entries. [ 62.007176] NUMA: Unknown node for memory at 0x680000000, assuming node 0 [ 62.052517] ------------[ cut here ]------------ [ 62.053211] kernel BUG at mm/memory_hotplug.c:1143! [ 62.053868] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP [ 62.054589] Modules linked in: [ 62.054999] CPU: 19 PID: 3275 Comm: bash Not tainted 5.1.0-rc2-00004-g28cea40b2683 #41 [ 62.056274] Hardware name: linux,dummy-virt (DT) [ 62.057166] pstate: 40400005 (nZcv daif +PAN -UAO) [ 62.058083] pc : add_memory_resource+0x1cc/0x1d8 [ 62.058961] lr : add_memory_resource+0x10c/0x1d8 [ 62.059842] sp : ffff0000168b3ce0 [ 62.060477] x29: ffff0000168b3ce0 x28: ffff8005db546c00 [ 62.061501] x27: 0000000000000000 x26: 0000000000000000 [ 62.062509] x25: ffff0000111ef000 x24: ffff0000111ef5d0 [ 62.063520] x23: 0000000000000000 x22: 00000006bfffffff [ 62.064540] x21: 00000000ffffffef x20: 00000000006c0000 [ 62.065558] x19: 0000000000680000 x18: 0000000000000024 [ 62.066566] x17: 0000000000000000 x16: 0000000000000000 [ 62.067579] x15: ffffffffffffffff x14: ffff8005e412e890 [ 62.068588] x13: ffff8005d6b105d8 x12: 0000000000000000 [ 62.069610] x11: ffff8005d6b10490 x10: 0000000000000040 [ 62.070615] x9 : ffff8005e412e898 x8 : ffff8005e412e890 [ 62.071631] x7 : ffff8005d6b105d8 x6 : ffff8005db546c00 [ 62.072640] x5 : 0000000000000001 x4 : 0000000000000002 [ 62.073654] x3 : ffff8005d7049480 x2 : 0000000000000002 [ 62.074666] x1 : 0000000000000003 x0 : 00000000ffffffef [ 62.075685] Process bash (pid: 3275, stack limit = 0x00000000d754280f) [ 62.076930] Call trace: [ 62.077411] add_memory_resource+0x1cc/0x1d8 [ 62.078227] __add_memory+0x70/0xa8 [ 62.078901] probe_store+0xa4/0xc8 [ 62.079561] dev_attr_store+0x18/0x28 [ 62.080270] sysfs_kf_write+0x40/0x58 [ 62.080992] kernfs_fop_write+0xcc/0x1d8 [ 62.081744] __vfs_write+0x18/0x40 [ 62.082400] vfs_write+0xa4/0x1b0 [ 62.083037] ksys_write+0x5c/0xc0 [ 62.083681] __arm64_sys_write+0x18/0x20 [ 62.084432] el0_svc_handler+0x88/0x100 [ 62.085177] el0_svc+0x8/0xc Re-ordering memblock_[free|remove]() with arch_remove_memory() solves the problem on arm64 as pfn_valid() behaves correctly and returns positive as memblock for the address range still exists. arch_remove_memory() removes applicable memory sections from zone with __remove_pages() and tears down kernel linear mapping. Removing memblock regions afterwards is safe because there is no other memblock (bootmem) allocator user that late. So nobody is going to allocate from the removed range just to blow up later. Also nobody should be using the bootmem allocated range else we wouldn't allow to remove it. So reordering is indeed safe. Reviewed-by: David Hildenbrand Reviewed-by: Oscar Salvador Acked-by: Mark Rutland Acked-by: Michal Hocko Signed-off-by: Anshuman Khandual --- mm/memory_hotplug.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index b9ba5b8..1635a89 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1772,13 +1772,13 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size) /* remove memmap entry */ firmware_map_remove(start, start + size, "System RAM"); - memblock_free(start, size); - memblock_remove(start, size); /* remove memory block devices before removing memory */ remove_memory_block_devices(start, size); arch_remove_memory(nid, start, size, NULL); + memblock_free(start, size); + memblock_remove(start, size); __release_memory_resource(start, size); try_offline_node(nid); From patchwork Mon Jul 15 06:17:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 11043389 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CD32114F6 for ; Mon, 15 Jul 2019 06:17:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BEB6822362 for ; Mon, 15 Jul 2019 06:17:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B250C26D08; Mon, 15 Jul 2019 06:17:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4842A22362 for ; Mon, 15 Jul 2019 06:17:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6460E6B0007; Mon, 15 Jul 2019 02:17:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5CEFC6B0008; Mon, 15 Jul 2019 02:17:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 497E16B000A; Mon, 15 Jul 2019 02:17:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id F1D036B0007 for ; Mon, 15 Jul 2019 02:17:38 -0400 (EDT) Received: by mail-ed1-f71.google.com with SMTP id b12so12896464eds.14 for ; Sun, 14 Jul 2019 23:17:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=GlOBlEcIXkjl+TiR1o4jafzqgo8fD8vSAH8C3FJFjlA=; b=ailewWR/Y8pJF7+2BkzApYFh/DMJfSanJGbLd7pjPKdyJ9SWk8W7jx9UB53F60oyDf iuodoUD8AgGelqch6pqkoucVOfuklAdxdY4wBj3gjVsDeR4oEvKnmQ4zmvkzXHQFyWb0 2IoUtbM4/nppLvtv7ID+Y/r8vuReFgK0uM/Cbl8ahhGw9iNUNch8b6SMnuH/w0RYHPkv QYBsAWjviHkKrW6Nhp9Ts2D+85+TbmZ/iv9Ea5xyPnjL2I3yZh2fCSYwXS/J8fyb8FT1 iIXRbju/rl1kmGLoahr65B9vItsjf+stZaZeCDNuifBRUwuu4ZhUNHRH2iPYd3GuaKLY u72w== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com X-Gm-Message-State: APjAAAVYHUBvCSF8WraWZJCtorucGLA3fnoAE/GEXOzCl/d5B1hO+pUm 9QAH5/p4Slc6xQu2DFT/YrQuuoTJdvo+xFZ4yOFrIR/zqr11uZKno2r1/iJ+C8XIXlIQEnPkO3H eJRSZ/1MzUc/XomS5y3dZcql+Y3fHPJ7e5EqNFgcl5P+TQax6zbC9r7xCfXbguoDCMw== X-Received: by 2002:a17:906:4e92:: with SMTP id v18mr19429697eju.57.1563171458558; Sun, 14 Jul 2019 23:17:38 -0700 (PDT) X-Google-Smtp-Source: APXvYqyRtEX/OGE5p3ySSv5ryBlTQoRYJftPFJmutFcabpFXE0eN5rKz8t70w+dwhroiZB9lNGGY X-Received: by 2002:a17:906:4e92:: with SMTP id v18mr19429660eju.57.1563171457801; Sun, 14 Jul 2019 23:17:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1563171457; cv=none; d=google.com; s=arc-20160816; b=EnSRHOgEPwHKxeYUsWbQenYsGaMdrZrar6AMdoN+IyUJGmq7SzEGviwzSf46jT2Uay 4u7JQIfUDyLSgYE0LbMlPbkGazQc0+Lf9lhekgnVQFanWVlvqny5Ux+Xfhsb3fF/o4iy m8uXcAVero/xPm9H/P/TCfPjSq4fx8vxXal1pw1l+G/LyajZ/+gJehG5AUZQ/hEz8/RA mXxEtANGLoqyzwfZa5KlMvAhkLCBeQvWSly+hfrFwD1De5+t3wuxFIrME7ncR3k5WKfM 5d+GDTBOZ+0ou41ykm152ukOwwk9uIIvZTfcZR+fXoqE5fFvezHAb+wQijQvFhhj5poJ Qasw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=GlOBlEcIXkjl+TiR1o4jafzqgo8fD8vSAH8C3FJFjlA=; b=TdUU1y8AerlO4ttSzMGx6i1jpQGP75jYz0y57YNQtnobSsWImttFiGkS/VgAmFwg+w pCvlf0hLJiP8LLDakPS0uDzTjonfF4zAzyDT3858mt5a4GbYA0b7cZZP0dqVZc9nel81 HWRJdUZg1ZmD/zNlgqtSsdSQKsfCG+dv5h3wNz7YCsd1luT8kugGuceVS2A0CkiLiVpd c3PAhWhrPjuw0UdBcEw5OaBJXg/kaDmCQAJhL/GRMb84JzGb828iZKmWHEzlS+00RvD3 +i8phgHkhgrMPHGJZvdrC6Z5vTEPzPGVpMpNjlPpQkkRwLHr4ZwaBNU2WIXbBx4FGoqV TeIA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from foss.arm.com (foss.arm.com. [217.140.110.172]) by mx.google.com with ESMTP id w21si6955929ejb.254.2019.07.14.23.17.37 for ; Sun, 14 Jul 2019 23:17:37 -0700 (PDT) Received-SPF: pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) client-ip=217.140.110.172; Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9BEA0337; Sun, 14 Jul 2019 23:17:36 -0700 (PDT) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.40.143]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6BE9F3F71F; Sun, 14 Jul 2019 23:19:30 -0700 (PDT) From: Anshuman Khandual To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, akpm@linux-foundation.org, catalin.marinas@arm.com, will.deacon@arm.com Cc: mark.rutland@arm.com, mhocko@suse.com, ira.weiny@intel.com, david@redhat.com, cai@lca.pw, logang@deltatee.com, james.morse@arm.com, cpandya@codeaurora.org, arunks@codeaurora.org, dan.j.williams@intel.com, mgorman@techsingularity.net, osalvador@suse.de, ard.biesheuvel@arm.com, steve.capper@arm.com Subject: [PATCH V6 RESEND 2/3] arm64/mm: Hold memory hotplug lock while walking for kernel page table dump Date: Mon, 15 Jul 2019 11:47:49 +0530 Message-Id: <1563171470-3117-3-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1563171470-3117-1-git-send-email-anshuman.khandual@arm.com> References: <1563171470-3117-1-git-send-email-anshuman.khandual@arm.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The arm64 page table dump code can race with concurrent modification of the kernel page tables. When a leaf entries are modified concurrently, the dump code may log stale or inconsistent information for a VA range, but this is otherwise not harmful. When intermediate levels of table are freed, the dump code will continue to use memory which has been freed and potentially reallocated for another purpose. In such cases, the dump code may dereference bogus addresses, leading to a number of potential problems. Intermediate levels of table may by freed during memory hot-remove, which will be enabled by a subsequent patch. To avoid racing with this, take the memory hotplug lock when walking the kernel page table. Acked-by: David Hildenbrand Acked-by: Mark Rutland Signed-off-by: Anshuman Khandual --- arch/arm64/mm/ptdump_debugfs.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/arm64/mm/ptdump_debugfs.c b/arch/arm64/mm/ptdump_debugfs.c index 064163f..b5eebc8 100644 --- a/arch/arm64/mm/ptdump_debugfs.c +++ b/arch/arm64/mm/ptdump_debugfs.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include #include #include @@ -7,7 +8,10 @@ static int ptdump_show(struct seq_file *m, void *v) { struct ptdump_info *info = m->private; + + get_online_mems(); ptdump_walk_pgd(m, info); + put_online_mems(); return 0; } DEFINE_SHOW_ATTRIBUTE(ptdump); From patchwork Mon Jul 15 06:17:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 11043391 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 40048112C for ; Mon, 15 Jul 2019 06:17:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2EFA322362 for ; Mon, 15 Jul 2019 06:17:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 21F6626D08; Mon, 15 Jul 2019 06:17:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2424F22362 for ; Mon, 15 Jul 2019 06:17:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 164776B0008; Mon, 15 Jul 2019 02:17:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0ED6D6B000A; Mon, 15 Jul 2019 02:17:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED1806B000C; Mon, 15 Jul 2019 02:17:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 997686B0008 for ; Mon, 15 Jul 2019 02:17:44 -0400 (EDT) Received: by mail-ed1-f70.google.com with SMTP id w25so12883183edu.11 for ; Sun, 14 Jul 2019 23:17:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=m/ch1EJnhlU3hP8sI64woDPhA16LL6MnmQw+Y8E+S98=; b=AqkMePOfEyRSQbZanrYwnSl80J/Tdrou+qmCgmQsg0WUYLcXr1FmmzaI0/pc9ViR9o 3x/xRLgaAb1IGUl3VToS3VSBf7eWycHJ6wqb1+5hqi8GxMT/IE+LL06EraaWwhwlbc4U fsnhf33zgrDWxSpfJWunsF4cwfDW/JEhgDZGfZ5OWrypp2oYImoofiZaasSpJG/lqEUy p7mkcUsLNO6PBDvfOlXqbtGHzZn99fDFR2zqhOmA6BCWhHhhln+2IQfdPs1oprzgUDwL j0oeKUQJnhCE5VJ5GwEWqNsv66LNH6mt3+ylDcYHvjf8ZYAR8LfncnjRJ+Hc1YSuWrlh rYZg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com X-Gm-Message-State: APjAAAXzUAeGTCJZjYtFNVZI0/01zHT6lnCnHBJRmVqb5QOor0E2hjCV 6JOhAzMr8WHbbeSHEkhtEzUmfHfp5n/L8+eyVjnamM4k230eRqS00nkhOOQT2FtY55+rHUOdnIN FvO/t1X6jRz28FdseOlZ3aczCoE2qlzNP1VhkV/h0L+iauaT6+KB1aoYKm+/dp6uIUQ== X-Received: by 2002:a50:89b4:: with SMTP id g49mr21300965edg.39.1563171464184; Sun, 14 Jul 2019 23:17:44 -0700 (PDT) X-Google-Smtp-Source: APXvYqxJ1eExHN0/dUeKEmDwCPXNSoJF2AXcoL0p5mUR7eIa7xMStSjixa2WVRQASSoACpXdGRcy X-Received: by 2002:a50:89b4:: with SMTP id g49mr21300908edg.39.1563171463221; Sun, 14 Jul 2019 23:17:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1563171463; cv=none; d=google.com; s=arc-20160816; b=ckhVgYtVt3YxmO2FIaNsWFpW17sIRfeZqA7aBTh5RUytj9pm1QmQd/sjYwKZXiI3La 0fOOknjZDK/+jACES5ikEyj1EVBgpYKXPGIW5PlT6/nwakKi1NQ8yL8t3PQz4wDRRswg +hiAUdpyGKtHt0CC1VXUNoX/PMveicstdNsWysZXaPTdio9GOEjnQ+HkLkvGY2fESwaJ lZio5TKjwIVwug+P4/YN2p4X5wGfyMHdZu6kGDBUaY9HHUAPtUCgKGusenXmsXtvVwbu tA5Yd5YKTIYESYB8Fa8kZjk3gnEoMbksBjSKvyLFhLNVG/Oyh5zB08VSYgYWJHfciyrH upmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=m/ch1EJnhlU3hP8sI64woDPhA16LL6MnmQw+Y8E+S98=; b=hDvYsM4w5UeOvbKP+jI+srPaD8eDq9+OJ4xgDaceaxwiYAriu+775tPkLOvAJJXnHI jImYI5mWjGP6KNtv1tSbJ/PBIoyJtLeo/lpzJpAY5Bahy9mTNOEa2rqju6pNCovutzcM if0E8UqaYy17jNYrNSazKZtpR+xL09748ueHhOjZw3Tr1HJD9be8Fh7sB5mvmP9EkXTn w8/Zez/m1Lgy7yBxELAK4dw6cHm0jA2gOgIenw1LOAF9d138ouJmD3FVqqXJZeZswITr rJxH0FzDCL31SmwZ+fAGR9uAtvioM16YsCjBKt5zRzHIaYB3sYRtPLVqrJdwb4qo4Lfu 5mCw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from foss.arm.com (foss.arm.com. [217.140.110.172]) by mx.google.com with ESMTP id t23si8882631eju.143.2019.07.14.23.17.42 for ; Sun, 14 Jul 2019 23:17:43 -0700 (PDT) Received-SPF: pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) client-ip=217.140.110.172; Authentication-Results: mx.google.com; spf=pass (google.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 46752337; Sun, 14 Jul 2019 23:17:42 -0700 (PDT) Received: from p8cg001049571a15.blr.arm.com (p8cg001049571a15.blr.arm.com [10.162.40.143]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 17ABF3F71F; Sun, 14 Jul 2019 23:19:35 -0700 (PDT) From: Anshuman Khandual To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, akpm@linux-foundation.org, catalin.marinas@arm.com, will.deacon@arm.com Cc: mark.rutland@arm.com, mhocko@suse.com, ira.weiny@intel.com, david@redhat.com, cai@lca.pw, logang@deltatee.com, james.morse@arm.com, cpandya@codeaurora.org, arunks@codeaurora.org, dan.j.williams@intel.com, mgorman@techsingularity.net, osalvador@suse.de, ard.biesheuvel@arm.com, steve.capper@arm.com Subject: [PATCH V6 RESEND 3/3] arm64/mm: Enable memory hot remove Date: Mon, 15 Jul 2019 11:47:50 +0530 Message-Id: <1563171470-3117-4-git-send-email-anshuman.khandual@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1563171470-3117-1-git-send-email-anshuman.khandual@arm.com> References: <1563171470-3117-1-git-send-email-anshuman.khandual@arm.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The arch code for hot-remove must tear down portions of the linear map and vmemmap corresponding to memory being removed. In both cases the page tables mapping these regions must be freed, and when sparse vmemmap is in use the memory backing the vmemmap must also be freed. This patch adds a new remove_pagetable() helper which can be used to tear down either region, and calls it from vmemmap_free() and ___remove_pgd_mapping(). The sparse_vmap argument determines whether the backing memory will be freed. remove_pagetable() makes two distinct passes over the kernel page table. In the first pass it unmaps, invalidates applicable TLB cache and frees backing memory if required (vmemmap) for each mapped leaf entry. In the second pass it looks for empty page table sections whose page table page can be unmapped, TLB invalidated and freed. While freeing intermediate level page table pages bail out if any of its entries are still valid. This can happen for partially filled kernel page table either from a previously attempted failed memory hot add or while removing an address range which does not span the entire page table page range. The vmemmap region may share levels of table with the vmalloc region. There can be conflicts between hot remove freeing page table pages with a concurrent vmalloc() walking the kernel page table. This conflict can not just be solved by taking the init_mm ptl because of existing locking scheme in vmalloc(). Hence unlike linear mapping, skip freeing page table pages while tearing down vmemmap mapping. While here update arch_add_memory() to handle __add_pages() failures by just unmapping recently added kernel linear mapping. Now enable memory hot remove on arm64 platforms by default with ARCH_ENABLE_MEMORY_HOTREMOVE. This implementation is overall inspired from kernel page table tear down procedure on X86 architecture. Acked-by: Steve Capper Acked-by: David Hildenbrand Signed-off-by: Anshuman Khandual --- arch/arm64/Kconfig | 3 + arch/arm64/mm/mmu.c | 290 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 284 insertions(+), 9 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7442edb..b94daec 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -273,6 +273,9 @@ config ZONE_DMA32 config ARCH_ENABLE_MEMORY_HOTPLUG def_bool y +config ARCH_ENABLE_MEMORY_HOTREMOVE + def_bool y + config SMP def_bool y diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 750a69d..282a4b2 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -722,6 +722,250 @@ int kern_addr_valid(unsigned long addr) return pfn_valid(pte_pfn(pte)); } + +#ifdef CONFIG_MEMORY_HOTPLUG +static void free_hotplug_page_range(struct page *page, size_t size) +{ + WARN_ON(!page || PageReserved(page)); + free_pages((unsigned long)page_address(page), get_order(size)); +} + +static void free_hotplug_pgtable_page(struct page *page) +{ + free_hotplug_page_range(page, PAGE_SIZE); +} + +static void free_pte_table(pmd_t *pmdp, unsigned long addr) +{ + struct page *page; + pte_t *ptep; + int i; + + ptep = pte_offset_kernel(pmdp, 0UL); + for (i = 0; i < PTRS_PER_PTE; i++) { + if (!pte_none(READ_ONCE(ptep[i]))) + return; + } + + page = pmd_page(READ_ONCE(*pmdp)); + pmd_clear(pmdp); + __flush_tlb_kernel_pgtable(addr); + free_hotplug_pgtable_page(page); +} + +static void free_pmd_table(pud_t *pudp, unsigned long addr) +{ + struct page *page; + pmd_t *pmdp; + int i; + + if (CONFIG_PGTABLE_LEVELS <= 2) + return; + + pmdp = pmd_offset(pudp, 0UL); + for (i = 0; i < PTRS_PER_PMD; i++) { + if (!pmd_none(READ_ONCE(pmdp[i]))) + return; + } + + page = pud_page(READ_ONCE(*pudp)); + pud_clear(pudp); + __flush_tlb_kernel_pgtable(addr); + free_hotplug_pgtable_page(page); +} + +static void free_pud_table(pgd_t *pgdp, unsigned long addr) +{ + struct page *page; + pud_t *pudp; + int i; + + if (CONFIG_PGTABLE_LEVELS <= 3) + return; + + pudp = pud_offset(pgdp, 0UL); + for (i = 0; i < PTRS_PER_PUD; i++) { + if (!pud_none(READ_ONCE(pudp[i]))) + return; + } + + page = pgd_page(READ_ONCE(*pgdp)); + pgd_clear(pgdp); + __flush_tlb_kernel_pgtable(addr); + free_hotplug_pgtable_page(page); +} + +static void unmap_hotplug_pte_range(pmd_t *pmdp, unsigned long addr, + unsigned long end, bool sparse_vmap) +{ + struct page *page; + pte_t *ptep, pte; + + do { + ptep = pte_offset_kernel(pmdp, addr); + pte = READ_ONCE(*ptep); + if (pte_none(pte)) + continue; + + WARN_ON(!pte_present(pte)); + page = sparse_vmap ? pte_page(pte) : NULL; + pte_clear(&init_mm, addr, ptep); + flush_tlb_kernel_range(addr, addr + PAGE_SIZE); + if (sparse_vmap) + free_hotplug_page_range(page, PAGE_SIZE); + } while (addr += PAGE_SIZE, addr < end); +} + +static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr, + unsigned long end, bool sparse_vmap) +{ + unsigned long next; + struct page *page; + pmd_t *pmdp, pmd; + + do { + next = pmd_addr_end(addr, end); + pmdp = pmd_offset(pudp, addr); + pmd = READ_ONCE(*pmdp); + if (pmd_none(pmd)) + continue; + + WARN_ON(!pmd_present(pmd)); + if (pmd_sect(pmd)) { + page = sparse_vmap ? pmd_page(pmd) : NULL; + pmd_clear(pmdp); + flush_tlb_kernel_range(addr, next); + if (sparse_vmap) + free_hotplug_page_range(page, PMD_SIZE); + continue; + } + WARN_ON(!pmd_table(pmd)); + unmap_hotplug_pte_range(pmdp, addr, next, sparse_vmap); + } while (addr = next, addr < end); +} + +static void unmap_hotplug_pud_range(pgd_t *pgdp, unsigned long addr, + unsigned long end, bool sparse_vmap) +{ + unsigned long next; + struct page *page; + pud_t *pudp, pud; + + do { + next = pud_addr_end(addr, end); + pudp = pud_offset(pgdp, addr); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) + continue; + + WARN_ON(!pud_present(pud)); + if (pud_sect(pud)) { + page = sparse_vmap ? pud_page(pud) : NULL; + pud_clear(pudp); + flush_tlb_kernel_range(addr, next); + if (sparse_vmap) + free_hotplug_page_range(page, PUD_SIZE); + continue; + } + WARN_ON(!pud_table(pud)); + unmap_hotplug_pmd_range(pudp, addr, next, sparse_vmap); + } while (addr = next, addr < end); +} + +static void unmap_hotplug_range(unsigned long addr, unsigned long end, + bool sparse_vmap) +{ + unsigned long next; + pgd_t *pgdp, pgd; + + do { + next = pgd_addr_end(addr, end); + pgdp = pgd_offset_k(addr); + pgd = READ_ONCE(*pgdp); + if (pgd_none(pgd)) + continue; + + WARN_ON(!pgd_present(pgd)); + unmap_hotplug_pud_range(pgdp, addr, next, sparse_vmap); + } while (addr = next, addr < end); +} + +static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, + unsigned long end) +{ + pte_t *ptep, pte; + + do { + ptep = pte_offset_kernel(pmdp, addr); + pte = READ_ONCE(*ptep); + WARN_ON(!pte_none(pte)); + } while (addr += PAGE_SIZE, addr < end); +} + +static void free_empty_pmd_table(pud_t *pudp, unsigned long addr, + unsigned long end) +{ + unsigned long next; + pmd_t *pmdp, pmd; + + do { + next = pmd_addr_end(addr, end); + pmdp = pmd_offset(pudp, addr); + pmd = READ_ONCE(*pmdp); + if (pmd_none(pmd)) + continue; + + WARN_ON(!pmd_present(pmd) || !pmd_table(pmd) || pmd_sect(pmd)); + free_empty_pte_table(pmdp, addr, next); + free_pte_table(pmdp, addr); + } while (addr = next, addr < end); +} + +static void free_empty_pud_table(pgd_t *pgdp, unsigned long addr, + unsigned long end) +{ + unsigned long next; + pud_t *pudp, pud; + + do { + next = pud_addr_end(addr, end); + pudp = pud_offset(pgdp, addr); + pud = READ_ONCE(*pudp); + if (pud_none(pud)) + continue; + + WARN_ON(!pud_present(pud) || !pud_table(pud) || pud_sect(pud)); + free_empty_pmd_table(pudp, addr, next); + free_pmd_table(pudp, addr); + } while (addr = next, addr < end); +} + +static void free_empty_tables(unsigned long addr, unsigned long end) +{ + unsigned long next; + pgd_t *pgdp, pgd; + + do { + next = pgd_addr_end(addr, end); + pgdp = pgd_offset_k(addr); + pgd = READ_ONCE(*pgdp); + if (pgd_none(pgd)) + continue; + + WARN_ON(!pgd_present(pgd)); + free_empty_pud_table(pgdp, addr, next); + free_pud_table(pgdp, addr); + } while (addr = next, addr < end); +} + +static void remove_pagetable(unsigned long start, unsigned long end, + bool sparse_vmap) +{ + unmap_hotplug_range(start, end, sparse_vmap); + free_empty_tables(start, end); +} +#endif + #ifdef CONFIG_SPARSEMEM_VMEMMAP #if !ARM64_SWAPPER_USES_SECTION_MAPS int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, @@ -769,6 +1013,27 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, void vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap) { +#ifdef CONFIG_MEMORY_HOTPLUG + /* + * FIXME: We should have called remove_pagetable(start, end, true). + * vmemmap and vmalloc virtual range might share intermediate kernel + * page table entries. Removing vmemmap range page table pages here + * can potentially conflict with a concurrent vmalloc() allocation. + * + * This is primarily because vmalloc() does not take init_mm ptl for + * the entire page table walk and it's modification. Instead it just + * takes the lock while allocating and installing page table pages + * via [p4d|pud|pmd|pte]_alloc(). A concurrently vanishing page table + * entry via memory hot remove can cause vmalloc() kernel page table + * walk pointers to be invalid on the fly which can cause corruption + * or worst, a crash. + * + * To avoid this problem, lets not free empty page table pages for + * given vmemmap range being hot-removed. Just unmap and free the + * range instead. + */ + unmap_hotplug_range(start, end, true); +#endif } #endif /* CONFIG_SPARSEMEM_VMEMMAP */ @@ -1060,10 +1325,18 @@ int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) } #ifdef CONFIG_MEMORY_HOTPLUG +static void __remove_pgd_mapping(pgd_t *pgdir, unsigned long start, u64 size) +{ + unsigned long end = start + size; + + WARN_ON(pgdir != init_mm.pgd); + remove_pagetable(start, end, false); +} + int arch_add_memory(int nid, u64 start, u64 size, struct mhp_restrictions *restrictions) { - int flags = 0; + int ret, flags = 0; if (rodata_full || debug_pagealloc_enabled()) flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; @@ -1071,9 +1344,14 @@ int arch_add_memory(int nid, u64 start, u64 size, __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), size, PAGE_KERNEL, __pgd_pgtable_alloc, flags); - return __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, + ret = __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, restrictions); + if (ret) + __remove_pgd_mapping(swapper_pg_dir, + __phys_to_virt(start), size); + return ret; } + void arch_remove_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap) { @@ -1081,14 +1359,8 @@ void arch_remove_memory(int nid, u64 start, u64 size, unsigned long nr_pages = size >> PAGE_SHIFT; struct zone *zone; - /* - * FIXME: Cleanup page tables (also in arch_add_memory() in case - * adding fails). Until then, this function should only be used - * during memory hotplug (adding memory), not for memory - * unplug. ARCH_ENABLE_MEMORY_HOTREMOVE must not be - * unlocked yet. - */ zone = page_zone(pfn_to_page(start_pfn)); __remove_pages(zone, start_pfn, nr_pages, altmap); + __remove_pgd_mapping(swapper_pg_dir, __phys_to_virt(start), size); } #endif