From patchwork Tue Sep 20 10:43:48 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Richter X-Patchwork-Id: 9341467 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 0504B601C2 for ; Tue, 20 Sep 2016 10:45:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E6EA3294AC for ; Tue, 20 Sep 2016 10:45:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D66DF29DF9; Tue, 20 Sep 2016 10:45:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED, T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 405D7294AC for ; Tue, 20 Sep 2016 10:45:52 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.85_2 #1 (Red Hat Linux)) id 1bmIXE-0005U9-Pq; Tue, 20 Sep 2016 10:44:20 +0000 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]) by bombadil.infradead.org with esmtps (Exim 4.85_2 #1 (Red Hat Linux)) id 1bmIX8-0005Re-Kl for linux-arm-kernel@lists.infradead.org; Tue, 20 Sep 2016 10:44:17 +0000 Received: by mail-wm0-x242.google.com with SMTP id 133so2709986wmq.2 for ; Tue, 20 Sep 2016 03:43:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=BbRue7gpgHq2BuORAmsPrGTl3rMTtO9yaoPbNwEIkrg=; b=gfG3XKoiFsjMGp8MXQTV6amiHa1ebdMk5I3VIYRFkDemvmxturp7MgEgeqHK1NpUH2 GNfJrG8Qf4p2tdxRAI1S6xpS54c8FyPdvPyKr4Pqns2lhx8IBUPtkW30YvlftGoMxfjz mZwU0Gtgg5Mp3wIm/1Oz5eTUla4gDHPhUqPz4eEp2FVOBvSGtEksyzjt+0ixpY8y/EN9 d5suUMFosZaMbbJ/jCfB2D5VRXqVBteTYseoWIp0TG6MO6GvqHWdpvZqt38zgy0QFRyL 6fNSocW2X29s6oYTbUuRFXel73kbchADXv4zK00VhjTx36Y4XKyH92zzmy5AFfHs/QPX VAXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=BbRue7gpgHq2BuORAmsPrGTl3rMTtO9yaoPbNwEIkrg=; b=e0XZZ4n4SllVt/WjJpA74P7JlW+UCy/Ylb/W2mIh6oE+A5rqRBGzgEXXVIkcFc4PYR keA8ull+e5zl6OqqUu5eMPNHRA41nUysikKNR93IpPxdygkWKNPhHehqQf+yVw/GFYo3 LVpDbvurgnDAInoKGfbzqm265VOCVTFtLtcbO2oP6nPv5mArBSOwdRqBXgYKeDvfO348 SwGzHnfQ0bZM+vJmecmdrC7ZlTHNBsuiNaAQz8Yp9cUf6lqXAzwOCnZ8QpqRH+L8VNl8 6g9NGjwNGuwsLvO6ZHbOST6zNKJIhq/ZtBso/TwnrVyK/XcElVJDrjFg7WGTnqFLPX+J p07A== X-Gm-Message-State: AE9vXwMLS4JNFkD3p3v5gvdqqRHvXHxQtPBm7bVVf2hz5IeEO3gAuIOBxFg0E70WXGc4VQ== X-Received: by 10.28.105.146 with SMTP id z18mr2544586wmh.109.1474368231651; Tue, 20 Sep 2016 03:43:51 -0700 (PDT) Received: from rric.localdomain (x5ce55349.dyn.telefonica.de. [92.229.83.73]) by smtp.gmail.com with ESMTPSA id e5sm26728452wma.13.2016.09.20.03.43.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 20 Sep 2016 03:43:50 -0700 (PDT) Date: Tue, 20 Sep 2016 12:43:48 +0200 From: Robert Richter To: David Daney Subject: Re: [PATCH] arm64, numa: Add cpu_to_node() implementation. Message-ID: <20160920104348.GP25086@rric.localdomain> References: <1474310970-21264-1-git-send-email-ddaney.cavm@gmail.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <1474310970-21264-1-git-send-email-ddaney.cavm@gmail.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20160920_034414_929383_1A927AA2 X-CRM114-Status: GOOD ( 20.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: David Daney , Marc Zyngier , "Rafael J. Wysocki" , Will Deacon , linux-kernel@vger.kernel.org, Hanjun Guo , Ganapatrao Kulkarni , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP David, On 19.09.16 11:49:30, David Daney wrote: > Fix by supplying a cpu_to_node() implementation that returns correct > node mappings. > +int cpu_to_node(int cpu) > +{ > + int nid; > + > + /* > + * Return 0 for unknown mapping so that we report something > + * sensible if firmware doesn't supply a proper mapping. > + */ > + if (cpu < 0 || cpu >= NR_CPUS) > + return 0; > + > + nid = cpu_to_node_map[cpu]; > + if (nid == NUMA_NO_NODE) > + nid = 0; > + return nid; > +} > +EXPORT_SYMBOL(cpu_to_node); this implementation fixes the per-cpu workqueue initialization, but I don't think a cpu_to_node() implementation private to arm64 is the proper solution. Apart from better using generic code, the cpu_to_node() function is called in the kernel's fast path. I think your implementation is too expensive and also does not consider per-cpu data access for the lookup as the generic code does. Secondly, numa_off is not considered at all. Instead we need to make sure the set_*numa_node() functions are called earlier before secondary cpus are booted. My suggested change for that is this: I have tested the code and it properly sets up all per-cpu workqueues. Unfortunately either your nor my code does fix the BUG_ON() I see with the numa kernel: kernel BUG at mm/page_alloc.c:1848! See below for the core dump. It looks like this happens due to moving a mem block where first and last page are mapped to different numa nodes, thus, triggering the BUG_ON(). Continuing with my investigations... -Robert [ 9.674272] ------------[ cut here ]------------ [ 9.678881] kernel BUG at mm/page_alloc.c:1848! [ 9.683406] Internal error: Oops - BUG: 0 [#1] SMP [ 9.688190] Modules linked in: [ 9.691247] CPU: 77 PID: 1 Comm: swapper/0 Tainted: G W 4.8.0-rc5.vanilla5-00030-ga2b86cb3ce72 #38 [ 9.701322] Hardware name: www.cavium.com ThunderX CRB-2S/ThunderX CRB-2S, BIOS 0.3 Aug 24 2016 [ 9.710008] task: ffff800fe4561400 task.stack: ffff800ffbe0c000 [ 9.715939] PC is at move_freepages+0x160/0x168 [ 9.720460] LR is at move_freepages+0x160/0x168 [ 9.724979] pc : [] lr : [] pstate: 600000c5 [ 9.732362] sp : ffff800ffbe0f510 [ 9.735666] x29: ffff800ffbe0f510 x28: ffff7fe043f80020 [ 9.740975] x27: ffff7fe043f80000 x26: 000000000000000c [ 9.746283] x25: 000000000000000c x24: ffff810ffffaf0e0 [ 9.751591] x23: 0000000000000001 x22: 0000000000000000 [ 9.756898] x21: ffff7fe043ffffc0 x20: ffff810ffffaeb00 [ 9.762206] x19: ffff7fe043f80000 x18: 0000000000000010 [ 9.767513] x17: 0000000000000000 x16: 0000000100000000 [ 9.772821] x15: ffff000088f03f37 x14: 6e2c303d64696e2c [ 9.778128] x13: 3038336566666666 x12: 6630303866666666 [ 9.783436] x11: 3d656e6f7a203a64 x10: 0000000000000536 [ 9.788744] x9 : 0000000000000060 x8 : 3030626561666666 [ 9.794051] x7 : 6630313866666666 x6 : ffff000008f03f97 [ 9.799359] x5 : 0000000000000006 x4 : 000000000000000c [ 9.804667] x3 : 0000000000010000 x2 : 0000000000010000 [ 9.809975] x1 : ffff000008da7be0 x0 : 0000000000000050 [ 10.517213] Call trace: [ 10.519651] Exception stack(0xffff800ffbe0f340 to 0xffff800ffbe0f470) [ 10.526081] f340: ffff7fe043f80000 0001000000000000 ffff800ffbe0f510 ffff0000081ec7d0 [ 10.533900] f360: ffff000008f03988 0000000008da7bc8 ffff800ffbe0f410 ffff0000081275fc [ 10.541718] f380: ffff800ffbe0f470 ffff000008ac5a00 ffff7fe043ffffc0 0000000000000000 [ 10.549536] f3a0: 0000000000000001 ffff810ffffaf0e0 000000000000000c 000000000000000c [ 10.557355] f3c0: ffff7fe043f80000 ffff7fe043f80020 0000000000000030 0000000000000000 [ 10.565173] f3e0: 0000000000000050 ffff000008da7be0 0000000000010000 0000000000010000 [ 10.572991] f400: 000000000000000c 0000000000000006 ffff000008f03f97 6630313866666666 [ 10.580809] f420: 3030626561666666 0000000000000060 0000000000000536 3d656e6f7a203a64 [ 10.588628] f440: 6630303866666666 3038336566666666 6e2c303d64696e2c ffff000088f03f37 [ 10.596446] f460: 0000000100000000 0000000000000000 [ 10.601316] [] move_freepages+0x160/0x168 [ 10.606879] [] move_freepages_block+0xa8/0xb8 [ 10.612788] [] __rmqueue+0x610/0x670 [ 10.617918] [] get_page_from_freelist+0x3cc/0xb40 [ 10.624174] [] __alloc_pages_nodemask+0x12c/0xd40 [ 10.630438] [] alloc_page_interleave+0x60/0xb0 [ 10.636434] [] alloc_pages_current+0x108/0x168 [ 10.642430] [] __page_cache_alloc+0x104/0x140 [ 10.648339] [] pagecache_get_page+0x118/0x2e8 [ 10.654248] [] grab_cache_page_write_begin+0x48/0x68 [ 10.660769] [] simple_write_begin+0x40/0x150 [ 10.666591] [] generic_perform_write+0xb8/0x1a0 [ 10.672674] [] __generic_file_write_iter+0x178/0x1c8 [ 10.679191] [] generic_file_write_iter+0xcc/0x1c8 [ 10.685448] [] __vfs_write+0xcc/0x140 [ 10.690663] [] vfs_write+0xa8/0x1c0 [ 10.695704] [] SyS_write+0x54/0xb0 [ 10.700666] [] xwrite+0x34/0x7c [ 10.705359] [] do_copy+0x9c/0xf4 [ 10.710140] [] write_buffer+0x34/0x50 [ 10.715354] [] flush_buffer+0x48/0xb8 [ 10.720579] [] __gunzip+0x27c/0x324 [ 10.725620] [] gunzip+0x18/0x20 [ 10.730314] [] unpack_to_rootfs+0x168/0x280 [ 10.736049] [] populate_rootfs+0x70/0x138 [ 10.741615] [] do_one_initcall+0x44/0x138 [ 10.747179] [] kernel_init_freeable+0x1ac/0x24c [ 10.753267] [] kernel_init+0x20/0xf8 [ 10.758395] [] ret_from_fork+0x10/0x50 [ 10.763698] Code: 17fffff2 b00046c0 91280000 97ffd47d (d4210000) [ 10.769834] ---[ end trace 972d622f64fd69c0 ]--- diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index d93d43352504..952365c2f100 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -204,7 +204,6 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) static void smp_store_cpu_info(unsigned int cpuid) { store_cpu_topology(cpuid); - numa_store_cpu_info(cpuid); } /* @@ -719,6 +718,7 @@ void __init smp_prepare_cpus(unsigned int max_cpus) continue; set_cpu_present(cpu, true); + numa_store_cpu_info(cpu); } }