From patchwork Tue May 29 21:52:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeremy Linton X-Patchwork-Id: 10437177 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D6937601C7 for ; Tue, 29 May 2018 21:52:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C4E34288F3 for ; Tue, 29 May 2018 21:52:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B7EBC28964; Tue, 29 May 2018 21:52:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4816C288F3 for ; Tue, 29 May 2018 21:52:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=c46PQnndL4av08tVkZvF6JAK18LHwrk1XlC41rPy6Yg=; b=G39DZ6NZov8eTT9eXNLrscza7 gZn1leYs/1mfJsqp5W2C/yskLp6umsRya/uY5Dt3QJV5lFqgNzQqLvHTvc04gxu/0F/+OSFR3nkeZ mohPLwKVBkljSI/GsJYHxvZj7ViG3HwdabA8LNsT5FlZ2k/IkV0lXLzLVDoZgfCaKDFHcPRfocn0Q TWDJr8bDOJVOqxyXKo4yWkhS7dvT8A0/FFZ0XfnqzW8OcPLRTq50QgnfsHmKRg4e1He0NgtsQifc4 m3Je6tJcy69A/Oop5pm6DYgGXTicoF2rJS+bvVhrchxbys8Y88jkP6WXWMRcFt8MAnblRmmu7A7+M 1oS+TKWVQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fNmXV-0004Wv-PR; Tue, 29 May 2018 21:52:21 +0000 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fNmXR-0004UU-S6; Tue, 29 May 2018 21:52:19 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CF98080D; Tue, 29 May 2018 14:52:06 -0700 (PDT) Received: from [10.119.48.76] (unknown [10.119.48.76]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 7FA973F557; Tue, 29 May 2018 14:52:05 -0700 (PDT) Subject: Re: [PATCH v9 00/12] Support PPTT for ARM64 To: Geert Uytterhoeven , Will Deacon References: <20180511235807.30834-1-jeremy.linton@arm.com> <20180517170523.h7tuvbzdfluuidcz@armageddon.cambridge.arm.com> <09fb3fe7-d703-43f1-74f7-f8cb5ff1f67a@arm.com> <551905a6-eaa8-97df-06ec-1ceedfbc164f@arm.com> <20180529150823.GD17159@arm.com> From: Jeremy Linton Message-ID: Date: Tue, 29 May 2018 16:52:04 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180529_145217_945428_99055FE8 X-CRM114-Status: GOOD ( 26.68 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , austinwc@codeaurora.org, tnowicki@caviumnetworks.com, Catalin Marinas , Palmer Dabbelt , linux-riscv@lists.infradead.org, wangxiongfeng2@huawei.com, vkilari@codeaurora.org, Lorenzo Pieralisi , jhugo@codeaurora.org, Morten.Rasmussen@arm.com, ACPI Devel Maling List , Len Brown , John Garry , Al Stone , Linux ARM , Ard Biesheuvel , Greg KH , "Rafael J. Wysocki" , Linux Kernel Mailing List , Linux-Renesas , Hanjun Guo , Sudeep Holla , Dietmar Eggemann Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Hi, On 05/29/2018 10:51 AM, Geert Uytterhoeven wrote: > Hi Will, > > On Tue, May 29, 2018 at 5:08 PM, Will Deacon wrote: >> On Tue, May 29, 2018 at 02:18:40PM +0100, Sudeep Holla wrote: >>> On 29/05/18 12:56, Geert Uytterhoeven wrote: >>>> On Tue, May 29, 2018 at 1:14 PM, Sudeep Holla wrote: >>>>> On 29/05/18 11:48, Geert Uytterhoeven wrote: >>>>>> System supend still works fine on systems with big cores only: >>>>>> >>>>>> R-Car H3 ES1.0 (4xCA57 (4xCA53 disabled in firmware)) >>>>>> R-Car M3-N (2xCA57) >>>>>> >>>>>> Reverting this commit fixes the issue for me. >>>>> >>>>> I can't find anything that relates to system suspend in these patches >>>>> unless they are messing with something during CPU hot plug-in back >>>>> during resume. >>>> >>>> It's only the last patch that introduces the breakage. >>>> >>> >>> As specified in the commit log, it won't change any behavior for DT >>> systems if it's non-NUMA or single node system. So I am still wondering >>> what could trigger this regression. >> >> I wonder if we're somehow giving an uninitialised/invalid NUMA configuration >> to the scheduler, although I can't see how this would happen. >> >> Geert -- if you enable CONFIG_DEBUG_PER_CPU_MAPS=y and apply the diff below >> do you see anything shouting in dmesg? > > Thanks, but unfortunately it doesn't help. > I added some debug code to print cpumask, but so far I don't see anything > suspicious. I suspect most of the problem is related to the node mask changing at unexpected times (particularly cores being removed from the mask). Once I understand that more, there may be a simpler patch. OTOH, I've been testing with this, and with it, I can't seem to duplicate the problem with CONFIG_NUMA disabled I found. diff --git a/arch/arm64/include/asm/topology.h b/arch/arm64/include/asm/topology.h index df48212f767b..7450ef5ed733 100644 --- a/arch/arm64/include/asm/topology.h +++ b/arch/arm64/include/asm/topology.h @@ -12,6 +12,7 @@ struct cpu_topology { cpumask_t thread_sibling; cpumask_t core_sibling; cpumask_t llc_siblings; + cpumask_t node_siblings; /* maintain a stable node sibling list */ }; extern struct cpu_topology cpu_topology[NR_CPUS]; diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index f3e2e3aec0b0..f4eb80852d78 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -677,8 +677,9 @@ void __init smp_prepare_cpus(unsigned int max_cpus) init_cpu_topology(); this_cpu = smp_processor_id(); - store_cpu_topology(this_cpu); numa_store_cpu_info(this_cpu); + store_cpu_topology(this_cpu); + /* * If UP is mandated by "nosmp" (which implies "maxcpus=0"), don't set diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index 7415c166281f..6819c764537d 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -215,7 +215,7 @@ EXPORT_SYMBOL_GPL(cpu_topology); const struct cpumask *cpu_coregroup_mask(int cpu) { - const cpumask_t *core_mask = cpumask_of_node(cpu_to_node(cpu)); + const cpumask_t *core_mask = &cpu_topology[cpu].node_siblings; /* Find the smaller of NUMA, core or LLC siblings */ if (cpumask_subset(&cpu_topology[cpu].core_sibling, core_mask)) { @@ -233,12 +233,16 @@ const struct cpumask *cpu_coregroup_mask(int cpu) static void update_siblings_masks(unsigned int cpuid) { struct cpu_topology *cpu_topo, *cpuid_topo = &cpu_topology[cpuid]; + int node = cpu_to_node(cpuid); int cpu; /* update core and thread sibling masks */ for_each_possible_cpu(cpu) { cpu_topo = &cpu_topology[cpu]; + if (cpu_to_node(cpu) == node) + cpumask_set_cpu(cpu, &cpu_topo->node_siblings); + if (cpuid_topo->llc_id == cpu_topo->llc_id) cpumask_set_cpu(cpu, &cpuid_topo->llc_siblings); @@ -311,6 +315,9 @@ static void __init reset_cpu_topology(void) cpumask_clear(&cpu_topo->llc_siblings); cpumask_set_cpu(cpu, &cpu_topo->llc_siblings); + cpumask_clear(&cpu_topo->node_siblings); + cpumask_set_cpu(cpu, &cpu_topo->node_siblings); + cpumask_clear(&cpu_topo->core_sibling); cpumask_set_cpu(cpu, &cpu_topo->core_sibling); cpumask_clear(&cpu_topo->thread_sibling);