From patchwork Fri Apr 7 16:57:14 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9669953 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E45A8602A0 for ; Fri, 7 Apr 2017 16:59:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D4598285C5 for ; Fri, 7 Apr 2017 16:59:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C90F328617; Fri, 7 Apr 2017 16:59:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3E117285C5 for ; Fri, 7 Apr 2017 16:59:21 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cwXCM-0001q4-TX; Fri, 07 Apr 2017 16:57:22 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cwXCK-0001on-Tq for xen-devel@lists.xenproject.org; Fri, 07 Apr 2017 16:57:21 +0000 Received: from [85.158.139.211] by server-10.bemta-5.messagelabs.com id A5/27-02186-0F4C7E85; Fri, 07 Apr 2017 16:57:20 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrDIsWRWlGSWpSXmKPExsXiVRvkpPvmyPM Ig/fzhS2+b5nM5MDocfjDFZYAxijWzLyk/IoE1ow/LT8YC46rVXyd1szawLhWpouRi0NIYBqj xJurf5hBHBaBNawSWxdNZOli5OSQELjEKrH0tgCEHSfxdPIsRgi7SuLzxIesILaQgIrEze2rm CAm/WKUaJqwBaxZWEBP4sjRH+wQdoDEpsaLbCA2m4CBxJsde8GaRQSUJO6tmgzUzMHBLBAusb qTGyTMIqAq8bOtixnE5hXwlnjzdxLYSE4BH4lnp2YxQez1llh3fg7YeFEBOYmVl1tYIeoFJU7 OfMICMVJTYv0ufZAws4C8xPa3c5gnMIrMQlI1C6FqFpKqBYzMqxjVi1OLylKLdM30kooy0zNK chMzc3QNDUz1clOLixPTU3MSk4r1kvNzNzECQ58BCHYwTm1wPsQoycGkJMqr4PMkQogvKT+lM iOxOCO+qDQntfgQowYHh8CEs3OnM0mx5OXnpSpJ8M44/DxCSLAoNT21Ii0zBxidMKUSHDxKIr w3QdK8xQWJucWZ6RCpU4y6HI9W/njPJAQ2Q0qctwKkSACkKKM0D24ELFFcYpSVEuZlBDpQiKc gtSg3swRV/hWjOAejkjDvIpApPJl5JXCbXgEdwQR0hM+tpyBHlCQipKQaGLX237h091DZiqri bQ6K92d3Nu5OV5dfs+3X6a7ThyoPC/QnV4q+c2M/kyfPYbPI/MbxA8V8FeVxLBvO/z//bOF22 e/XedWPBOls/vDwINPrK+ffTzNeflxpQ956g6rtXzQT4xXXPAv51/dAN4/JqEus89zc+TPU1h oxMf3+vUcs/mLlzeu/2I8qsRRnJBpqMRcVJwIAKC2KBQ8DAAA= X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-12.tower-206.messagelabs.com!1491584236!56186092!1 X-Originating-IP: [74.125.82.66] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.4.12; banners=-,-,- X-VirusChecked: Checked Received: (qmail 32100 invoked from network); 7 Apr 2017 16:57:16 -0000 Received: from mail-wm0-f66.google.com (HELO mail-wm0-f66.google.com) (74.125.82.66) by server-12.tower-206.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 7 Apr 2017 16:57:16 -0000 Received: by mail-wm0-f66.google.com with SMTP id d79so646682wmi.2 for ; Fri, 07 Apr 2017 09:57:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=gN8O3UJVI14s4OPLGgPGFDhcDn8PKFWI28py/I8zYgU=; b=Qkvb/vOJfNkHUT1pmObxm8twCamSSkekKhHoDGveiivFDY6iNKjoHmWOhoxYpwSOXY by2xAd3dUIpJNkWvv7Ce2dqJUoa5dfE4TktDgwxU1sd2qk0X0ZlTpB1KJr8FBst0JmNB wAjEVUeoIzLLqsKOVTa4/OsX1dhiRofcRBFDlQLp762t8EoPyl3b44lory5bSZsqcKp7 ISvxI5jIb+YSm6HT+0hGSCh3CuQWvA6s8/QLUDL4n0xq8IuyFRLFNBTLoNEX1FPE66Xk 6qIIYNwAU7TfST//j369Es/uVEXU1UisrkTSj9mVJQsMKsiD1v3dZv2AemrNA9g2Uf7x RRmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=gN8O3UJVI14s4OPLGgPGFDhcDn8PKFWI28py/I8zYgU=; b=URIj35It0aDBxM511OkQDzWYarLFP9Q025zXdy/XxgH0o5etEpJMzX0ui/fV9BMR6g QTsdcmvcAc2EWDnCyo9PZBcsbtvArYD5y2zUTGr7K8Li2NhbgcTG+5m/QpLv/37LbVC7 mW7Zty1W5w9aaxqReRya81NnDU4fU3yR+OUFL87g4lRBl2mKo6YPqLX+W4xAzwnVBU0P YjvfiztpTqnNUAM3RnoTx3QSuOyVlkZ0UoPhG9T70YnhIxtmPMF+G7lIzgCG+DyvMMRK MOUdLSQocE3Kgc4m3EHvfI2zXEBg5/DCtrGGCCnTtWhawXZ6FuOPZx1Jkm0GRdqEk6A5 xvfg== X-Gm-Message-State: AN3rC/5rKPWExEOMG/nPZOriAPj/Wr/b/t1tKMa8/68rhIR2KgWSiGIA 13jgRoPGPbWrtw== X-Received: by 10.28.153.140 with SMTP id b134mr290281wme.124.1491584235882; Fri, 07 Apr 2017 09:57:15 -0700 (PDT) Received: from Solace.fritz.box ([80.66.223.217]) by smtp.gmail.com with ESMTPSA id b82sm30902752wmh.4.2017.04.07.09.57.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 07 Apr 2017 09:57:15 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 07 Apr 2017 18:57:14 +0200 Message-ID: <149158423409.32558.7946100494361201979.stgit@Solace.fritz.box> In-Reply-To: <149158365254.32558.7658440932477066488.stgit@Solace.fritz.box> References: <149158365254.32558.7658440932477066488.stgit@Solace.fritz.box> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: Anshul Makkar , George Dunlap Subject: [Xen-devel] [PATCH v3 7/7] xen: credit1: treat pCPUs more evenly during balancing. X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Right now, we use cpumask_first() for going through the bus pCPUs in csched_load_balance(). This means not all pCPUs have equal chances of seeing their pending work stolen. It also means there is more runqueue lock pressure on lower ID pCPUs. To avoid all this, let's record and remember, for each NUMA node, from what pCPU we have stolen for last, and start from that the following time. Signed-off-by: Dario Faggioli Acked-by: George Dunlap --- Cc: Anshul Makkar --- xen/common/sched_credit.c | 37 +++++++++++++++++++++++++++++++++---- 1 file changed, 33 insertions(+), 4 deletions(-) diff --git a/xen/common/sched_credit.c b/xen/common/sched_credit.c index a0ad167..93658dc 100644 --- a/xen/common/sched_credit.c +++ b/xen/common/sched_credit.c @@ -229,6 +229,7 @@ struct csched_private { uint32_t credit; int credit_balance; uint32_t runq_sort; + uint32_t *balance_bias; unsigned ratelimit_us; /* Period of master and tick in milliseconds */ unsigned tslice_ms, tick_period_us, ticks_per_tslice; @@ -560,6 +561,7 @@ csched_deinit_pdata(const struct scheduler *ops, void *pcpu, int cpu) { struct csched_private *prv = CSCHED_PRIV(ops); struct csched_pcpu *spc = pcpu; + unsigned int node = cpu_to_node(cpu); unsigned long flags; /* @@ -583,6 +585,12 @@ csched_deinit_pdata(const struct scheduler *ops, void *pcpu, int cpu) prv->master = cpumask_first(prv->cpus); migrate_timer(&prv->master_ticker, prv->master); } + if ( prv->balance_bias[node] == cpu ) + { + cpumask_and(cpumask_scratch, prv->cpus, &node_to_cpumask(node)); + if ( !cpumask_empty(cpumask_scratch) ) + prv->balance_bias[node] = cpumask_first(cpumask_scratch); + } kill_timer(&spc->ticker); if ( prv->ncpus == 0 ) kill_timer(&prv->master_ticker); @@ -622,6 +630,10 @@ init_pdata(struct csched_private *prv, struct csched_pcpu *spc, int cpu) NOW() + MILLISECS(prv->tslice_ms)); } + cpumask_and(cpumask_scratch, prv->cpus, &node_to_cpumask(cpu_to_node(cpu))); + if ( cpumask_weight(cpumask_scratch) == 1 ) + prv->balance_bias[cpu_to_node(cpu)] = cpu; + init_timer(&spc->ticker, csched_tick, (void *)(unsigned long)cpu, cpu); set_timer(&spc->ticker, NOW() + MICROSECS(prv->tick_period_us) ); @@ -1735,7 +1747,7 @@ csched_load_balance(struct csched_private *prv, int cpu, struct csched_vcpu *speer; cpumask_t workers; cpumask_t *online; - int peer_cpu, peer_node, bstep; + int peer_cpu, first_cpu, peer_node, bstep; int node = cpu_to_node(cpu); BUG_ON( cpu != snext->vcpu->processor ); @@ -1779,9 +1791,10 @@ csched_load_balance(struct csched_private *prv, int cpu, cpumask_and(&workers, &workers, &node_to_cpumask(peer_node)); __cpumask_clear_cpu(cpu, &workers); - peer_cpu = cpumask_first(&workers); - if ( peer_cpu >= nr_cpu_ids ) + first_cpu = cpumask_cycle(prv->balance_bias[peer_node], &workers); + if ( first_cpu >= nr_cpu_ids ) goto next_node; + peer_cpu = first_cpu; do { spinlock_t *lock; @@ -1845,13 +1858,19 @@ csched_load_balance(struct csched_private *prv, int cpu, if ( speer != NULL ) { *stolen = 1; + /* + * Next time we'll look for work to steal on this node, we + * will start from the next pCPU, with respect to this one, + * so we don't risk stealing always from the same ones. + */ + prv->balance_bias[peer_node] = peer_cpu; return speer; } next_cpu: peer_cpu = cpumask_cycle(peer_cpu, &workers); - } while( peer_cpu != cpumask_first(&workers) ); + } while( peer_cpu != first_cpu ); next_node: peer_node = cycle_node(peer_node, node_online_map); @@ -2204,10 +2223,19 @@ csched_init(struct scheduler *ops) prv = xzalloc(struct csched_private); if ( prv == NULL ) return -ENOMEM; + + prv->balance_bias = xzalloc_array(uint32_t, MAX_NUMNODES); + if ( prv->balance_bias == NULL ) + { + xfree(prv); + return -ENOMEM; + } + if ( !zalloc_cpumask_var(&prv->cpus) || !zalloc_cpumask_var(&prv->idlers) ) { free_cpumask_var(prv->cpus); + xfree(prv->balance_bias); xfree(prv); return -ENOMEM; } @@ -2253,6 +2281,7 @@ csched_deinit(struct scheduler *ops) ops->sched_data = NULL; free_cpumask_var(prv->cpus); free_cpumask_var(prv->idlers); + xfree(prv->balance_bias); xfree(prv); } }