From patchwork Fri Mar 18 19:05:30 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 8623341 Return-Path: X-Original-To: patchwork-xen-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 79A22C0553 for ; Fri, 18 Mar 2016 19:07:56 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 75EEE20204 for ; Fri, 18 Mar 2016 19:07:55 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6FA3420123 for ; Fri, 18 Mar 2016 19:07:54 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1agziJ-0000eF-JB; Fri, 18 Mar 2016 19:05:35 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1agziJ-0000dd-3K for xen-devel@lists.xenproject.org; Fri, 18 Mar 2016 19:05:35 +0000 Received: from [85.158.137.68] by server-2.bemta-3.messagelabs.com id 23/F9-03293-E715CE65; Fri, 18 Mar 2016 19:05:34 +0000 X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-15.tower-31.messagelabs.com!1458327933!29792352!1 X-Originating-IP: [74.125.82.66] X-SpamReason: No, hits=0.2 required=7.0 tests=RCVD_ILLEGAL_IP X-StarScan-Received: X-StarScan-Version: 8.11; banners=-,-,- X-VirusChecked: Checked Received: (qmail 56675 invoked from network); 18 Mar 2016 19:05:33 -0000 Received: from mail-wm0-f66.google.com (HELO mail-wm0-f66.google.com) (74.125.82.66) by server-15.tower-31.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 18 Mar 2016 19:05:33 -0000 Received: by mail-wm0-f66.google.com with SMTP id l68so8157701wml.3 for ; Fri, 18 Mar 2016 12:05:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=t86sqvr+nYthFIPNZCGMA6aHii3ghTEV0JXDlL/8NvY=; b=DOWUGvNLeoJtcHP0zoIRwoCgaNqGK/g1aEqcx3v6vfvMYj2Qprd3jd1x1uQUjYSyn0 oglPuyBeef0RikJyEBvPYobKxyNjpajX3fdP1wbOJnFiwDgkBxelKB5Oy+Q387Z2FsCZ U7vDY99K3IF+XHQrIiFUUr5YmLARoea0ocQ7wD5bGK4sHSKNT9NSNjpRZa42P0P1JDXY r9e0YjdqR3UspUK6EoY9odBWT7X6grOaNseW2KY548LsasKjQhylYdSF6lR0+RfMVjIQ TgFHIQSe2adI6KxBG/mIOmkFMRb+GtSPmqTIJMk7kl5+zNwjNt7kRE2lukD9/+lkTgAb u+Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=t86sqvr+nYthFIPNZCGMA6aHii3ghTEV0JXDlL/8NvY=; b=VK+A3i1TXsTpzmFjr01JXLazaFrTl5AtJz3xxHQL5bWrK0YvpQL5hMBpDXuBdXVDmp b4yj75Nc4ciOumEHCTBqP1A+6X1KISYTAIxD2OegTSHJrQZZmiaJTXHzGq8ygJmbCe59 DC7whvuyFM6M0ObAMlg1o+uLI1HSnGFY8Av9ObBu49zmw4wfWR+pk+laAyM6MWM+OCNx Gx1d9hlWmKGRAXr1fwPqlfE9q5aqk/GuKQBwd8ifchMpxkuGZOx/g8GV2EpVUxKa7ztx sYLl30eAgWEedCvsPDe6vecK/j4UnBKCeNZWA/PQVlYY27Jx7mNoEcfWdLyWFC+kUHiW hEug== X-Gm-Message-State: AD7BkJLPsqTDjjrcR/bHp59ejYBrYZkKtmP5a91wKcdo0IgnH7YXGPPbOxFA3BcI7LtULg== X-Received: by 10.194.94.138 with SMTP id dc10mr20010537wjb.37.1458327933210; Fri, 18 Mar 2016 12:05:33 -0700 (PDT) Received: from Solace.station (net-2-35-170-8.cust.vodafonedsl.it. [2.35.170.8]) by smtp.gmail.com with ESMTPSA id hh8sm13360328wjc.42.2016.03.18.12.05.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Mar 2016 12:05:32 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 18 Mar 2016 20:05:30 +0100 Message-ID: <20160318190530.8117.62203.stgit@Solace.station> In-Reply-To: <20160318185524.8117.74837.stgit@Solace.station> References: <20160318185524.8117.74837.stgit@Solace.station> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: George Dunlap , Justin Weaver Subject: [Xen-devel] [PATCH 12/16] xen: sched: fix per-socket runqueue creation in credit2 X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Spam-Status: No, score=-4.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The credit2 scheduler tries to setup runqueues in such a way that there is one of them per each socket. However, that does not work. The issue is described in bug #36 "credit2 only uses one runqueue instead of one runq per socket" (http://bugs.xenproject.org/xen/bug/36), and a solution has been attempted by an old patch series: http://lists.xen.org/archives/html/xen-devel/2014-08/msg02168.html Here, we take advantage of the fact that now initialization happens (for all schedulers) during CPU_STARTING, so we have all the topology information available when necessary. This is true for all the pCPUs _except_ the boot CPU. That is not an issue, though. In fact, no runqueue exists yet when the boot CPU is initialized, so we can just create one and put the boot CPU in there. Signed-off-by: Dario Faggioli Reviewed-by: George Dunlap --- Cc: George Dunlap Cc: Justin Weaver --- xen/common/sched_credit2.c | 59 ++++++++++++++++++++++++++++++++------------ 1 file changed, 43 insertions(+), 16 deletions(-) diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c index 4ff26c9..456b9ea 100644 --- a/xen/common/sched_credit2.c +++ b/xen/common/sched_credit2.c @@ -53,7 +53,6 @@ * http://wiki.xen.org/wiki/Credit2_Scheduler_Development * TODO: * + Multiple sockets - * - Detect cpu layout and make runqueue map, one per L2 (make_runq_map()) * - Simple load balancer / runqueue assignment * - Runqueue load measurement * - Load-based load balancer @@ -1972,6 +1971,48 @@ static void deactivate_runqueue(struct csched2_private *prv, int rqi) cpumask_clear_cpu(rqi, &prv->active_queues); } +static unsigned int +cpu_to_runqueue(struct csched2_private *prv, unsigned int cpu) +{ + struct csched2_runqueue_data *rqd; + unsigned int rqi; + + for ( rqi = 0; rqi < nr_cpu_ids; rqi++ ) + { + unsigned int peer_cpu; + + /* + * As soon as we come across an uninitialized runqueue, use it. + * In fact, either: + * - we are initializing the first cpu, and we assign it to + * runqueue 0. This is handy, especially if we are dealing + * with the boot cpu (if credit2 is the default scheduler), + * as we would not be able to use cpu_to_socket() and similar + * helpers anyway (they're result of which is not reliable yet); + * - we have gone through all the active runqueues, and have not + * found anyone whose cpus' topology matches the one we are + * dealing with, so ativating a new runqueue is what we want. + */ + if ( prv->rqd[rqi].id == -1 ) + break; + + rqd = prv->rqd + rqi; + BUG_ON(cpumask_empty(&rqd->active)); + + peer_cpu = cpumask_first(&rqd->active); + BUG_ON(cpu_to_socket(cpu) == XEN_INVALID_SOCKET_ID || + cpu_to_socket(peer_cpu) == XEN_INVALID_SOCKET_ID); + + if ( cpu_to_socket(cpumask_first(&rqd->active)) == cpu_to_socket(cpu) ) + break; + } + + /* We really expect to be able to assign each cpu to a runqueue. */ + BUG_ON(rqi >= nr_cpu_ids); + + return rqi; +} + /* Returns the ID of the runqueue the cpu is assigned to. */ static unsigned init_pdata(struct csched2_private *prv, unsigned int cpu) @@ -1983,21 +2024,7 @@ init_pdata(struct csched2_private *prv, unsigned int cpu) ASSERT(!cpumask_test_cpu(cpu, &prv->initialized)); /* Figure out which runqueue to put it in */ - rqi = 0; - - /* Figure out which runqueue to put it in */ - /* NB: cpu 0 doesn't get a STARTING callback, so we hard-code it to runqueue 0. */ - if ( cpu == 0 ) - rqi = 0; - else - rqi = cpu_to_socket(cpu); - - if ( rqi == XEN_INVALID_SOCKET_ID ) - { - printk("%s: cpu_to_socket(%d) returned %d!\n", - __func__, cpu, rqi); - BUG(); - } + rqi = cpu_to_runqueue(prv, cpu); rqd = prv->rqd + rqi;