From patchwork Thu Jul 27 12:06:00 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9866699 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 76BCD60382 for ; Thu, 27 Jul 2017 12:08:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6923A28789 for ; Thu, 27 Jul 2017 12:08:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5CA29287FE; Thu, 27 Jul 2017 12:08:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 273FA28789 for ; Thu, 27 Jul 2017 12:08:07 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dahYM-0005c6-5r; Thu, 27 Jul 2017 12:06:06 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dahYL-0005ba-3h for xen-devel@lists.xenproject.org; Thu, 27 Jul 2017 12:06:05 +0000 Received: from [85.158.137.68] by server-7.bemta-3.messagelabs.com id ED/36-02177-C27D9795; Thu, 27 Jul 2017 12:06:04 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrPIsWRWlGSWpSXmKPExsXiVRvkoqt9vTL S4H+zusX3LZOZHBg9Dn+4whLAGMWamZeUX5HAmvHh9DzGgvlyFSuX7WJtYLws3sXIxSEkMINR YsHmr8wgDovAGlaJiydmMIE4EgKXWCUauvrYuxg5gZw4iUdTrrNB2BUSTy/uYgaxhQRUJG5uX 8UEMeoHo8SOx6vAioQF9CSOHP3BDmHHSrSt/gkWZxMwkHizYy8riC0ioCRxb9VkJhCbWSBUYs 67M2A1LAKqEnOWPAVbwCvgKHG3FWImp4CTxPYLTWwQix0lbp/9CTZfVEBOYuXlFlaIekGJkzO fsHQxcgDN1JRYv0sfYry8xPa3c5gnMIrMQlI1C6FqFpKqBYzMqxg1ilOLylKLdI1M9ZKKMtMz SnITM3N0DQ2M9XJTi4sT01NzEpOK9ZLzczcxAiOgnoGBcQdj6wm/Q4ySHExKoryTTCsihfiS8 lMqMxKLM+KLSnNSiw8xanBwCEw4O3c6kxRLXn5eqpIEr9S1ykghwaLU9NSKtMwcYIzClEpw8C iJ8DqApHmLCxJzizPTIVKnGHU5Jh3Y/oVJCGyGlDjvs6tARQIgRRmleXAjYOniEqOslDAvIwM DgxBPQWpRbmYJqvwrRnEORiVhXiGQVTyZeSVwm14BHcEEdMTEJrAjShIRUlINjA0NgmtzGpv9 zp+M4586xXOWouDNXUf4m31F08ozdf/NlxOzVepbr7DKZ92sMqWLUgF5yyJWsjgXunzu1Pkp3 tsZxSnHdTO6eeqbhxu13J4eD9xREh3+huvxeQ/2LVYfDX0PXGKr32j8ZBHLZtETKSYHDOOt0t odH2bGfGt0zUyYpHj0/pVOJZbijERDLeai4kQAzMJCohIDAAA= X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-15.tower-31.messagelabs.com!1501157163!103351118!1 X-Originating-IP: [74.125.82.68] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.4.25; banners=-,-,- X-VirusChecked: Checked Received: (qmail 30153 invoked from network); 27 Jul 2017 12:06:03 -0000 Received: from mail-wm0-f68.google.com (HELO mail-wm0-f68.google.com) (74.125.82.68) by server-15.tower-31.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 27 Jul 2017 12:06:03 -0000 Received: by mail-wm0-f68.google.com with SMTP id 184so23446122wmo.3 for ; Thu, 27 Jul 2017 05:06:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=73htqAp7NjafrwXorqax4+KntmL0TEi2I9ypiQ0hSnY=; b=eiOkQUA2s0BoVKH4RsEuNiSNJA82hrG2XPBZAmxp5ChnnialU7oCECYx1fmoynYy2i 2329c7zLWLFOEgSk+IpPUKIfWqFzhHnjX+yzg4FBPdaSQ+IefYZc8bWk+gfi6X0bloil oqPTlw7jlOtG/cXSDmV7+PXDXXFTOXxhRh05yNR/3+2Gk9edK8X2Cwkbn6YlnV3RDwRx CNXtrRZRC6vENd5VW6qblsKvylPyviXhogSVawrjQ3o0YMD2KOX9hILhRRU43fHDZwHc 4JBtwUBhDV16V6ldIcvnra3Z2T0KxWx14f1oXvM3+jE+bwe3EYnDTJrnCL5LiybHiFUk mpJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=73htqAp7NjafrwXorqax4+KntmL0TEi2I9ypiQ0hSnY=; b=X2mr6kGpHFmFI9q8pTHIg4bIE0a43wVh++oTN2LIVwNx+eyrnoMHJNoV+4gWxaerte 8IkohYr1bjb09bj1MYeRv4GHa6HVq4G3nU4hWUlMeiUIKG1P0WX6JyXzjatSCCPZEu8m Y4b8tc8cZjtIRrWmYVO5xj3kCbelNx23oyqkkkC4fqFKAkgXPVoLCbWQ9XeEETI89THu kSO7wBmyDREhoxWUoQMZ+TXdWrK/gqdT2D8gCEuMD7tzdvGsFi4R7mIhADmitZgG9Gc0 NpGQGQci+lYdL+lzVLQPgGQtlC+yX+vyorM+IM3jqeuE1ZeIa1p3noU5zkQc1POQGjrP mbWw== X-Gm-Message-State: AIVw110QALEfpbRRfKAsFUf2c+zsNLBXrZLU28drUkblwWGd30xWc5Pz bz2FPG/P+rftyVYW X-Received: by 10.28.140.66 with SMTP id o63mr2980138wmd.95.1501157162957; Thu, 27 Jul 2017 05:06:02 -0700 (PDT) Received: from [192.168.0.31] ([80.66.223.212]) by smtp.gmail.com with ESMTPSA id v41sm4417035wrc.65.2017.07.27.05.06.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 27 Jul 2017 05:06:02 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Thu, 27 Jul 2017 14:06:00 +0200 Message-ID: <150115716086.6767.12547868316668472877.stgit@Solace> In-Reply-To: <150115657192.6767.15778617807307106582.stgit@Solace> References: <150115657192.6767.15778617807307106582.stgit@Solace> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: George Dunlap , Anshul Makkar Subject: [Xen-devel] [PATCH v2 4/6] xen: credit2: kick away vcpus not running within their soft-affinity X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP If, during scheduling, we realize that the current vcpu is running outside of its own soft-affinity, it would be preferable to send it somewhere else. Of course, that may not be possible, and if we're too strict, we risk having vcpus sit in runqueues, even if there are idle pcpus (violating work-conservingness). In fact, what about there are no pcpus, from the soft affinity mask of the vcpu in question, where it can run? To make sure we don't fall in the above described trap, only actually de-schedule the vcpu if there are idle and not already tickled cpus from its soft affinity where it can run immediately. If there is (at least one) of such cpus, we let current be preempted, so that csched2_context_saved() will put it back in the runq, and runq_tickle() will wake (one of) the cpu. If there is not even one, we let current run where it is, as running outside its soft-affinity is still better than not running at all. Signed-off-by: Dario Faggioli Reviewed-by: George Dunlap --- Cc: Anshul Makkar --- xen/common/sched_credit2.c | 40 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c index 8237a0a..3f10b4b 100644 --- a/xen/common/sched_credit2.c +++ b/xen/common/sched_credit2.c @@ -2675,6 +2675,7 @@ runq_candidate(struct csched2_runqueue_data *rqd, struct csched2_vcpu *snext = NULL; struct csched2_private *prv = csched2_priv(per_cpu(scheduler, cpu)); bool yield = __test_and_clear_bit(__CSFLAG_vcpu_yield, &scurr->flags); + bool soft_aff_preempt = false; *skipped = 0; @@ -2708,8 +2709,43 @@ runq_candidate(struct csched2_runqueue_data *rqd, return scurr; } - /* Default to current if runnable, idle otherwise */ - if ( vcpu_runnable(scurr->vcpu) ) + /* If scurr has a soft-affinity, let's check whether cpu is part of it */ + if ( !is_idle_vcpu(scurr->vcpu) && + has_soft_affinity(scurr->vcpu, scurr->vcpu->cpu_hard_affinity) ) + { + affinity_balance_cpumask(scurr->vcpu, BALANCE_SOFT_AFFINITY, + cpumask_scratch); + if ( unlikely(!cpumask_test_cpu(cpu, cpumask_scratch)) ) + { + cpumask_t *online = cpupool_domain_cpumask(scurr->vcpu->domain); + + /* Ok, is any of the pcpus in scurr soft-affinity idle? */ + cpumask_and(cpumask_scratch, cpumask_scratch, &rqd->idle); + cpumask_andnot(cpumask_scratch, cpumask_scratch, &rqd->tickled); + soft_aff_preempt = cpumask_intersects(cpumask_scratch, online); + } + } + + /* + * If scurr is runnable, and this cpu is in its soft-affinity, default to + * it. We also default to it, even if cpu is not in its soft-affinity, if + * there aren't any idle and not tickled cpu in its soft-affinity. In + * fact, we don't want to risk leaving scurr in the runq and this cpu idle + * only because scurr is running outside of its soft-affinity. + * + * On the other hand, if cpu is not in scurr's soft-affinity, and there + * looks to be better options, go for them. That happens by defaulting to + * idle here, which means scurr will be preempted, put back in runq, and + * one of those idle and not tickled cpus from its soft-affinity will be + * tickled to pick it up. + * + * Finally, if scurr does not have a valid soft-affinity, we also let it + * continue to run here (in fact, soft_aff_preempt will still be false, + * in this case). + * + * Of course, we also default to idle also if scurr is not runnable. + */ + if ( vcpu_runnable(scurr->vcpu) && !soft_aff_preempt ) snext = scurr; else snext = csched2_vcpu(idle_vcpu[cpu]);