From patchwork Mon Jul 3 16:17:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haoran Li X-Patchwork-Id: 9823671 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id F0D7560246 for ; Mon, 3 Jul 2017 16:20:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D78032861E for ; Mon, 3 Jul 2017 16:20:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CC22528627; Mon, 3 Jul 2017 16:20:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_MED, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2C72A2861E for ; Mon, 3 Jul 2017 16:20:27 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dS43D-0001Sc-SG; Mon, 03 Jul 2017 16:18:15 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dS43C-0001SR-JV for xen-devel@lists.xenproject.org; Mon, 03 Jul 2017 16:18:14 +0000 Received: from [85.158.137.68] by server-2.bemta-3.messagelabs.com id F0/14-22472-54E6A595; Mon, 03 Jul 2017 16:18:13 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprMIsWRWlGSWpSXmKPExsVyMfSas65rXlS kwdJ7/Bbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8bzF1wFbwwrHnfdZGlgnK3excjFISQwjVFi yfZmJhCHReASi8S21SdYQRwJgXcsEme69gBlOICcOIkPvQJdjJxAZqXExuWb2UBsIQElialHe 5khJs1ikpiw6TEjSIJNQFXi3O8fLCC2CFDRvVWTmUBsZoEwiafPJrKD2MICKhKnl/5jBrFZgO o3v7oDVsMr4CRx6WkfE8QyOYmTxyazTmDkW8DIsIpRozi1qCy1SNfQQC+pKDM9oyQ3MTMHyDP Wy00tLk5MT81JTCrWS87P3cQIDJR6BgbGHYzbupwPMUpyMCmJ8rrejIwU4kvKT6nMSCzOiC8q zUktPsQow8GhJMF7NTcqUkiwKDU9tSItMwcYsjBpCQ4eJRHeLYZAad7igsTc4sx0iNQpRkuOK 1fWfWHimHJgO5B8NeH/NyYhlrz8vFQpcd6jIPMEQBoySvPgxsHi6hKjrJQwLyMDA4MQT0FqUW 5mCar8K0ZxDkYlYd5OkCk8mXklcFtfAR3EBHRQQ08EyEEliQgpqQbGLv7Oup4E0fnP1gtuX38 q1oHn3r1Nz3lOBqxfP2fCarHZMwyjf20y2VP+XP7HzLC5Bbb/txxewih/0pN3Ue/P65lX7Kzq H0Zw3dba5uU9r32Chavq5WkTTsvf9dr/VHemdYgfe1n9iZUBKx4vWX1q9a4djKWNLhuy/Aq/G s0rW3H1fI/xy/mP1imxFGckGmoxFxUnAgD3WFo9pgIAAA== X-Env-Sender: naroahlee@gmail.com X-Msg-Ref: server-15.tower-31.messagelabs.com!1499098692!99681755!1 X-Originating-IP: [209.85.214.67] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG X-StarScan-Received: X-StarScan-Version: 9.4.19; banners=-,-,- X-VirusChecked: Checked Received: (qmail 31737 invoked from network); 3 Jul 2017 16:18:13 -0000 Received: from mail-it0-f67.google.com (HELO mail-it0-f67.google.com) (209.85.214.67) by server-15.tower-31.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 3 Jul 2017 16:18:13 -0000 Received: by mail-it0-f67.google.com with SMTP id k3so8562509ita.3 for ; Mon, 03 Jul 2017 09:18:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=3HerRDH5XQYZdaNsxG8+dt8xGsy4FpKbT8WZL6wFtTY=; b=E7IY7U5RLa4WwR0yApvWyA/Rc5i9Nm+ZMP43HARsOnWhzh05AqwIWBHeLrR3CT+LkA doOKDM6bcHAOmE7PLsQakbtzIjxa7efYkh3/vm/LRBl08FOpfbk9XYEF3Im5MKUtTkBc fDaY/JHgQXa8pj1DRgCpsswDOuT4yU6FqyGVdfkMtHXmz5M2oTTa//vQtODoCEEnA+I2 0kBqdah+aLv7xQCuiEzMfl8aHurq+dAOrDdMhJZhNpYp0iZnjb6k+5bj/K/FC3TQwWyq 9ye1mEOVOMmwF78UFyMk8HsFRk57ogziGIoJz2eeXEiWYWyy6JqCsrYCtAJMp8fAtvnm g2xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=3HerRDH5XQYZdaNsxG8+dt8xGsy4FpKbT8WZL6wFtTY=; b=HxLZcBJmPABFjDIDsqx+aGxqeLChST4I4j+r6LFz0jei7f4sT+/ibevsKB9dTlmdQ5 P3buN15Uwqkx5CRPIPC0bcY/LNx8oWycjkW4/xRtsjq8Lbn+axkdC4qzOIb0XFaIcuax uMXcWQX9QlXwOOa+RfPlf4PE60dyUNt2pURyu+AJWcHyOtBTXvXeIWIR5WkAd/YNdbLA awPwxqJbcELFumZHNNjhJNmZIcT2/iXwz1+5y9jnObIzcul3n9q/Js/7cOawX1xQKCUR jQjRB22WOA6RztNMn2blWVFLMRqZkm8x8qZa8nDfTrS2OLbpRwh6yKMENZfCQ27JRJLq wPmQ== X-Gm-Message-State: AIVw113WS11b/fYRRto3z847OGeiUj06maIkt10UC03MU7E1lMHi+U/U rUCn9IBMonhzUEoF X-Received: by 10.36.44.74 with SMTP id i71mr11862435iti.112.1499098691254; Mon, 03 Jul 2017 09:18:11 -0700 (PDT) Received: from E5-2863v4.seas.wustl.edu (admin998.cec.wustl.edu. [128.252.20.193]) by smtp.googlemail.com with ESMTPSA id o130sm10496299ito.2.2017.07.03.09.18.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 03 Jul 2017 09:18:10 -0700 (PDT) From: Haoran Li To: xen-devel@lists.xenproject.org Date: Mon, 3 Jul 2017 11:17:36 -0500 Message-Id: <1499098656-1608-1-git-send-email-naroahlee@gmail.com> X-Mailer: git-send-email 1.9.1 Cc: dario.faggioli@citrix.com, mengxu@cis.upenn.edu, naroahlee Subject: [Xen-devel] [RTDS Patch v3 for Xen4.8] X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: naroahlee When more than one idle VCPUs that have the same PCPU as their previous running core invoke runq_tickle(), they will tickle the same PCPU. The tickled PCPU will only pick at most one VCPU, i.e., the highest-priority one, to execute. The other VCPUs will not be scheduled for a period, even when there is an idle core, making these VCPUs unnecessarily starve for one period. Therefore, always make sure that we only tickle PCPUs that have not been tickled already. Signed-off-by: Haoran Li Reviewed-by: Meng Xu --- xen/common/sched_rt.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/xen/common/sched_rt.c b/xen/common/sched_rt.c index 1b30014..b3d55d8 100644 --- a/xen/common/sched_rt.c +++ b/xen/common/sched_rt.c @@ -1144,12 +1144,11 @@ rt_vcpu_sleep(const struct scheduler *ops, struct vcpu *vc) * Called by wake() and context_saved() * We have a running candidate here, the kick logic is: * Among all the cpus that are within the cpu affinity - * 1) if the new->cpu is idle, kick it. This could benefit cache hit - * 2) if there are any idle vcpu, kick it. - * 3) now all pcpus are busy; + * 1) if there are any idle vcpu, kick it. + * For cache benefit, we first search new->cpu. + * 2) now all pcpus are busy; * among all the running vcpus, pick lowest priority one * if snext has higher priority, kick it. - * * TODO: * 1) what if these two vcpus belongs to the same domain? * replace a vcpu belonging to the same domain introduces more overhead @@ -1174,17 +1173,11 @@ runq_tickle(const struct scheduler *ops, struct rt_vcpu *new) cpumask_and(¬_tickled, online, new->vcpu->cpu_hard_affinity); cpumask_andnot(¬_tickled, ¬_tickled, &prv->tickled); - /* 1) if new's previous cpu is idle, kick it for cache benefit */ - if ( is_idle_vcpu(curr_on_cpu(new->vcpu->processor)) ) - { - SCHED_STAT_CRANK(tickled_idle_cpu); - cpu_to_tickle = new->vcpu->processor; - goto out; - } - - /* 2) if there are any idle pcpu, kick it */ + /* 1) if there are any idle pcpu, kick it */ /* The same loop also find the one with lowest priority */ - for_each_cpu(cpu, ¬_tickled) + /* For cache benefit, we search new->cpu first */ + cpu = cpumask_test_or_cycle(new->vcpu->processor, ¬_tickled); + while ( cpu != nr_cpu_ids ) { iter_vc = curr_on_cpu(cpu); if ( is_idle_vcpu(iter_vc) ) @@ -1197,9 +1190,12 @@ runq_tickle(const struct scheduler *ops, struct rt_vcpu *new) if ( latest_deadline_vcpu == NULL || iter_svc->cur_deadline > latest_deadline_vcpu->cur_deadline ) latest_deadline_vcpu = iter_svc; + + cpumask_clear_cpu(cpu, ¬_tickled); + cpu = cpumask_cycle(cpu, ¬_tickled); } - /* 3) candicate has higher priority, kick out lowest priority vcpu */ + /* 2) candicate has higher priority, kick out lowest priority vcpu */ if ( latest_deadline_vcpu != NULL && new->cur_deadline < latest_deadline_vcpu->cur_deadline ) { @@ -1207,7 +1203,6 @@ runq_tickle(const struct scheduler *ops, struct rt_vcpu *new) cpu_to_tickle = latest_deadline_vcpu->vcpu->processor; goto out; } - /* didn't tickle any cpu */ SCHED_STAT_CRANK(tickled_no_cpu); return;