From patchwork Wed Feb 3 22:34:23 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Dalessandro X-Patchwork-Id: 8210001 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id C94ACBEEE5 for ; Wed, 3 Feb 2016 22:34:57 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id C721A202C8 for ; Wed, 3 Feb 2016 22:34:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2C3FB20253 for ; Wed, 3 Feb 2016 22:34:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756782AbcBCWe1 (ORCPT ); Wed, 3 Feb 2016 17:34:27 -0500 Received: from mga01.intel.com ([192.55.52.88]:5479 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756757AbcBCWeY (ORCPT ); Wed, 3 Feb 2016 17:34:24 -0500 Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP; 03 Feb 2016 14:34:24 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.22,392,1449561600"; d="scan'208";a="907980854" Received: from scymds01.sc.intel.com ([10.82.194.37]) by fmsmga002.fm.intel.com with ESMTP; 03 Feb 2016 14:34:23 -0800 Received: from scvm10.sc.intel.com (scvm10.sc.intel.com [10.82.195.27]) by scymds01.sc.intel.com with ESMTP id u13MYNVc007883; Wed, 3 Feb 2016 14:34:23 -0800 Received: from scvm10.sc.intel.com (localhost [127.0.0.1]) by scvm10.sc.intel.com with ESMTP id u13MYNFD010215; Wed, 3 Feb 2016 14:34:23 -0800 Subject: [PATCH 27/54] staging/rdma/hfi1: Allow a fair scheduling of QPs To: dledford@redhat.com From: Dennis Dalessandro Cc: linux-rdma@vger.kernel.org, Mike Marciniszyn , Vennila Megavannan , Jubin John Date: Wed, 03 Feb 2016 14:34:23 -0800 Message-ID: <20160203223420.5923.85376.stgit@scvm10.sc.intel.com> In-Reply-To: <20160203222512.5923.30980.stgit@scvm10.sc.intel.com> References: <20160203222512.5923.30980.stgit@scvm10.sc.intel.com> User-Agent: StGit/0.16 MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-7.3 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Vennila Megavannan This patch fixes the fairness issues in QP scheduling - the timeout for cond_resched is changed to a ratio of qp->timeout_jiffies - workqueue_congested is used to determine if qp needs to reschedule itself Reviewed-by: Mike Marciniszyn Signed-off-by: Vennila Megavannan Signed-off-by: Jubin John --- drivers/staging/rdma/hfi1/ruc.c | 52 +++++++++++++++++++++++++++++---------- 1 files changed, 38 insertions(+), 14 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/staging/rdma/hfi1/ruc.c b/drivers/staging/rdma/hfi1/ruc.c index ae28b85..f09badb 100644 --- a/drivers/staging/rdma/hfi1/ruc.c +++ b/drivers/staging/rdma/hfi1/ruc.c @@ -822,29 +822,42 @@ void _hfi1_do_send(struct work_struct *work) void hfi1_do_send(struct rvt_qp *qp) { struct hfi1_pkt_state ps; + struct hfi1_qp_priv *priv = qp->priv; int (*make_req)(struct rvt_qp *qp); unsigned long flags; unsigned long timeout; + unsigned long timeout_int; + int cpu; ps.dev = to_idev(qp->ibqp.device); ps.ibp = to_iport(qp->ibqp.device, qp->port_num); ps.ppd = ppd_from_ibp(ps.ibp); - if ((qp->ibqp.qp_type == IB_QPT_RC || - qp->ibqp.qp_type == IB_QPT_UC) && - !loopback && - (qp->remote_ah_attr.dlid & ~((1 << ps.ppd->lmc) - 1)) == - ps.ppd->lid) { - ruc_loopback(qp); - return; - } - - if (qp->ibqp.qp_type == IB_QPT_RC) + switch (qp->ibqp.qp_type) { + case IB_QPT_RC: + if (!loopback && ((qp->remote_ah_attr.dlid & ~((1 << ps.ppd->lmc + ) - 1)) == + ps.ppd->lid)) { + ruc_loopback(qp); + return; + } make_req = hfi1_make_rc_req; - else if (qp->ibqp.qp_type == IB_QPT_UC) + timeout_int = (qp->timeout_jiffies); + break; + case IB_QPT_UC: + if (!loopback && ((qp->remote_ah_attr.dlid & ~((1 << ps.ppd->lmc + ) - 1)) == + ps.ppd->lid)) { + ruc_loopback(qp); + return; + } make_req = hfi1_make_uc_req; - else + timeout_int = SEND_RESCHED_TIMEOUT; + break; + default: make_req = hfi1_make_ud_req; + timeout_int = SEND_RESCHED_TIMEOUT; + } spin_lock_irqsave(&qp->s_lock, flags); @@ -858,7 +871,9 @@ void hfi1_do_send(struct rvt_qp *qp) spin_unlock_irqrestore(&qp->s_lock, flags); - timeout = jiffies + SEND_RESCHED_TIMEOUT; + timeout = jiffies + (timeout_int) / 8; + cpu = priv->s_sde ? priv->s_sde->cpu : + cpumask_first(cpumask_of_node(ps.ppd->dd->node)); do { /* Check for a constructed packet to be sent. */ if (qp->s_hdrwords != 0) { @@ -874,9 +889,18 @@ void hfi1_do_send(struct rvt_qp *qp) /* allow other tasks to run */ if (unlikely(time_after(jiffies, timeout))) { + if (workqueue_congested(cpu, ps.ppd->hfi1_wq)) { + spin_lock_irqsave(&qp->s_lock, flags); + qp->s_flags &= ~RVT_S_BUSY; + hfi1_schedule_send(qp); + spin_unlock_irqrestore(&qp->s_lock, + flags); + this_cpu_inc(*ps.ppd->dd->send_schedule); + return; + } cond_resched(); this_cpu_inc(*ps.ppd->dd->send_schedule); - timeout = jiffies + SEND_RESCHED_TIMEOUT; + timeout = jiffies + (timeout_int) / 8; } } while (make_req(qp)); }