From patchwork Thu Dec 10 21:52:30 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 7822511 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id E9709BEEE1 for ; Thu, 10 Dec 2015 21:52:45 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 08D05205B1 for ; Thu, 10 Dec 2015 21:52:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 14A1D2056D for ; Thu, 10 Dec 2015 21:52:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751799AbbLJVwm (ORCPT ); Thu, 10 Dec 2015 16:52:42 -0500 Received: from mga14.intel.com ([192.55.52.115]:43312 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754021AbbLJVwm (ORCPT ); Thu, 10 Dec 2015 16:52:42 -0500 Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP; 10 Dec 2015 13:52:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,410,1444719600"; d="scan'208";a="858384831" Received: from phlsvsds.ph.intel.com ([10.228.195.38]) by fmsmga001.fm.intel.com with ESMTP; 10 Dec 2015 13:52:38 -0800 Received: from phlsvsds.ph.intel.com (localhost.localdomain [127.0.0.1]) by phlsvsds.ph.intel.com (8.13.8/8.13.8) with ESMTP id tBALqZSl001852; Thu, 10 Dec 2015 16:52:36 -0500 Received: (from iweiny@localhost) by phlsvsds.ph.intel.com (8.13.8/8.13.8/Submit) id tBALqZLw001849; Thu, 10 Dec 2015 16:52:35 -0500 X-Authentication-Warning: phlsvsds.ph.intel.com: iweiny set sender to ira.weiny@intel.com using -f From: ira.weiny@intel.com To: dledford@redhat.com Cc: linux-rdma@vger.kernel.org, Dean Luick Subject: [PATCH] IB/mad: Ensure fairness in ib_mad_completion_handler Date: Thu, 10 Dec 2015 16:52:30 -0500 Message-Id: <1449784350-30214-1-git-send-email-ira.weiny@intel.com> X-Mailer: git-send-email 1.7.1 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Dean Luick It was found that when a process was rapidly sending MADs other processes could be hung in their unregister calls. This would happen when process A was injecting packets fast enough that the single threaded workqueue was never exiting ib_mad_completion_handler. Therefore when process B called flush_workqueue via the unregister call it would hang until process A stopped sending MADs. The fix is to periodically reschedule ib_mad_completion_handler after processing a large number of completions. The number of completions chosen was decided based on the defaults for the recv queue size. However, it was kept fixed such that increasing those queue sizes would not adversely affect fairness in the future. Reviewed-by: Ira Weiny Signed-off-by: Dean Luick --- drivers/infiniband/core/mad.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c index 2281de122038..d4d2a618fd66 100644 --- a/drivers/infiniband/core/mad.c +++ b/drivers/infiniband/core/mad.c @@ -61,6 +61,18 @@ MODULE_PARM_DESC(send_queue_size, "Size of send queue in number of work requests module_param_named(recv_queue_size, mad_recvq_size, int, 0444); MODULE_PARM_DESC(recv_queue_size, "Size of receive queue in number of work requests"); +/* + * Define a limit on the number of completions which will be processed by the + * worker thread in a single work item. This ensures that other work items + * (potentially from other users) are processed fairly. + * + * The number of completions was derived from the default queue sizes above. + * We use a value which is double the larger of the 2 queues (receive @ 512) + * but keep it fixed such that an increase in that value does not introduce + * unfairness. + */ +#define MAD_COMPLETION_PROC_LIMIT 1024 + static struct list_head ib_mad_port_list; static u32 ib_mad_client_id = 0; @@ -2555,6 +2567,7 @@ static void ib_mad_completion_handler(struct work_struct *work) { struct ib_mad_port_private *port_priv; struct ib_wc wc; + int count = 0; port_priv = container_of(work, struct ib_mad_port_private, work); ib_req_notify_cq(port_priv->cq, IB_CQ_NEXT_COMP); @@ -2574,6 +2587,11 @@ static void ib_mad_completion_handler(struct work_struct *work) } } else mad_error_handler(port_priv, &wc); + + if (++count > MAD_COMPLETION_PROC_LIMIT) { + queue_work(port_priv->wq, &port_priv->work); + break; + } } }