From patchwork Fri Nov 6 02:41:31 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ira Weiny X-Patchwork-Id: 7566181 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id BF881C05C6 for ; Fri, 6 Nov 2015 02:41:40 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id BAD5320805 for ; Fri, 6 Nov 2015 02:41:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 858AB20802 for ; Fri, 6 Nov 2015 02:41:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1031121AbbKFClh (ORCPT ); Thu, 5 Nov 2015 21:41:37 -0500 Received: from mga11.intel.com ([192.55.52.93]:56419 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1031088AbbKFClg (ORCPT ); Thu, 5 Nov 2015 21:41:36 -0500 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga102.fm.intel.com with ESMTP; 05 Nov 2015 18:41:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,250,1444719600"; d="scan'208";a="843766671" Received: from phlsvsds.ph.intel.com ([10.228.195.38]) by orsmga002.jf.intel.com with ESMTP; 05 Nov 2015 18:41:34 -0800 Received: from phlsvsds.ph.intel.com (localhost.localdomain [127.0.0.1]) by phlsvsds.ph.intel.com (8.13.8/8.13.8) with ESMTP id tA62fXgj017724; Thu, 5 Nov 2015 21:41:33 -0500 Received: (from iweiny@localhost) by phlsvsds.ph.intel.com (8.13.8/8.13.8/Submit) id tA62fXuH017721; Thu, 5 Nov 2015 21:41:33 -0500 X-Authentication-Warning: phlsvsds.ph.intel.com: iweiny set sender to ira.weiny@intel.com using -f From: ira.weiny@intel.com To: gregkh@linuxfoundation.org, devel@driverdev.osuosl.org Cc: dledford@redhat.com, linux-rdma@vger.kernel.org, dennis.dalessandro@intel.com, mike.marciniszyn@intel.com, Vennila Megavannan , Ira Weiny Subject: [PATCH v2 3/3] staging/rdma/hfi1: Method to toggle "fast ECN" detection Date: Thu, 5 Nov 2015 21:41:31 -0500 Message-Id: <1446777691-17460-1-git-send-email-ira.weiny@intel.com> X-Mailer: git-send-email 1.7.1 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Vennila Megavannan Add a module parameter to toggle prescan/Fast ECN Detection. The config is maintained to be able to fully remove the code to obtain the maximum receive performance _if_ the customer knows that they will never need the prescanning. This will result in the maximum performance on the RX path. The default of enabling the code with the config option but having the module parameter set to disabled, introduces a _slight_ performance impact but allows the user to determine if this congestion control fix should be active or not at run time. Testing showed that this was the best compromise for most systems in terms of performance and flexibility. This is because prescanning is not required most of the time. Should a customer find that their topology and/or workload requires prescanning they can then enable this option at run time. Enabling prescanning at run time will result in a significant reduction of performance at the node level but may result in better overall fabric/cluster performance. Reviewed-by: Arthur Kepner Reviewed-by: Mike Marciniszyn Signed-off-by: Vennila Megavannan Signed-off-by: Ira Weiny --- Changes from V1: Redo commit message as well as Kconfig help to make it clear what the compile and module options do. --- drivers/staging/rdma/hfi1/Kconfig | 24 +++++++++++++++++------- drivers/staging/rdma/hfi1/driver.c | 24 +++++++++++++++++------- 2 files changed, 34 insertions(+), 14 deletions(-) diff --git a/drivers/staging/rdma/hfi1/Kconfig b/drivers/staging/rdma/hfi1/Kconfig index fd25078ee923..b534cd0e348a 100644 --- a/drivers/staging/rdma/hfi1/Kconfig +++ b/drivers/staging/rdma/hfi1/Kconfig @@ -26,12 +26,22 @@ config SDMA_VERBOSITY This is a configuration flag to enable verbose SDMA debug config PRESCAN_RXQ - bool "Enable prescanning of the RX queue for ECNs" + bool "Enable code for prescanning of the RX queue for ECNs" depends on INFINIBAND_HFI1 - default n + default y ---help--- - This option toggles the prescanning of the receive queue for - Explicit Congestion Notifications. If an ECN is detected, it - is processed as quickly as possible, the ECN is toggled off. - After the prescanning step, the receive queue is processed as - usual. + This option enables code for the prescanning of the receive queue for + Explicit Congestion Notifications. Pre-scanning can be controlled via a + module option at run time. If an ECN is detected, it is processed as + quickly as possible, the ECN is toggled off. After the prescanning + step, the receive queue is processed as usual. + + If you know that you will never Explicit Congestion Notification, + setting this option to "no" will result in maximum performance. + + The default of "yes" results in a _slight_ performance impact but + allows the user to enable faster ECN detection at run time. This + allows for the most flexible tuning of an entire system at run time. + + If in doubt say "yes" + diff --git a/drivers/staging/rdma/hfi1/driver.c b/drivers/staging/rdma/hfi1/driver.c index 9a4ec09af020..4c1dd7130d7a 100644 --- a/drivers/staging/rdma/hfi1/driver.c +++ b/drivers/staging/rdma/hfi1/driver.c @@ -83,6 +83,14 @@ unsigned int hfi1_cu = 1; module_param_named(cu, hfi1_cu, uint, S_IRUGO); MODULE_PARM_DESC(cu, "Credit return units"); +#ifdef CONFIG_PRESCAN_RXQ +static unsigned int prescan_rx_queue; +module_param_named(prescan_rxq, prescan_rx_queue, uint, + S_IRUGO | S_IWUSR); +MODULE_PARM_DESC(prescan_rxq, + "Used to toggle rx prescan. Set to 1 to enable prescan"); +#endif /* CONFIG_PRESCAN_RXQ */ + unsigned long hfi1_cap_mask = HFI1_CAP_MASK_DEFAULT; static int hfi1_caps_set(const char *, const struct kernel_param *); static int hfi1_caps_get(char *, const struct kernel_param *); @@ -435,10 +443,8 @@ static inline void init_packet(struct hfi1_ctxtdata *rcd, } #ifndef CONFIG_PRESCAN_RXQ -static void prescan_rxq(struct hfi1_packet *packet) {} +#define prescan_rxq(packet) #else /* !CONFIG_PRESCAN_RXQ */ -static int prescan_receive_queue; - static void process_ecn(struct hfi1_qp *qp, struct hfi1_ib_header *hdr, struct hfi1_other_headers *ohdr, u64 rhf, u32 bth1, struct ib_grh *grh) @@ -541,15 +547,19 @@ static inline void update_ps_mdata(struct ps_mdata *mdata) * containing Excplicit Congestion Notifications (FECNs, or BECNs). * When an ECN is found, process the Congestion Notification, and toggle * it off. + * This is declared as a macro to allow quick checking of the module param and + * avoid the overhead of a function call if not enabled. */ -static void prescan_rxq(struct hfi1_packet *packet) +#define prescan_rxq(packet) \ + do { \ + if (prescan_rx_queue) \ + __prescan_rxq(packet); \ + } while (0) +static void __prescan_rxq(struct hfi1_packet *packet) { struct hfi1_ctxtdata *rcd = packet->rcd; struct ps_mdata mdata; - if (!prescan_receive_queue) - return; - init_ps_mdata(&mdata, packet); while (1) {