From patchwork Wed Nov 28 19:03:11 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hal Rosenstock X-Patchwork-Id: 1817891 X-Patchwork-Delegate: alexne@voltaire.com Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork2.kernel.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by patchwork2.kernel.org (Postfix) with ESMTP id B1247DF26F for ; Wed, 28 Nov 2012 19:03:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754613Ab2K1TDQ (ORCPT ); Wed, 28 Nov 2012 14:03:16 -0500 Received: from mail-bk0-f46.google.com ([209.85.214.46]:61131 "EHLO mail-bk0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754421Ab2K1TDP (ORCPT ); Wed, 28 Nov 2012 14:03:15 -0500 Received: by mail-bk0-f46.google.com with SMTP id q16so5900875bkw.19 for ; Wed, 28 Nov 2012 11:03:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :content-type:content-transfer-encoding:x-gm-message-state; bh=YuLivxxVonbiw+oHuA57yGrjPqfGwdCQRYY0mXPJJUY=; b=ouqggiO4u/mcBhIwOdZOXrheGY20rWIb4y0FYQ3xS+ZnJ7NagoaSQQqMqvfTQnJLYb ER9gAHNhOriimCKa86T5Oyxp2W0PvZ9xkuLSUkXPs6HfBKrnV75rbCEkiAg/DtxpsJ9j y0qJ4AJdJMgf59po9MPuMQxBU0E80AQmO61IV2w1sczUiKu+da9lvr4mwKDvSPUM5Vb7 qUL8hMLM5GybgL7Icj07Uo+T7yp5Fh4Q/G6jwFaZxROCOkF7MSCGFDdBrvmYxXp3xaUI dI5Dh9H7b1YUKhuiI/S44DDbxP6w9pu9PFFu6r/K6eOfc8LqqM380jpLqVHDHvIM6JOf wc6w== Received: by 10.204.147.67 with SMTP id k3mr5916348bkv.117.1354129394537; Wed, 28 Nov 2012 11:03:14 -0800 (PST) Received: from [192.168.1.102] (c-71-192-10-85.hsd1.ma.comcast.net. [71.192.10.85]) by mx.google.com with ESMTPS id l17sm13540486bkw.12.2012.11.28.11.03.12 (version=SSLv3 cipher=OTHER); Wed, 28 Nov 2012 11:03:13 -0800 (PST) Message-ID: <50B65FEF.4070408@dev.mellanox.co.il> Date: Wed, 28 Nov 2012 14:03:11 -0500 From: Hal Rosenstock User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:9.0) Gecko/20111222 Thunderbird/9.0.1 MIME-Version: 1.0 To: Alex Netes CC: "linux-rdma (linux-rdma@vger.kernel.org)" , Jim Schutt Subject: [PATCH 3/8] opensm/osm_torus.c: Require only 2 data VLs supported (PortInfo.VLCap) and use VLs 0-1 on CA links X-Gm-Message-State: ALoCoQlxYMiquZidG1HG6KDkKQ4r5v71PwwFfwj50lqWHiQ4gNbvhgBbK33MuJSmi2Lv2oSsCfY8 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org CA links preserve SLs 0-7 for QoS level 0 and SLs 8-15 for QoS level 1. First group of SLs are mapped to VL0 and second group are mapped to VL1. Signed-off-by: Jim Schutt Signed-off-by: Hal Rosenstock --- opensm/osm_torus.c | 124 ++++++++++++++++++++++++++++++++++++++++++++-------- 1 files changed, 105 insertions(+), 19 deletions(-) diff --git a/opensm/osm_torus.c b/opensm/osm_torus.c index 5bd7639..9d41297 100644 --- a/opensm/osm_torus.c +++ b/opensm/osm_torus.c @@ -7042,27 +7042,27 @@ bool verify_setup(struct torus *t, struct fabric *f) "ERR 4E20: missing required torus size specification!\n"); goto out; } - if (t->osm->subn.min_data_vls < 2 || t->osm->subn.min_sw_data_vls < 2) + if (t->osm->subn.min_sw_data_vls < 2) OSM_LOG(&t->osm->log, OSM_LOG_INFO, "Warning: Too few data VLs to support torus routing " - "without credit loops (have endport %d switchport %d " - "need 2)\n", - (int)t->osm->subn.min_data_vls, + "without credit loops (have switchport %d need 2)\n", (int)t->osm->subn.min_sw_data_vls); - if (t->osm->subn.min_data_vls < 4 || t->osm->subn.min_sw_data_vls < 4) + if (t->osm->subn.min_sw_data_vls < 4) OSM_LOG(&t->osm->log, OSM_LOG_INFO, "Warning: Too few data VLs to support torus routing " "with a failed switch without credit loops " - "(have endport %d switchport %d need 4)\n", - (int)t->osm->subn.min_data_vls, + "(have switchport %d need 4)\n", (int)t->osm->subn.min_sw_data_vls); - if (t->osm->subn.min_data_vls < 8 || t->osm->subn.min_sw_data_vls < 8) + if (t->osm->subn.min_sw_data_vls < 8) OSM_LOG(&t->osm->log, OSM_LOG_INFO, "Warning: Too few data VLs to support torus routing " - "with two QoS levels (have endport %d switchport %d " - "need 8)\n", - (int)t->osm->subn.min_data_vls, + "with two QoS levels (have switchport %d need 8)\n", (int)t->osm->subn.min_sw_data_vls); + if (t->osm->subn.min_data_vls < 2) + OSM_LOG(&t->osm->log, OSM_LOG_INFO, + "Warning: Too few data VLs to support torus routing " + "with two QoS levels (have endport %d need 2)\n", + (int)t->osm->subn.min_data_vls); /* * Be sure all the switches in the torus support the port * ordering that might have been configured. @@ -7944,6 +7944,7 @@ unsigned sl_get_qos(unsigned sl) * Functions to encode routing/QoS info into VL bits. Combine the resuts of * these functions with bitwise or to get final VL. * + * For interswitch links: * VL bit 0 encodes whether we need to leave on the "loop" VL. * * VL bit 1 encodes whether turn is XYZ DOR or ZYX DOR. A 3d mesh/torus @@ -7958,6 +7959,9 @@ unsigned sl_get_qos(unsigned sl) * * VL bit 2 encodes QoS level. * + * For end port links: + * VL bit 0 encodes QoS level. + * * Note that if VL bit encodings are changed here, the available fabric VL * verification in verify_setup() needs to be updated as well. */ @@ -7974,6 +7978,12 @@ unsigned vl_set_qos_vl(unsigned qos) } static inline +unsigned vl_set_ca_qos_vl(unsigned qos) +{ + return qos & 0x1; +} + +static inline unsigned vl_set_turn_vl(unsigned in_coord_dir, unsigned out_coord_dir) { unsigned vl = 0; @@ -8006,15 +8016,19 @@ unsigned sl2vl_entry(struct torus *t, struct t_switch *sw, data_vls = t->osm->subn.min_sw_data_vls; else data_vls = t->osm->subn.min_data_vls; - vl = 0; - - if (data_vls >= 2) - vl |= vl_set_loop_vl(sl_get_use_loop_vl(sl, od)); - if (data_vls >= 4) - vl |= vl_set_turn_vl(id, od); - if (data_vls >= 8) - vl |= vl_set_qos_vl(sl_get_qos(sl)); + vl = 0; + if (sw && od != TORUS_MAX_DIM) { + if (data_vls >= 2) + vl |= vl_set_loop_vl(sl_get_use_loop_vl(sl, od)); + if (data_vls >= 4) + vl |= vl_set_turn_vl(id, od); + if (data_vls >= 8) + vl |= vl_set_qos_vl(sl_get_qos(sl)); + } else { + if (data_vls >= 2) + vl |= vl_set_ca_qos_vl(sl_get_qos(sl)); + } return vl; } @@ -8048,6 +8062,73 @@ void torus_update_osm_sl2vl(void *context, osm_physp_t *osm_phys_port, } } +static +void torus_update_osm_vlarb(void *context, osm_physp_t *osm_phys_port, + uint8_t port_num, ib_vl_arb_table_t *block, + unsigned block_length, unsigned block_num) +{ + osm_node_t *node = osm_physp_get_node_ptr(osm_phys_port); + struct torus_context *ctx = context; + struct t_switch *sw = NULL; + unsigned i, next; + + if (node->sw) { + sw = node->sw->priv; + if (sw && sw->osm_switch != node->sw) { + osm_log_t *log = &ctx->osm->log; + guid_t guid; + + guid = osm_node_get_node_guid(node); + OSM_LOG(log, OSM_LOG_INFO, + "Error: osm_switch (GUID 0x%04"PRIx64") " + "not in our fabric description\n", + cl_ntoh64(guid)); + return; + } + } + + /* + * If osm_phys_port is a switch port that connects to a CA, then + * we're using at most VL 0 (for QoS level 0) and VL 1 (for QoS + * level 1). We've been passed the VLarb values for a switch + * external port, so we need to fix them up to avoid unexpected + * results depending on how the switch handles VLarb values for + * unprogrammed VLs. + * + * For inter-switch links torus-2QoS uses VLs 0-3 to implement + * QoS level 0, and VLs 4-7 to implement QoS level 1. + * + * So, leave VL 0 alone, remap VL 4 to VL 1, zero out the rest, + * and compress out the zero entries to the end. + */ + if (!sw || !port_num || + sw->port[port_num]->pgrp->port_grp != 2 * TORUS_MAX_DIM) + return; + + next = 0; + for (i = 0; i < block_length; i++) { + switch (block->vl_entry[i].vl) { + case 4: + block->vl_entry[i].vl = 1; + /* fall through */ + case 0: + block->vl_entry[next].vl = block->vl_entry[i].vl; + block->vl_entry[next].weight = block->vl_entry[i].weight; + next++; + /* + * If we didn't update vl_entry[i] in place, + * fall through to zero it out. + */ + if (next > i) + break; + default: + block->vl_entry[i].vl = 0; + block->vl_entry[i].weight = 0; + break; + } + } +} + /* * Computes the path lengths *vl0_len and *vl1_len to get from src * to dst on a ring with count switches. @@ -9219,6 +9300,9 @@ void check_qos_config(osm_qos_options_t *opt, bool tgt_is_default, "Warning: full torus-2QoS functionality not available " "for configured %s_max_vls = %d\n", str, opt->max_vls); + if (!strcmp(str, "qos_ca")) + goto check_sl2vl; + if (opt->vlarb_high) { is_default = false; vlarb_str = opt->vlarb_high; @@ -9244,6 +9328,7 @@ void check_qos_config(osm_qos_options_t *opt, bool tgt_is_default, if (!is_default || tgt_is_default) check_vlarb_config(vlarb_str, is_default, str, "low", log); +check_sl2vl: if (opt->sl2vl) OSM_LOG(log, OSM_LOG_INFO, "Warning: torus-2QoS must override configured " @@ -9354,6 +9439,7 @@ int osm_ucast_torus2QoS_setup(struct osm_routing_engine *r, r->context = ctx; r->ucast_build_fwd_tables = torus_build_lfts; r->update_sl2vl = torus_update_osm_sl2vl; + r->update_vlarb = torus_update_osm_vlarb; r->path_sl = torus_path_sl; r->mcast_build_stree = torus_mcast_stree; r->destroy = torus_context_delete;