From patchwork Fri Jun 21 21:48:31 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Coulter, Susan K" X-Patchwork-Id: 2764751 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id B7A4D9F756 for ; Fri, 21 Jun 2013 22:09:52 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id AAED5201D8 for ; Fri, 21 Jun 2013 22:09:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 72E73201D6 for ; Fri, 21 Jun 2013 22:09:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1946207Ab3FUWJt (ORCPT ); Fri, 21 Jun 2013 18:09:49 -0400 Received: from proofpoint4.lanl.gov ([204.121.3.52]:53195 "EHLO proofpoint4.lanl.gov" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1946172Ab3FUWJs convert rfc822-to-8bit (ORCPT ); Fri, 21 Jun 2013 18:09:48 -0400 X-Greylist: delayed 1275 seconds by postgrey-1.27 at vger.kernel.org; Fri, 21 Jun 2013 18:09:48 EDT Received: from mailrelay1.lanl.gov (mailrelay1.lanl.gov [128.165.4.101]) by proofpoint4.lanl.gov (8.14.4/8.14.4) with ESMTP id r5LLmWDj024596 for ; Fri, 21 Jun 2013 15:48:32 -0600 Received: from localhost (localhost.localdomain [127.0.0.1]) by mailrelay1.lanl.gov (Postfix) with ESMTP id 8F93414148CE for ; Fri, 21 Jun 2013 15:48:32 -0600 (MDT) X-NIE-2-Virus-Scanner: amavisd-new at mailrelay1.lanl.gov Received: from ECS-EXG-P-CH03.win.lanl.gov (ecs-exg-p-ch03.win.lanl.gov [128.165.106.13]) by mailrelay1.lanl.gov (Postfix) with ESMTP id 7651914148C2 for ; Fri, 21 Jun 2013 15:48:32 -0600 (MDT) Received: from ECS-EXG-P-MB01.win.lanl.gov ([169.254.1.239]) by ECS-EXG-P-CH03.win.lanl.gov ([128.165.106.13]) with mapi id 14.03.0123.003; Fri, 21 Jun 2013 15:48:32 -0600 From: "Coulter, Susan K" To: "linux-rdma (linux-rdma@vger.kernel.org)" Subject: partitions / ib_read_bw Thread-Topic: partitions / ib_read_bw Thread-Index: AQHObskSsAiErUAtZUmdbaQOs/KnPA== Date: Fri, 21 Jun 2013 21:48:31 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [128.165.184.250] Content-ID: <910178C59F69F84083C65395A57D339D@win.lanl.gov> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8794, 1.0.431, 0.0.0000 definitions=2013-06-21_07:2013-06-21, 2013-06-21, 1970-01-01 signatures=0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-8.4 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP One of my test clusters is running with multiple partitions. The partitions look correct (using "smpquery pkey") and the path records look correct (using "saquery pr"). I'd like a simple user-level test to validate who is allowed to talk to who; and hoped to use ib_read_bw. Alas, ib_read_bw does not honor partitions. Below is a patch I wrote to try and add pkey support to ib_read_bw. No joy. The server shows it is populating ibv_qp_attr with the pkey(s) of any full partition it is a member of, but the client does not get that same pkey in ibv_wc. What am I missing? ==================================== Susan Coulter HPC-3 Network/Infrastructure 505-667-8425 Increase the Peace... An eye for an eye leaves the whole world blind ==================================== --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html --- read_bw.c.orig 2013-01-29 12:27:20.000000000 -0700 +++ read_bw.c 2013-06-07 17:15:33.931392279 -0600 @@ -37,11 +37,13 @@ # include #endif /* HAVE_CONFIG_H */ +#include +#include +#include #include #include -#include #include -#include +#include #include "get_clock.h" #include "perftest_resources.h" @@ -53,14 +55,42 @@ cycles_t *tposted; cycles_t *tcompleted; +struct full_pkey_entry { + int idx; + struct full_pkey_entry *next; +}; +static struct full_pkey_entry *fpk_head; + /****************************************************************************** * ******************************************************************************/ static int pp_connect_ctx(struct pingpong_context *ctx,int my_psn,int my_out_reads, struct pingpong_dest *dest,struct perftest_parameters *user_parm) { + struct full_pkey_entry *p; struct ibv_qp_attr attr; memset(&attr, 0, sizeof attr); + + attr.pkey_index = 0; + attr.alt_pkey_index = 0; + p = fpk_head; + while (p != NULL) { + if (attr.pkey_index == 0) + attr.pkey_index = p->idx; + else if (attr.alt_pkey_index == 0) + attr.alt_pkey_index = p->idx; + p=p->next; + } + fprintf(stdout,"skc: pkey_index is %i\n", attr.pkey_index); + fprintf(stdout,"skc: alt_pkey_index is %i\n", attr.alt_pkey_index); + +/* check pkey value */ + + ret = ibv_query_pkey(ctx, user_parm->ib_port, attr.pkey_index, pkey_n); + if (!ret) { + pkey = ntohs(*pkey_n); + fprintf(stdout,"skc: pkey_index value is %u\n", pkey); + } attr.qp_state = IBV_QPS_RTR; attr.path_mtu = user_parm->curr_mtu; @@ -166,6 +196,8 @@ ALLOCATE(wc , struct ibv_wc , DEF_WC_SIZE); + fprintf(stdout,"skc: ibv_wc allocated\n"); + list.addr = (uintptr_t)ctx->buf; list.length = user_param->size; list.lkey = ctx->mr->lkey; @@ -219,6 +251,8 @@ if (ne > 0) { for (i = 0; i < ne; i++) { + fprintf(stdout,"skc: completion queue %i polled, pkey is %i\n", i, wc[i].pkey_index); + if (wc[i].status != IBV_WC_SUCCESS) NOTIFY_COMP_ERROR_SEND(wc[i],scnt,ccnt); @@ -243,12 +277,48 @@ return 0; } +int check_pkey(struct ibv_context *context, struct perftest_parameters *parms) { + + int idx,ret; + uint16_t *pkey_n,pkey; + struct ibv_port_attr pattr; + struct full_pkey_entry *fpk, *p; + + ret = ibv_query_port(context, parms->ib_port, &pattr); + + if(!ret) { + pkey_n = malloc(sizeof *pkey_n); + for (idx=0; idx < pattr.pkey_tbl_len; idx++) { + ret = ibv_query_pkey(context, parms->ib_port, idx, pkey_n); + if (!ret) { + pkey = ntohs(*pkey_n); + if (pkey > 32767) { + fprintf(stdout,"skc: full member pkey is %u at index %i\n", pkey, idx); + fpk = (struct full_pkey_entry *) malloc(sizeof(struct full_pkey_entry)); + fpk->idx = idx; + fpk->next = NULL; + if (fpk_head == NULL) { + fpk_head = fpk; + } else { + for (p=fpk_head; p->next != NULL; p=p->next); + p->next = fpk; + } + } + } else { + ret = EPERM; + } + } + } + return ret; +} + /****************************************************************************** * ******************************************************************************/ int main(int argc, char *argv[]) { int i = 0; + int ret = 0; struct ibv_device *ib_dev = NULL; struct pingpong_context ctx; struct pingpong_dest my_dest,rem_dest; @@ -280,6 +350,12 @@ return 1; } + ret = check_pkey(ctx.context, &user_param); + if (ret) { + fprintf(stderr, "PKey stuff borked\n"); + return FAILURE; + } + // See if MTU and link type are valid and supported. if (check_link_and_mtu(ctx.context,&user_param)) { fprintf(stderr, " Couldn't get context for the device\n"); @@ -362,7 +438,7 @@ if (ctx_hand_shake(&user_comm,&my_dest,&rem_dest)) { fprintf(stderr,"Failed to exchange date between server and clients\n"); return 1; - } + } // For half duplex tests, server just waits for client to exit if (user_param.machine == SERVER && !user_param.duplex) { @@ -390,6 +466,8 @@ ALLOCATE(tposted , cycles_t , user_param.iters); ALLOCATE(tcompleted , cycles_t , user_param.iters); + fprintf(stdout,"skc: ready to run iterations\n"); + if (user_param.all == ON) { for (i = 1; i < 24 ; ++i) {