From patchwork Fri Sep 5 13:18:43 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sreedhar Kodali X-Patchwork-Id: 4852051 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 8B7DB9F2EC for ; Fri, 5 Sep 2014 13:18:54 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 59C26201FB for ; Fri, 5 Sep 2014 13:18:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8E964201F7 for ; Fri, 5 Sep 2014 13:18:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932296AbaIENSr (ORCPT ); Fri, 5 Sep 2014 09:18:47 -0400 Received: from e34.co.us.ibm.com ([32.97.110.152]:34934 "EHLO e34.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932184AbaIENSr (ORCPT ); Fri, 5 Sep 2014 09:18:47 -0400 Received: from /spool/local by e34.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 5 Sep 2014 07:18:46 -0600 Received: from d03dlp03.boulder.ibm.com (9.17.202.179) by e34.co.us.ibm.com (192.168.1.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 5 Sep 2014 07:18:45 -0600 Received: from b03cxnp08028.gho.boulder.ibm.com (b03cxnp08028.gho.boulder.ibm.com [9.17.130.20]) by d03dlp03.boulder.ibm.com (Postfix) with ESMTP id C05DD19D8039 for ; Fri, 5 Sep 2014 07:18:31 -0600 (MDT) Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by b03cxnp08028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id s85DIipR18481384 for ; Fri, 5 Sep 2014 15:18:44 +0200 Received: from d03av02.boulder.ibm.com (localhost [127.0.0.1]) by d03av02.boulder.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id s85DIi2q030707 for ; Fri, 5 Sep 2014 07:18:44 -0600 Received: from ltcweb.rtp.raleigh.ibm.com (ltcweb.rtp.raleigh.ibm.com [9.37.210.204]) by d03av02.boulder.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id s85DIhm8030677; Fri, 5 Sep 2014 07:18:43 -0600 Received: from ltc.linux.ibm.com (localhost.localdomain [127.0.0.1]) by ltcweb.rtp.raleigh.ibm.com (Postfix) with ESMTP id 9E328C0103; Fri, 5 Sep 2014 09:18:43 -0400 (EDT) MIME-Version: 1.0 Date: Fri, 05 Sep 2014 18:48:43 +0530 From: Sreedhar Kodali To: linux-rdma@vger.kernel.org Cc: sean.hefty@intel.com, pradeeps@linux.vnet.ibm.com Subject: [PATCH v2 3/4] rsockets: distribute completion queue vectors among multiple cores Message-ID: X-Sender: srkodali@linux.vnet.ibm.com User-Agent: Roundcube Webmail/1.0.1 X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 14090513-1542-0000-0000-0000048F0B8D Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-8.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Sreedhar Kodali Distribute interrupt vectors among multiple cores while processing completion events. By default the existing mechanism always defaults to core 0 for comp vector processing during the creation of a completion queue. If the workload is very high, then this results in bottleneck at core 0 because the same core is used for both event and task processing. A '/comp_vector' option is exposed, the value of which is a range or comma separated list of cores for distributing interrupt vectors. If not set, the existing mechanism prevails where in comp vector processing is directed to core 0. Signed-off-by: Sreedhar Kodali --- --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/src/rsocket.c b/src/rsocket.c index b70d56a..ffea0ca 100644 --- a/src/rsocket.c +++ b/src/rsocket.c @@ -116,6 +116,8 @@ static uint32_t def_mem = (1 << 17); static uint32_t def_wmem = (1 << 17); static uint32_t polling_time = 10; static uint16_t restart_onintr = 0; +static uint16_t next_comp_vector = 0; +static uint64_t comp_vector_mask = 0; /* * Immediate data format is determined by the upper bits @@ -548,6 +550,37 @@ void rs_configure(void) (void) fscanf(f, "%hu", &restart_onintr); fclose(f); } + + if ((f = fopen(RS_CONF_DIR "/comp_vector", "r"))) { + char vbuf[256]; + char *vptr; + vptr = fgets(vbuf, sizeof(vbuf), f); + fclose(f); + if (vptr) { + char *tok, *save, *tmp, *str, *tok2; + int lvect, uvect, vect; + + for (str = vptr; ; str = NULL) { + tok = strtok_r(str, ",", &save); + if (tok == NULL) { + break; + } + if (!(tmp = strpbrk(tok, "-"))) { + lvect = uvect = atoi(tok); + } else { + tok2 = tmp + 1; + *tmp = '\0'; + lvect = atoi(tok); + uvect = atoi(tok2); + } + lvect = (lvect < 0) ? 0 : ((lvect > 63) ? 63 : lvect); + uvect = (uvect < 0) ? 0 : ((uvect > 63) ? 63 : uvect); + for (vect = lvect; vect <= uvect; vect++) { + comp_vector_mask |= ((uint64_t)1 << vect); + } + } + } + } init = 1; out: pthread_mutex_unlock(&mut); @@ -762,12 +795,27 @@ static int ds_init_bufs(struct ds_qp *qp) */ static int rs_create_cq(struct rsocket *rs, struct rdma_cm_id *cm_id) { + int vector = 0; + cm_id->recv_cq_channel = ibv_create_comp_channel(cm_id->verbs); if (!cm_id->recv_cq_channel) return -1; + if (comp_vector_mask) { + int found = 0; + while (found == 0) { + if (comp_vector_mask & ((uint64_t) 1 << next_comp_vector)) { + found = 1; + vector = next_comp_vector; + } + if (++next_comp_vector == 64) { + next_comp_vector = 0; + } + } + } + cm_id->recv_cq = ibv_create_cq(cm_id->verbs, rs->sq_size + rs->rq_size, - cm_id, cm_id->recv_cq_channel, 0); + cm_id, cm_id->recv_cq_channel, vector); if (!cm_id->recv_cq) goto err1;