From patchwork Mon Sep 8 12:48:26 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sreedhar Kodali X-Patchwork-Id: 4861741 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 73E54C0338 for ; Mon, 8 Sep 2014 12:48:39 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 690C920108 for ; Mon, 8 Sep 2014 12:48:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E58BF200E5 for ; Mon, 8 Sep 2014 12:48:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753033AbaIHMsa (ORCPT ); Mon, 8 Sep 2014 08:48:30 -0400 Received: from e35.co.us.ibm.com ([32.97.110.153]:47124 "EHLO e35.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752665AbaIHMsa (ORCPT ); Mon, 8 Sep 2014 08:48:30 -0400 Received: from /spool/local by e35.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 8 Sep 2014 06:48:29 -0600 Received: from d03dlp03.boulder.ibm.com (9.17.202.179) by e35.co.us.ibm.com (192.168.1.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Mon, 8 Sep 2014 06:48:28 -0600 Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by d03dlp03.boulder.ibm.com (Postfix) with ESMTP id 752BB19D8039 for ; Mon, 8 Sep 2014 06:48:14 -0600 (MDT) Received: from b03eismtpcl100.ahe.boulder.ibm.com (b03eismtpcl100.ahe.boulder.ibm.com [9.17.195.167]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id s88CmRED27918372 for ; Mon, 8 Sep 2014 14:48:27 +0200 Received: from b03eismtpcl100.ahe.boulder.ibm.com (localhost [127.0.0.1]) by b03eismtpcl100.ahe.boulder.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id s88CmQAF001557 for ; Mon, 8 Sep 2014 06:48:27 -0600 Received: from ltcweb.rtp.raleigh.ibm.com (ltcweb.rtp.raleigh.ibm.com [9.37.210.204]) by b03eismtpcl100.ahe.boulder.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id s88CmQkk001479; Mon, 8 Sep 2014 06:48:26 -0600 Received: from ltc.linux.ibm.com (localhost.localdomain [127.0.0.1]) by ltcweb.rtp.raleigh.ibm.com (Postfix) with ESMTP id 3B3EDC0103; Mon, 8 Sep 2014 08:48:26 -0400 (EDT) MIME-Version: 1.0 Date: Mon, 08 Sep 2014 18:18:26 +0530 From: Sreedhar Kodali To: linux-rdma@vger.kernel.org Cc: sean.hefty@intel.com, pradeeps@linux.vnet.ibm.com, sithirug@linux.vnet.ibm.com Subject: [PATCH v3 3/4] rsockets: distribute completion queue vectors Message-ID: <33dc77c8c6006b5ac067e6c6f485df60@imap.linux.ibm.com> X-Sender: srkodali@linux.vnet.ibm.com User-Agent: Roundcube Webmail/1.0.1 X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 14090812-6688-0000-0000-000004975078 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-9.4 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Sreedhar Kodali Change note from v2 to v3: description is changed as suggested by Bart Date: Mon Sep 8 14:32:18 2014 +0530 Distribute completion vectors while creating completion queues. The existing mechanism always uses 0 for the completion vector. Mapping of a completion vector to a particular CPU core is decided by the smp_affinity factor that is set at the system level for the corresponding irq number. While driving a large workload this may result in bottleneck at the mapped core because the same core could be used for both event and task processing. A '/comp_vector' option is exposed, the value of which is a range or comma separated list of completion vectors. The specified completion vectors are equally distributed among the created completion queues with each new connection picking up the next indexed completion vector in the list with index wrapping up to the beginning if the list end is reached. If this option is not set, the existing mechanism prevails where in completion vectors are set to 0 for all the connections. Signed-off-by: Sreedhar Kodali --- --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/src/rsocket.c b/src/rsocket.c index b70d56a..ffea0ca 100644 --- a/src/rsocket.c +++ b/src/rsocket.c @@ -116,6 +116,8 @@ static uint32_t def_mem = (1 << 17); static uint32_t def_wmem = (1 << 17); static uint32_t polling_time = 10; static uint16_t restart_onintr = 0; +static uint16_t next_comp_vector = 0; +static uint64_t comp_vector_mask = 0; /* * Immediate data format is determined by the upper bits @@ -548,6 +550,37 @@ void rs_configure(void) (void) fscanf(f, "%hu", &restart_onintr); fclose(f); } + + if ((f = fopen(RS_CONF_DIR "/comp_vector", "r"))) { + char vbuf[256]; + char *vptr; + vptr = fgets(vbuf, sizeof(vbuf), f); + fclose(f); + if (vptr) { + char *tok, *save, *tmp, *str, *tok2; + int lvect, uvect, vect; + + for (str = vptr; ; str = NULL) { + tok = strtok_r(str, ",", &save); + if (tok == NULL) { + break; + } + if (!(tmp = strpbrk(tok, "-"))) { + lvect = uvect = atoi(tok); + } else { + tok2 = tmp + 1; + *tmp = '\0'; + lvect = atoi(tok); + uvect = atoi(tok2); + } + lvect = (lvect < 0) ? 0 : ((lvect > 63) ? 63 : lvect); + uvect = (uvect < 0) ? 0 : ((uvect > 63) ? 63 : uvect); + for (vect = lvect; vect <= uvect; vect++) { + comp_vector_mask |= ((uint64_t)1 << vect); + } + } + } + } init = 1; out: pthread_mutex_unlock(&mut); @@ -762,12 +795,27 @@ static int ds_init_bufs(struct ds_qp *qp) */ static int rs_create_cq(struct rsocket *rs, struct rdma_cm_id *cm_id) { + int vector = 0; + cm_id->recv_cq_channel = ibv_create_comp_channel(cm_id->verbs); if (!cm_id->recv_cq_channel) return -1; + if (comp_vector_mask) { + int found = 0; + while (found == 0) { + if (comp_vector_mask & ((uint64_t) 1 << next_comp_vector)) { + found = 1; + vector = next_comp_vector; + } + if (++next_comp_vector == 64) { + next_comp_vector = 0; + } + } + } + cm_id->recv_cq = ibv_create_cq(cm_id->verbs, rs->sq_size + rs->rq_size, - cm_id, cm_id->recv_cq_channel, 0); + cm_id, cm_id->recv_cq_channel, vector); if (!cm_id->recv_cq) goto err1;