From patchwork Fri Sep 5 13:23:11 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sreedhar Kodali X-Patchwork-Id: 4852071 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id CEF68C0338 for ; Fri, 5 Sep 2014 13:23:28 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 35E1D20200 for ; Fri, 5 Sep 2014 13:23:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3CEA4201FB for ; Fri, 5 Sep 2014 13:23:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754050AbaIENXP (ORCPT ); Fri, 5 Sep 2014 09:23:15 -0400 Received: from e33.co.us.ibm.com ([32.97.110.151]:42573 "EHLO e33.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751276AbaIENXP (ORCPT ); Fri, 5 Sep 2014 09:23:15 -0400 Received: from /spool/local by e33.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 5 Sep 2014 07:23:14 -0600 Received: from d03dlp02.boulder.ibm.com (9.17.202.178) by e33.co.us.ibm.com (192.168.1.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 5 Sep 2014 07:23:13 -0600 Received: from b03cxnp08028.gho.boulder.ibm.com (b03cxnp08028.gho.boulder.ibm.com [9.17.130.20]) by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id DBF9E3E40048 for ; Fri, 5 Sep 2014 07:23:12 -0600 (MDT) Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by b03cxnp08028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id s85DNCak28180546 for ; Fri, 5 Sep 2014 15:23:12 +0200 Received: from d03av02.boulder.ibm.com (localhost [127.0.0.1]) by d03av02.boulder.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id s85DNC7C018766 for ; Fri, 5 Sep 2014 07:23:12 -0600 Received: from ltcweb.rtp.raleigh.ibm.com (ltcweb.rtp.raleigh.ibm.com [9.37.210.204]) by d03av02.boulder.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id s85DNB72018606; Fri, 5 Sep 2014 07:23:11 -0600 Received: from ltc.linux.ibm.com (localhost.localdomain [127.0.0.1]) by ltcweb.rtp.raleigh.ibm.com (Postfix) with ESMTP id 27A03C0103; Fri, 5 Sep 2014 09:23:11 -0400 (EDT) MIME-Version: 1.0 Date: Fri, 05 Sep 2014 18:53:11 +0530 From: Sreedhar Kodali To: linux-rdma@vger.kernel.org Cc: sean.hefty@intel.com, pradeeps@linux.vnet.ibm.com Subject: [PATCH v2 4/4] rsockets: acknowledge completion queue events in batch Message-ID: X-Sender: srkodali@linux.vnet.ibm.com User-Agent: Roundcube Webmail/1.0.1 X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 14090513-0928-0000-0000-00000497A46C Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-8.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Sreedhar Kodali Expose '/unackcqe_default' variable so completion event acknowledgment can be done in batch instead of singles to minimize locking overheads. The default value of this variable is 1 so the existing functionality is preserved. Having a higher value delays acknowledging until the specified limit is reached for retrieved completion events. Signed-off-by: Sreedhar Kodali --- fastlock_init(&rs->cq_lock); @@ -1044,8 +1055,11 @@ static void rs_free(struct rsocket *rs) if (rs->cm_id) { rs_free_iomappings(rs); - if (rs->cm_id->qp) + if (rs->cm_id->qp) { + if (rs->unackcqe > 0) + ibv_ack_cq_events(rs->cm_id->recv_cq, rs->unackcqe); rdma_destroy_qp(rs->cm_id); + } rdma_destroy_id(rs->cm_id); } @@ -2026,7 +2040,11 @@ static int rs_get_cq_event(struct rsocket *rs) resume_get_cq_event: ret = ibv_get_cq_event(rs->cm_id->recv_cq_channel, &cq, &context); if (!ret) { - ibv_ack_cq_events(rs->cm_id->recv_cq, 1); + rs->unackcqe += 1; + if (rs->unackcqe == def_unackcqe) { + ibv_ack_cq_events(rs->cm_id->recv_cq, rs->unackcqe); + rs->unackcqe = 0; + } rs->cq_armed = 0; } else if (restart_onintr == 1 && errno == EINTR) { errno = 0; --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/src/rsocket.c b/src/rsocket.c index ffea0ca..3c39da7 100644 --- a/src/rsocket.c +++ b/src/rsocket.c @@ -118,6 +118,7 @@ static uint32_t polling_time = 10; static uint16_t restart_onintr = 0; static uint16_t next_comp_vector = 0; static uint64_t comp_vector_mask = 0; +static uint32_t def_unackcqe = 1; /* * Immediate data format is determined by the upper bits @@ -384,6 +385,7 @@ struct rsocket { dlist_entry iomap_list; dlist_entry iomap_queue; int iomap_pending; + int unackcqe; }; #define DS_UDP_TAG 0x55555555 @@ -581,6 +583,14 @@ void rs_configure(void) } } } + + if ((f = fopen(RS_CONF_DIR "/unackcqe_default", "r"))) { + (void) fscanf(f, "%u", &def_unackcqe); + fclose(f); + if (def_unackcqe < 1) { + def_unackcqe = 1; + } + } init = 1; out: pthread_mutex_unlock(&mut); @@ -638,6 +648,7 @@ static struct rsocket *rs_alloc(struct rsocket *inherited_rs, int type) rs->target_iomap_size = def_iomap_size; } } + rs->unackcqe = 0; fastlock_init(&rs->slock); fastlock_init(&rs->rlock);