From patchwork Thu Sep 18 10:38:33 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sreedhar Kodali X-Patchwork-Id: 4930351 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id E5AA49F350 for ; Thu, 18 Sep 2014 10:37:55 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id A3CC3201B9 for ; Thu, 18 Sep 2014 10:38:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 584102017D for ; Thu, 18 Sep 2014 10:38:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756735AbaIRKih (ORCPT ); Thu, 18 Sep 2014 06:38:37 -0400 Received: from e9.ny.us.ibm.com ([32.97.182.139]:49181 "EHLO e9.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755755AbaIRKih (ORCPT ); Thu, 18 Sep 2014 06:38:37 -0400 Received: from /spool/local by e9.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 18 Sep 2014 06:38:36 -0400 Received: from d01dlp02.pok.ibm.com (9.56.250.167) by e9.ny.us.ibm.com (192.168.1.109) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Thu, 18 Sep 2014 06:38:34 -0400 Received: from b01cxnp23034.gho.pok.ibm.com (b01cxnp23034.gho.pok.ibm.com [9.57.198.29]) by d01dlp02.pok.ibm.com (Postfix) with ESMTP id 086D56E8029 for ; Thu, 18 Sep 2014 06:38:22 -0400 (EDT) Received: from d01av01.pok.ibm.com (d01av01.pok.ibm.com [9.56.224.215]) by b01cxnp23034.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id s8IAcYMI63439098 for ; Thu, 18 Sep 2014 10:38:34 GMT Received: from d01av01.pok.ibm.com (localhost [127.0.0.1]) by d01av01.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id s8IAcXiJ011697 for ; Thu, 18 Sep 2014 06:38:33 -0400 Received: from ltcweb.rtp.raleigh.ibm.com (ltcweb.rtp.raleigh.ibm.com [9.37.210.204]) by d01av01.pok.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id s8IAcX4U011688; Thu, 18 Sep 2014 06:38:33 -0400 Received: from ltc.linux.ibm.com (localhost.localdomain [127.0.0.1]) by ltcweb.rtp.raleigh.ibm.com (Postfix) with ESMTP id 3F014C0103; Thu, 18 Sep 2014 06:38:33 -0400 (EDT) MIME-Version: 1.0 Date: Thu, 18 Sep 2014 16:08:33 +0530 From: Sreedhar Kodali To: sean.hefty@intel.com, sean.hefty@intel.com Cc: linux-rdma@vger.kernel.org, pradeeps@linux.vnet.ibm.com Subject: [PATCH v5 2/4] rsockets: acknowledge completion queue events in batch Message-ID: <18bdfd1d731439cbe604263b37379cb6@imap.linux.ibm.com> X-Sender: srkodali@linux.vnet.ibm.com User-Agent: Roundcube Webmail/1.0.1 X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 14091810-7182-0000-0000-0000007C3F0B Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-7.5 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, T_TVD_MIME_EPI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Note: This patch is being resent as suggested by Sean Also attached the patch file for convenience From: Sreedhar Kodali Date: Thu Sep 18 14:29:48 2014 +0530 Perform completion event acknowledgments in batches instead of individually to minimze locking overheads. Size of the completion queue decides the size of a batch. Signed-off-by: Sreedhar Kodali --- fastlock_init(&rs->cq_lock); @@ -990,8 +992,11 @@ static void rs_free(struct rsocket *rs) if (rs->cm_id) { rs_free_iomappings(rs); - if (rs->cm_id->qp) + if (rs->cm_id->qp) { + if (rs->unack_cqe > 0) + ibv_ack_cq_events(rs->cm_id->recv_cq, rs->unack_cqe); rdma_destroy_qp(rs->cm_id); + } rdma_destroy_id(rs->cm_id); } @@ -1965,7 +1970,11 @@ static int rs_get_cq_event(struct rsocket *rs) ret = ibv_get_cq_event(rs->cm_id->recv_cq_channel, &cq, &context); if (!ret) { - ibv_ack_cq_events(rs->cm_id->recv_cq, 1); + rs->unack_cqe += 1; + if (rs->unack_cqe == rs->sq_size + rs->rq_size) { + ibv_ack_cq_events(rs->cm_id->recv_cq, rs->unack_cqe); + rs->unack_cqe = 0; + } rs->cq_armed = 0; } else if (errno != EAGAIN) { rs->state = rs_error; commit 490a2aa74e7dcdefe684e850d65b68bde3516ec4 Author: Sreedhar Kodali Date: Thu Sep 18 14:29:48 2014 +0530 Perform completion event acknowledgments in batches instead of individually to minimze locking overheads. Size of the completion queue decides the size of a batch. Signed-off-by: Sreedhar Kodali --- diff --git a/src/rsocket.c b/src/rsocket.c index 00d9a18..39c35cf 100644 --- a/src/rsocket.c +++ b/src/rsocket.c @@ -381,6 +381,7 @@ struct rsocket { dlist_entry iomap_list; dlist_entry iomap_queue; int iomap_pending; + int unack_cqe; }; #define DS_UDP_TAG 0x55555555 @@ -599,6 +600,7 @@ static struct rsocket *rs_alloc(struct rsocket *inherited_rs, int type) rs->target_iomap_size = def_iomap_size; } } + rs->unack_cqe = 0; fastlock_init(&rs->slock); fastlock_init(&rs->rlock); fastlock_init(&rs->cq_lock); @@ -990,8 +992,11 @@ static void rs_free(struct rsocket *rs) if (rs->cm_id) { rs_free_iomappings(rs); - if (rs->cm_id->qp) + if (rs->cm_id->qp) { + if (rs->unack_cqe > 0) + ibv_ack_cq_events(rs->cm_id->recv_cq, rs->unack_cqe); rdma_destroy_qp(rs->cm_id); + } rdma_destroy_id(rs->cm_id); } @@ -1965,7 +1970,11 @@ static int rs_get_cq_event(struct rsocket *rs) ret = ibv_get_cq_event(rs->cm_id->recv_cq_channel, &cq, &context); if (!ret) { - ibv_ack_cq_events(rs->cm_id->recv_cq, 1); + rs->unack_cqe += 1; + if (rs->unack_cqe == rs->sq_size + rs->rq_size) { + ibv_ack_cq_events(rs->cm_id->recv_cq, rs->unack_cqe); + rs->unack_cqe = 0; + } rs->cq_armed = 0; } else if (errno != EAGAIN) { rs->state = rs_error; diff --git a/src/rsocket.c b/src/rsocket.c index 00d9a18..39c35cf 100644 --- a/src/rsocket.c +++ b/src/rsocket.c @@ -381,6 +381,7 @@ struct rsocket { dlist_entry iomap_list; dlist_entry iomap_queue; int iomap_pending; + int unack_cqe; }; #define DS_UDP_TAG 0x55555555 @@ -599,6 +600,7 @@ static struct rsocket *rs_alloc(struct rsocket *inherited_rs, int type) rs->target_iomap_size = def_iomap_size; } } + rs->unack_cqe = 0; fastlock_init(&rs->slock); fastlock_init(&rs->rlock);