From patchwork Sat Jan 16 15:53:42 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 8049221 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id D2924BEEE5 for ; Sat, 16 Jan 2016 15:56:45 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id D878020361 for ; Sat, 16 Jan 2016 15:56:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D5AA420357 for ; Sat, 16 Jan 2016 15:56:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752151AbcAPP4h (ORCPT ); Sat, 16 Jan 2016 10:56:37 -0500 Received: from mail-wm0-f52.google.com ([74.125.82.52]:35402 "EHLO mail-wm0-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751425AbcAPP41 (ORCPT ); Sat, 16 Jan 2016 10:56:27 -0500 Received: by mail-wm0-f52.google.com with SMTP id r129so4873927wmr.0 for ; Sat, 16 Jan 2016 07:56:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leon-nu.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; bh=wp+yeXwmV3/6y3vggnYMkX4C0vJ0JHw4Emrvyv/lbZ4=; b=BYN7M6alf5r/tlaEhjZ75ROmwC4mAal2Lu1Nk4re1hjgkSlMk82NLHKBthPkQpYINY La4TM2trh+wZE9FFFlPHze3zS88RLyUH6t7P1+LdAJcRlTJuTGxkzpyzylXh2MkHPU4L FmGHWktQowMvkEKfckdezjFbGgVVBxNIc/hmcy23+9Y0/AxTwtNi9kunFmKsc7IOwQU8 GeFPPmyMSObyF6gnimm3gmi2agdQdNspavSFzaRz819OTtr4xj1HRCrMW3a5lfM2q8Ef EbK7HEKZ3iXK/i37YF1AJQnQgeNYyT322SRD51c8VCLY/7ikpHyqbmjT6dEdiX9gJPMt uqDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-type:content-transfer-encoding; bh=wp+yeXwmV3/6y3vggnYMkX4C0vJ0JHw4Emrvyv/lbZ4=; b=JyQGxaxnZeGWQaOYUHIdWZYo+qNs+5y9jT29E+dFJeyT1GsOpQoa4HXYDcfir/6Atq fryACFhowLxJUj4byoD7qLcX3m6UCpmNVN3RGNwWJBUzIZvC/E6Z4FaHGrdc3NIl1m14 H2HWDuoogIi+GuKI0phm4In8HvOmii9k+o/hqxJdyAkHQn+afZhVx34QRlQyrrs7YbVV BlInl9FqJWZa34ztAe9iWNZXyToc2CpgCx/uzTEohrv5PGk6Xs+/gvCxR4sNLrM3UL96 gn1eJ99DuzH4VAjUW2I8aRZDd6I4F8ka+6QtidhCpvA4a0l7W8ulWdp+aLQlea0tj1BQ iV9Q== X-Gm-Message-State: ALoCoQkbkM7iU/Emdo8OpDJ3M8NK2PpOgt7NMqVM2GXB514/yMt7b6yOnAj9lF3lRba6PVLTWCI6N38YarPllnuwMIESwwc+yQ== X-Received: by 10.194.80.200 with SMTP id t8mr15543009wjx.74.1452959786566; Sat, 16 Jan 2016 07:56:26 -0800 (PST) Received: from localhost ([213.57.247.249]) by smtp.gmail.com with ESMTPSA id x6sm15430488wje.38.2016.01.16.07.56.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 16 Jan 2016 07:56:25 -0800 (PST) From: Leon Romanovsky To: dledford@redhat.com Cc: linux-rdma@vger.kernel.org, Leon Romanovsky Subject: [PATCH libibverbs V1 3/5] Add cross-channel work requests primitives Date: Sat, 16 Jan 2016 17:53:42 +0200 Message-Id: <1452959624-29454-4-git-send-email-leon@leon.nu> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1452959624-29454-1-git-send-email-leon@leon.nu> References: <1452959624-29454-1-git-send-email-leon@leon.nu> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Leon Romanovsky The cross-channel feature relies on special primitives to send and receive work requests. * CQE wait - This work request holds execution of subsequent requests on that queue until this number of completions of a CQ is met. • WQE enable - This work request specifies value of work requests on the controlled send/receive queue. It enables the execution of all WQEs up to the work request which is marked by IBV_SEND_WAIT_EN_LAST. Signed-off-by: Leon Romanovsky Reviewed-by: Sagi Grimberg --- include/infiniband/verbs.h | 16 ++++++++++++++-- man/ibv_post_send.3 | 25 +++++++++++++++++++++++-- 2 files changed, 37 insertions(+), 4 deletions(-) diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h index 7e85de08bfc9..d91dd8a1376e 100644 --- a/include/infiniband/verbs.h +++ b/include/infiniband/verbs.h @@ -884,7 +884,10 @@ enum ibv_wr_opcode { IBV_WR_SEND_WITH_IMM, IBV_WR_RDMA_READ, IBV_WR_ATOMIC_CMP_AND_SWP, - IBV_WR_ATOMIC_FETCH_AND_ADD + IBV_WR_ATOMIC_FETCH_AND_ADD, + IBV_WR_SEND_ENABLE, + IBV_WR_RECV_ENABLE, + IBV_WR_CQE_WAIT }; enum ibv_send_flags { @@ -892,7 +895,8 @@ enum ibv_send_flags { IBV_SEND_SIGNALED = 1 << 1, IBV_SEND_SOLICITED = 1 << 2, IBV_SEND_INLINE = 1 << 3, - IBV_SEND_IP_CSUM = 1 << 4 + IBV_SEND_IP_CSUM = 1 << 4, + IBV_SEND_WAIT_EN_LAST = 1 << 5 }; struct ibv_sge { @@ -925,6 +929,14 @@ struct ibv_send_wr { uint32_t remote_qpn; uint32_t remote_qkey; } ud; + struct { + struct ibv_cq *cq; + int32_t cq_count; + } cqe_wait; + struct { + struct ibv_qp *qp; + int32_t wqe_count; + } wqe_enable; } wr; union { struct { diff --git a/man/ibv_post_send.3 b/man/ibv_post_send.3 index eeea0787fa80..28dd43bfa3d9 100644 --- a/man/ibv_post_send.3 +++ b/man/ibv_post_send.3 @@ -1,6 +1,6 @@ .\" -*- nroff -*- .\" -.TH IBV_POST_SEND 3 2006-10-31 libibverbs "Libibverbs Programmer's Manual" +.TH IBV_POST_SEND 3 2015-12-26 libibverbs "Libibverbs Programmer's Manual" .SH "NAME" ibv_post_send \- post a list of work requests (WRs) to a send queue .SH "SYNOPSIS" @@ -58,6 +58,18 @@ uint32_t remote_qpn; /* QP number of the destination QP */ uint32_t remote_qkey; /* Q_Key number of the destination QP */ .in -8 } ud; +struct { +.in +8 +struct ibv_cq *cq; /* CQ for the wait on */ +int32_t cq_count; /* Count of completions of a CQ to met */ +.in -8 +} cqe_wait; +struct { +.in +8 +struct ibv_qp *qp; /* QP for the release */ +int32_t wqe_count; /* Number of work requests to release */ +.in -8 +} wqe_enable; .in -8 } wr; .in -8 @@ -85,6 +97,9 @@ IBV_WR_RDMA_WRITE_WITH_IMM | | X | X IBV_WR_RDMA_READ | | | X IBV_WR_ATOMIC_CMP_AND_SWP | | | X IBV_WR_ATOMIC_FETCH_AND_ADD | | | X +IBV_WR_SEND_ENABLE | | | X +IBV_WR_RECV_ENABLE | | | X +IBV_WR_CQE_WAIT | | | X .fi .PP The attribute send_flags describes the properties of the \s-1WR\s0. It is either 0 or the bitwise \s-1OR\s0 of one or more of the following flags: @@ -102,6 +117,8 @@ in a send WQE. Valid only for Send and RDMA Write. The L_Key will not be check .B IBV_SEND_IP_CSUM \fR Offload the IPv4 and TCP/UDP checksum calculation. Valid only when \fBdevice_cap_flags\fR in device_attr indicates current QP is supported by checksum offload. +.TP +.B IBV_SEND_WAIT_EN_LAST \fR Mark this work request as a last one in the cross-channel offloaded sequence. .SH "RETURN VALUE" .B ibv_post_send() returns 0 on success, or the value of errno on failure (which indicates the failure reason). @@ -124,4 +141,8 @@ after the call returns. .BR ibv_poll_cq (3) .SH "AUTHORS" .TP -Dotan Barak +Dotan Barak +.RI < dotanba@gmail.com > +.TP +Leon Romanovsky +.RI < leonro@mellanox.com >