From patchwork Thu Feb 5 12:28:33 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Majd Dibbiny X-Patchwork-Id: 5784221 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id DB25F9F30C for ; Thu, 5 Feb 2015 12:28:56 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 178A820268 for ; Thu, 5 Feb 2015 12:28:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DAA2320270 for ; Thu, 5 Feb 2015 12:28:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757091AbbBEM2x (ORCPT ); Thu, 5 Feb 2015 07:28:53 -0500 Received: from mailp.voltaire.com ([193.47.165.129]:60167 "EHLO mellanox.co.il" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1757244AbbBEM2x (ORCPT ); Thu, 5 Feb 2015 07:28:53 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from majd@mellanox.com) with ESMTPS (AES256-SHA encrypted); 5 Feb 2015 14:28:34 +0200 Received: from dev-l-vrt-196.mtl.labs.mlnx (dev-l-vrt-196.mtl.labs.mlnx [10.134.196.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id t15CSX1q001448; Thu, 5 Feb 2015 14:28:34 +0200 From: Majd Dibbiny To: Yishaih@mellanox.com Cc: majd@mellanox.com, Roland@kernel.org, linux-rdma@vger.kernel.org Subject: [PATCH libmlx4 2/2] Implement Send with invalidate Date: Thu, 5 Feb 2015 14:28:33 +0200 Message-Id: <1423139313-17418-3-git-send-email-majd@mellanox.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1423139313-17418-1-git-send-email-majd@mellanox.com> References: <1423139313-17418-1-git-send-email-majd@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Send with invalidate can be used to invalidate a remote memory region or memory window. The complete details are specified in the InfiniBand Architecture Specifications, section 9.4.1.1. According to the specs, the following objects can be invalidated: 1. Memory Window type 2 2. Physical MR 3. Fast register MR The invalidation is done by posting a send work request, where the opcode is IBV_WR_SEND_WITH_INV and the immediate data contain's the R_key. Upon success, the responder's work completion will contain the invalidated R_key in it's immediate data. Signed-off-by: Majd Dibbiny --- src/cq.c | 8 ++++++++ src/qp.c | 4 ++++ 2 files changed, 12 insertions(+), 0 deletions(-) diff --git a/src/cq.c b/src/cq.c index 77eb6c2..c52e6c7 100644 --- a/src/cq.c +++ b/src/cq.c @@ -294,6 +294,9 @@ static int mlx4_poll_one(struct mlx4_cq *cq, case MLX4_OPCODE_BIND_MW: wc->opcode = IBV_WC_BIND_MW; break; + case MLX4_OPCODE_SEND_INVAL: + wc->opcode = IBV_WC_SEND; + break; default: /* assume it's a send completion */ wc->opcode = IBV_WC_SEND; @@ -308,6 +311,11 @@ static int mlx4_poll_one(struct mlx4_cq *cq, wc->wc_flags = IBV_WC_WITH_IMM; wc->imm_data = cqe->immed_rss_invalid; break; + case MLX4_RECV_OPCODE_SEND_INVAL: + wc->opcode = IBV_WC_RECV; + wc->wc_flags |= IBV_WC_WITH_INV; + wc->imm_data = ntohl(cqe->immed_rss_invalid); + break; case MLX4_RECV_OPCODE_SEND: wc->opcode = IBV_WC_RECV; wc->wc_flags = 0; diff --git a/src/qp.c b/src/qp.c index 80a21da..2612748 100644 --- a/src/qp.c +++ b/src/qp.c @@ -56,6 +56,7 @@ static const uint32_t mlx4_ib_opcode[] = { [IBV_WR_ATOMIC_FETCH_AND_ADD] = MLX4_OPCODE_ATOMIC_FA, [IBV_WR_LOCAL_INV] = MLX4_OPCODE_LOCAL_INVAL, [IBV_WR_BIND_MW] = MLX4_OPCODE_BIND_MW, + [IBV_WR_SEND_WITH_INV] = MLX4_OPCODE_SEND_INVAL, }; static void *get_recv_wqe(struct mlx4_qp *qp, int n) @@ -332,6 +333,9 @@ int mlx4_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr, size += sizeof (struct mlx4_wqe_bind_seg) / 16; break; + case IBV_WR_SEND_WITH_INV: + ctrl->imm = htonl(wr->imm_data); + break; default: /* No extra segments required for sends */