From patchwork Thu Feb 16 19:23:02 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 9578159 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 47735600C5 for ; Thu, 16 Feb 2017 19:23:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3C0F32865E for ; Thu, 16 Feb 2017 19:23:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 310A028666; Thu, 16 Feb 2017 19:23:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B93562865E for ; Thu, 16 Feb 2017 19:23:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933040AbdBPTXX (ORCPT ); Thu, 16 Feb 2017 14:23:23 -0500 Received: from quartz.orcorp.ca ([184.70.90.242]:51063 "EHLO quartz.orcorp.ca" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932933AbdBPTXR (ORCPT ); Thu, 16 Feb 2017 14:23:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=obsidianresearch.com; s=rsa1; h=References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From; bh=A9bdPVwa57yK3s4D9b0F4VQ6LyU13IB/M5Mv3s4L9Ho=; b=0vNhNTR3V2oNHjr1ntV3X49TeyK+3aDFmOGJLzpCS0ULv1U20Z7GOdN9lGGOEWzhsx/gSd1+QEbbeiJqgzwEhV+52X7CgZyOV/c32o9FZ9YngEeQNQVTRalRhGeTZ7pJn7dGq/L5spODlQBvMcv6wwoVK07zNR5mehJb+vkHbCY=; Received: from [10.0.0.156] (helo=jggl.edm.orcorp.ca) by quartz.orcorp.ca with esmtps (TLS1.2:ECDHE_RSA_AES_128_CBC_SHA256:128) (Exim 4.84_2) (envelope-from ) id 1ceRe8-00010K-Cf; Thu, 16 Feb 2017 12:23:16 -0700 From: Jason Gunthorpe To: linux-rdma@vger.kernel.org Cc: Yishai Hadas Subject: [PATCH rdma-core 07/14] mlx4: Update to use new udma write barriers Date: Thu, 16 Feb 2017 12:23:02 -0700 Message-Id: <1487272989-8215-8-git-send-email-jgunthorpe@obsidianresearch.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1487272989-8215-1-git-send-email-jgunthorpe@obsidianresearch.com> References: <1487272989-8215-1-git-send-email-jgunthorpe@obsidianresearch.com> X-Broken-Reverse-DNS: no host name found for IP address 10.0.0.156 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The mlx4 comments are good so these translate fairly directly. - Added barrier at the top of mlx4_post_send, this makes the driver ready for a change to a stronger udma_to_device_barrier / weaker udma_order_write_barrier() which would make the post loop a bit faster. No change on x86-64 - The wmb() directly before the BF copy is upgraded to a wc_wmb(), this is consistent with what mlx5 does and makes sense. Signed-off-by: Jason Gunthorpe --- providers/mlx4/cq.c | 6 +++--- providers/mlx4/qp.c | 19 +++++++++++-------- providers/mlx4/srq.c | 2 +- 3 files changed, 15 insertions(+), 12 deletions(-) diff --git a/providers/mlx4/cq.c b/providers/mlx4/cq.c index 6a5cf8be218892..14f8cbce6d75ed 100644 --- a/providers/mlx4/cq.c +++ b/providers/mlx4/cq.c @@ -222,7 +222,7 @@ static inline int mlx4_get_next_cqe(struct mlx4_cq *cq, * Make sure we read CQ entry contents after we've checked the * ownership bit. */ - rmb(); + udma_from_device_barrier(); *pcqe = cqe; @@ -698,7 +698,7 @@ int mlx4_arm_cq(struct ibv_cq *ibvcq, int solicited) * Make sure that the doorbell record in host memory is * written before ringing the doorbell via PCI MMIO. */ - wmb(); + udma_to_device_barrier(); doorbell[0] = htonl(sn << 28 | cmd | cq->cqn); doorbell[1] = htonl(ci); @@ -764,7 +764,7 @@ void __mlx4_cq_clean(struct mlx4_cq *cq, uint32_t qpn, struct mlx4_srq *srq) * Make sure update of buffer contents is done before * updating consumer index. */ - wmb(); + udma_to_device_barrier(); mlx4_update_cons_index(cq); } } diff --git a/providers/mlx4/qp.c b/providers/mlx4/qp.c index a607326c7c452c..77a4a34576cb69 100644 --- a/providers/mlx4/qp.c +++ b/providers/mlx4/qp.c @@ -204,7 +204,7 @@ static void set_data_seg(struct mlx4_wqe_data_seg *dseg, struct ibv_sge *sg) * chunk and get a valid (!= * 0xffffffff) byte count but * stale data, and end up sending the wrong data. */ - wmb(); + udma_ordering_write_barrier(); if (likely(sg->length)) dseg->byte_count = htonl(sg->length); @@ -228,6 +228,9 @@ int mlx4_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr, pthread_spin_lock(&qp->sq.lock); + /* Get all user DMA buffers ready to go */ + udma_to_device_barrier(); + /* XXX check that state is OK to post send */ ind = qp->sq.head; @@ -400,7 +403,7 @@ int mlx4_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr, wqe += to_copy; addr += to_copy; seg_len += to_copy; - wmb(); /* see comment below */ + udma_ordering_write_barrier(); /* see comment below */ seg->byte_count = htonl(MLX4_INLINE_SEG | seg_len); seg_len = 0; seg = wqe; @@ -428,7 +431,7 @@ int mlx4_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr, * data, and end up sending the wrong * data. */ - wmb(); + udma_ordering_write_barrier(); seg->byte_count = htonl(MLX4_INLINE_SEG | seg_len); } @@ -450,7 +453,7 @@ int mlx4_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr, * setting ownership bit (because HW can start * executing as soon as we do). */ - wmb(); + udma_ordering_write_barrier(); ctrl->owner_opcode = htonl(mlx4_ib_opcode[wr->opcode]) | (ind & qp->sq.wqe_cnt ? htonl(1 << 31) : 0); @@ -478,7 +481,7 @@ out: * Make sure that descriptor is written to memory * before writing to BlueFlame page. */ - wmb(); + mmio_wc_start(); ++qp->sq.head; @@ -486,7 +489,7 @@ out: mlx4_bf_copy(ctx->bf_page + ctx->bf_offset, (unsigned long *) ctrl, align(size * 16, 64)); - wc_wmb(); + mmio_flush_writes(); ctx->bf_offset ^= ctx->bf_buf_size; @@ -498,7 +501,7 @@ out: * Make sure that descriptors are written before * doorbell record. */ - wmb(); + udma_to_device_barrier(); mmio_writel((unsigned long)(ctx->uar + MLX4_SEND_DOORBELL), qp->doorbell_qpn); @@ -566,7 +569,7 @@ out: * Make sure that descriptors are written before * doorbell record. */ - wmb(); + udma_to_device_barrier(); *qp->db = htonl(qp->rq.head & 0xffff); } diff --git a/providers/mlx4/srq.c b/providers/mlx4/srq.c index 4f90efdf927209..6e4ff5663d019b 100644 --- a/providers/mlx4/srq.c +++ b/providers/mlx4/srq.c @@ -113,7 +113,7 @@ int mlx4_post_srq_recv(struct ibv_srq *ibsrq, * Make sure that descriptors are written before * we write doorbell record. */ - wmb(); + udma_to_device_barrier(); *srq->db = htonl(srq->counter); }