From patchwork Mon Dec 25 13:56:58 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10132569 X-Patchwork-Delegate: leon@leon.nu Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 739AE60753 for ; Mon, 25 Dec 2017 13:57:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 65A372F135 for ; Mon, 25 Dec 2017 13:57:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5A80D2F140; Mon, 25 Dec 2017 13:57:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 066242F135 for ; Mon, 25 Dec 2017 13:57:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752414AbdLYN5d (ORCPT ); Mon, 25 Dec 2017 08:57:33 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:37328 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752467AbdLYN51 (ORCPT ); Mon, 25 Dec 2017 08:57:27 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 25 Dec 2017 15:57:19 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id vBPDvIro016317; Mon, 25 Dec 2017 15:57:18 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id vBPDvI1V010669; Mon, 25 Dec 2017 15:57:18 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id vBPDvIM9010668; Mon, 25 Dec 2017 15:57:18 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, Alexr@mellanox.com, jgg@mellanox.com, majd@mellanox.com Subject: [PATCH rdma-core 6/9] mlx5: Add support for dynamic UAR allocation Date: Mon, 25 Dec 2017 15:56:58 +0200 Message-Id: <1514210221-10466-7-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1514210221-10466-1-git-send-email-yishaih@mellanox.com> References: <1514210221-10466-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch adds support to dynamically allocate a UAR by introducing a new mmap command to be issued with the mlx5 kernel driver. The allocated UAR will be managed by the mlx5 user space driver and its internal BF registers will be used upon allocating an ibv_thread_domain. To enable allocating more than 256 UARs the page index is encoded in an extra one byte just after the command byte. This functionality will be used in downstream patches from this series. Signed-off-by: Yishai Hadas --- providers/mlx5/mlx5.c | 13 ++++++++++--- providers/mlx5/mlx5.h | 11 ++++++++++- 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c index 4c872c8..bdad9ba 100644 --- a/providers/mlx5/mlx5.c +++ b/providers/mlx5/mlx5.c @@ -782,7 +782,12 @@ static off_t get_uar_mmap_offset(int idx, int page_size, int command) off_t offset = 0; set_command(command, &offset); - set_index(idx, &offset); + + if (command == MLX5_MMAP_ALLOC_WC && + idx >= (1 << MLX5_IB_MMAP_CMD_SHIFT)) + set_extended_index(idx, &offset); + else + set_index(idx, &offset); return offset * page_size; } @@ -793,8 +798,8 @@ static off_t uar_type_to_cmd(int uar_type) MLX5_MMAP_GET_REGULAR_PAGES_CMD; } -static void *mlx5_mmap(struct mlx5_uar_info *uar, int index, - int cmd_fd, int page_size, int uar_type) +void *mlx5_mmap(struct mlx5_uar_info *uar, int index, int cmd_fd, int page_size, + int uar_type) { off_t offset; @@ -813,6 +818,8 @@ static void *mlx5_mmap(struct mlx5_uar_info *uar, int index, * MLX5_MMAP_GET_NC_PAGES_CMD mmap command. */ offset = get_uar_mmap_offset(index, page_size, + (uar_type == MLX5_UAR_TYPE_REGULAR_DYN) ? + MLX5_MMAP_ALLOC_WC : MLX5_MMAP_GET_REGULAR_PAGES_CMD); uar->reg = mmap(NULL, page_size, PROT_WRITE, MAP_SHARED, cmd_fd, offset); diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h index 8d5d193..7a9434b 100644 --- a/providers/mlx5/mlx5.h +++ b/providers/mlx5/mlx5.h @@ -58,7 +58,8 @@ enum { enum { MLX5_MMAP_GET_CONTIGUOUS_PAGES_CMD = 1, - MLX5_MMAP_GET_CORE_CLOCK_CMD = 5 + MLX5_MMAP_GET_CORE_CLOCK_CMD = 5, + MLX5_MMAP_ALLOC_WC = 6, }; enum { @@ -222,6 +223,7 @@ struct mlx5_spinlock { enum mlx5_uar_type { MLX5_UAR_TYPE_REGULAR, MLX5_UAR_TYPE_NC, + MLX5_UAR_TYPE_REGULAR_DYN, }; struct mlx5_uar_info { @@ -800,6 +802,8 @@ struct ibv_pd *mlx5_alloc_parent_domain(struct ibv_context *context, struct ibv_parent_domain_init_attr *attr); +void *mlx5_mmap(struct mlx5_uar_info *uar, int index, + int cmd_fd, int page_size, int uar_type); static inline void *mlx5_find_uidx(struct mlx5_context *ctx, uint32_t uidx) { int tind = uidx >> MLX5_UIDX_TABLE_SHIFT; @@ -873,6 +877,11 @@ static inline void set_index(int index, off_t *offset) set_arg(index, offset); } +static inline void set_extended_index(int index, off_t *offset) +{ + *offset |= (index & 0xff) | ((index >> 8) << 16); +} + static inline uint8_t calc_sig(void *wqe, int size) { int i;