From patchwork Thu Apr 26 18:40:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rohit Zambre X-Patchwork-Id: 10366675 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id F3611602DC for ; Thu, 26 Apr 2018 18:40:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E0C912923C for ; Thu, 26 Apr 2018 18:40:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D9F7029213; Thu, 26 Apr 2018 18:40:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0B11A29231 for ; Thu, 26 Apr 2018 18:40:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751496AbeDZSkb (ORCPT ); Thu, 26 Apr 2018 14:40:31 -0400 Received: from mail-it0-f67.google.com ([209.85.214.67]:34674 "EHLO mail-it0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751269AbeDZSka (ORCPT ); Thu, 26 Apr 2018 14:40:30 -0400 Received: by mail-it0-f67.google.com with SMTP id c5-v6so449275itj.1 for ; Thu, 26 Apr 2018 11:40:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uci-edu.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id; bh=rye85APIBHcel3hrew4UiTAjCyft6Rm0gvFBbOmXfeo=; b=KjVQ+YRndlopokCRpLZqrFi9wF3vnb/okWAHR1hswGOA+2WzOK9Gqu2XsB7paCeyDr DVxWXquyUlEIz9srkwIijgYU07tqrPpH/06AAeDqu2rS+2NUoxELR/1gCJ/DsmivzIoS /zl203hyk2Cf7W7PTBaXeZcsEaBIroSZYJoGH0ciGtCvK4Tn0eNm4jsb9egg8/xF4ejR pMplqDaU5V6bBtuouhrsC9ZkiqByrM7nfQGanaidYfb5qAqoFPkVTZwPRJucvRy4uErh +fO2qxml5dNMmTf9bSLIDInuJwUO/NyUib46cmCEngJI1KTCuRGaQ0tWciiF9OCJZb13 EcaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=rye85APIBHcel3hrew4UiTAjCyft6Rm0gvFBbOmXfeo=; b=lY1o8FkDvGR3vTPGJfwIm7U1QqsMDMgN/46i6sdXTfOXPy6D3kN+vLFjwHRXSKD0tA L9qgp26o6W+H8VMEyaFhNU/KywKrAnuwUVLRBZ5pomf5c+nvBZDWXa4CeSq2n9p0bcV3 a20PSn807JXM1W9oG3PVsFT4Af6IGpYV5qsYxiDqxzJwjqxGd+xPpI5T/XHSU/VCamjQ PDEE8XM6fDJowrr9b96AnfIJZE9V8hYc6hNrZ5M4YmmLbL+CHD+pf4HrO6F15e8XTUvO XRpA8rS+AEjShAabxqBHjsboZvaDKVrl370ujccnrOXK6uJxsQGzXUSESsnlW1truv8q KZ6Q== X-Gm-Message-State: ALQs6tBXDkTnvwFSwiNA3T9aKSIg0kw1Kq+pEUskdYQrWexSAgtTvvsu ooyJsZEDtbZdC5l18e64w621MXv7 X-Google-Smtp-Source: AB8JxZo+pcwPsTYmLBqETkqPQZ+ACUZDdoqQCeIKQWGx6pfXwrW94SWQzXUUt8S/BRidvB7KoMiQPw== X-Received: by 2002:a24:f9cc:: with SMTP id l195-v6mr6205698ith.132.1524768030359; Thu, 26 Apr 2018 11:40:30 -0700 (PDT) Received: from jlselogin1.ftm.alcf.anl.gov (jlselogin1.pub.jlse.anl.gov. [140.221.96.130]) by smtp.gmail.com with ESMTPSA id s2-v6sm9709162iod.56.2018.04.26.11.40.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Apr 2018 11:40:29 -0700 (PDT) From: Rohit Zambre To: dledford@redhat.com, jgg@mellanox.com Cc: linux-rdma@vger.kernel.org, Rohit Zambre Subject: [PATCH rdma-next 1/1] IB/mlx5: Check for at least 1 medium bfreg Date: Thu, 26 Apr 2018 18:40:22 +0000 Message-Id: <1524768022-14878-1-git-send-email-rzambre@uci.edu> X-Mailer: git-send-email 1.8.3.1 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The problem occurs when the num_low_latency_bfregs equals total_num_bfregs-1. In this case, the current bfreg-to-QP mapping policy maps the first QP of the context to bfreg_1. However, the userspace mlx5 driver designates bfreg_1 as a medium bfreg; need_lock for bf_1 evaluates to 1. Hence, the kernel is allocatng a medium bfreg to the first QP of the context, which is incorrect behavior. More importantly, for any combination of total_num_bfregs and num_low_latency_bfregs, there must be at least 1 medium bfreg to accommodate for the QPs created after the first #num_low_latency_bfregs QPs in the context. For example, if the user requests for 4 total bfregs and 3 low latency bfregs, the driver cannot map any QP after the 3rd one without breaking the definition of a low latency bfreg: one that is mapped to only 1 QP and hence doesn't need a lock. Currently, the driver allows for such a user-request to pass. The current check only accounts for the 0th bfreg. This fix implements the check for the constraint on the minimum number of medium bfregs required. Signed-off-by: Rohit Zambre --- drivers/infiniband/hw/mlx5/main.c | 5 +++-- drivers/infiniband/hw/mlx5/qp.c | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index daa919e..75c4b26 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -1525,7 +1525,7 @@ static int calc_total_bfregs(struct mlx5_ib_dev *dev, bool lib_uar_4k, bfregs_per_sys_page = uars_per_sys_page * MLX5_NON_FP_BFREGS_PER_UAR; /* This holds the required static allocation asked by the user */ req->total_num_bfregs = ALIGN(req->total_num_bfregs, bfregs_per_sys_page); - if (req->num_low_latency_bfregs > req->total_num_bfregs - 1) + if (req->num_low_latency_bfregs > req->total_num_bfregs - 2) return -EINVAL; bfregi->num_static_sys_pages = req->total_num_bfregs / bfregs_per_sys_page; @@ -1669,7 +1669,8 @@ static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev, req.total_num_bfregs = ALIGN(req.total_num_bfregs, MLX5_NON_FP_BFREGS_PER_UAR); - if (req.num_low_latency_bfregs > req.total_num_bfregs - 1) + /* account for the 0th bfreg and at least one medium bfreg */ + if (req.num_low_latency_bfregs > req.total_num_bfregs - 2) return ERR_PTR(-EINVAL); resp.qp_tab_size = 1 << MLX5_CAP_GEN(dev->mdev, log_max_qp); diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c index 7ed4b70..ad09174 100644 --- a/drivers/infiniband/hw/mlx5/qp.c +++ b/drivers/infiniband/hw/mlx5/qp.c @@ -507,7 +507,7 @@ static int num_med_bfreg(struct mlx5_ib_dev *dev, n = max_bfregs(dev, bfregi) - bfregi->num_low_latency_bfregs - NUM_NON_BLUE_FLAME_BFREGS; - return n >= 0 ? n : 0; + return n > 0 ? n : 1; } static int first_hi_bfreg(struct mlx5_ib_dev *dev,