From patchwork Sun Jan 27 12:01:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10782875 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A14B0186D for ; Sun, 27 Jan 2019 12:40:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 918AE2B296 for ; Sun, 27 Jan 2019 12:40:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 80DCB2B2E9; Sun, 27 Jan 2019 12:40:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E42BD2B2B4 for ; Sun, 27 Jan 2019 12:40:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726535AbfA0MkW (ORCPT ); Sun, 27 Jan 2019 07:40:22 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:54140 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726493AbfA0MkV (ORCPT ); Sun, 27 Jan 2019 07:40:21 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 27 Jan 2019 14:40:13 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0RCeDxk017527; Sun, 27 Jan 2019 14:40:13 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id x0RC1kmT016764; Sun, 27 Jan 2019 14:01:46 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id x0RC1kL5016763; Sun, 27 Jan 2019 14:01:46 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, michaelgur@mellanox.com, jgg@mellanox.com, majd@mellanox.com Subject: [PATCH rdma-core 3/3] mlx5: Report PCI atomic capabilities Date: Sun, 27 Jan 2019 14:01:34 +0200 Message-Id: <1548590494-16684-4-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1548590494-16684-1-git-send-email-yishaih@mellanox.com> References: <1548590494-16684-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Michael Guralnik Query the PCI atomic capabilities using the DEVX infrastructure and report it through device query. Signed-off-by: Michael Guralnik Signed-off-by: Yishai Hadas --- providers/mlx5/mlx5_ifc.h | 100 ++++++++++++++++++++++++++++++++++++++++++++++ providers/mlx5/verbs.c | 31 ++++++++++++++ 2 files changed, 131 insertions(+) create mode 100644 providers/mlx5/mlx5_ifc.h diff --git a/providers/mlx5/mlx5_ifc.h b/providers/mlx5/mlx5_ifc.h new file mode 100644 index 0000000..5cf89d6 --- /dev/null +++ b/providers/mlx5/mlx5_ifc.h @@ -0,0 +1,100 @@ +/* + * Copyright (c) 2019 Mellanox Technologies, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#define u8 uint8_t + +enum mlx5_cap_mode { + HCA_CAP_OPMOD_GET_CUR = 1, +}; + +enum { + MLX5_CMD_OP_QUERY_HCA_CAP = 0x100, +}; + +struct mlx5_ifc_atomic_caps_bits { + u8 reserved_at_0[0x40]; + + u8 atomic_req_8B_endianness_mode[0x2]; + u8 reserved_at_42[0x4]; + u8 supported_atomic_req_8B_endianness_mode_1[0x1]; + + u8 reserved_at_47[0x19]; + + u8 reserved_at_60[0x20]; + + u8 reserved_at_80[0x10]; + u8 atomic_operations[0x10]; + + u8 reserved_at_a0[0x10]; + u8 atomic_size_qp[0x10]; + + u8 reserved_at_c0[0x10]; + u8 atomic_size_dc[0x10]; + + u8 reserved_at_e0[0x1a0]; + + u8 fetch_add_pci_atomic[0x10]; + u8 swap_pci_atomic[0x10]; + u8 compare_swap_pci_atomic[0x10]; + + u8 reserved_at_2b0[0x550]; +}; + +union mlx5_ifc_hca_cap_union_bits { + struct mlx5_ifc_atomic_caps_bits atomic_caps; + u8 reserved_at_0[0x8000]; +}; + +struct mlx5_ifc_query_hca_cap_out_bits { + u8 status[0x8]; + u8 reserved_at_8[0x18]; + + u8 syndrome[0x20]; + + u8 reserved_at_40[0x40]; + + union mlx5_ifc_hca_cap_union_bits capability; +}; + +struct mlx5_ifc_query_hca_cap_in_bits { + u8 opcode[0x10]; + u8 reserved_at_10[0x10]; + + u8 reserved_at_20[0x10]; + u8 op_mod[0x10]; + + u8 reserved_at_40[0x40]; +}; + +enum mlx5_cap_type { + MLX5_CAP_ATOMIC = 3, +}; diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c index bab675f..4bc3eac 100644 --- a/providers/mlx5/verbs.c +++ b/providers/mlx5/verbs.c @@ -55,6 +55,7 @@ #include "mlx5.h" #include "mlx5-abi.h" #include "wqe.h" +#include "mlx5_ifc.h" int mlx5_single_threaded = 0; @@ -2816,6 +2817,32 @@ err: return NULL; } +static void get_pci_atomic_caps(struct ibv_context *context, + struct ibv_device_attr_ex *attr) +{ + uint32_t in[DEVX_ST_SZ_DW(query_hca_cap_in)] = {}; + uint32_t out[DEVX_ST_SZ_DW(query_hca_cap_out)] = {}; + uint16_t opmod = (MLX5_CAP_ATOMIC << 1) | HCA_CAP_OPMOD_GET_CUR; + int ret; + + DEVX_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP); + DEVX_SET(query_hca_cap_in, in, op_mod, opmod); + + ret = mlx5dv_devx_general_cmd(context, in, sizeof(in), out, + sizeof(out)); + if (!ret) { + attr->pci_atomic_caps.fetch_add = + DEVX_GET(query_hca_cap_out, out, + capability.atomic_caps.fetch_add_pci_atomic); + attr->pci_atomic_caps.swap = + DEVX_GET(query_hca_cap_out, out, + capability.atomic_caps.swap_pci_atomic); + attr->pci_atomic_caps.compare_swap = + DEVX_GET(query_hca_cap_out, out, + capability.atomic_caps.compare_swap_pci_atomic); + } +} + int mlx5_query_device_ex(struct ibv_context *context, const struct ibv_query_device_ex_input *input, struct ibv_device_attr_ex *attr, @@ -2894,6 +2921,10 @@ int mlx5_query_device_ex(struct ibv_context *context, snprintf(a->fw_ver, sizeof(a->fw_ver), "%d.%d.%04d", major, minor, sub_minor); + if (attr_size >= offsetof(struct ibv_device_attr_ex, pci_atomic_caps) + + sizeof(attr->pci_atomic_caps)) + get_pci_atomic_caps(context, attr); + return 0; }