From patchwork Tue Jul 9 15:01:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 13728204 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CDB2D19CD11 for ; Tue, 9 Jul 2024 15:01:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720537288; cv=none; b=ZssKrVz0yyklJxKhWi7LBtXm1zTqcef/XbTCXdiMksBBxZBcBzhOlE2GepgjN/p2kLr3jgofgnSP0olz4bhM9ZY8lbiNIzBiEGvE7QAYDvwJJYDd7aDtLnYvXzRgBjq1Wlvp0YfRyDdvOWLuAQ3+YJSHk8B5RAL2xlbde8A+KYQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720537288; c=relaxed/simple; bh=c0KoLXoRQKOE+iw7pZVO9/OP4bNiSGfP3rLJ7YXxCd8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QTyJs7OWr1wd37pKE0Ns1DK6NUhg1o4aSuZ0jrA7+9KzCIGaXa+ARxqX6B5FOhNcR0FKQgiY39/exNEqTg4agrxK7Cgd/h43sxm6Kw11+Wrq7XTnqvWFq81r6J0u65sKQNEr7H2E48YDyCFPhwYHQVNe+0dyBz1aYjBI2vRSjck= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=dNWNUutl; arc=none smtp.client-ip=209.85.208.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="dNWNUutl" Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-58b0dddab8cso7789889a12.0 for ; Tue, 09 Jul 2024 08:01:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1720537282; x=1721142082; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yBXPBGTYOojItKAzX7jkgiz3uyYlSKioXedMWKxUIW0=; b=dNWNUutll4jdsxGrtO/Wsw0xadz4TCSmkCHTV6RGnc8HuGcqEfUY68pYxtRMLmgMDF B1jpRmqn0XJUGAQNX9N61axbHreZi+hrTg4NcQnLbygNMARXDAnMGGKCK0DzQWMAYLQT WGXCnsmOxTbWaEDm9dB4uf563cyHCRXdj8uHE/HalxFWtRlzCEm2WqEeLzmGhPhEP+w+ imlRUd6ieq7a8n9tH3/ibmh/5hmJykYLedHUHo2586yHL2Gqp86keNVsbH7Q7MQT/YNl 87vtiYyC/3VlVlUDxpT3e+HVEgI3NZPdJqog2ox5Zhk7O4lqTAHNEEKDoEfooBQBoe3q QRtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720537282; x=1721142082; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yBXPBGTYOojItKAzX7jkgiz3uyYlSKioXedMWKxUIW0=; b=tH4bXluQVCW7w9imnMCEau5hGeFqDEuQFhgQhpYFHSU256q9cVuXXl+vc4MAHf016Y kQ3FnyOZ7WzVdzuhZc/jaut1mza6cTc3dwxQLAwH+A4O5uf7VAWbI574ftQldRyUN1Es v9vGciJu3Tx8ttVS3lglaggJiudivXgbOxioHTTZ9C+lXJJvBiIflcI2e5oLI4Bf8uay AvP4Z0WEtcMJn0DCpZoDzRQlCFzyfZa4yY8Krdzo7v+27Hf/3bBh9qhmeT7hVLjn4lar qeTbP0ZISLkyNI1S7iDztRqFqUj84n4wNAOBtLeCbH5tYiSVzXl6Cs87Pn3dYz85mHmk 8kMw== X-Gm-Message-State: AOJu0Yx7G2hVG/cVCJDBAppenzEFxnK3edo4jOWWHEaOocXjtERysDY6 6T10rG/9Px+c//BwqUzWLaAWaSQeXjWhye9Ce2Elal1m5ITi2AOXcvGp14UpGm/xCrf+6RtVRMw S X-Google-Smtp-Source: AGHT+IFTXd9d/plY2kD+hBHfpwW3y2c32FIynIbEXS35LPP21Pn+EC9Sx7dtspDACaaklXmfzPtvMQ== X-Received: by 2002:aa7:da0f:0:b0:57a:2ccb:b3f2 with SMTP id 4fb4d7f45d1cf-594bb18046cmr1624859a12.16.1720537282623; Tue, 09 Jul 2024 08:01:22 -0700 (PDT) Received: from lb02065.fritz.box ([2001:9e8:1436:4f00:2ca:d136:a29a:bb96]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-594bc4e80c6sm1166172a12.46.2024.07.09.08.01.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Jul 2024 08:01:21 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, jgg@ziepe.ca, selvin.xavier@broadcom.com, haris.iqbal@ionos.com, jinpu.wang@ionos.com Subject: [PATCH for-next 1/2] bnxt_re: Fix imm_data endianness Date: Tue, 9 Jul 2024 17:01:18 +0200 Message-Id: <20240709150119.29937-2-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240709150119.29937-1-jinpu.wang@ionos.com> References: <20240709150119.29937-1-jinpu.wang@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 When map a device between servers with MLX and BCM RoCE nics, RTRS server complain about unknown imm type, and can't map the device, After more debug, it seems bnxt_re wrongly handle the imm_data, this patch fixed the compat issue with MLX for us. In offlist discussion, Selvin confirm HW is working in little endian format and all data needs to be converted to LE while providing. This patch fix the endianness for imm_data Fixes: 1ac5a4047975 ("RDMA/bnxt_re: Add bnxt_re RoCE driver") Signed-off-by: Jack Wang --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 8 ++++---- drivers/infiniband/hw/bnxt_re/qplib_fp.h | 6 +++--- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index e453ca701e87..c5080028247e 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -2479,7 +2479,7 @@ static int bnxt_re_build_send_wqe(struct bnxt_re_qp *qp, break; case IB_WR_SEND_WITH_IMM: wqe->type = BNXT_QPLIB_SWQE_TYPE_SEND_WITH_IMM; - wqe->send.imm_data = wr->ex.imm_data; + wqe->send.imm_data = cpu_to_le32(be32_to_cpu(wr->ex.imm_data)); break; case IB_WR_SEND_WITH_INV: wqe->type = BNXT_QPLIB_SWQE_TYPE_SEND_WITH_INV; @@ -2509,7 +2509,7 @@ static int bnxt_re_build_rdma_wqe(const struct ib_send_wr *wr, break; case IB_WR_RDMA_WRITE_WITH_IMM: wqe->type = BNXT_QPLIB_SWQE_TYPE_RDMA_WRITE_WITH_IMM; - wqe->rdma.imm_data = wr->ex.imm_data; + wqe->rdma.imm_data = cpu_to_le32(be32_to_cpu(wr->ex.imm_data)); break; case IB_WR_RDMA_READ: wqe->type = BNXT_QPLIB_SWQE_TYPE_RDMA_READ; @@ -3582,7 +3582,7 @@ static void bnxt_re_process_res_shadow_qp_wc(struct bnxt_re_qp *gsi_sqp, wc->byte_len = orig_cqe->length; wc->qp = &gsi_qp->ib_qp; - wc->ex.imm_data = orig_cqe->immdata; + wc->ex.imm_data = cpu_to_be32(le32_to_cpu(orig_cqe->immdata)); wc->src_qp = orig_cqe->src_qp; memcpy(wc->smac, orig_cqe->smac, ETH_ALEN); if (bnxt_re_is_vlan_pkt(orig_cqe, &vlan_id, &sl)) { @@ -3727,7 +3727,7 @@ int bnxt_re_poll_cq(struct ib_cq *ib_cq, int num_entries, struct ib_wc *wc) (unsigned long)(cqe->qp_handle), struct bnxt_re_qp, qplib_qp); wc->qp = &qp->ib_qp; - wc->ex.imm_data = cqe->immdata; + wc->ex.imm_data = cpu_to_be32(le32_to_cpu(cqe->immdata)); wc->src_qp = cqe->src_qp; memcpy(wc->smac, cqe->smac, ETH_ALEN); wc->port_num = 1; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.h b/drivers/infiniband/hw/bnxt_re/qplib_fp.h index 4aaac84c1b1b..1fcaba0f680b 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.h @@ -164,7 +164,7 @@ struct bnxt_qplib_swqe { /* Send, with imm, inval key */ struct { union { - __be32 imm_data; + __le32 imm_data; u32 inv_key; }; u32 q_key; @@ -182,7 +182,7 @@ struct bnxt_qplib_swqe { /* RDMA write, with imm, read */ struct { union { - __be32 imm_data; + __le32 imm_data; u32 inv_key; }; u64 remote_va; @@ -389,7 +389,7 @@ struct bnxt_qplib_cqe { u16 cfa_meta; u64 wr_id; union { - __be32 immdata; + __le32 immdata; u32 invrkey; }; u64 qp_handle; From patchwork Tue Jul 9 15:01:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 13728205 Received: from mail-ed1-f49.google.com (mail-ed1-f49.google.com [209.85.208.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E91C19D890 for ; Tue, 9 Jul 2024 15:01:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720537289; cv=none; b=ESnuFPAgpL2UXxbFBn4pMXdmGxsYUweyjlnLTIG42PQzX8qBkRdeqwAnXdmeUKAVIUBUmP+5+2vrc0mM+dnGiw3YaTlASPFi84e0YRi0OmzVZby+yzzN6EHVTmXcWn4uPFnrEdrN2xOouLYYQPPD5F+3g0Abr8RG0tBFPXiPf2E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720537289; c=relaxed/simple; bh=FW6fIdGit+pyP1OewBZblyVS76v/fNq0887mMCejRwQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tV2l14k0N1Qs3iyd/tvccVUwo0FzWvC8IhMCAUafpuZ5F6gc1x9t5ej5D/yr1GiWGLlm43zDqQrK58GHR9xtFqkPbqQ5FEVpjISl8nS0sFn8cqzm5u7kf9ij6XoEiAwJfEMYQovPYeErILkwG0Q8sUSSdXbdM8liW9ptm+Tqc4U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com; spf=pass smtp.mailfrom=ionos.com; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b=C/tML7au; arc=none smtp.client-ip=209.85.208.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=ionos.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ionos.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ionos.com header.i=@ionos.com header.b="C/tML7au" Received: by mail-ed1-f49.google.com with SMTP id 4fb4d7f45d1cf-58bac81f419so6974878a12.0 for ; Tue, 09 Jul 2024 08:01:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; t=1720537284; x=1721142084; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OBqw8jE35wR5me+P1Q3829kCDIv4M/IogxwaM5m70tg=; b=C/tML7audLB7jXMRa/fOhbo5V1KRVaRPJVHDNwna8zMRa3dbMGvkR7Dqn5oYxT2w9c UqN8Iu6wzb+AoCQZpOZoFodMeb681Ac+6pfrAxAY/qiT7WSKunW65zGJ77T+gBRYPLR9 Y1s61AkUgUsgDUdjjq5yYLQ9jb0KZfQS5kOdISkyAFV4OjVzhUNeeVaQlIWLd5Om+pcj D0GZ3JBVNeIM6ToW9nrAKYL+kk7MoHw+h9KFcVWh/ityR/AggpZvuXHGWkh1JwpF6hiv sV9ONjWbxKrzio47ewP799n0+IS2cdOMRHemHaljzhD/Nh5MTvN2SqG1t5CmZoH/tlRg rp7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720537284; x=1721142084; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OBqw8jE35wR5me+P1Q3829kCDIv4M/IogxwaM5m70tg=; b=DVtrM89U7VLRdduc+WX/KLvskMxyT8QaVn/Ph12aDXlA1rzotLqarKOAqgXymxbq69 fj6RISzruCpXR91z+Kpab6PpcW4S3gluPwEKj/bi1UmKXglcWrmVk94q6iJ8S5lRVdaW i6GNiqeAh/Qpu+/BYKMOJgkykFFwqdEofQA5stv4vFoxDo6s1Y+7K9kz0nUDjTWFCOQg JcJ/zkJqeIdzM0joCEy9Jh7+C/XkiDlFRgR5mhXbRq1MyRSu4ndlL6UUKQhgW8poZIQB sq2ry0ohWBPX9kse7osnm8/G1G3yBuTd6Vnwtfssil1AoFQH5zxPKgpDje8EeSNQO4LR 1Vpw== X-Gm-Message-State: AOJu0Yymb0eyA7WO4gZNSEJbIwfAwQjPZOXG7IYa4Mj1F6ialc+jbSe+ WzdjdFOD7dPQC8MBg4VKV+zsW8OQ9cHCJjuAU6i+VUgs04d6vIy78WffN1iIYB16ALoxNHESS5f a X-Google-Smtp-Source: AGHT+IFfv+v12ijQeCSAGiop1O2oF8oiE2Qrai/EXIpvbGLWRPUybiOojHqNPrRP7f26faqbejZVoA== X-Received: by 2002:a05:6402:33ce:b0:58e:dbd:65cd with SMTP id 4fb4d7f45d1cf-594bc7c7e1amr1849928a12.26.1720537284156; Tue, 09 Jul 2024 08:01:24 -0700 (PDT) Received: from lb02065.fritz.box ([2001:9e8:1436:4f00:2ca:d136:a29a:bb96]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-594bc4e80c6sm1166172a12.46.2024.07.09.08.01.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Jul 2024 08:01:23 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, jgg@ziepe.ca, selvin.xavier@broadcom.com, haris.iqbal@ionos.com, jinpu.wang@ionos.com Subject: [PATCH for-next 2/2] bnxt_re: Fix inv_key endianness Date: Tue, 9 Jul 2024 17:01:19 +0200 Message-Id: <20240709150119.29937-3-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240709150119.29937-1-jinpu.wang@ionos.com> References: <20240709150119.29937-1-jinpu.wang@ionos.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Similar like previous patch, this change the endianness for inv_key, hw expect LE, so change the type accordingly. Fixes: 1ac5a4047975 ("RDMA/bnxt_re: Add bnxt_re RoCE driver") Signed-off-by: Jack Wang --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 6 +++--- drivers/infiniband/hw/bnxt_re/qplib_fp.h | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index c5080028247e..cdc8ebcf3a76 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -2483,7 +2483,7 @@ static int bnxt_re_build_send_wqe(struct bnxt_re_qp *qp, break; case IB_WR_SEND_WITH_INV: wqe->type = BNXT_QPLIB_SWQE_TYPE_SEND_WITH_INV; - wqe->send.inv_key = wr->ex.invalidate_rkey; + wqe->send.inv_key = cpu_to_le32(wr->ex.invalidate_rkey); break; default: return -EINVAL; @@ -2513,7 +2513,7 @@ static int bnxt_re_build_rdma_wqe(const struct ib_send_wr *wr, break; case IB_WR_RDMA_READ: wqe->type = BNXT_QPLIB_SWQE_TYPE_RDMA_READ; - wqe->rdma.inv_key = wr->ex.invalidate_rkey; + wqe->rdma.inv_key = cpu_to_le32(wr->ex.invalidate_rkey); break; default: return -EINVAL; @@ -2563,7 +2563,7 @@ static int bnxt_re_build_inv_wqe(const struct ib_send_wr *wr, struct bnxt_qplib_swqe *wqe) { wqe->type = BNXT_QPLIB_SWQE_TYPE_LOCAL_INV; - wqe->local_inv.inv_l_key = wr->ex.invalidate_rkey; + wqe->local_inv.inv_l_key = cpu_to_le32(wr->ex.invalidate_rkey); if (wr->send_flags & IB_SEND_SIGNALED) wqe->flags |= BNXT_QPLIB_SWQE_FLAGS_SIGNAL_COMP; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.h b/drivers/infiniband/hw/bnxt_re/qplib_fp.h index 1fcaba0f680b..813332b2c872 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.h @@ -165,7 +165,7 @@ struct bnxt_qplib_swqe { struct { union { __le32 imm_data; - u32 inv_key; + __le32 inv_key; }; u32 q_key; u32 dst_qp; @@ -183,7 +183,7 @@ struct bnxt_qplib_swqe { struct { union { __le32 imm_data; - u32 inv_key; + __le32 inv_key; }; u64 remote_va; u32 r_key; @@ -199,7 +199,7 @@ struct bnxt_qplib_swqe { /* Local Invalidate */ struct { - u32 inv_l_key; + __le32 inv_l_key; } local_inv; /* FR-PMR */