From patchwork Sat Aug 24 03:19:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nikolova, Tatyana E" X-Patchwork-Id: 13776190 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AFEB722092; Sat, 24 Aug 2024 03:20:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.16 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724469642; cv=none; b=SfPypGy0Qi0Oe+3NtowahHEuEOOtfgBqHyKImM5q2HovsbBRI/2pBaqktGHAbMq91xWiWzmOzVSQOtbRx5zqNSHKm8ZQgnNMKRgsDFgQWqdJtKk1okmVx8rksrt+0J0P2tRogtiksoooXCQI60cgl5/oc/AHFjHGIg990S0H8Pk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724469642; c=relaxed/simple; bh=GxRHMsKtQlsvD0pV0UY5vbT82OsgqfrD9ebFFJq1mV8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kHq9eR0RjM5Yu+pgYwO97gkPLTPwB2M+pkQLXoS0o6njlx9TJmzX96nYHpWqZNrplOjdELx4NliUvltoQtu3id2tF5kQJdRmCOU+xKOozLbHHaWDwZhUzd/fZ14sGKFCXTSOMdpKP2XTAkOSYZDDOTfkkbOMFQZtGYszq3ejhkQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=VDXLC7kq; arc=none smtp.client-ip=192.198.163.16 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="VDXLC7kq" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1724469640; x=1756005640; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=GxRHMsKtQlsvD0pV0UY5vbT82OsgqfrD9ebFFJq1mV8=; b=VDXLC7kqbz8dHtydz8/FpGeA7gg8yAUIqErmjg64kSyHCwAdzfVSxrMc jvEKdf3fb7QsU8Toi4+sXmt1cqFi8IQPZaQUlGgLKA+kWjVNLZKFZSGGn 5Dnbqx66yDDxtqsvaRK+UwYwI7yd8OVeYRCHDYXv1APxAYnnkvP96HS11 XsUsghQ3+mU8HbxXq38NIydikl0gZ3rmXsj3WMPMv1yWttNJ/P+vWmZvw 2It5/peNh3KJiTDj5Rako79smk9sqKe9KXo9U4QMiz4HcJD1eeZuDFW2V U9AiZ68PlbcQqqFPQpmSrcELwTvZdfzrpOs+l0/yzKc+4znsagPc/ElQ7 A==; X-CSE-ConnectionGUID: qO5WkonjTRezhkgKFPjavw== X-CSE-MsgGUID: pc6et48eR+Ol5XWyGgCxww== X-IronPort-AV: E=McAfee;i="6700,10204,11173"; a="13187770" X-IronPort-AV: E=Sophos;i="6.10,172,1719903600"; d="scan'208";a="13187770" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Aug 2024 20:20:39 -0700 X-CSE-ConnectionGUID: R4pFilnnQmOPX7QuQ2n/iQ== X-CSE-MsgGUID: CSSRTcz6RLWfTijH1sWOeg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,172,1719903600"; d="scan'208";a="99492074" Received: from tenikolo-mobl1.amr.corp.intel.com ([10.124.36.66]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Aug 2024 20:20:38 -0700 From: Tatyana Nikolova To: jgg@nvidia.com, leon@kernel.org Cc: linux-rdma@vger.kernel.org, netdev@vger.kernel.org, Joshua Hay , Tatyana Nikolova Subject: [RFC v2 02/25] idpf: use reserved rdma vectors from control plane Date: Fri, 23 Aug 2024 22:19:01 -0500 Message-Id: <20240824031924.421-3-tatyana.e.nikolova@intel.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20240824031924.421-1-tatyana.e.nikolova@intel.com> References: <20240824031924.421-1-tatyana.e.nikolova@intel.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Joshua Hay Fetch the number of reserved rdma vectors from the control plane. Adjust the number of reserved lan vectors if necessary. Adjust the minimum number of vectors the OS should reserve to include rdma; and fail if the OS cannot reserve enough vectors for the minimum number of lan and rdma vectors required. Create a separate msix table for the reserved rdma vectors, which will just get handed off to the rdma core device to do with what it will. v2: * Use default minimum if control plane does not provide any reserved rdma vecs. Otherwise, if control plane does not provide enough reserved rdma vecs, fail. * Use the actual number of vectors when getting vecids. Signed-off-by: Joshua Hay Signed-off-by: Tatyana Nikolova --- drivers/net/ethernet/intel/idpf/idpf.h | 24 ++++++- drivers/net/ethernet/intel/idpf/idpf_lib.c | 76 +++++++++++++++++---- drivers/net/ethernet/intel/idpf/idpf_txrx.h | 1 + drivers/net/ethernet/intel/idpf/virtchnl2.h | 5 +- 4 files changed, 88 insertions(+), 18 deletions(-) diff --git a/drivers/net/ethernet/intel/idpf/idpf.h b/drivers/net/ethernet/intel/idpf/idpf.h index 2c31ad87587a..85503fb5dd69 100644 --- a/drivers/net/ethernet/intel/idpf/idpf.h +++ b/drivers/net/ethernet/intel/idpf/idpf.h @@ -489,10 +489,11 @@ struct idpf_vc_xn_manager; * @flags: See enum idpf_flags * @reset_reg: See struct idpf_reset_reg * @hw: Device access data - * @num_req_msix: Requested number of MSIX vectors * @num_avail_msix: Available number of MSIX vectors * @num_msix_entries: Number of entries in MSIX table * @msix_entries: MSIX table + * @num_rdma_msix_entries: Available number of MSIX vectors for RDMA + * @rdma_msix_entries: RDMA MSIX table * @req_vec_chunks: Requested vector chunk data * @mb_vector: Mailbox vector data * @vector_stack: Stack to store the msix vector indexes @@ -542,10 +543,11 @@ struct idpf_adapter { DECLARE_BITMAP(flags, IDPF_FLAGS_NBITS); struct idpf_reset_reg reset_reg; struct idpf_hw hw; - u16 num_req_msix; u16 num_avail_msix; u16 num_msix_entries; struct msix_entry *msix_entries; + u16 num_rdma_msix_entries; + struct msix_entry *rdma_msix_entries; struct virtchnl2_alloc_vectors *req_vec_chunks; struct idpf_q_vector mb_vector; struct idpf_vector_lifo vector_stack; @@ -609,6 +611,15 @@ static inline int idpf_is_queue_model_split(u16 q_model) bool idpf_is_capability_ena(struct idpf_adapter *adapter, bool all, enum idpf_cap_field field, u64 flag); +/** + * idpf_is_rdma_cap_ena - Determine if RDMA is supported + * @adapter: private data struct + */ +static inline bool idpf_is_rdma_cap_ena(struct idpf_adapter *adapter) +{ + return idpf_is_cap_ena(adapter, IDPF_OTHER_CAPS, VIRTCHNL2_CAP_RDMA); +} + #define IDPF_CAP_RSS (\ VIRTCHNL2_CAP_RSS_IPV4_TCP |\ VIRTCHNL2_CAP_RSS_IPV4_TCP |\ @@ -663,6 +674,15 @@ static inline u16 idpf_get_reserved_vecs(struct idpf_adapter *adapter) return le16_to_cpu(adapter->caps.num_allocated_vectors); } +/** + * idpf_get_reserved_rdma_vecs - Get reserved RDMA vectors + * @adapter: private data struct + */ +static inline u16 idpf_get_reserved_rdma_vecs(struct idpf_adapter *adapter) +{ + return le16_to_cpu(adapter->caps.num_rdma_allocated_vectors); +} + /** * idpf_get_default_vports - Get default number of vports * @adapter: private data struct diff --git a/drivers/net/ethernet/intel/idpf/idpf_lib.c b/drivers/net/ethernet/intel/idpf/idpf_lib.c index 5dbf2b4ba1b0..e985f27051de 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_lib.c +++ b/drivers/net/ethernet/intel/idpf/idpf_lib.c @@ -87,6 +87,8 @@ void idpf_intr_rel(struct idpf_adapter *adapter) idpf_deinit_vector_stack(adapter); kfree(adapter->msix_entries); adapter->msix_entries = NULL; + kfree(adapter->rdma_msix_entries); + adapter->rdma_msix_entries = NULL; } /** @@ -314,13 +316,33 @@ int idpf_req_rel_vector_indexes(struct idpf_adapter *adapter, */ int idpf_intr_req(struct idpf_adapter *adapter) { + u16 num_lan_vecs, min_lan_vecs, num_rdma_vecs = 0, min_rdma_vecs = 0; u16 default_vports = idpf_get_default_vports(adapter); int num_q_vecs, total_vecs, num_vec_ids; int min_vectors, v_actual, err; unsigned int vector; u16 *vecids; + int i; total_vecs = idpf_get_reserved_vecs(adapter); + num_lan_vecs = total_vecs; + if (idpf_is_rdma_cap_ena(adapter)) { + num_rdma_vecs = idpf_get_reserved_rdma_vecs(adapter); + min_rdma_vecs = IDPF_MIN_RDMA_VEC; + + if (!num_rdma_vecs) { + /* If idpf_get_reserved_rdma_vecs is 0, vectors are + * pulled from the LAN pool. + */ + num_rdma_vecs = min_rdma_vecs; + } else if (num_rdma_vecs < min_rdma_vecs) { + dev_err(&adapter->pdev->dev, + "Not enough vectors reserved for rdma (min: %u, current: %u)\n", + min_rdma_vecs, num_rdma_vecs); + return -EINVAL; + } + } + num_q_vecs = total_vecs - IDPF_MBX_Q_VEC; err = idpf_send_alloc_vectors_msg(adapter, num_q_vecs); @@ -331,27 +353,44 @@ int idpf_intr_req(struct idpf_adapter *adapter) return -EAGAIN; } - min_vectors = IDPF_MBX_Q_VEC + IDPF_MIN_Q_VEC * default_vports; + min_lan_vecs = IDPF_MBX_Q_VEC + IDPF_MIN_Q_VEC * default_vports; + min_vectors = min_lan_vecs + min_rdma_vecs; v_actual = pci_alloc_irq_vectors(adapter->pdev, min_vectors, total_vecs, PCI_IRQ_MSIX); if (v_actual < min_vectors) { - dev_err(&adapter->pdev->dev, "Failed to allocate MSIX vectors: %d\n", + dev_err(&adapter->pdev->dev, "Failed to allocate minimum MSIX vectors required: %d\n", v_actual); err = -EAGAIN; goto send_dealloc_vecs; } - adapter->msix_entries = kcalloc(v_actual, sizeof(struct msix_entry), - GFP_KERNEL); + if (idpf_is_rdma_cap_ena(adapter)) { + if (v_actual < total_vecs) { + dev_warn(&adapter->pdev->dev, + "Warning: not enough vectors available. Defaulting to minimum for RDMA and remaining for LAN.\n"); + num_rdma_vecs = IDPF_MIN_RDMA_VEC; + } + adapter->rdma_msix_entries = + kcalloc(num_rdma_vecs, + sizeof(struct msix_entry), GFP_KERNEL); + if (!adapter->rdma_msix_entries) { + err = -ENOMEM; + goto free_irq; + } + } + + num_lan_vecs = v_actual - num_rdma_vecs; + adapter->msix_entries = kcalloc(num_lan_vecs, sizeof(struct msix_entry), + GFP_KERNEL); if (!adapter->msix_entries) { err = -ENOMEM; - goto free_irq; + goto free_rdma_msix; } idpf_set_mb_vec_id(adapter); - vecids = kcalloc(total_vecs, sizeof(u16), GFP_KERNEL); + vecids = kcalloc(v_actual, sizeof(u16), GFP_KERNEL); if (!vecids) { err = -ENOMEM; goto free_msix; @@ -364,32 +403,36 @@ int idpf_intr_req(struct idpf_adapter *adapter) ac = adapter->req_vec_chunks; vchunks = &ac->vchunks; - num_vec_ids = idpf_get_vec_ids(adapter, vecids, total_vecs, + num_vec_ids = idpf_get_vec_ids(adapter, vecids, v_actual, vchunks); if (num_vec_ids < v_actual) { err = -EINVAL; goto free_vecids; } } else { - int i; - for (i = 0; i < v_actual; i++) vecids[i] = i; } - for (vector = 0; vector < v_actual; vector++) { - adapter->msix_entries[vector].entry = vecids[vector]; - adapter->msix_entries[vector].vector = + for (i = 0, vector = 0; vector < num_lan_vecs; vector++, i++) { + adapter->msix_entries[i].entry = vecids[vector]; + adapter->msix_entries[i].vector = + pci_irq_vector(adapter->pdev, vector); + } + for (i = 0; i < num_rdma_vecs; vector++, i++) { + adapter->rdma_msix_entries[i].entry = vecids[vector]; + adapter->rdma_msix_entries[i].vector = pci_irq_vector(adapter->pdev, vector); } - adapter->num_req_msix = total_vecs; - adapter->num_msix_entries = v_actual; /* 'num_avail_msix' is used to distribute excess vectors to the vports * after considering the minimum vectors required per each default * vport */ - adapter->num_avail_msix = v_actual - min_vectors; + adapter->num_avail_msix = num_lan_vecs - min_lan_vecs; + adapter->num_msix_entries = num_lan_vecs; + if (idpf_is_rdma_cap_ena(adapter)) + adapter->num_rdma_msix_entries = num_rdma_vecs; /* Fill MSIX vector lifo stack with vector indexes */ err = idpf_init_vector_stack(adapter); @@ -411,6 +454,9 @@ int idpf_intr_req(struct idpf_adapter *adapter) free_msix: kfree(adapter->msix_entries); adapter->msix_entries = NULL; +free_rdma_msix: + kfree(adapter->rdma_msix_entries); + adapter->rdma_msix_entries = NULL; free_irq: pci_free_irq_vectors(adapter->pdev); send_dealloc_vecs: diff --git a/drivers/net/ethernet/intel/idpf/idpf_txrx.h b/drivers/net/ethernet/intel/idpf/idpf_txrx.h index 6215dbee5546..63f3ba7d1ab3 100644 --- a/drivers/net/ethernet/intel/idpf/idpf_txrx.h +++ b/drivers/net/ethernet/intel/idpf/idpf_txrx.h @@ -57,6 +57,7 @@ /* Default vector sharing */ #define IDPF_MBX_Q_VEC 1 #define IDPF_MIN_Q_VEC 1 +#define IDPF_MIN_RDMA_VEC 4 #define IDPF_DFLT_TX_Q_DESC_COUNT 512 #define IDPF_DFLT_TX_COMPLQ_DESC_COUNT 512 diff --git a/drivers/net/ethernet/intel/idpf/virtchnl2.h b/drivers/net/ethernet/intel/idpf/virtchnl2.h index 63deb120359c..80c17e4a394e 100644 --- a/drivers/net/ethernet/intel/idpf/virtchnl2.h +++ b/drivers/net/ethernet/intel/idpf/virtchnl2.h @@ -473,6 +473,8 @@ VIRTCHNL2_CHECK_STRUCT_LEN(8, virtchnl2_version_info); * segment offload. * @max_hdr_buf_per_lso: Max number of header buffers that can be used for * an LSO. + * @num_rdma_allocated_vectors: Maximum number of allocated RDMA vectors for + * the device. * @pad1: Padding for future extensions. * * Dataplane driver sends this message to CP to negotiate capabilities and @@ -520,7 +522,8 @@ struct virtchnl2_get_capabilities { __le32 device_type; u8 min_sso_packet_len; u8 max_hdr_buf_per_lso; - u8 pad1[10]; + __le16 num_rdma_allocated_vectors; + u8 pad1[8]; }; VIRTCHNL2_CHECK_STRUCT_LEN(80, virtchnl2_get_capabilities);