From patchwork Wed Nov 13 23:56:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13874388 X-Patchwork-Delegate: kuba@kernel.org Received: from alln-iport-4.cisco.com (alln-iport-4.cisco.com [173.37.142.91]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C809218BB82; Wed, 13 Nov 2024 23:56:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.142.91 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542218; cv=none; b=j8FkDelmo5TlUXBNoegKKWSlepsK6ZtQugFoEUztrCzX+FQVhz7KJLzUxNZ3x1segOwWLbwEE7jg6jvDNszedA6y/5c8GQcJeYZI3zfUDI4EnTzKpZLUZidwLcefcHZcu2PxBmZCUbynAMti1r0THj0WV4sUW906YWkqpjwhXDw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542218; c=relaxed/simple; bh=hrWeRdXAZbogkSLfeHXbAXj699i49M8/2VzfMC2r0zo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=G1u2urNMGOtSVicdaV2Zb3iavoaDom9J7sTw53AZQ1cu8+EhUn+Ib8yKczPPpWuNgWzUFt+31qm5l2Vm+yXO0qCCsnYVZbBI2LkmDOWQaZRkgSlFZfON3Bygb5cojrzn+Js/ZHGQ7IXQj0UCBzb1d+1noqHTsF89ItKBQfKMfrc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=hw8OL7fM; arc=none smtp.client-ip=173.37.142.91 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="hw8OL7fM" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=18988; q=dns/txt; s=iport; t=1731542214; x=1732751814; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=0ed6HZKdaPIq4JhbhBLvsC2gmejf3nvsoXzSRrm0gmg=; b=hw8OL7fMLm8IUsVTpyfnMUnq0Me7SfcfUK4TapD41BIQPU0KU80JBI3U nVKW0USAMvSCi+XhgVotJWu4tfqmdiCP3EG/rgxyrgSPVIeKoRy8UoQyk fHits96yyF1JFxqopWpK0GvIoeVoXgN7d/AWrOKXetwoGzUP8s5C3C8RD Y=; X-CSE-ConnectionGUID: 1EZmtvDkSwifxkjG+gdy7w== X-CSE-MsgGUID: 6E+5tM1OT1WqOm2NdrAe/Q== X-IPAS-Result: A0ABAADHOzVnj5IQJK1aGQEBAQEBAQEBAQEBAQEBAQEBARIBAQEBAQEBAQEBAQFAgT8EAQEBAQELAYQaQkiEVYgdhzCCIYt1kiMUgREDVg8BAQEPRAQBAYUHAopFAiY0CQ4BAgQBAQEBAwIDAQEBAQEBAQEBAQELAQEFAQEBAgEHBRQBAQEBAQE5BQ47hgiGWwIBAyMEUhAlAiYCAisbEAYBEoMBgmUCAbBmen8zgQGEe9k4gW2BGi4BiEsBgWyDfTuEPCcbgUlEglCCLYQqg3SCaQSDO4JbUnYlgRMCAgIHAoN8g3mZKQk/gQUcA1khEQFVEw0KCwcFY1c8AyJvalx6K4EOgRc6Q4E7gSIvGyELXIE3gRoUBhUEgQ5GP4JKaUs6Ag0CNoIkJFmCT4UahG2EZoJULx1AAwsYDUgRLDUGDhsGPQFuB55lRoMyexR/gRQsBSY4klwJAQeDa40+gh+fTIQkoVwzqk0eEJhJIqQchGaBZzqBWzMaCBsVgyJSGQ+OKgMNCRaTAAG2P0M1OwIHCwEBAwmGS4keYAEB IronPort-Data: A9a23:qoFvmaACzPhguBVW/7zjw5YqxClBgxIJ4kV8jS/XYbTApGlzhj0Pz msfWj+EO/6OZGP2f9F+O4+19RkEvcPTyoJiOVdlrnsFo1CmBibm6XV1Cm+qYkt+++WaFBoPA /02M4eGdIZsCCeB/n9BC5C5xVFkz6aEW7HgP+DNPyF1VGdMRTwo4f5Zs7ZRbrVA357gWWthh fuo+5eDYQb8gGYvWo4pw/vrRC1H7ayaVAww5jTSVdgT1HfCmn8cCo4oJK3ZBxPQXolOE+emc P3Ixbe/83mx109F5gSNy+uTnuUiG9Y+DCDW4pZkc/HKbitq+kTe5p0G2M80Mi+7vdkmc+dZk 72hvbToIesg0zaldO41C3G0GAkmVUFKFSOuzXWX6aSuI0P6n3TEmthcCV0oEY4j5ed5EWJM2 r8XJCFScUXW7w626OrTpuhEj8AnKozveYgYoHwllWCfBvc9SpeFSKLPjTNa9G5v3YYVQ7CHO YxANWQHgBfoO3WjPn8RBZ8ll+Cij1H0ciZTrxSeoq9fD237l1wpjei0b4aKEjCMbZt43RuEg zvhwz3GGzYfbvqT0xOItUv504cjmgugBdpNT+fnnhJwu3Wfz3IeDTUaXEW2pP2+hFL4Xd9DQ 2QZ9jcrpLo/6GSkSd7yWxD+q3mB1jYcXMBVCMU55RuLx66S5ByWbkAHUzRIQN8rrsk7QXotz FDht9rvCSZir/6TRG6R+6m8qS60P24eLQcqfSYOQA0Ey8PurIE6klTESdMLOKq0iMDlXDL92 TaHqAAgiLgJy80GzaO2+RbAmT3Ejp7EUgI4+C3JUW+/qAB0foioY8qv81ez0BpbBI+dSl/Eu D0PnNKTqbhfS5qMjyeKBu4KGdlF+sppLhWFrGxqA4cN9A+y+nCzfoJ/oz5yNRh2Z5NslSDSX GffvgZY5Zl2NXSsbLNqb4/ZNyjM5fa8fTgCfq6JBueicqRMmBm7EDaCjHN8PlwBcmBwwcnT2 r/CLa5A6Er274w8k1JaoM9GjdcWKtgWnz+7eHwC503PPUCiTHCUU6wZF1CFc/o06qiJyC2Mr I0GapHQlUUPDbSkCsUyzWL1BQ1URZTcLc2nw/G7isbac2KK5Ul4UaaImuJ7E2Cbt/8Fz7yWl p1CZqOo4AGi3SKcc1rihoFLY7L0VpE3tmMgISEpJh6p3XNlCbtDH49BH6bbiYIPrbQ5pdYtF qFtU5zZXpxnFG+dkxxDNsaVkWCXXEjw7e54F3b+OGBnF3OhLiSVkuLZkvzHr3lSUHHv7pRh8 tVNFGrzGPI+euirN+6OANrH8r97lSF1dD5aN6cQHuRuRQ== IronPort-HdrOrdr: A9a23:RhDw9KMddDZzn8BcTsejsMiBIKoaSvp037B87TEWdfU1SL39qy nAppgmPHPP5wr5HUtBpTmMAsK9qBDnhPtICOsqXItKBzOWwVdARbsKheGO/9SjIVycygc379 YDT0ERMrPN5VYTt7ec3DWF X-Talos-CUID: 9a23:w8EiFGFDcT47aLLEqmI+2XwRBt4JcUbCzXnAIE2XSmllWeKKHAo= X-Talos-MUID: 9a23:J+46BgyKpG/1o0OhUMUFXhuw3M6aqPqDSx0RtMwhh5C7LAt1YRGRlh26BbZyfw== X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,152,1728950400"; d="scan'208";a="380581747" Received: from alln-l-core-09.cisco.com ([173.36.16.146]) by alln-iport-4.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 13 Nov 2024 23:56:53 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by alln-l-core-09.cisco.com (Postfix) with ESMTPS id 498B018000441; Wed, 13 Nov 2024 23:56:53 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id C5339CC128E; Wed, 13 Nov 2024 23:56:52 +0000 (GMT) From: Nelson Escobar Date: Wed, 13 Nov 2024 23:56:33 +0000 Subject: [PATCH net-next v4 1/7] enic: Create enic_wq/rq structures to bundle per wq/rq data Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241113-remove_vic_resource_limits-v4-1-a34cf8570c67@cisco.com> References: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> In-Reply-To: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731542212; l=19533; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=hrWeRdXAZbogkSLfeHXbAXj699i49M8/2VzfMC2r0zo=; b=5zh65VENJHypMD/X7lA60ML16jcRVicTRtK1D9EJWpllO+7XYxuyxWncZnfjlxRLkd8obro+B Sln6p+aHZyOAqtP1vX0pqWnpNb1LPpZj/i8Cnh9G41Lq5LarPe3ILXl X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Patchwork-Delegate: kuba@kernel.org Bundling the wq/rq specific data into dedicated enic_wq/rq structures cleans up the enic structure and simplifies future changes related to wq/rq. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman Reviewed-by: Vadim Fedorenko --- drivers/net/ethernet/cisco/enic/enic.h | 18 ++-- drivers/net/ethernet/cisco/enic/enic_ethtool.c | 4 +- drivers/net/ethernet/cisco/enic/enic_main.c | 120 ++++++++++++------------- drivers/net/ethernet/cisco/enic/enic_res.c | 12 +-- 4 files changed, 81 insertions(+), 73 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index 0cc3644ee8554f52401a0be7f44a1475ab2ea2b9..07459eac2592ce5185321a619577404232cfbc2c 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -162,6 +162,17 @@ struct enic_rq_stats { u64 desc_skip; /* Rx pkt went into later buffer */ }; +struct enic_wq { + spinlock_t lock; /* spinlock for wq */ + struct vnic_wq vwq; + struct enic_wq_stats stats; +} ____cacheline_aligned; + +struct enic_rq { + struct vnic_rq vrq; + struct enic_rq_stats stats; +} ____cacheline_aligned; + /* Per-instance private data structure */ struct enic { struct net_device *netdev; @@ -194,16 +205,13 @@ struct enic { struct enic_port_profile *pp; /* work queue cache line section */ - ____cacheline_aligned struct vnic_wq wq[ENIC_WQ_MAX]; - spinlock_t wq_lock[ENIC_WQ_MAX]; - struct enic_wq_stats wq_stats[ENIC_WQ_MAX]; + ____cacheline_aligned struct enic_wq wq[ENIC_WQ_MAX]; unsigned int wq_count; u16 loop_enable; u16 loop_tag; /* receive queue cache line section */ - ____cacheline_aligned struct vnic_rq rq[ENIC_RQ_MAX]; - struct enic_rq_stats rq_stats[ENIC_RQ_MAX]; + ____cacheline_aligned struct enic_rq rq[ENIC_RQ_MAX]; unsigned int rq_count; struct vxlan_offload vxlan; struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX]; diff --git a/drivers/net/ethernet/cisco/enic/enic_ethtool.c b/drivers/net/ethernet/cisco/enic/enic_ethtool.c index f7986f2b6a1794144909ffad9e3e09c32ea44c93..909d6f7000e160cf2e15de4660c1034cad7d51ba 100644 --- a/drivers/net/ethernet/cisco/enic/enic_ethtool.c +++ b/drivers/net/ethernet/cisco/enic/enic_ethtool.c @@ -337,7 +337,7 @@ static void enic_get_ethtool_stats(struct net_device *netdev, for (i = 0; i < NUM_ENIC_GEN_STATS; i++) *(data++) = ((u64 *)&enic->gen_stats)[enic_gen_stats[i].index]; for (i = 0; i < enic->rq_count; i++) { - struct enic_rq_stats *rqstats = &enic->rq_stats[i]; + struct enic_rq_stats *rqstats = &enic->rq[i].stats; int index; for (j = 0; j < NUM_ENIC_PER_RQ_STATS; j++) { @@ -346,7 +346,7 @@ static void enic_get_ethtool_stats(struct net_device *netdev, } } for (i = 0; i < enic->wq_count; i++) { - struct enic_wq_stats *wqstats = &enic->wq_stats[i]; + struct enic_wq_stats *wqstats = &enic->wq[i].stats; int index; for (j = 0; j < NUM_ENIC_PER_WQ_STATS; j++) { diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index ffed14b63d41d1737c577fe1662eb1c2c8aea808..eb00058b6c68ec5c1ac433b54b5bc6f3fb613777 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -342,8 +342,8 @@ static void enic_wq_free_buf(struct vnic_wq *wq, { struct enic *enic = vnic_dev_priv(wq->vdev); - enic->wq_stats[wq->index].cq_work++; - enic->wq_stats[wq->index].cq_bytes += buf->len; + enic->wq[wq->index].stats.cq_work++; + enic->wq[wq->index].stats.cq_bytes += buf->len; enic_free_wq_buf(wq, buf); } @@ -352,20 +352,20 @@ static int enic_wq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, { struct enic *enic = vnic_dev_priv(vdev); - spin_lock(&enic->wq_lock[q_number]); + spin_lock(&enic->wq[q_number].lock); - vnic_wq_service(&enic->wq[q_number], cq_desc, + vnic_wq_service(&enic->wq[q_number].vwq, cq_desc, completed_index, enic_wq_free_buf, opaque); if (netif_tx_queue_stopped(netdev_get_tx_queue(enic->netdev, q_number)) && - vnic_wq_desc_avail(&enic->wq[q_number]) >= + vnic_wq_desc_avail(&enic->wq[q_number].vwq) >= (MAX_SKB_FRAGS + ENIC_DESC_MAX_SPLITS)) { netif_wake_subqueue(enic->netdev, q_number); - enic->wq_stats[q_number].wake++; + enic->wq[q_number].stats.wake++; } - spin_unlock(&enic->wq_lock[q_number]); + spin_unlock(&enic->wq[q_number].lock); return 0; } @@ -377,7 +377,7 @@ static bool enic_log_q_error(struct enic *enic) bool err = false; for (i = 0; i < enic->wq_count; i++) { - error_status = vnic_wq_error_status(&enic->wq[i]); + error_status = vnic_wq_error_status(&enic->wq[i].vwq); err |= error_status; if (error_status) netdev_err(enic->netdev, "WQ[%d] error_status %d\n", @@ -385,7 +385,7 @@ static bool enic_log_q_error(struct enic *enic) } for (i = 0; i < enic->rq_count; i++) { - error_status = vnic_rq_error_status(&enic->rq[i]); + error_status = vnic_rq_error_status(&enic->rq[i].vrq); err |= error_status; if (error_status) netdev_err(enic->netdev, "RQ[%d] error_status %d\n", @@ -598,9 +598,9 @@ static int enic_queue_wq_skb_vlan(struct enic *enic, struct vnic_wq *wq, err = enic_queue_wq_skb_cont(enic, wq, skb, len_left, loopback); /* The enic_queue_wq_desc() above does not do HW checksum */ - enic->wq_stats[wq->index].csum_none++; - enic->wq_stats[wq->index].packets++; - enic->wq_stats[wq->index].bytes += skb->len; + enic->wq[wq->index].stats.csum_none++; + enic->wq[wq->index].stats.packets++; + enic->wq[wq->index].stats.bytes += skb->len; return err; } @@ -634,9 +634,9 @@ static int enic_queue_wq_skb_csum_l4(struct enic *enic, struct vnic_wq *wq, if (!eop) err = enic_queue_wq_skb_cont(enic, wq, skb, len_left, loopback); - enic->wq_stats[wq->index].csum_partial++; - enic->wq_stats[wq->index].packets++; - enic->wq_stats[wq->index].bytes += skb->len; + enic->wq[wq->index].stats.csum_partial++; + enic->wq[wq->index].stats.packets++; + enic->wq[wq->index].stats.bytes += skb->len; return err; } @@ -699,11 +699,11 @@ static int enic_queue_wq_skb_tso(struct enic *enic, struct vnic_wq *wq, if (skb->encapsulation) { hdr_len = skb_inner_tcp_all_headers(skb); enic_preload_tcp_csum_encap(skb); - enic->wq_stats[wq->index].encap_tso++; + enic->wq[wq->index].stats.encap_tso++; } else { hdr_len = skb_tcp_all_headers(skb); enic_preload_tcp_csum(skb); - enic->wq_stats[wq->index].tso++; + enic->wq[wq->index].stats.tso++; } /* Queue WQ_ENET_MAX_DESC_LEN length descriptors @@ -757,8 +757,8 @@ static int enic_queue_wq_skb_tso(struct enic *enic, struct vnic_wq *wq, pkts = len / mss; if ((len % mss) > 0) pkts++; - enic->wq_stats[wq->index].packets += pkts; - enic->wq_stats[wq->index].bytes += (len + (pkts * hdr_len)); + enic->wq[wq->index].stats.packets += pkts; + enic->wq[wq->index].stats.bytes += (len + (pkts * hdr_len)); return 0; } @@ -792,9 +792,9 @@ static inline int enic_queue_wq_skb_encap(struct enic *enic, struct vnic_wq *wq, if (!eop) err = enic_queue_wq_skb_cont(enic, wq, skb, len_left, loopback); - enic->wq_stats[wq->index].encap_csum++; - enic->wq_stats[wq->index].packets++; - enic->wq_stats[wq->index].bytes += skb->len; + enic->wq[wq->index].stats.encap_csum++; + enic->wq[wq->index].stats.packets++; + enic->wq[wq->index].stats.bytes += skb->len; return err; } @@ -812,7 +812,7 @@ static inline int enic_queue_wq_skb(struct enic *enic, /* VLAN tag from trunking driver */ vlan_tag_insert = 1; vlan_tag = skb_vlan_tag_get(skb); - enic->wq_stats[wq->index].add_vlan++; + enic->wq[wq->index].stats.add_vlan++; } else if (enic->loop_enable) { vlan_tag = enic->loop_tag; loopback = 1; @@ -859,11 +859,11 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, struct netdev_queue *txq; txq_map = skb_get_queue_mapping(skb) % enic->wq_count; - wq = &enic->wq[txq_map]; + wq = &enic->wq[txq_map].vwq; if (skb->len <= 0) { dev_kfree_skb_any(skb); - enic->wq_stats[wq->index].null_pkt++; + enic->wq[wq->index].stats.null_pkt++; return NETDEV_TX_OK; } @@ -878,19 +878,19 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, skb_shinfo(skb)->nr_frags + 1 > ENIC_NON_TSO_MAX_DESC && skb_linearize(skb)) { dev_kfree_skb_any(skb); - enic->wq_stats[wq->index].skb_linear_fail++; + enic->wq[wq->index].stats.skb_linear_fail++; return NETDEV_TX_OK; } - spin_lock(&enic->wq_lock[txq_map]); + spin_lock(&enic->wq[txq_map].lock); if (vnic_wq_desc_avail(wq) < skb_shinfo(skb)->nr_frags + ENIC_DESC_MAX_SPLITS) { netif_tx_stop_queue(txq); /* This is a hard error, log it */ netdev_err(netdev, "BUG! Tx ring full when queue awake!\n"); - spin_unlock(&enic->wq_lock[txq_map]); - enic->wq_stats[wq->index].desc_full_awake++; + spin_unlock(&enic->wq[txq_map].lock); + enic->wq[wq->index].stats.desc_full_awake++; return NETDEV_TX_BUSY; } @@ -899,14 +899,14 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, if (vnic_wq_desc_avail(wq) < MAX_SKB_FRAGS + ENIC_DESC_MAX_SPLITS) { netif_tx_stop_queue(txq); - enic->wq_stats[wq->index].stopped++; + enic->wq[wq->index].stats.stopped++; } skb_tx_timestamp(skb); if (!netdev_xmit_more() || netif_xmit_stopped(txq)) vnic_wq_doorbell(wq); error: - spin_unlock(&enic->wq_lock[txq_map]); + spin_unlock(&enic->wq[txq_map].lock); return NETDEV_TX_OK; } @@ -941,9 +941,9 @@ static void enic_get_stats(struct net_device *netdev, net_stats->multicast = stats->rx.rx_multicast_frames_ok; for (i = 0; i < ENIC_RQ_MAX; i++) { - struct enic_rq_stats *rqs = &enic->rq_stats[i]; + struct enic_rq_stats *rqs = &enic->rq[i].stats; - if (!enic->rq->ctrl) + if (!enic->rq[i].vrq.ctrl) break; pkt_truncated += rqs->pkt_truncated; bad_fcs += rqs->bad_fcs; @@ -1313,7 +1313,7 @@ static int enic_rq_alloc_buf(struct vnic_rq *rq) } skb = netdev_alloc_skb_ip_align(netdev, len); if (!skb) { - enic->rq_stats[rq->index].no_skb++; + enic->rq[rq->index].stats.no_skb++; return -ENOMEM; } @@ -1366,7 +1366,7 @@ static void enic_rq_indicate_buf(struct vnic_rq *rq, struct net_device *netdev = enic->netdev; struct sk_buff *skb; struct vnic_cq *cq = &enic->cq[enic_cq_rq(enic, rq->index)]; - struct enic_rq_stats *rqstats = &enic->rq_stats[rq->index]; + struct enic_rq_stats *rqstats = &enic->rq[rq->index].stats; u8 type, color, eop, sop, ingress_port, vlan_stripped; u8 fcoe, fcoe_sof, fcoe_fc_crc_ok, fcoe_enc_error, fcoe_eof; @@ -1512,7 +1512,7 @@ static int enic_rq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, { struct enic *enic = vnic_dev_priv(vdev); - vnic_rq_service(&enic->rq[q_number], cq_desc, + vnic_rq_service(&enic->rq[q_number].vrq, cq_desc, completed_index, VNIC_RQ_RETURN_DESC, enic_rq_indicate_buf, opaque); @@ -1609,7 +1609,7 @@ static int enic_poll(struct napi_struct *napi, int budget) 0 /* don't unmask intr */, 0 /* don't reset intr timer */); - err = vnic_rq_fill(&enic->rq[0], enic_rq_alloc_buf); + err = vnic_rq_fill(&enic->rq[0].vrq, enic_rq_alloc_buf); /* Buffer allocation failed. Stay in polling * mode so we can try to fill the ring again. @@ -1621,7 +1621,7 @@ static int enic_poll(struct napi_struct *napi, int budget) /* Call the function which refreshes the intr coalescing timer * value based on the traffic. */ - enic_calc_int_moderation(enic, &enic->rq[0]); + enic_calc_int_moderation(enic, &enic->rq[0].vrq); if ((rq_work_done < budget) && napi_complete_done(napi, rq_work_done)) { @@ -1630,11 +1630,11 @@ static int enic_poll(struct napi_struct *napi, int budget) */ if (enic->rx_coalesce_setting.use_adaptive_rx_coalesce) - enic_set_int_moderation(enic, &enic->rq[0]); + enic_set_int_moderation(enic, &enic->rq[0].vrq); vnic_intr_unmask(&enic->intr[intr]); - enic->rq_stats[0].napi_complete++; + enic->rq[0].stats.napi_complete++; } else { - enic->rq_stats[0].napi_repoll++; + enic->rq[0].stats.napi_repoll++; } return rq_work_done; @@ -1683,7 +1683,7 @@ static int enic_poll_msix_wq(struct napi_struct *napi, int budget) struct net_device *netdev = napi->dev; struct enic *enic = netdev_priv(netdev); unsigned int wq_index = (napi - &enic->napi[0]) - enic->rq_count; - struct vnic_wq *wq = &enic->wq[wq_index]; + struct vnic_wq *wq = &enic->wq[wq_index].vwq; unsigned int cq; unsigned int intr; unsigned int wq_work_to_do = ENIC_WQ_NAPI_BUDGET; @@ -1737,7 +1737,7 @@ static int enic_poll_msix_rq(struct napi_struct *napi, int budget) 0 /* don't unmask intr */, 0 /* don't reset intr timer */); - err = vnic_rq_fill(&enic->rq[rq], enic_rq_alloc_buf); + err = vnic_rq_fill(&enic->rq[rq].vrq, enic_rq_alloc_buf); /* Buffer allocation failed. Stay in polling mode * so we can try to fill the ring again. @@ -1749,7 +1749,7 @@ static int enic_poll_msix_rq(struct napi_struct *napi, int budget) /* Call the function which refreshes the intr coalescing timer * value based on the traffic. */ - enic_calc_int_moderation(enic, &enic->rq[rq]); + enic_calc_int_moderation(enic, &enic->rq[rq].vrq); if ((work_done < budget) && napi_complete_done(napi, work_done)) { @@ -1758,11 +1758,11 @@ static int enic_poll_msix_rq(struct napi_struct *napi, int budget) */ if (enic->rx_coalesce_setting.use_adaptive_rx_coalesce) - enic_set_int_moderation(enic, &enic->rq[rq]); + enic_set_int_moderation(enic, &enic->rq[rq].vrq); vnic_intr_unmask(&enic->intr[intr]); - enic->rq_stats[rq].napi_complete++; + enic->rq[rq].stats.napi_complete++; } else { - enic->rq_stats[rq].napi_repoll++; + enic->rq[rq].stats.napi_repoll++; } return work_done; @@ -1989,10 +1989,10 @@ static int enic_open(struct net_device *netdev) for (i = 0; i < enic->rq_count; i++) { /* enable rq before updating rq desc */ - vnic_rq_enable(&enic->rq[i]); - vnic_rq_fill(&enic->rq[i], enic_rq_alloc_buf); + vnic_rq_enable(&enic->rq[i].vrq); + vnic_rq_fill(&enic->rq[i].vrq, enic_rq_alloc_buf); /* Need at least one buffer on ring to get going */ - if (vnic_rq_desc_used(&enic->rq[i]) == 0) { + if (vnic_rq_desc_used(&enic->rq[i].vrq) == 0) { netdev_err(netdev, "Unable to alloc receive buffers\n"); err = -ENOMEM; goto err_out_free_rq; @@ -2000,7 +2000,7 @@ static int enic_open(struct net_device *netdev) } for (i = 0; i < enic->wq_count; i++) - vnic_wq_enable(&enic->wq[i]); + vnic_wq_enable(&enic->wq[i].vwq); if (!enic_is_dynamic(enic) && !enic_is_sriov_vf(enic)) enic_dev_add_station_addr(enic); @@ -2027,9 +2027,9 @@ static int enic_open(struct net_device *netdev) err_out_free_rq: for (i = 0; i < enic->rq_count; i++) { - ret = vnic_rq_disable(&enic->rq[i]); + ret = vnic_rq_disable(&enic->rq[i].vrq); if (!ret) - vnic_rq_clean(&enic->rq[i], enic_free_rq_buf); + vnic_rq_clean(&enic->rq[i].vrq, enic_free_rq_buf); } enic_dev_notify_unset(enic); err_out_free_intr: @@ -2071,12 +2071,12 @@ static int enic_stop(struct net_device *netdev) enic_dev_del_station_addr(enic); for (i = 0; i < enic->wq_count; i++) { - err = vnic_wq_disable(&enic->wq[i]); + err = vnic_wq_disable(&enic->wq[i].vwq); if (err) return err; } for (i = 0; i < enic->rq_count; i++) { - err = vnic_rq_disable(&enic->rq[i]); + err = vnic_rq_disable(&enic->rq[i].vrq); if (err) return err; } @@ -2086,9 +2086,9 @@ static int enic_stop(struct net_device *netdev) enic_free_intr(enic); for (i = 0; i < enic->wq_count; i++) - vnic_wq_clean(&enic->wq[i], enic_free_wq_buf); + vnic_wq_clean(&enic->wq[i].vwq, enic_free_wq_buf); for (i = 0; i < enic->rq_count; i++) - vnic_rq_clean(&enic->rq[i], enic_free_rq_buf); + vnic_rq_clean(&enic->rq[i].vrq, enic_free_rq_buf); for (i = 0; i < enic->cq_count; i++) vnic_cq_clean(&enic->cq[i]); for (i = 0; i < enic->intr_count; i++) @@ -2576,7 +2576,7 @@ static void enic_get_queue_stats_rx(struct net_device *dev, int idx, struct netdev_queue_stats_rx *rxs) { struct enic *enic = netdev_priv(dev); - struct enic_rq_stats *rqstats = &enic->rq_stats[idx]; + struct enic_rq_stats *rqstats = &enic->rq[idx].stats; rxs->bytes = rqstats->bytes; rxs->packets = rqstats->packets; @@ -2590,7 +2590,7 @@ static void enic_get_queue_stats_tx(struct net_device *dev, int idx, struct netdev_queue_stats_tx *txs) { struct enic *enic = netdev_priv(dev); - struct enic_wq_stats *wqstats = &enic->wq_stats[idx]; + struct enic_wq_stats *wqstats = &enic->wq[idx].stats; txs->bytes = wqstats->bytes; txs->packets = wqstats->packets; @@ -2993,7 +2993,7 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) INIT_WORK(&enic->change_mtu_work, enic_change_mtu_work); for (i = 0; i < enic->wq_count; i++) - spin_lock_init(&enic->wq_lock[i]); + spin_lock_init(&enic->wq[i].lock); /* Register net device */ diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 1c48aebdbab02b88293544dcabda2d90d1a71a70..60be09acb9fd56b642b7cabc77fac01f526b29a2 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -176,9 +176,9 @@ void enic_free_vnic_resources(struct enic *enic) unsigned int i; for (i = 0; i < enic->wq_count; i++) - vnic_wq_free(&enic->wq[i]); + vnic_wq_free(&enic->wq[i].vwq); for (i = 0; i < enic->rq_count; i++) - vnic_rq_free(&enic->rq[i]); + vnic_rq_free(&enic->rq[i].vrq); for (i = 0; i < enic->cq_count; i++) vnic_cq_free(&enic->cq[i]); for (i = 0; i < enic->intr_count; i++) @@ -233,7 +233,7 @@ void enic_init_vnic_resources(struct enic *enic) for (i = 0; i < enic->rq_count; i++) { cq_index = i; - vnic_rq_init(&enic->rq[i], + vnic_rq_init(&enic->rq[i].vrq, cq_index, error_interrupt_enable, error_interrupt_offset); @@ -241,7 +241,7 @@ void enic_init_vnic_resources(struct enic *enic) for (i = 0; i < enic->wq_count; i++) { cq_index = enic->rq_count + i; - vnic_wq_init(&enic->wq[i], + vnic_wq_init(&enic->wq[i].vwq, cq_index, error_interrupt_enable, error_interrupt_offset); @@ -322,7 +322,7 @@ int enic_alloc_vnic_resources(struct enic *enic) */ for (i = 0; i < enic->wq_count; i++) { - err = vnic_wq_alloc(enic->vdev, &enic->wq[i], i, + err = vnic_wq_alloc(enic->vdev, &enic->wq[i].vwq, i, enic->config.wq_desc_count, sizeof(struct wq_enet_desc)); if (err) @@ -330,7 +330,7 @@ int enic_alloc_vnic_resources(struct enic *enic) } for (i = 0; i < enic->rq_count; i++) { - err = vnic_rq_alloc(enic->vdev, &enic->rq[i], i, + err = vnic_rq_alloc(enic->vdev, &enic->rq[i].vrq, i, enic->config.rq_desc_count, sizeof(struct rq_enet_desc)); if (err) From patchwork Wed Nov 13 23:56:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13874387 X-Patchwork-Delegate: kuba@kernel.org Received: from alln-iport-5.cisco.com (alln-iport-5.cisco.com [173.37.142.92]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D03818C01A; Wed, 13 Nov 2024 23:56:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.142.92 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542217; cv=none; b=UlaTdjZuUPbEPJm4e2DNbJklzwCipcO/pnkHWaopKLAJFRgtTv/4Css5fHxvw1ipQK0EcNdwviFLUee1Zt89lX7n71eQfJnIcenFobcyNB4EBN2uyfjTLqziX+g5U630b/3R0ZX8sESCb3XVf72av/yShKAz3r+0JGa+Dsg1Rzs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542217; c=relaxed/simple; bh=bjDYEaN1jnLUpoSt2mCpRky26CJzLy56FeZuHu3EI4M=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Pkh7MjkGeE+xPYADBSUNPV4de/ENCZG8xDH83Fi0upJC9neS4p4DV4ggmeF/B9Hujdazx+sv6vam3PuUcDglFVkSdJ0CGU/R1ZAafUdDzwcAQIz2iMtSPco6bc2UoFr/eG2vwseQEb2cJgrKMgDAocsgx9SqX0wY7qqGcBmbfgc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=iWdPgm/S; arc=none smtp.client-ip=173.37.142.92 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="iWdPgm/S" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=3562; q=dns/txt; s=iport; t=1731542215; x=1732751815; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=otUS7+T9V88sqMYIwIvCRgKR9l6KDUgEA7HwdHgcsl4=; b=iWdPgm/SDehEywARFjrWYoOeB5W0UPRHzKu19FaETUnW8uTYt+A/ajcm TYb8mX+rujJVQyVN4w97ilQvJYHef/6xlUdZX5cTBLKrjlCIUuT6JiICE i3ajxX2iSDplb8i7XcGP6nrgyCY2Id0tO9GxumkLTI1YqUObcNEbblD1r 4=; X-CSE-ConnectionGUID: 49hTXwsMTuiKsYzAhknrJQ== X-CSE-MsgGUID: 044Pnhr6SQG7+68S8/jGYQ== X-IPAS-Result: A0ACAADHOzVnj5EQJK1aGgEBAQEBAQEBAQEDAQEBARIBAQEBAgIBAQEBQIE/BQEBAQELAYQaQkiEVYgdhzCCIYt1kiOBJQNWDwEBAQ9EBAEBhQcCikUCJjQJDgECBAEBAQEDAgMBAQEBAQEBAQEBAQsBAQUBAQECAQcFFAEBAQEBATkFDjuGCIZbAgEDIwRSECUCJgICKxsQBgESgwGCZQIBsGZ6fzOBAYR72TiBbYEaLgGISwGBbIN9O4Q8JxuBSUSBFAGDaIgegmkEh14lhR6DeZkpCT+BBRwDWSERAVUTDQoLBwVjVzwDIm9qXHorgQ6BFzpDgTuBIi8bIQtcgTeBGhQGFQSBDkY/gkppSzoCDQI2giQkWYJPhRqEbYRmglQvHUADCxgNSBEsNQYOGwY9AW4HnmVGgysHexSBLgJAPxCXO40+oWuEJKFcM6pNmHcipByEZoFnOoFbMxoIGxWDIlIZD44tDQkWkwABtj9DNTsCBwsBAQMJkEkBAQ IronPort-Data: A9a23:9jWB967t4XtBwQ/LbazhgwxRtK3HchMFZxGqfqrLsTDasY5as4F+v jcfWW6Fa6yIazb0ctwlbNi+8ksCvZXXmtFrSgY5+C9nZn8b8sCt6fZ1gavT04J+CuWZESqLO u1HMoGowPgcFyGa/lH1dOC89RGQ7InQLpLkEunIJyttcgFtTSYlmHpLlvUw6mJSqYDR7zil5 5Wr+aUzBHf/g2QoaztNsPrYwP9SlK2aVA0w7wRWic9j5Dcyp1FNZLoDKKe4KWfPQ4U8NoaSW +bZwbilyXjS9hErB8nNuu6TnpoiG+O60aCm0xK6aoD66vRwjnVaPpUTaJLwXXxqZwChxLid/ jniWauYEm/FNoWU8AgUvoIx/ytWZcWq85efSZSzXFD6I0DuKxPRL/tS4E4eYdwYxtZoGkRy6 P07MwEfRxCAgfiU+efuIgVsrpxLwMjDNYcbvDRkiDreF/tjGcCFSKTR7tge1zA17ixMNa+BP IxCN3w2MlKZP0An1lQ/UPrSmM+ui3TkeDpSoXqepLE85C7YywkZPL3Fa4qMI4HSGJsP9qqej jqao1z/AgtED+Wwlnnbokn8we3grBquDer+E5Xjq6Y12wfMroAJMzUaXEW2pNG1g1CzXtZYJ VBS/CcyxYA/+FGuR8vwQzW3p3mLuhNaUN1Ve8U67xuI0YLX7hyfC2xCSSROAPQlqcU/bT8nz FmEm5XuHzMHmLSTRWiQ6fSSoC++NDY9KXIEY2kPTWMt+9DprYcypgjCQtZqDOi+ididMTXxx S2a6SsznbMeieYV2Kihu1PKmTShot7OVAFdzgPaQm6o8Ctna4O/IY+l817W6bBHNonxc7Wal HEAn87b6KUFCouA0XTdBu4MB7quof2CNVUwnGKDAbF9pjSRy2G4WLxN4RJlFVVAMsUCYx3QN Rq7VRxq2LdfO36jbKlSao23Ctg3waWIKTgDfq6IBjapSsYsHDJr7B1TiVisM3cBeXXAcJ3T2 7/HIa5A7l5DVcyLKQZaoc9Gj9fHIQhlmQvuqWjTlUjP7FZnTCf9pU05GFWPdPsly6iPvR/Y9 d1SX+PTlE4GDbekPHmHrNdJRbzvEZTdLc2owyCwXrPSSjeK5El4Wpc9PJt4IdU8xPUP/gs21 irtAxQGoLYAuZE3AV7XMi84MuyHsWdXpnMgNitkJkezx3Umes6u6qxZH6bbjpF5nNGPOcVcF qFfE+3ZW6wnYm2ep1w1M8KnxKQ8L0vDuO57F3b+CNTJV8I7H1SRkjIlFyOznBQz4t2f7pti/ eb4jVyEEPLuhW1KVa7rVR5m9Hvp1VB1pQ64dxKgzgV7EKk0zLVXFg== IronPort-HdrOrdr: A9a23:02naXqBxOM/qwIflHelh55DYdb4zR+YMi2TDGXoBLiC9Ffb5qy nOppUmPHDP5Ar5NEtMpTnEAtjkfZq+z/BICPcqTNSftWDd0QPCRr2Kr7GSoQEIcBeQygcy79 YFT4FOTPD9ElR+i9/3+02bH8ZI+qj+zImYwcrT0HtpSxhncOVb7wl/AhuGCUEefng+OXNALu v72iKCzADQA0j+qa+AdwI4Y9Q= X-Talos-CUID: 9a23:6xNuYmCLFa0dmyT6E3U29HIkBdsATmDc72zhMWa4CmBzaoTAHA== X-Talos-MUID: 9a23:rOl+/QxzusnQ6KAGl8zOEyIqj+SaqPzyVHgLkY9YgPCram8uawzNyzaYc7Zyfw== X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,152,1728950400"; d="scan'208";a="381150562" Received: from alln-l-core-08.cisco.com ([173.36.16.145]) by alln-iport-5.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 13 Nov 2024 23:56:53 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by alln-l-core-08.cisco.com (Postfix) with ESMTPS id 4A1AE1800022B; Wed, 13 Nov 2024 23:56:53 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id CACC6CC128F; Wed, 13 Nov 2024 23:56:52 +0000 (GMT) From: Nelson Escobar Date: Wed, 13 Nov 2024 23:56:34 +0000 Subject: [PATCH net-next v4 2/7] enic: Make MSI-X I/O interrupts come after the other required ones Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241113-remove_vic_resource_limits-v4-2-a34cf8570c67@cisco.com> References: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> In-Reply-To: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731542212; l=3619; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=bjDYEaN1jnLUpoSt2mCpRky26CJzLy56FeZuHu3EI4M=; b=lMbf8PX3anuiIBjxFEGXWJKqDeQFTx6rDIaYdYU9Q92pwiXTFyigaJ+++7Z60dgfN1nwwwyt2 nSTJMqkcbxyCxJS1/Lg31bjiI23/Sj9Zt+yIwYEFAxtPYAndQ0Ifr6b X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Outbound-SMTP-Client: 171.70.41.192, neescoba-vicdev.cisco.com X-Outbound-Node: alln-l-core-08.cisco.com X-Patchwork-Delegate: kuba@kernel.org The VIC hardware has a constraint that the MSIX interrupt used for errors be specified as a 7 bit number. Before this patch, it was allocated after the I/O interrupts, which would cause a problem if 128 or more I/O interrupts are in use. So make the required interrupts come before the I/O interrupts to guarantee the error interrupt offset never exceeds 7 bits. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Reviewed-by: Simon Horman Signed-off-by: Nelson Escobar Reviewed-by: Vadim Fedorenko --- drivers/net/ethernet/cisco/enic/enic.h | 20 +++++++++++++++----- drivers/net/ethernet/cisco/enic/enic_res.c | 11 +++++++---- 2 files changed, 22 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index 07459eac2592ce5185321a619577404232cfbc2c..ec83a273d1ca40ae89f3c193207cf26814f6b277 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -280,18 +280,28 @@ static inline unsigned int enic_msix_wq_intr(struct enic *enic, return enic->cq[enic_cq_wq(enic, wq)].interrupt_offset; } -static inline unsigned int enic_msix_err_intr(struct enic *enic) -{ - return enic->rq_count + enic->wq_count; -} +/* MSIX interrupts are organized as the error interrupt, then the notify + * interrupt followed by all the I/O interrupts. The error interrupt needs + * to fit in 7 bits due to hardware constraints + */ +#define ENIC_MSIX_RESERVED_INTR 2 +#define ENIC_MSIX_ERR_INTR 0 +#define ENIC_MSIX_NOTIFY_INTR 1 +#define ENIC_MSIX_IO_INTR_BASE ENIC_MSIX_RESERVED_INTR +#define ENIC_MSIX_MIN_INTR (ENIC_MSIX_RESERVED_INTR + 2) #define ENIC_LEGACY_IO_INTR 0 #define ENIC_LEGACY_ERR_INTR 1 #define ENIC_LEGACY_NOTIFY_INTR 2 +static inline unsigned int enic_msix_err_intr(struct enic *enic) +{ + return ENIC_MSIX_ERR_INTR; +} + static inline unsigned int enic_msix_notify_intr(struct enic *enic) { - return enic->rq_count + enic->wq_count + 1; + return ENIC_MSIX_NOTIFY_INTR; } static inline bool enic_is_err_intr(struct enic *enic, int intr) diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 60be09acb9fd56b642b7cabc77fac01f526b29a2..72b51e9d8d1a26a2cd18df9c9d702e5b11993b70 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -221,9 +221,12 @@ void enic_init_vnic_resources(struct enic *enic) switch (intr_mode) { case VNIC_DEV_INTR_MODE_INTX: + error_interrupt_enable = 1; + error_interrupt_offset = ENIC_LEGACY_ERR_INTR; + break; case VNIC_DEV_INTR_MODE_MSIX: error_interrupt_enable = 1; - error_interrupt_offset = enic->intr_count - 2; + error_interrupt_offset = enic_msix_err_intr(enic); break; default: error_interrupt_enable = 0; @@ -249,15 +252,15 @@ void enic_init_vnic_resources(struct enic *enic) /* Init CQ resources * - * CQ[0 - n+m-1] point to INTR[0] for INTx, MSI - * CQ[0 - n+m-1] point to INTR[0 - n+m-1] for MSI-X + * All CQs point to INTR[0] for INTx, MSI + * CQ[i] point to INTR[ENIC_MSIX_IO_INTR_BASE + i] for MSI-X */ for (i = 0; i < enic->cq_count; i++) { switch (intr_mode) { case VNIC_DEV_INTR_MODE_MSIX: - interrupt_offset = i; + interrupt_offset = ENIC_MSIX_IO_INTR_BASE + i; break; default: interrupt_offset = 0; From patchwork Wed Nov 13 23:56:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13874389 X-Patchwork-Delegate: kuba@kernel.org Received: from alln-iport-1.cisco.com (alln-iport-1.cisco.com [173.37.142.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3647191466; Wed, 13 Nov 2024 23:57:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.142.88 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542222; cv=none; b=aAOD3fr9+AJ+vqVDi3125PM0mAA0UteL1zLY2TZTzMU278qUWZukToqm7HSqTbPJyIisEC10GbaXH76ly92C7mA18ujLQJX5bFumj12cwpuJ+Hv5ZFWbqFKPAPQJUqMgRL0WDNWumgC38WShR9lvHcmYa/cVjvt+eTLJ8c7Jmyw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542222; c=relaxed/simple; bh=51HgZ51PPJWb3IXVJ68mkV9bh9OMM4AxdY0gpO3hPNs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Sc1P9p7xeQH6KhtgLyhrQjrN97suC4ANwUQfIgTBaEKdeQGLT/bhRFjVIZ+5Ggk+t0rokF10RWIipD3E7GtOOUXm6edccb822rowjaRPXEJ8Wxe9Dl4OasfwaSK8YjPLy0cQ1DJQYzEhoV+GhJAjAdTYu+Rr4I7DepDjJmTkjE4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=MIxP48Y4; arc=none smtp.client-ip=173.37.142.88 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="MIxP48Y4" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=4143; q=dns/txt; s=iport; t=1731542220; x=1732751820; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=lrKZ3TBoxKRQYQGUg6LTUTzsyeGOVtpLFpggJFh1SfY=; b=MIxP48Y4lMRj9sFInzWLXWzLZNZ5IF7Xl3VYRaUPkLEb+yNE+4bO8kew mUxct0xWlLmrVnw4rFHA8eWrPiLFQuLH3DbrQNNnOCIYoSXFwG5Tmxbd8 pS+Yv9tT4nt/7gXxLrTfithL9e85JutmVg5t0lc+88uIkC/ANv6HovYR4 A=; X-CSE-ConnectionGUID: MRObatH8TkCeBNlz8sq56A== X-CSE-MsgGUID: bOpgorpbQIGkShFEYSH01g== X-IPAS-Result: A0ATAADHOzVnj5UQJK1aHAEBAQEBAQcBARIBAQQEAQFAgT8HAQELAYQaQkiEVYgdhzCCIYt1kiOBJQNWDwEBAQ9EBAEBhQcCikUCJjQJDgECBAEBAQEDAgMBAQEBAQEBAQEBAQsBAQUBAQECAQcFFAEBAQEBATkFDjuGCIZbAgEDIwRSECUCJgICKxsQBgESgwGCZQIBsGZ6fzOBAYR72TiBbYEaLgGISwGBbIN9O4Q8JxuBSUSCUIE+b4gegmkEhmh2hUODeZkpCT+BBRwDWSERAVUTDQoLBwVjVzwDIm9qXHorgQ6BFzpDgTuBIi8bIQtcgTeBGhQGFQSBDkY/gkppSzoCDQI2giQkWYJPhRqEbYRmglQvHUADCxgNSBEsNQYOGwY9AW4HnmVGgysHgQ+CEyxjlliNPqFrhCShXDOEBJQBkkiYdyKkHIRmgWc6gVszGggbFYMiUhkPji0NCZMWAbY/QzU7AgcLAQEDCZBJAQE IronPort-Data: A9a23:LjhLrq0hJBPZIr8iDfbD5UZxkn2cJEfYwER7XKvMYLTBsI5bp2FSm GEfC2iCa/qJYjb0coh3aNm/oBgGvJKAztdmGVZr3Hw8FHgiRegpqji6wuYcGwvIc6UvmWo+t 512huHodZ5yFjmE4E/watANlFEkvYmQXL3wFeXYDS54QA5gWU8JhAlq8wIDqtYAbeORXUXU4 7sen+WFYAX5gmctaTpPg06+gEoHUMra6WtwUmMWPZinjHeG/1EJAZQWI72GLneQauF8Au6gS u/f+6qy92Xf8g1FIovNfmHTKxBirhb6ZGBiu1IOM0SQqkEqSh8ajs7XAMEhhXJ/0F1lqTzeJ OJl7vRcQS9xVkHFdX90vxNwS0mSNoUekFPLzOTWXcG7lyX7n3XQL/pGKXsRN4QE2rZLAkpD3 9ocOmAWd03drrfjqF67YrEEasULJc3vOsYb/3pn1zycVKxgSpHYSKKM7thdtNsyrpkRRrCFO YxAN3w2MEWojx5nYj/7DLo4keqzjX71ehVTqUmeouw85G27IAlZi+izaYGKJ4bSLSlTtky/v m3B5XzzOzA1N+ef5xe0yVL03+CayEsXX6pJSeXnraQ16LGJ/UQfAQMbUHO3qOe0j0q5Vc4ZL UEIkgIjobU3/V6mUvHyWBq3pHPCtRkZM/JQFPc/8ymOx7DS7gLfAXILJhZCddYvnMw7Xzon0 hmOhdyBLTVpvKeYVjGb+6uYoC2aPTUTKykJZUcsVQIP7t/iiJs+ghLGUpBoF6vdptn0Hyzgh jOHti4zg50NgsMRkaa251bKh3SrvJehZgg4+gnaQEq74Q5jIo2ofYql7R7c9/koEWqCZlCFu H5Bn42V6/oDSMjX0ieMW+4KWrqu4p5pLQEwn3ZKJb4r/iiNq0KgVo16xjVsdFxwb5gtLGqBj FDohStd45paPX2PZKBxYp6sB8lC8UQGPYq5PhwzRoQTCqWdZDO6EDdSiVl8Nl0BcXTAc4lja f93ku71UR727JiLKhLtGI/xNpdwmkgDKZv7H8yT8vhe+eP2iISpYbkEKkCSSesy8bmJpg7Ym /4GaJDTlk8FDr2uP3CNmWL2EbzsBSVlbXwRg5EGHtNv3iI/SQnN9teIm+p4IN0/90irvryYp yrjMqOn9LYPrSaacVrRMC8LhELHVpdkpnVzJj03IVutwDAiZ43phJrzhLNpFYTLANdLlKYuJ 9FcIp3oKq0WFlzvpW9HBbGj99MKSfherV7VV8ZTSGRkJ8Y4L+EIk/e4FjbSGN4mVXrr65Bm+ OL/h2s2g/MrHmxfMSofU9r3p3vZgJTXsLsas5fgSjWLRHjRzQ== IronPort-HdrOrdr: A9a23:7BTTEKrOfIiymdsWgNNQRBwaV5rmeYIsimQD101hICG9vPbo8P xG+8566faUslcssR4b9exoVJPsfZqYz+8R3WBzB9mftXfdyQiVxehZhOOIqQEIWReOlNK1vp 0OT0ERMqyVMXFKyev3/wW8Fc8t252k/LDAv5an815dCSxndK1k6R50EUKgEkNwTBRbHpZRLu vk2iM+nUvHRZzSBf7LfEXsmIP41qb2qK4= X-Talos-CUID: 9a23:2CHe7W47NHTS9atlL9ss0XQzEMk5Ui3hz1zdCWm8GzZyUeDLYArF X-Talos-MUID: 9a23:0noMZAa/rvMYgOBTvRj0izZ4LM1RwoutM0AjvKQ9m5HdOnkl X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,152,1728950400"; d="scan'208";a="386137520" Received: from alln-l-core-12.cisco.com ([173.36.16.149]) by alln-iport-1.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 13 Nov 2024 23:56:53 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by alln-l-core-12.cisco.com (Postfix) with ESMTPS id 541571800022A; Wed, 13 Nov 2024 23:56:53 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id D07D4CC1290; Wed, 13 Nov 2024 23:56:52 +0000 (GMT) From: Nelson Escobar Date: Wed, 13 Nov 2024 23:56:35 +0000 Subject: [PATCH net-next v4 3/7] enic: Save resource counts we read from HW Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241113-remove_vic_resource_limits-v4-3-a34cf8570c67@cisco.com> References: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> In-Reply-To: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731542212; l=4213; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=51HgZ51PPJWb3IXVJ68mkV9bh9OMM4AxdY0gpO3hPNs=; b=CUxuTb2UJUQSZbaKS5gIF+2TFF7hHLsl+0fviPx8D3m6Wsaf6pnQqY+mYy5kM3s7FoM9j4eMA iG97E8jSG0bBko6pDYeQz/0D5T5j+U9eXwSxogVE/5lSIBVPjGBPVTK X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Outbound-SMTP-Client: 171.70.41.192, neescoba-vicdev.cisco.com X-Outbound-Node: alln-l-core-12.cisco.com X-Patchwork-Delegate: kuba@kernel.org Save the resources counts for wq,rq,cq, and interrupts in *_avail variables so that we don't lose the information when adjusting the counts we are actually using. Report the wq_avail and rq_avail as the channel maximums in 'ethtool -l' output. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman Reviewed-by: Vadim Fedorenko --- drivers/net/ethernet/cisco/enic/enic.h | 4 ++++ drivers/net/ethernet/cisco/enic/enic_ethtool.c | 4 ++-- drivers/net/ethernet/cisco/enic/enic_res.c | 19 ++++++++++++------- 3 files changed, 18 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index ec83a273d1ca40ae89f3c193207cf26814f6b277..98d6e0b825525a92f12776259c1c9c9730cb8a1e 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -206,23 +206,27 @@ struct enic { /* work queue cache line section */ ____cacheline_aligned struct enic_wq wq[ENIC_WQ_MAX]; + unsigned int wq_avail; unsigned int wq_count; u16 loop_enable; u16 loop_tag; /* receive queue cache line section */ ____cacheline_aligned struct enic_rq rq[ENIC_RQ_MAX]; + unsigned int rq_avail; unsigned int rq_count; struct vxlan_offload vxlan; struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX]; /* interrupt resource cache line section */ ____cacheline_aligned struct vnic_intr intr[ENIC_INTR_MAX]; + unsigned int intr_avail; unsigned int intr_count; u32 __iomem *legacy_pba; /* memory-mapped */ /* completion queue cache line section */ ____cacheline_aligned struct vnic_cq cq[ENIC_CQ_MAX]; + unsigned int cq_avail; unsigned int cq_count; struct enic_rfs_flw_tbl rfs_h; u32 rx_copybreak; diff --git a/drivers/net/ethernet/cisco/enic/enic_ethtool.c b/drivers/net/ethernet/cisco/enic/enic_ethtool.c index 909d6f7000e160cf2e15de4660c1034cad7d51ba..d607b4f0542ceaef09e9528a591ca27177986143 100644 --- a/drivers/net/ethernet/cisco/enic/enic_ethtool.c +++ b/drivers/net/ethernet/cisco/enic/enic_ethtool.c @@ -695,8 +695,8 @@ static void enic_get_channels(struct net_device *netdev, switch (vnic_dev_get_intr_mode(enic->vdev)) { case VNIC_DEV_INTR_MODE_MSIX: - channels->max_rx = ENIC_RQ_MAX; - channels->max_tx = ENIC_WQ_MAX; + channels->max_rx = min(enic->rq_avail, ENIC_RQ_MAX); + channels->max_tx = min(enic->wq_avail, ENIC_WQ_MAX); channels->rx_count = enic->rq_count; channels->tx_count = enic->wq_count; break; diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 72b51e9d8d1a26a2cd18df9c9d702e5b11993b70..1261251998330c8b8363c4dd2db1ccc25847476c 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -187,16 +187,21 @@ void enic_free_vnic_resources(struct enic *enic) void enic_get_res_counts(struct enic *enic) { - enic->wq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_WQ); - enic->rq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_RQ); - enic->cq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_CQ); - enic->intr_count = vnic_dev_get_res_count(enic->vdev, - RES_TYPE_INTR_CTRL); + enic->wq_avail = vnic_dev_get_res_count(enic->vdev, RES_TYPE_WQ); + enic->rq_avail = vnic_dev_get_res_count(enic->vdev, RES_TYPE_RQ); + enic->cq_avail = vnic_dev_get_res_count(enic->vdev, RES_TYPE_CQ); + enic->intr_avail = vnic_dev_get_res_count(enic->vdev, + RES_TYPE_INTR_CTRL); + + enic->wq_count = enic->wq_avail; + enic->rq_count = enic->rq_avail; + enic->cq_count = enic->cq_avail; + enic->intr_count = enic->intr_avail; dev_info(enic_get_dev(enic), "vNIC resources avail: wq %d rq %d cq %d intr %d\n", - enic->wq_count, enic->rq_count, - enic->cq_count, enic->intr_count); + enic->wq_avail, enic->rq_avail, + enic->cq_avail, enic->intr_avail); } void enic_init_vnic_resources(struct enic *enic) From patchwork Wed Nov 13 23:56:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13874390 X-Patchwork-Delegate: kuba@kernel.org Received: from alln-iport-3.cisco.com (alln-iport-3.cisco.com [173.37.142.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C06721917C4; Wed, 13 Nov 2024 23:57:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.142.90 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542223; cv=none; b=A7/zHBqoZQDMDP9LLDWCdLd9NWoc2SoLK4XyaKsRCDJkHL2WqCPCVF4mlIXV+R0oxZ73pBV+JeBL/fPXRoMTabGRyH3HdYjXPaUB4QMaMRdEuMLHCKFSD09HM7+qpjTj5q57fmETDo8MHMss7RhTI0BuecwxuvECCLAf3eDGVc8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542223; c=relaxed/simple; bh=5owxqDXstMN6SHOLEqqiW5l7qDBaLbB+d3MirCHt/8w=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=M4TGRx6U241XTvdAEL0I5TPqzf4kHR+cFqRhJqcPoqGQDM+YcloYZ94EiagE6eZC/dmlWCjJo5hlrxiGxxvGZf+emATnmaDhVinv42cNmOuoKmYk8wT+fLOpbmJQ3G9s4dgHFcpIZOv8ju1FXsJlDRNV/qI6atC8VueSnHPnD3I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=dUEh0PJh; arc=none smtp.client-ip=173.37.142.90 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="dUEh0PJh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=6960; q=dns/txt; s=iport; t=1731542220; x=1732751820; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=xKLIfSrBTMEdou9XF+9pCY9FiWhEmlG0ZHo76FyHKjo=; b=dUEh0PJhNU9g2cYgWldTMPOmZmeBORl/C9vOYcIGsQSpLolcx+7Kqjgr 3jIOQE2ZDJoVN1CrFn8elVXvG6SMQH/jMITliH0CsdECsFc42GOc99kmP 1PgLTrcq4J2A2JuMM0A/30vJWB7jXBZCQUWI4eYfe3kvlwxy9d0bz2N9B k=; X-CSE-ConnectionGUID: OnRzWLl7QLGGOGC+q/XR5A== X-CSE-MsgGUID: Khr1ApE2Td2RbfZWkB4qMg== X-IPAS-Result: A0AFAABLPDVn/4oQJK1aGQEBAQEBAQEBAQEBAQEBAQEBARIBAQEBAQEBAQEBAQFAgT8EAQEBAQELAYJKgVBCSIRViB2HMIIhi3WSIxSBEQNWDwEBAQ9EBAEBhQcCikUCJjQJDgECBAEBAQEDAgMBAQEBAQEBAQEBAQsBAQUBAQECAQcFgQ4ThgiGWwIBAyMEUhAlAiYCAisbEAYBEoMBgmUCAbBten8zgQGEe9k4gW2BGi4BiEsBgWyDfTuEPCcbgUlEgRWBO4E+b4Qqg3SCaQSCQYNVUnYlgRMCAgIHAoN8g3lbmE4JP4EFHANZIREBVRMNCgsHBWNXPAMib2pceiuBDoEXOkOBO4EiLxshC1yBN4EZARQGFQSBDkY/gkppSzoCDQI2giQkWYJPhRqEbYRmglQvHUADCxgNSBEsNQYOGwY9AW4HnmVGgysHgQ+BJgcDYyxjklwKg3KNPoIfn0yEJKFcM6pNmHcio05OhGaBZzyBWTMaCBsVgyJSGQ+OLRYWkwABtj9DNTsCBwsBAQMJkEkBAQ IronPort-Data: A9a23:t4p2qKNl01GsegPvrR12lsFynXyQoLVcMsEvi/4bfWQNrUorhjYDy zceWm3UPPreZTT2KNh1bt6+8kMEvsDVmIdkTXM5pCpnJ55oRWUpJjg4wmPYZX76whjrFRo/h ykmQoCeaphyFjmE+0/F3oHJ9RFUzbuPSqf3FNnKMyVwQR4MYCo6gHqPocZh6mJTqYb/WlnlV e/a+ZWFZAb8gmUsaQr41orawP9RlKWq0N8nlgRWicBj5Df2i3QTBZQDEqC9R1OQapVUBOOzW 9HYx7i/+G7Dlz91Yj9yuu+mGqGiaue60Tmm0hK6aYD76vRxjnBaPpIACRYpQRw/ZwNlMDxG4 I4lWZSYEW/FN0BX8QgXe0Ew/ypWZcWq9FJbSJSymZT78qHIT5fj69JoNlMJGYlHw8JYLkZh3 r8BbwA3MznW0opawJrjIgVtrs0nKM+uOMYUvWttiGmAS/0nWpvEBa7N4Le03h9p2ZsIRqiYP pRfMGE/BPjDS0Un1lM/Dp8zh+yvjHDXeDxDo1XTrq0yi4TW5FcgiuK3YICOJLRmQ+12nlzBj 3rrpVj1WCsQOu6gyhDZ61ej07qncSTTHdh6+KeD3vJjnlCW7mAaFhATUVy1vb+/h1LWc99TN kkd6Ccyhac180OvQ5/2WBjQiH6DpBsHc9ldCes37EeK0KW8ywWEDGEsTTNbbtEi8sgsSlQC3 1mFhd72RjpirLGYV1qZ67GS6ziyUQANJGUPYy4sVwYJ49D/5oo0i3rnStdlDb7wjdDvHzz06 y6FoTJ4hLgJi8MPkaKh8jjvhT+wqpXXZhA66x+RXW+/6A59Iom/aOSVBUPz5PJEKsOdC1KGp nVBw5HY5+EVBpbLnyuIKAkQIIyUCz++GGW0qTZS81MJrlxBJ1bLkVhs3QxD IronPort-HdrOrdr: A9a23:SsX8f62IMJ/+Pn7+uzPTkAqjBH8kLtp133Aq2lEZdPWaSL3+qy nIppQmPH7P6Qr5N0tNpTntAsS9qDbnhPtICOoqU4tKIjOW21dARbsKheCJ/9SjIVydygc378 hdmsZFebnNJGk/oMrk7Ay/Cto6hPuK4MmT9J/j5kYoYA10Z6Rn9gtjTjyaHEp/WRVcCfMCZe OhD7J81lydkbB9VLXAOpHDNNKz3OH2qA== X-Talos-CUID: 9a23:MPQQxWusp6MSAlzYUhChlAbq6IsCdEfP81PUfHacJklOZ5bWUBigwv5Nxp8= X-Talos-MUID: 9a23:YFB5VAnbQE5pVn9GgHPGdnpBNN0xyryDUnwBy68Mtfu7NgYsIhKS2WE= X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,152,1728950400"; d="scan'208";a="392228236" Received: from alln-l-core-01.cisco.com ([173.36.16.138]) by alln-iport-3.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 13 Nov 2024 23:56:54 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by alln-l-core-01.cisco.com (Postfix) with ESMTPS id 5BA671800019A; Wed, 13 Nov 2024 23:56:53 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id D5BCACC12A5; Wed, 13 Nov 2024 23:56:52 +0000 (GMT) From: Nelson Escobar Date: Wed, 13 Nov 2024 23:56:36 +0000 Subject: [PATCH net-next v4 4/7] enic: Allocate arrays in enic struct based on VIC config Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241113-remove_vic_resource_limits-v4-4-a34cf8570c67@cisco.com> References: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> In-Reply-To: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731542212; l=7119; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=5owxqDXstMN6SHOLEqqiW5l7qDBaLbB+d3MirCHt/8w=; b=z5IZXmvKbmF9fvwHuFOjYqalykDRw8sn9jfD3FxtKqwNWWB4vl65qryJAj/Mew7SOeAig8QXy TA+kIiA/Tu7D1jGJnHRGwVQf6VbrlWOsOkfXIbkYOh7bwOxTHZ4PrJY X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Outbound-SMTP-Client: 171.70.41.192, neescoba-vicdev.cisco.com X-Outbound-Node: alln-l-core-01.cisco.com X-Patchwork-Delegate: kuba@kernel.org Allocate wq, rq, cq, intr, and napi arrays based on the number of resources configured in the VIC. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman Reviewed-by: Vadim Fedorenko --- drivers/net/ethernet/cisco/enic/enic.h | 24 ++++----- drivers/net/ethernet/cisco/enic/enic_main.c | 84 ++++++++++++++++++++++++++--- 2 files changed, 87 insertions(+), 21 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index 98d6e0b825525a92f12776259c1c9c9730cb8a1e..10b7e02ba4d0ee37bf492eebf1b2556a745bee21 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -23,10 +23,8 @@ #define ENIC_BARS_MAX 6 -#define ENIC_WQ_MAX 8 -#define ENIC_RQ_MAX 8 -#define ENIC_CQ_MAX (ENIC_WQ_MAX + ENIC_RQ_MAX) -#define ENIC_INTR_MAX (ENIC_CQ_MAX + 2) +#define ENIC_WQ_MAX 256 +#define ENIC_RQ_MAX 256 #define ENIC_WQ_NAPI_BUDGET 256 @@ -184,8 +182,8 @@ struct enic { struct work_struct reset; struct work_struct tx_hang_reset; struct work_struct change_mtu_work; - struct msix_entry msix_entry[ENIC_INTR_MAX]; - struct enic_msix_entry msix[ENIC_INTR_MAX]; + struct msix_entry *msix_entry; + struct enic_msix_entry *msix; u32 msg_enable; spinlock_t devcmd_lock; u8 mac_addr[ETH_ALEN]; @@ -204,28 +202,24 @@ struct enic { bool enic_api_busy; struct enic_port_profile *pp; - /* work queue cache line section */ - ____cacheline_aligned struct enic_wq wq[ENIC_WQ_MAX]; + struct enic_wq *wq; unsigned int wq_avail; unsigned int wq_count; u16 loop_enable; u16 loop_tag; - /* receive queue cache line section */ - ____cacheline_aligned struct enic_rq rq[ENIC_RQ_MAX]; + struct enic_rq *rq; unsigned int rq_avail; unsigned int rq_count; struct vxlan_offload vxlan; - struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX]; + struct napi_struct *napi; - /* interrupt resource cache line section */ - ____cacheline_aligned struct vnic_intr intr[ENIC_INTR_MAX]; + struct vnic_intr *intr; unsigned int intr_avail; unsigned int intr_count; u32 __iomem *legacy_pba; /* memory-mapped */ - /* completion queue cache line section */ - ____cacheline_aligned struct vnic_cq cq[ENIC_CQ_MAX]; + struct vnic_cq *cq; unsigned int cq_avail; unsigned int cq_count; struct enic_rfs_flw_tbl rfs_h; diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index eb00058b6c68ec5c1ac433b54b5bc6f3fb613777..564202e81a711a6791bef7e848627f0a439cc6f3 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -940,7 +940,7 @@ static void enic_get_stats(struct net_device *netdev, net_stats->rx_errors = stats->rx.rx_errors; net_stats->multicast = stats->rx.rx_multicast_frames_ok; - for (i = 0; i < ENIC_RQ_MAX; i++) { + for (i = 0; i < enic->rq_count; i++) { struct enic_rq_stats *rqs = &enic->rq[i].stats; if (!enic->rq[i].vrq.ctrl) @@ -1792,7 +1792,7 @@ static void enic_free_intr(struct enic *enic) free_irq(enic->pdev->irq, enic); break; case VNIC_DEV_INTR_MODE_MSIX: - for (i = 0; i < ARRAY_SIZE(enic->msix); i++) + for (i = 0; i < enic->intr_count; i++) if (enic->msix[i].requested) free_irq(enic->msix_entry[i].vector, enic->msix[i].devid); @@ -1859,7 +1859,7 @@ static int enic_request_intr(struct enic *enic) enic->msix[intr].isr = enic_isr_msix_notify; enic->msix[intr].devid = enic; - for (i = 0; i < ARRAY_SIZE(enic->msix); i++) + for (i = 0; i < enic->intr_count; i++) enic->msix[i].requested = 0; for (i = 0; i < enic->intr_count; i++) { @@ -2456,8 +2456,7 @@ static int enic_set_intr_mode(struct enic *enic) * (the last INTR is used for notifications) */ - BUG_ON(ARRAY_SIZE(enic->msix_entry) < n + m + 2); - for (i = 0; i < n + m + 2; i++) + for (i = 0; i < enic->intr_avail; i++) enic->msix_entry[i].entry = i; /* Use multiple RQs if RSS is enabled @@ -2674,6 +2673,71 @@ static const struct netdev_stat_ops enic_netdev_stat_ops = { .get_base_stats = enic_get_base_stats, }; +static void enic_free_enic_resources(struct enic *enic) +{ + kfree(enic->wq); + enic->wq = NULL; + + kfree(enic->rq); + enic->rq = NULL; + + kfree(enic->cq); + enic->cq = NULL; + + kfree(enic->napi); + enic->napi = NULL; + + kfree(enic->msix_entry); + enic->msix_entry = NULL; + + kfree(enic->msix); + enic->msix = NULL; + + kfree(enic->intr); + enic->intr = NULL; +} + +static int enic_alloc_enic_resources(struct enic *enic) +{ + enic->wq = kcalloc(enic->wq_avail, sizeof(struct enic_wq), GFP_KERNEL); + if (!enic->wq) + goto free_queues; + + enic->rq = kcalloc(enic->rq_avail, sizeof(struct enic_rq), GFP_KERNEL); + if (!enic->rq) + goto free_queues; + + enic->cq = kcalloc(enic->cq_avail, sizeof(struct vnic_cq), GFP_KERNEL); + if (!enic->cq) + goto free_queues; + + enic->napi = kcalloc(enic->wq_avail + enic->rq_avail, + sizeof(struct napi_struct), GFP_KERNEL); + if (!enic->napi) + goto free_queues; + + enic->msix_entry = kcalloc(enic->intr_avail, sizeof(struct msix_entry), + GFP_KERNEL); + if (!enic->msix_entry) + goto free_queues; + + enic->msix = kcalloc(enic->intr_avail, sizeof(struct enic_msix_entry), + GFP_KERNEL); + if (!enic->msix) + goto free_queues; + + enic->intr = kcalloc(enic->intr_avail, sizeof(struct vnic_intr), + GFP_KERNEL); + if (!enic->intr) + goto free_queues; + + return 0; + +free_queues: + enic_free_enic_resources(enic); + return -ENOMEM; +} + static void enic_dev_deinit(struct enic *enic) { unsigned int i; @@ -2691,6 +2755,7 @@ static void enic_dev_deinit(struct enic *enic) enic_free_vnic_resources(enic); enic_clear_intr_mode(enic); enic_free_affinity_hint(enic); + enic_free_enic_resources(enic); } static void enic_kdump_kernel_config(struct enic *enic) @@ -2734,6 +2799,12 @@ static int enic_dev_init(struct enic *enic) enic_get_res_counts(enic); + err = enic_alloc_enic_resources(enic); + if (err) { + dev_err(dev, "Failed to allocate enic resources\n"); + return err; + } + /* modify resource count if we are in kdump_kernel */ enic_kdump_kernel_config(enic); @@ -2746,7 +2817,7 @@ static int enic_dev_init(struct enic *enic) if (err) { dev_err(dev, "Failed to set intr mode based on resource " "counts and system capabilities, aborting\n"); - return err; + goto err_out_free_vnic_resources; } /* Allocate and configure vNIC resources @@ -2788,6 +2859,7 @@ static int enic_dev_init(struct enic *enic) enic_free_affinity_hint(enic); enic_clear_intr_mode(enic); enic_free_vnic_resources(enic); + enic_free_enic_resources(enic); return err; } From patchwork Wed Nov 13 23:56:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13874394 X-Patchwork-Delegate: kuba@kernel.org Received: from alln-iport-2.cisco.com (alln-iport-2.cisco.com [173.37.142.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8792118DF91; Wed, 13 Nov 2024 23:58:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.142.89 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542285; cv=none; b=MV11Ask9pyJwnS/YtFxBZXVXT5FOoMcuheK895d+7dL8v81Ri1J2XVJqogWEdXN72Gn1IHxxKtdG0VSCLU/avQXQEdPAFOhiWjCKUpxzowx0nXVHDa+7/4sZZsfiKODH3aNeNcT8uFM5caImTPid5DjEmfVXVNxrNOzB4aS0lQs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542285; c=relaxed/simple; bh=BaAMDPDZS2YT6jrU8+AmVF8L/RZgKYuGDX5Aoh7FqDg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=glGK535gxQ3z5ODWkXG0kvoT/K2ytGy9+hJPWFfpR/9ct0lorntLeNqSoYh9Boo4TfQSMVtAkMfUTCntlLBxWsCEQF6bBuqO9NtLEujYR8/eQiWRuVCH8ls+kFpxR3kScC+D3oCDh1+aDmefKQ0LnAVgQ2IU5yNX7m1cpQoYiIU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=lUCb2rG2; arc=none smtp.client-ip=173.37.142.89 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="lUCb2rG2" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=5894; q=dns/txt; s=iport; t=1731542283; x=1732751883; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=7faiQRGgmp7GL149R+szlt5/BOnQOPfLqxK1XaSDFqE=; b=lUCb2rG2qtfwSQYnyOdQdsSJdANMCWp4fIP/yxyCjAvRv70b7BcCmM9i zm4sQoio4SrcQ9iT5a3liMFYQM5UMMuuhL1t/IZg1oi5R1Utlp2lWeQrO jf+eO7xNuCdrfmVgtawrEvy/cFxBjZhE6Q8taDml/mj43U9q3RIjMVNhp g=; X-CSE-ConnectionGUID: FafY/dQyQBCNKHq8cJzebQ== X-CSE-MsgGUID: Opc5s8N3R0+vSp7+1roukQ== X-IPAS-Result: A0AGAADHOzVnj44QJK1aGwEBAQEBAQEBBQEBARIBAQEDAwEBAUCBPwYBAQELAYQaQkiEVYgdhzCOFpIjgSUDVg8BAQEPRAQBAYUHAopFAiY0CQ4BAgQBAQEBAwIDAQEBAQEBAQEBAQELAQEFAQEBAgEHBRQBAQEBAQE5BQ47hgiGWwIBAyMEUhAlAiYCAisbEAYBEoMBgmUCAbBmen8zgQHeM4FtgRouAYhLAYFsg307hDwnG4FJRIEUAYNohBuEA4JpBIQtA4I4diWFHoN5W49RiH0JP4EFHANZIREBVRMNCgsHBWNXPAMib2pceiuBDoEXOkOBO4EiLxshC1yBN4EaFAYVBIEORj+CSmlLOgINAjaCJCRZgk+FGoRthGaCVC8dQAMLGA1IESw1Bg4bBj0BbgeeZUaDMoEOgScHgRJjkm2Da40+oWuEJKFcM6pNmHcipByEZoFnOoFbMxoIGxWDIlIZD44tDQkWkwABtj9DNTsCBwsBAQMJjk2BfAEB IronPort-Data: A9a23:986sfazTWRfacTBMbdl6t+eixirEfRIJ4+MujC+fZmUNrF6WrkUHn GNKW2yHO/uKM2ajeo92b4zl8xwP7cPVxtZkQVE+r1hgHilAwSbn6Xt1DatR0we6dJCroJdPt p1GAjX4BJlqCCea/lH1b+CJQUBUjcmgXqD7BPPPJhd/TAplTDZJoR94kobVuKYw6TSCK13L4 ImaT/H3Ygf/h2ctazlMsMpvlTs21BjMkGJA1rABTagjUG/2zxE9EJ8ZLKetGHr0KqE8NvK6X evK0Iai9Wrf+Ro3Yvv9+losWhRXKlJ6FVHmZkt+A8BOsDAbzsAB+vpT2M4nVKtio27hc+adZ zl6ncfYpQ8BZsUgkQmGOvVSO3kW0aZuoNcrLZUj2CCe5xWuTpfi/xlhJG8/M7xf9t9mOEtl0 984MWAGTAKOjdvjldpXSsE07igiBMDvOIVavjRryivUSK57B5vCWK7No9Rf2V/chOgXQq2YP JRfMGQpNUiaC/FMEg9/5JYWmuqlnXL4eTRwo1OOrq1x6G/WpOB0+OOyYIOJIIPSHq25mG6/v mmb5DTfWStCd8Kk0WXGq2irm+z2yHaTtIU6T+DgqaUw3zV/3Fc7BBQIWF6TrfCnh0u6XNxDb UoZ5kIGoKQv8UW5Q8XVUBq/r3qJ+BUbXrJ4E+og7RqlyafO5QudQG8eQVZpa8Esvec1SCYs2 1vPmMnmbRRmtrGPRG3e8LqIoT6sESwIK2lEbi9sZRMM6dTloakpgx7PR8olG6mw5vXxGDft0 3WJoTI4irE7k8EGzeO48ErBjjbqoYLGJiYz6xnbU3yN8Ax0fsimapau5Fyd6uxPRLt1VXGIu HwC3szb5+cUANTVyWqGQf4GG/ei4PPt3CDgbUBHMZAvxnOhwm+aV6t2wxFndExLIMsOQGq8C KPMgj956JhWNXqsSKZ4ZYOtFsgnpZQM8/y7Cpg4ifIQP/BMmB+7wc14WaKHM4nQfKkQfUMXZ cfznSWEVCpy5UFbINyeHLx1PVgDnXxW+I8rbcqnpylLKJLHDJJvdZ8LMUGVcscy576erQPe/ r53bpTRkkgFCrOuMnSOq+b/yGzmy1BmW/gaTOQKJ4a+zvZOQTBJ5wL5mOl4Itc0xcy5aM+Tr ijmBCe0N2YTdVWcdF3VMSo8AF8edZ1+tnk8dTc9Jkql3mNrYICkqs8im2gfI9EaGBhY5acsF ZEtIpzYatwWE2iv02pGN/HV8tc9HClHcCrSZEJJlhBjJMY4H2QkO7bMImPSycX5JnHn6ZVl+ +b/iVKzrFhqb10KMfs6ocmHlzuZ1UXxUsorN6cUCrG/oHnRzbU= IronPort-HdrOrdr: A9a23:/HD1GawDuJ0DiUOGHjbCKrPwT71zdoMgy1knxilNoH1uEvBw8v rEoB1173LJYVoqMk3I+urgBED/exzhHPdOiOEs1NyZMDUO1lHHEL1f X-Talos-CUID: 9a23:XjTGBmgXsAT73AVH4XBCATuE0TJuXHTT/TDcGl6BI01Pa52MGVSy6q5uqp87 X-Talos-MUID: 9a23:9lcODQyaVVWWihoY0giD/r8XhbWaqIO2Ax0WwbNdgZHHKxdoZHTEljOqQbZyfw== X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,152,1728950400"; d="scan'208";a="378142703" Received: from alln-l-core-05.cisco.com ([173.36.16.142]) by alln-iport-2.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 13 Nov 2024 23:56:54 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by alln-l-core-05.cisco.com (Postfix) with ESMTPS id E234718000154; Wed, 13 Nov 2024 23:56:53 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id DB17CCC12AC; Wed, 13 Nov 2024 23:56:52 +0000 (GMT) From: Nelson Escobar Date: Wed, 13 Nov 2024 23:56:37 +0000 Subject: [PATCH net-next v4 5/7] enic: Adjust used MSI-X wq/rq/cq/interrupt resources in a more robust way Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241113-remove_vic_resource_limits-v4-5-a34cf8570c67@cisco.com> References: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> In-Reply-To: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731542212; l=6224; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=BaAMDPDZS2YT6jrU8+AmVF8L/RZgKYuGDX5Aoh7FqDg=; b=HgsBZACqfp16Yl9sxU5GphVHAlD2V68FhjLESi8FK1rVsCFClszHWGGi6FJdGABvoMNiSIe1n FqrDRJFP4lCDJ8zcGYVYtlRzi3D2zXfgdNKxKyysF5Al6MGR5TvjJqR X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Outbound-SMTP-Client: 171.70.41.192, neescoba-vicdev.cisco.com X-Outbound-Node: alln-l-core-05.cisco.com X-Patchwork-Delegate: kuba@kernel.org Instead of failing to use MSI-X if resources aren't configured exactly right, use the resources we do have. Since we could start using large numbers of rq resources, we do limit the rq count to what netif_get_num_default_rss_queues() recommends. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Reviewed-by: Simon Horman Signed-off-by: Nelson Escobar Reviewed-by: Vadim Fedorenko --- drivers/net/ethernet/cisco/enic/enic_main.c | 116 ++++++++++++++-------------- 1 file changed, 57 insertions(+), 59 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index 564202e81a711a6791bef7e848627f0a439cc6f3..8b07899462d0671843579d16c8c935d9ebbe447b 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -2442,85 +2442,86 @@ static void enic_tx_hang_reset(struct work_struct *work) static int enic_set_intr_mode(struct enic *enic) { - unsigned int n = min_t(unsigned int, enic->rq_count, ENIC_RQ_MAX); - unsigned int m = min_t(unsigned int, enic->wq_count, ENIC_WQ_MAX); unsigned int i; + int num_intr; /* Set interrupt mode (INTx, MSI, MSI-X) depending * on system capabilities. * - * Try MSI-X first + * We need a minimum of 1 RQ, 1 WQ, and 2 CQs * - * We need n RQs, m WQs, n+m CQs, and n+m+2 INTRs - * (the second to last INTR is used for WQ/RQ errors) - * (the last INTR is used for notifications) */ - for (i = 0; i < enic->intr_avail; i++) - enic->msix_entry[i].entry = i; - - /* Use multiple RQs if RSS is enabled - */ - - if (ENIC_SETTING(enic, RSS) && - enic->config.intr_mode < 1 && - enic->rq_count >= n && - enic->wq_count >= m && - enic->cq_count >= n + m && - enic->intr_count >= n + m + 2) { - - if (pci_enable_msix_range(enic->pdev, enic->msix_entry, - n + m + 2, n + m + 2) > 0) { - - enic->rq_count = n; - enic->wq_count = m; - enic->cq_count = n + m; - enic->intr_count = n + m + 2; - - vnic_dev_set_intr_mode(enic->vdev, - VNIC_DEV_INTR_MODE_MSIX); - - return 0; - } + if (enic->rq_avail < 1 || enic->wq_avail < 1 || enic->cq_avail < 2) { + dev_err(enic_get_dev(enic), + "Not enough resources available rq: %d wq: %d cq: %d\n", + enic->rq_avail, enic->wq_avail, + enic->cq_avail); + return -ENOSPC; } + /* if RSS isn't set, then we can only use one RQ */ + if (!ENIC_SETTING(enic, RSS)) + enic->rq_avail = 1; + + /* Try MSI-X first */ if (enic->config.intr_mode < 1 && - enic->rq_count >= 1 && - enic->wq_count >= m && - enic->cq_count >= 1 + m && - enic->intr_count >= 1 + m + 2) { - if (pci_enable_msix_range(enic->pdev, enic->msix_entry, - 1 + m + 2, 1 + m + 2) > 0) { - - enic->rq_count = 1; - enic->wq_count = m; - enic->cq_count = 1 + m; - enic->intr_count = 1 + m + 2; + enic->intr_avail >= ENIC_MSIX_MIN_INTR) { + unsigned int max_queues; + unsigned int rq_default; + unsigned int rq_avail; + unsigned int wq_avail; + + for (i = 0; i < enic->intr_avail; i++) + enic->msix_entry[i].entry = i; + + num_intr = pci_enable_msix_range(enic->pdev, enic->msix_entry, + ENIC_MSIX_MIN_INTR, + enic->intr_avail); + if (num_intr > 0) { + wq_avail = min(enic->wq_avail, ENIC_WQ_MAX); + rq_default = netif_get_num_default_rss_queues(); + rq_avail = min3(enic->rq_avail, ENIC_RQ_MAX, rq_default); + max_queues = min(enic->cq_avail, + enic->intr_avail - ENIC_MSIX_RESERVED_INTR); + + if (wq_avail + rq_avail <= max_queues) { + enic->rq_count = rq_avail; + enic->wq_count = wq_avail; + } else { + /* recalculate wq/rq count */ + if (rq_avail < wq_avail) { + enic->rq_count = min(rq_avail, max_queues / 2); + enic->wq_count = max_queues - enic->rq_count; + } else { + enic->wq_count = min(wq_avail, max_queues / 2); + enic->rq_count = max_queues - enic->wq_count; + } + } + enic->cq_count = enic->rq_count + enic->wq_count; + enic->intr_count = enic->cq_count + ENIC_MSIX_RESERVED_INTR; vnic_dev_set_intr_mode(enic->vdev, - VNIC_DEV_INTR_MODE_MSIX); - + VNIC_DEV_INTR_MODE_MSIX); + enic->intr_avail = num_intr; return 0; } } /* Next try MSI * - * We need 1 RQ, 1 WQ, 2 CQs, and 1 INTR + * We need 1 INTR */ if (enic->config.intr_mode < 2 && - enic->rq_count >= 1 && - enic->wq_count >= 1 && - enic->cq_count >= 2 && - enic->intr_count >= 1 && + enic->intr_avail >= 1 && !pci_enable_msi(enic->pdev)) { enic->rq_count = 1; enic->wq_count = 1; enic->cq_count = 2; enic->intr_count = 1; - + enic->intr_avail = 1; vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_MSI); return 0; @@ -2528,23 +2529,20 @@ static int enic_set_intr_mode(struct enic *enic) /* Next try INTx * - * We need 1 RQ, 1 WQ, 2 CQs, and 3 INTRs + * We need 3 INTRs * (the first INTR is used for WQ/RQ) * (the second INTR is used for WQ/RQ errors) * (the last INTR is used for notifications) */ if (enic->config.intr_mode < 3 && - enic->rq_count >= 1 && - enic->wq_count >= 1 && - enic->cq_count >= 2 && - enic->intr_count >= 3) { + enic->intr_avail >= 3) { enic->rq_count = 1; enic->wq_count = 1; enic->cq_count = 2; enic->intr_count = 3; - + enic->intr_avail = 3; vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_INTX); return 0; @@ -2762,8 +2760,8 @@ static void enic_kdump_kernel_config(struct enic *enic) { if (is_kdump_kernel()) { dev_info(enic_get_dev(enic), "Running from within kdump kernel. Using minimal resources\n"); - enic->rq_count = 1; - enic->wq_count = 1; + enic->rq_avail = 1; + enic->wq_avail = 1; enic->config.rq_desc_count = ENIC_MIN_RQ_DESCS; enic->config.wq_desc_count = ENIC_MIN_WQ_DESCS; enic->config.mtu = min_t(u16, 1500, enic->config.mtu); From patchwork Wed Nov 13 23:56:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13874392 X-Patchwork-Delegate: kuba@kernel.org Received: from alln-iport-3.cisco.com (alln-iport-3.cisco.com [173.37.142.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DEEA1946BC; Wed, 13 Nov 2024 23:57:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.142.90 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542225; cv=none; b=fCATG7VN7CCQOnGu447G+z9S7Ou6cfXPvVVAINcNeY6pxOwg+9O/dz5f0NSWIGT+PFcZ2M59OGs3gBB7Hw8yzjg8aswH04PX7RA/e6zra8LDNWS29RYCPP25gUauSq/Hm/r/rmRegr1UZDjukyDoCM7EKxGciCg7wDxfwp/kqiI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542225; c=relaxed/simple; bh=MvXjUBEvjB7eEJ/Bu4ljMT1zyOIo5CsddFBMJVwXGt4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=aHRvc1mI63FwdkBdQtZmG7YbfSfHWxt/3Yj1Z/OpA+uT+qt914JnxIo856PuRZGPBK8i6DQx/okZmoLr77/bgwigw2gjytVNOaUO14w4lCvjhOPYxgm5ZHBcS5YvpX3TOKww5KduFK1wO7JPELBqLEDKXhHqHWLuc28F/N7CPDY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=X1Y3S/kg; arc=none smtp.client-ip=173.37.142.90 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="X1Y3S/kg" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=6237; q=dns/txt; s=iport; t=1731542223; x=1732751823; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=qftcgMzV0lob1J7dAZRYb3pGAgaYy5n0qlEWTtW7384=; b=X1Y3S/kgyjXjPUSeOYCuTlKdln1kVEmHADoueDMJWt0DjKIU4FhteWgt 4SErKZ2b5UxZBY5wk/9JB/oq7qUSZiExZEM+A9R+q260nEkKvFPSI/klu hL6DZOLM/ot12RtqFuaou/UxbDfPLVnx3btsAy+GFTzwBsozLFtDb36K+ k=; X-CSE-ConnectionGUID: 8+eQtAfQQhuA6AjI8I9wJA== X-CSE-MsgGUID: na9MQZNdSZyTzNH2iE9tAg== X-IPAS-Result: A0AHAABLPDVn/40QJK1aGgEBAQEBAQEBAQEDAQEBARIBAQEBAgIBAQEBQIE/BQEBAQELAYJKgVBCSIRViB2HMI4WkiOBJQNWDwEBAQ9EBAEBhQcCikUCJjQJDgECBAEBAQEDAgMBAQEBAQEBAQEBAQsBAQUBAQECAQcFgQ4ThgiGWwIBAyMEUhAlAiYCAisbEAYBEoMBgmUCAbBten8zgQGEe9k4gW2BGi4BiEsBgWyDfTuEPCcbgUlEgRSDaYgegmkEh14lhR6DeZAsiH0JP4EFHANZIREBVRMNCgsHBWNXPAMib2pceiuBDoEXOkOBO4EiLxshC1yBN4EZARQGFQSBDkY/gkppSzoCDQI2giQkWYJPhRqEbYRmglQvHUADCxgNSBEsNQYOGwY9AW4HnmVGgysHgQ6BMYEPkz8Rg2uNPoIfn0yEJKFcM6pNmHcipByEZoFnPIFZMxoIGxWDIlIZD44tFhaTAAG2P0M1OwIHCwEBAwmQSQEB IronPort-Data: A9a23:OZe/CajAjdkV7S2f/wOAUaeLX161sxEKZh0ujC45NGQN5FlHY01je htvXW+FbvvZYDTzfN0gPI22900AvpeBy9RlGQM5+Sk2QiNjpJueD7x1DKtf0wB+jyHnZBg6h ynLQoCYdKjYdleF+FH1dOCn9SQkvU2xbuKUIPbePSxsThNTRi4kiBZy88Y0mYcAbeKRW2thg vus5ZSFULOZ82QsaD5NsvvY8EgHUMna4Vv0gHRvPZing3eG/5UlJMp3Db28KXL+Xr5VEoaSL 87fzKu093/u5BwkDNWoiN7TKiXmlZaLYGBiIlIPM0STqkAqSh4ai87XB9JAAatjsAhlqvgqo Dl7WTNcfi9yVkHEsLx1vxC1iEiSN4UekFPMCSDXXcB+UyQqflO0q8iCAn3aMqVIuetOHUhAz MUYNRlTRxKi18Pn35CSH7wEasQLdKEHPasWvnVmiDWcBvE8TNWbH+PB5MRT23E7gcUm8fT2P pVCL2ExKk2eJUQTYz/7C7pm9Ausrn/yfiZTr1icjaE2+GPUigd21dABNfKOK4bQH5UPwhzwS mTu+W2oJygoEtCk+yu54FSM3O7izD/SV9dHfFG/3rsw6LGJ/UQfAQMbUHO3qOe0j0q5Vc4ZL UEIkgIjobU3/V6mUvHyWBq3pHPCtRkZM/JQFPc/8ymOx7DS7gLfAXILJhZCddYvnMw7Xzon0 hmOhdyBLTVpvKeYVjGb+6uYoC2aPTUTKykJZUcsVQIP7t/iiJs+ghLGUpBoF6vdptn0Hyzgh jOHti4zg50NgsMRkaa251bKh3SrvJehZgg4+gnaQEq74Q5jIo2ofYql7R7c9/koEWqCZlCFu H5Bn42V6/oDSMnR0ieMW+4KWrqu4p5pLQHhvLKmJLF5nxzFxpJpVdo4DO1WTKuxDvs5RA== IronPort-HdrOrdr: A9a23:D0EVna3+/xVfHJZHYgywGgqjBH8kLtp133Aq2lEZdPWaSL3+qy nIppQmPH7P6Qr5N0tNpTntAsS9qDbnhPtICOoqU4tKIjOW21dARbsKheCJ/9SjIVydygc378 hdmsZFebnNJGk/oMrk7Ay/Cto6hPuK4MmT9J/j5kYoYA10Z6Rn9gtjTjyaHEp/WRVcCfMCZe OhD7J81lydkbB9VLXAOpHDNNKz3OH2qA== X-Talos-CUID: 9a23:L3DX0GHLjExQN9KgqmI3t0MtBukrS0Td0V6OCk2ZWUc4V5+8HAo= X-Talos-MUID: 9a23:x3eqKgke2RKInEMme2DUdnpkPZZN2paIBHs/gKkUneneNAEhJx6S2WE= X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,152,1728950400"; d="scan'208";a="392228242" Received: from alln-l-core-04.cisco.com ([173.36.16.141]) by alln-iport-3.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 13 Nov 2024 23:56:54 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by alln-l-core-04.cisco.com (Postfix) with ESMTPS id E29831800019B; Wed, 13 Nov 2024 23:56:53 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id E05D2CC12AD; Wed, 13 Nov 2024 23:56:52 +0000 (GMT) From: Nelson Escobar Date: Wed, 13 Nov 2024 23:56:38 +0000 Subject: [PATCH net-next v4 6/7] enic: Move enic resource adjustments to separate function Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241113-remove_vic_resource_limits-v4-6-a34cf8570c67@cisco.com> References: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> In-Reply-To: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731542212; l=6495; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=MvXjUBEvjB7eEJ/Bu4ljMT1zyOIo5CsddFBMJVwXGt4=; b=0+l8IXnRLM++s0wDhN0hoSr9zcpo0gFpjIgv2G4j3yiyVFCOy9zjxvhJrGPUVTv0Jwxe3ISLb vQIST5l5ZggDkf9xSLqy9kdBkyBB0HYTjvCBDvmiMAEbN7aKga2vFIq X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Outbound-SMTP-Client: 171.70.41.192, neescoba-vicdev.cisco.com X-Outbound-Node: alln-l-core-04.cisco.com X-Patchwork-Delegate: kuba@kernel.org Move the enic resource adjustments out of enic_set_intr_mode() and into its own function, enic_adjust_resources(). Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Reviewed-by: Simon Horman Signed-off-by: Nelson Escobar Reviewed-by: Vadim Fedorenko --- drivers/net/ethernet/cisco/enic/enic_main.c | 127 +++++++++++++++------------- 1 file changed, 70 insertions(+), 57 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index 8b07899462d0671843579d16c8c935d9ebbe447b..84e85c9e2bf52f0089c0a8d03fb6d22ada4d086c 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -2448,30 +2448,11 @@ static int enic_set_intr_mode(struct enic *enic) /* Set interrupt mode (INTx, MSI, MSI-X) depending * on system capabilities. * - * We need a minimum of 1 RQ, 1 WQ, and 2 CQs - * + * Try MSI-X first */ - if (enic->rq_avail < 1 || enic->wq_avail < 1 || enic->cq_avail < 2) { - dev_err(enic_get_dev(enic), - "Not enough resources available rq: %d wq: %d cq: %d\n", - enic->rq_avail, enic->wq_avail, - enic->cq_avail); - return -ENOSPC; - } - - /* if RSS isn't set, then we can only use one RQ */ - if (!ENIC_SETTING(enic, RSS)) - enic->rq_avail = 1; - - /* Try MSI-X first */ if (enic->config.intr_mode < 1 && enic->intr_avail >= ENIC_MSIX_MIN_INTR) { - unsigned int max_queues; - unsigned int rq_default; - unsigned int rq_avail; - unsigned int wq_avail; - for (i = 0; i < enic->intr_avail; i++) enic->msix_entry[i].entry = i; @@ -2479,28 +2460,6 @@ static int enic_set_intr_mode(struct enic *enic) ENIC_MSIX_MIN_INTR, enic->intr_avail); if (num_intr > 0) { - wq_avail = min(enic->wq_avail, ENIC_WQ_MAX); - rq_default = netif_get_num_default_rss_queues(); - rq_avail = min3(enic->rq_avail, ENIC_RQ_MAX, rq_default); - max_queues = min(enic->cq_avail, - enic->intr_avail - ENIC_MSIX_RESERVED_INTR); - - if (wq_avail + rq_avail <= max_queues) { - enic->rq_count = rq_avail; - enic->wq_count = wq_avail; - } else { - /* recalculate wq/rq count */ - if (rq_avail < wq_avail) { - enic->rq_count = min(rq_avail, max_queues / 2); - enic->wq_count = max_queues - enic->rq_count; - } else { - enic->wq_count = min(wq_avail, max_queues / 2); - enic->rq_count = max_queues - enic->wq_count; - } - } - enic->cq_count = enic->rq_count + enic->wq_count; - enic->intr_count = enic->cq_count + ENIC_MSIX_RESERVED_INTR; - vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_MSIX); enic->intr_avail = num_intr; @@ -2516,14 +2475,8 @@ static int enic_set_intr_mode(struct enic *enic) if (enic->config.intr_mode < 2 && enic->intr_avail >= 1 && !pci_enable_msi(enic->pdev)) { - - enic->rq_count = 1; - enic->wq_count = 1; - enic->cq_count = 2; - enic->intr_count = 1; enic->intr_avail = 1; vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_MSI); - return 0; } @@ -2537,14 +2490,8 @@ static int enic_set_intr_mode(struct enic *enic) if (enic->config.intr_mode < 3 && enic->intr_avail >= 3) { - - enic->rq_count = 1; - enic->wq_count = 1; - enic->cq_count = 2; - enic->intr_count = 3; enic->intr_avail = 3; vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_INTX); - return 0; } @@ -2569,6 +2516,67 @@ static void enic_clear_intr_mode(struct enic *enic) vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_UNKNOWN); } +static int enic_adjust_resources(struct enic *enic) +{ + unsigned int max_queues; + unsigned int rq_default; + unsigned int rq_avail; + unsigned int wq_avail; + + if (enic->rq_avail < 1 || enic->wq_avail < 1 || enic->cq_avail < 2) { + dev_err(enic_get_dev(enic), + "Not enough resources available rq: %d wq: %d cq: %d\n", + enic->rq_avail, enic->wq_avail, + enic->cq_avail); + return -ENOSPC; + } + + /* if RSS isn't set, then we can only use one RQ */ + if (!ENIC_SETTING(enic, RSS)) + enic->rq_avail = 1; + + switch (vnic_dev_get_intr_mode(enic->vdev)) { + case VNIC_DEV_INTR_MODE_INTX: + case VNIC_DEV_INTR_MODE_MSI: + enic->rq_count = 1; + enic->wq_count = 1; + enic->cq_count = 2; + enic->intr_count = enic->intr_avail; + break; + case VNIC_DEV_INTR_MODE_MSIX: + /* Adjust the number of wqs/rqs/cqs/interrupts that will be + * used based on which resource is the most constrained + */ + wq_avail = min(enic->wq_avail, ENIC_WQ_MAX); + rq_default = netif_get_num_default_rss_queues(); + rq_avail = min3(enic->rq_avail, ENIC_RQ_MAX, rq_default); + max_queues = min(enic->cq_avail, + enic->intr_avail - ENIC_MSIX_RESERVED_INTR); + if (wq_avail + rq_avail <= max_queues) { + enic->rq_count = rq_avail; + enic->wq_count = wq_avail; + } else { + /* recalculate wq/rq count */ + if (rq_avail < wq_avail) { + enic->rq_count = min(rq_avail, max_queues / 2); + enic->wq_count = max_queues - enic->rq_count; + } else { + enic->wq_count = min(wq_avail, max_queues / 2); + enic->rq_count = max_queues - enic->wq_count; + } + } + enic->cq_count = enic->rq_count + enic->wq_count; + enic->intr_count = enic->cq_count + ENIC_MSIX_RESERVED_INTR; + + break; + default: + dev_err(enic_get_dev(enic), "Unknown interrupt mode\n"); + return -EINVAL; + } + + return 0; +} + static void enic_get_queue_stats_rx(struct net_device *dev, int idx, struct netdev_queue_stats_rx *rxs) { @@ -2807,9 +2815,7 @@ static int enic_dev_init(struct enic *enic) */ enic_kdump_kernel_config(enic); - /* Set interrupt mode based on resource counts and system - * capabilities - */ + /* Set interrupt mode based on system capabilities */ err = enic_set_intr_mode(enic); if (err) { @@ -2818,6 +2824,13 @@ static int enic_dev_init(struct enic *enic) goto err_out_free_vnic_resources; } + /* Adjust resource counts based on most constrained resources */ + err = enic_adjust_resources(enic); + if (err) { + dev_err(dev, "Failed to adjust resources\n"); + goto err_out_free_vnic_resources; + } + /* Allocate and configure vNIC resources */ From patchwork Wed Nov 13 23:56:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Nelson Escobar (neescoba)" X-Patchwork-Id: 13874391 X-Patchwork-Delegate: kuba@kernel.org Received: from alln-iport-1.cisco.com (alln-iport-1.cisco.com [173.37.142.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20BAB193061; Wed, 13 Nov 2024 23:57:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=173.37.142.88 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542224; cv=none; b=VbB7z6KEAcDTPtDQz6IaKiqSJPRCuzD19rshMLO8Fl0DhbnuehWOsx6r59MT0YPoYMQN6WPNQLL3nwZqhYW3O25VdFFGKgaQroZ0NFYjnyKtGPR0KwGTpcldrkQa7FaWnGKKs6zpR7mUA0Q4FqmIPuCfHv5UKAbbtvbKfZePogg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731542224; c=relaxed/simple; bh=EKr4vTzAC596mh6OAe51sUzgy1wDv2Oe1xck8TnlJ0U=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=NFnHCfHrO9fb4QIzjFPLRZ0l/4onRmis9YquO3x7P78P++/uArDmFOCRXq65C+0sK7H5NBkJEdBBdeiKtUfUb2l7uJwJor4LsdM1r940Ff1bGqlhz5Qn6C1ctcMIu8gqWin6/ZPi0+XmgEkOJonalBLaHimvg34AYkXe5560t+4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com; spf=pass smtp.mailfrom=cisco.com; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b=Vuq0sRej; arc=none smtp.client-ip=173.37.142.88 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=cisco.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cisco.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=cisco.com header.i=@cisco.com header.b="Vuq0sRej" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=2306; q=dns/txt; s=iport; t=1731542223; x=1732751823; h=from:date:subject:mime-version:content-transfer-encoding: message-id:references:in-reply-to:to:cc; bh=+a/hUgADXfQgRFbN1jhyeqhTNqk5xC9hKt4NZ9QC7+c=; b=Vuq0sRejbh3IqSThpxjiEBj6zsUaHAXHs6wAxaTrgpdcf39fqQUD+Zhj w3lJjLlIxx2W5wVVejdC/bSuglDeWhYJW1ykzBM3mXSjwNkt363y5NxBI yjIUaihUj/YX5qX1Vvz6HvgfeY2n/M4a+GthN1F0qsoTi4rPCoifnWg4j 8=; X-CSE-ConnectionGUID: Ieu90L4SSdKvJVkVYWJouw== X-CSE-MsgGUID: v3lxAPX6R+GGkUDK/i/3aA== X-IPAS-Result: A0AGAADHOzVnj5MQJK1aGwEBAQEBAQEBBQEBARIBAQEDAwEBAUCBPwYBAQELAYQaQkiEVYgdhzCOFoxHhVyBJQNWDwEBAQ9EBAEBhQcCikUCJjQJDgECBAEBAQEDAgMBAQEBAQEBAQEBAQsBAQUBAQECAQcFFAEBAQEBATkFDjuGCIZbAgEDIwRSECUCJgICKxsQBgESgwGCZQIBsGZ6fzOBAYR72TiBbYEaLgGISwGBbINkGTuEPCcbgUlEhH2IHoJpBIQtA4MuhUODeZkpCT+BBRwDWSERAVUTDQoLBwVjVzwDIm9qXHorgQ6BFzpDgTuBIi8bIQtcgTeBGhQGFQSBDkY/gkppSzoCDQI2giQkWYJPhRqEbYRmglQvHUADCxgNSBEsNQYOGwY9AW4HnmVGgzIBgQ2CQJNQg2uNPqFrhCShXDOqTS6HZJBlIqQchGaBZzqBWzMaCBsVgyJSGQ+OLQ0JkxYBtj9DNTsCBwsBAQMJkEkBAQ IronPort-Data: A9a23:v0WPe61fzGy16YTCj/bD5UZxkn2cJEfYwER7XKvMYLTBsI5bp2FRz 2YXUD2Bafzba2Skc91zO4y39x4FsJCGn942HFM63Hw8FHgiRegpqji6wuYcGwvIc6UvmWo+t 512huHodZ5yFjmE4E/watANlFEkvYmQXL3wFeXYDS54QA5gWU8JhAlq8wIDqtYAbeORXUXU4 7sen+WFYAX5gmctaTpPg06+gEoHUMra6WtwUmMWPZinjHeG/1EJAZQWI72GLneQauF8Au6gS u/f+6qy92Xf8g1FIovNfmHTKxBirhb6ZGBiu1IOM0SQqkEqSh8ajs7XAMEhhXJ/0F1lqTzeJ OJl7vRcQS9xVkHFdX90vxNwS0mSNoUekFPLzOTWXcG7lyX7n3XQL/pGLUwrbNU8pt1MDUJQ1 8IbFgEyTBeemLfjqF67YrEEasULJc3vOsYb/3pn1zycVK5gSpHYSKKM7thdtNsyrpkRRrCFO YxAN3w2MEuojx5nYj/7DLo4keqzjX71ehVTqUmeouw85G27IAlZi+izaYGKJ4fSLSlTtljDo nLvx1SnOFI1G9+i5RnC8zX3j8aayEsXX6pJSeXnraQ16LGJ/UQfAQMbUHO3qOe0j0q5Vc4ZL UEIkgIjobU3/V6mUvHyWBq3pHPCtRkZM/JQFPc/8ymOx7DS7gLfAXILJhZCddYvnMw7Xzon0 hmOhdyBLTVpvKeYVjGb+6uYoC2aPTUTKykJZUcsVQIP7t/iiJs+ghLGUpBoF6vdptn0Hyzgh jOHti4zg50NgsMRkaa251bKh3SrvJehZgg4+gnaQEq74Q5jIo2ofYql7R7c9/koEWqCZlCFu H5Bn42V6/oDSMjV0ieMW+4KWrqu4p5pLQEwn3ZOEoRwrxDxwEWvXp5y3jJAFXhLNdsbLGqBj FDohStd45paPX2PZKBxYp6sB8lC8UQGPYq5PhwzRoQTCqWdZDO6EDdSiVl8Nl0BcXTAc4lja f93ku71UR727JiLKhLtGY/xNpdwmkgDKZv7H8yT8vhe+eP2iISpYbkEKkCSSesy8bmJpg7Ym /4GaJDTlk8FDr2uP3CNmWL2EbzsBSVlbXwRg5EGHtNv3iI/SQnN9teIm+p4IN0/90irvryYp yrjMqOn9LYPrSaacVrRMC8LhELHVpdkpnVzJj03IVutwDAiZ43phJrzhLNpFYTLANdLlKYuJ 9FcIp3oKq0WFlzvpW9HBbGj99MKSfherV7VV8ZTSGRkJ8Y4L+EIk/e4FjbSGN4mVXrr65Bm+ OL/h2s2g/MrHmxfMSofU9r3p3vZgJTXsLsas5fgSjWLRHjRzQ== IronPort-HdrOrdr: A9a23:IiC8jarGiGCMJL5ppaHcWsEaV5rmeYIsimQD101hICG9vPbo8P xG+8566faUslcssR4b9exoVJPsfZqYz+8R3WBzB9mftXfdyQiVxehZhOOIqQEIWReOlNK1vp 0OT0ERMqyVMXFKyev3/wW8Fc8t252k/LDAv5an815dCSxndK1k6R50EUKgEkNwTBRbHpZRLu vk2iM+nUvHRZzSBf7LfEXsmIP41qb2qK4= X-Talos-CUID: 9a23:pvVfd2uKXCrFsuxDXOx9AUUz6IsEf2CMlFOMGHS0KnhLdr6reGCy/757xp8= X-Talos-MUID: 9a23:O1jzGgTW5hJ8xiMXRXTy3zVyHf9YvZ7tJ28OrLkl/MjDbiVZbmI= X-IronPort-Anti-Spam-Filtered: true X-IronPort-AV: E=Sophos;i="6.12,152,1728950400"; d="scan'208";a="386137530" Received: from alln-l-core-10.cisco.com ([173.36.16.147]) by alln-iport-1.cisco.com with ESMTP/TLS/TLS_AES_256_GCM_SHA384; 13 Nov 2024 23:56:55 +0000 Received: from neescoba-vicdev.cisco.com (neescoba-vicdev.cisco.com [171.70.41.192]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by alln-l-core-10.cisco.com (Postfix) with ESMTPS id 60DED1800026E; Wed, 13 Nov 2024 23:56:54 +0000 (GMT) Received: by neescoba-vicdev.cisco.com (Postfix, from userid 412739) id E59C7CC12B6; Wed, 13 Nov 2024 23:56:52 +0000 (GMT) From: Nelson Escobar Date: Wed, 13 Nov 2024 23:56:39 +0000 Subject: [PATCH net-next v4 7/7] enic: Move kdump check into enic_adjust_resources() Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241113-remove_vic_resource_limits-v4-7-a34cf8570c67@cisco.com> References: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> In-Reply-To: <20241113-remove_vic_resource_limits-v4-0-a34cf8570c67@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1731542212; l=2338; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=EKr4vTzAC596mh6OAe51sUzgy1wDv2Oe1xck8TnlJ0U=; b=CB4HU7454gJHpqTOfwkzGZOvsq5kj+WNbydAhzUAhCn3N980d7ss0O64Cd486EACE/CrYNfHt 5uLh1IHgJOUDFHWGUv5sn2TOjveRvWdmpRfQ+a8lP2JjP6Ae+QJ8UCt X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Outbound-SMTP-Client: 171.70.41.192, neescoba-vicdev.cisco.com X-Outbound-Node: alln-l-core-10.cisco.com X-Patchwork-Delegate: kuba@kernel.org Move the kdump check into enic_adjust_resources() so that everything that modifies resources is in the same function. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Reviewed-by: Simon Horman Signed-off-by: Nelson Escobar Reviewed-by: Vadim Fedorenko --- drivers/net/ethernet/cisco/enic/enic_main.c | 25 +++++++++---------------- 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index 84e85c9e2bf52f0089c0a8d03fb6d22ada4d086c..9913952ccb42f2017037a81a8e2c42daa7b53ec3 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -2531,6 +2531,15 @@ static int enic_adjust_resources(struct enic *enic) return -ENOSPC; } + if (is_kdump_kernel()) { + dev_info(enic_get_dev(enic), "Running from within kdump kernel. Using minimal resources\n"); + enic->rq_avail = 1; + enic->wq_avail = 1; + enic->config.rq_desc_count = ENIC_MIN_RQ_DESCS; + enic->config.wq_desc_count = ENIC_MIN_WQ_DESCS; + enic->config.mtu = min_t(u16, 1500, enic->config.mtu); + } + /* if RSS isn't set, then we can only use one RQ */ if (!ENIC_SETTING(enic, RSS)) enic->rq_avail = 1; @@ -2764,18 +2773,6 @@ static void enic_dev_deinit(struct enic *enic) enic_free_enic_resources(enic); } -static void enic_kdump_kernel_config(struct enic *enic) -{ - if (is_kdump_kernel()) { - dev_info(enic_get_dev(enic), "Running from within kdump kernel. Using minimal resources\n"); - enic->rq_avail = 1; - enic->wq_avail = 1; - enic->config.rq_desc_count = ENIC_MIN_RQ_DESCS; - enic->config.wq_desc_count = ENIC_MIN_WQ_DESCS; - enic->config.mtu = min_t(u16, 1500, enic->config.mtu); - } -} - static int enic_dev_init(struct enic *enic) { struct device *dev = enic_get_dev(enic); @@ -2811,10 +2808,6 @@ static int enic_dev_init(struct enic *enic) return err; } - /* modify resource count if we are in kdump_kernel - */ - enic_kdump_kernel_config(enic); - /* Set interrupt mode based on system capabilities */ err = enic_set_intr_mode(enic);