From patchwork Fri Oct 25 01:19:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nelson Escobar via B4 Relay X-Patchwork-Id: 13849897 X-Patchwork-Delegate: kuba@kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C76CB101C4; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729819263; cv=none; b=WDS1jmWUQUUDMf0AhDGaD3Q/Cft7jtHihojRVbgXAcaRXkQ85zNvbN8J1HJmgQZnpOlsPUDhRVAcYkNwDS2HERJC50ZoyceBM8JnpGdtAkesHDB482DtVWGbtvO8MgiPOVLi5rH6Or5PnNNOW4ah7pghVZhvpQdmOpd9ju4u49g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729819263; c=relaxed/simple; bh=L1XE8PZoAUPUzZtSnw2PFB2Lg7bJUKovoB5JtkuiFXw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=rxo4tfTSXbFcp1Na/SSEAWrk7kaSzRHc0jpuMS1YHBBtm0AawINnnlRJopOcNmWk6efJ6d7Po6e9OuvnnDx05phOd06GL9+nA1rO7T95o54ui8GQH64YvBx67TCdsJze1okMtAQjaxW4UK9zuGr4TVbCEraIou2urtydRN9HQaw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RiKP/j4E; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RiKP/j4E" Received: by smtp.kernel.org (Postfix) with ESMTPS id 73AB7C4CEE3; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1729819263; bh=L1XE8PZoAUPUzZtSnw2PFB2Lg7bJUKovoB5JtkuiFXw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=RiKP/j4ENxGtnn6xjbLMn23kMV/lca67TdME8O/RCQ4LdtS/ZDsLxGFLYs5X7qxmq m8YyWcMBKRL686kRZYMtTOfi9/NyW4vFyk+RVoujdE19Db0ZqQb83rEs1OBJnU8HSr 03kcTSNWfwqvHsaN1JnzpGSeKid1bC5VnniQTicy9HDflaybtIsLR7t0fIUVwEXXcz xcPTC64+FtkepLOdwmWT+F0Iq8j2iAPmmyjBhwR6tBbq6m3zPlKLGCZB01bjbd9e4O VfdetoXQuJH2v9umEbeDayo7FOTdueEo8ihsCbgKzsgeFdqeJgDdMH5qGD6Lcxvwqy FBuv2w9S1DrUw== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FE5FD10399; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) From: Nelson Escobar via B4 Relay Date: Thu, 24 Oct 2024 18:19:43 -0700 Subject: [PATCH net-next v2 1/5] enic: Create enic_wq/rq structures to bundle per wq/rq data Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241024-remove_vic_resource_limits-v2-1-039b8cae5fdd@cisco.com> References: <20241024-remove_vic_resource_limits-v2-0-039b8cae5fdd@cisco.com> In-Reply-To: <20241024-remove_vic_resource_limits-v2-0-039b8cae5fdd@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1729819262; l=19489; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=qNHE+/jqwWBnXG5CtX8XIEpIakCWbedk122EG1yjJa0=; b=1H3PS5MNwetkW917+2PJ6N3z/PRNih8FVZBGkMGPsz2OubaOfyJEx5pfKh0NxdWy0VPznTA4d gHpLTDiVuV2BlRIztNb/ua42kEVI78sVqGr9RLvuTz1wbxfvci9Xhqw X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Endpoint-Received: by B4 Relay for neescoba@cisco.com/20241023 with auth_id=255 X-Original-From: Nelson Escobar Reply-To: neescoba@cisco.com X-Patchwork-Delegate: kuba@kernel.org From: Nelson Escobar Bundling the wq/rq specific data into dedicated enic_wq/rq structures cleans up the enic structure and simplifies future changes related to wq/rq. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic.h | 18 ++-- drivers/net/ethernet/cisco/enic/enic_ethtool.c | 4 +- drivers/net/ethernet/cisco/enic/enic_main.c | 120 ++++++++++++------------- drivers/net/ethernet/cisco/enic/enic_res.c | 12 +-- 4 files changed, 81 insertions(+), 73 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index 0cc3644ee8554f52401a0be7f44a1475ab2ea2b9..e6edb43515b97feeb21a9b55a1eeaa9b9381183f 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -162,6 +162,17 @@ struct enic_rq_stats { u64 desc_skip; /* Rx pkt went into later buffer */ }; +struct enic_wq { + struct vnic_wq vwq; + struct enic_wq_stats stats; + spinlock_t lock; /* spinlock for wq */ +}; + +struct enic_rq { + struct vnic_rq vrq; + struct enic_rq_stats stats; +}; + /* Per-instance private data structure */ struct enic { struct net_device *netdev; @@ -194,16 +205,13 @@ struct enic { struct enic_port_profile *pp; /* work queue cache line section */ - ____cacheline_aligned struct vnic_wq wq[ENIC_WQ_MAX]; - spinlock_t wq_lock[ENIC_WQ_MAX]; - struct enic_wq_stats wq_stats[ENIC_WQ_MAX]; + ____cacheline_aligned struct enic_wq wq[ENIC_WQ_MAX]; unsigned int wq_count; u16 loop_enable; u16 loop_tag; /* receive queue cache line section */ - ____cacheline_aligned struct vnic_rq rq[ENIC_RQ_MAX]; - struct enic_rq_stats rq_stats[ENIC_RQ_MAX]; + ____cacheline_aligned struct enic_rq rq[ENIC_RQ_MAX]; unsigned int rq_count; struct vxlan_offload vxlan; struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX]; diff --git a/drivers/net/ethernet/cisco/enic/enic_ethtool.c b/drivers/net/ethernet/cisco/enic/enic_ethtool.c index f7986f2b6a1794144909ffad9e3e09c32ea44c93..909d6f7000e160cf2e15de4660c1034cad7d51ba 100644 --- a/drivers/net/ethernet/cisco/enic/enic_ethtool.c +++ b/drivers/net/ethernet/cisco/enic/enic_ethtool.c @@ -337,7 +337,7 @@ static void enic_get_ethtool_stats(struct net_device *netdev, for (i = 0; i < NUM_ENIC_GEN_STATS; i++) *(data++) = ((u64 *)&enic->gen_stats)[enic_gen_stats[i].index]; for (i = 0; i < enic->rq_count; i++) { - struct enic_rq_stats *rqstats = &enic->rq_stats[i]; + struct enic_rq_stats *rqstats = &enic->rq[i].stats; int index; for (j = 0; j < NUM_ENIC_PER_RQ_STATS; j++) { @@ -346,7 +346,7 @@ static void enic_get_ethtool_stats(struct net_device *netdev, } } for (i = 0; i < enic->wq_count; i++) { - struct enic_wq_stats *wqstats = &enic->wq_stats[i]; + struct enic_wq_stats *wqstats = &enic->wq[i].stats; int index; for (j = 0; j < NUM_ENIC_PER_WQ_STATS; j++) { diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index ffed14b63d41d1737c577fe1662eb1c2c8aea808..eb00058b6c68ec5c1ac433b54b5bc6f3fb613777 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -342,8 +342,8 @@ static void enic_wq_free_buf(struct vnic_wq *wq, { struct enic *enic = vnic_dev_priv(wq->vdev); - enic->wq_stats[wq->index].cq_work++; - enic->wq_stats[wq->index].cq_bytes += buf->len; + enic->wq[wq->index].stats.cq_work++; + enic->wq[wq->index].stats.cq_bytes += buf->len; enic_free_wq_buf(wq, buf); } @@ -352,20 +352,20 @@ static int enic_wq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, { struct enic *enic = vnic_dev_priv(vdev); - spin_lock(&enic->wq_lock[q_number]); + spin_lock(&enic->wq[q_number].lock); - vnic_wq_service(&enic->wq[q_number], cq_desc, + vnic_wq_service(&enic->wq[q_number].vwq, cq_desc, completed_index, enic_wq_free_buf, opaque); if (netif_tx_queue_stopped(netdev_get_tx_queue(enic->netdev, q_number)) && - vnic_wq_desc_avail(&enic->wq[q_number]) >= + vnic_wq_desc_avail(&enic->wq[q_number].vwq) >= (MAX_SKB_FRAGS + ENIC_DESC_MAX_SPLITS)) { netif_wake_subqueue(enic->netdev, q_number); - enic->wq_stats[q_number].wake++; + enic->wq[q_number].stats.wake++; } - spin_unlock(&enic->wq_lock[q_number]); + spin_unlock(&enic->wq[q_number].lock); return 0; } @@ -377,7 +377,7 @@ static bool enic_log_q_error(struct enic *enic) bool err = false; for (i = 0; i < enic->wq_count; i++) { - error_status = vnic_wq_error_status(&enic->wq[i]); + error_status = vnic_wq_error_status(&enic->wq[i].vwq); err |= error_status; if (error_status) netdev_err(enic->netdev, "WQ[%d] error_status %d\n", @@ -385,7 +385,7 @@ static bool enic_log_q_error(struct enic *enic) } for (i = 0; i < enic->rq_count; i++) { - error_status = vnic_rq_error_status(&enic->rq[i]); + error_status = vnic_rq_error_status(&enic->rq[i].vrq); err |= error_status; if (error_status) netdev_err(enic->netdev, "RQ[%d] error_status %d\n", @@ -598,9 +598,9 @@ static int enic_queue_wq_skb_vlan(struct enic *enic, struct vnic_wq *wq, err = enic_queue_wq_skb_cont(enic, wq, skb, len_left, loopback); /* The enic_queue_wq_desc() above does not do HW checksum */ - enic->wq_stats[wq->index].csum_none++; - enic->wq_stats[wq->index].packets++; - enic->wq_stats[wq->index].bytes += skb->len; + enic->wq[wq->index].stats.csum_none++; + enic->wq[wq->index].stats.packets++; + enic->wq[wq->index].stats.bytes += skb->len; return err; } @@ -634,9 +634,9 @@ static int enic_queue_wq_skb_csum_l4(struct enic *enic, struct vnic_wq *wq, if (!eop) err = enic_queue_wq_skb_cont(enic, wq, skb, len_left, loopback); - enic->wq_stats[wq->index].csum_partial++; - enic->wq_stats[wq->index].packets++; - enic->wq_stats[wq->index].bytes += skb->len; + enic->wq[wq->index].stats.csum_partial++; + enic->wq[wq->index].stats.packets++; + enic->wq[wq->index].stats.bytes += skb->len; return err; } @@ -699,11 +699,11 @@ static int enic_queue_wq_skb_tso(struct enic *enic, struct vnic_wq *wq, if (skb->encapsulation) { hdr_len = skb_inner_tcp_all_headers(skb); enic_preload_tcp_csum_encap(skb); - enic->wq_stats[wq->index].encap_tso++; + enic->wq[wq->index].stats.encap_tso++; } else { hdr_len = skb_tcp_all_headers(skb); enic_preload_tcp_csum(skb); - enic->wq_stats[wq->index].tso++; + enic->wq[wq->index].stats.tso++; } /* Queue WQ_ENET_MAX_DESC_LEN length descriptors @@ -757,8 +757,8 @@ static int enic_queue_wq_skb_tso(struct enic *enic, struct vnic_wq *wq, pkts = len / mss; if ((len % mss) > 0) pkts++; - enic->wq_stats[wq->index].packets += pkts; - enic->wq_stats[wq->index].bytes += (len + (pkts * hdr_len)); + enic->wq[wq->index].stats.packets += pkts; + enic->wq[wq->index].stats.bytes += (len + (pkts * hdr_len)); return 0; } @@ -792,9 +792,9 @@ static inline int enic_queue_wq_skb_encap(struct enic *enic, struct vnic_wq *wq, if (!eop) err = enic_queue_wq_skb_cont(enic, wq, skb, len_left, loopback); - enic->wq_stats[wq->index].encap_csum++; - enic->wq_stats[wq->index].packets++; - enic->wq_stats[wq->index].bytes += skb->len; + enic->wq[wq->index].stats.encap_csum++; + enic->wq[wq->index].stats.packets++; + enic->wq[wq->index].stats.bytes += skb->len; return err; } @@ -812,7 +812,7 @@ static inline int enic_queue_wq_skb(struct enic *enic, /* VLAN tag from trunking driver */ vlan_tag_insert = 1; vlan_tag = skb_vlan_tag_get(skb); - enic->wq_stats[wq->index].add_vlan++; + enic->wq[wq->index].stats.add_vlan++; } else if (enic->loop_enable) { vlan_tag = enic->loop_tag; loopback = 1; @@ -859,11 +859,11 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, struct netdev_queue *txq; txq_map = skb_get_queue_mapping(skb) % enic->wq_count; - wq = &enic->wq[txq_map]; + wq = &enic->wq[txq_map].vwq; if (skb->len <= 0) { dev_kfree_skb_any(skb); - enic->wq_stats[wq->index].null_pkt++; + enic->wq[wq->index].stats.null_pkt++; return NETDEV_TX_OK; } @@ -878,19 +878,19 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, skb_shinfo(skb)->nr_frags + 1 > ENIC_NON_TSO_MAX_DESC && skb_linearize(skb)) { dev_kfree_skb_any(skb); - enic->wq_stats[wq->index].skb_linear_fail++; + enic->wq[wq->index].stats.skb_linear_fail++; return NETDEV_TX_OK; } - spin_lock(&enic->wq_lock[txq_map]); + spin_lock(&enic->wq[txq_map].lock); if (vnic_wq_desc_avail(wq) < skb_shinfo(skb)->nr_frags + ENIC_DESC_MAX_SPLITS) { netif_tx_stop_queue(txq); /* This is a hard error, log it */ netdev_err(netdev, "BUG! Tx ring full when queue awake!\n"); - spin_unlock(&enic->wq_lock[txq_map]); - enic->wq_stats[wq->index].desc_full_awake++; + spin_unlock(&enic->wq[txq_map].lock); + enic->wq[wq->index].stats.desc_full_awake++; return NETDEV_TX_BUSY; } @@ -899,14 +899,14 @@ static netdev_tx_t enic_hard_start_xmit(struct sk_buff *skb, if (vnic_wq_desc_avail(wq) < MAX_SKB_FRAGS + ENIC_DESC_MAX_SPLITS) { netif_tx_stop_queue(txq); - enic->wq_stats[wq->index].stopped++; + enic->wq[wq->index].stats.stopped++; } skb_tx_timestamp(skb); if (!netdev_xmit_more() || netif_xmit_stopped(txq)) vnic_wq_doorbell(wq); error: - spin_unlock(&enic->wq_lock[txq_map]); + spin_unlock(&enic->wq[txq_map].lock); return NETDEV_TX_OK; } @@ -941,9 +941,9 @@ static void enic_get_stats(struct net_device *netdev, net_stats->multicast = stats->rx.rx_multicast_frames_ok; for (i = 0; i < ENIC_RQ_MAX; i++) { - struct enic_rq_stats *rqs = &enic->rq_stats[i]; + struct enic_rq_stats *rqs = &enic->rq[i].stats; - if (!enic->rq->ctrl) + if (!enic->rq[i].vrq.ctrl) break; pkt_truncated += rqs->pkt_truncated; bad_fcs += rqs->bad_fcs; @@ -1313,7 +1313,7 @@ static int enic_rq_alloc_buf(struct vnic_rq *rq) } skb = netdev_alloc_skb_ip_align(netdev, len); if (!skb) { - enic->rq_stats[rq->index].no_skb++; + enic->rq[rq->index].stats.no_skb++; return -ENOMEM; } @@ -1366,7 +1366,7 @@ static void enic_rq_indicate_buf(struct vnic_rq *rq, struct net_device *netdev = enic->netdev; struct sk_buff *skb; struct vnic_cq *cq = &enic->cq[enic_cq_rq(enic, rq->index)]; - struct enic_rq_stats *rqstats = &enic->rq_stats[rq->index]; + struct enic_rq_stats *rqstats = &enic->rq[rq->index].stats; u8 type, color, eop, sop, ingress_port, vlan_stripped; u8 fcoe, fcoe_sof, fcoe_fc_crc_ok, fcoe_enc_error, fcoe_eof; @@ -1512,7 +1512,7 @@ static int enic_rq_service(struct vnic_dev *vdev, struct cq_desc *cq_desc, { struct enic *enic = vnic_dev_priv(vdev); - vnic_rq_service(&enic->rq[q_number], cq_desc, + vnic_rq_service(&enic->rq[q_number].vrq, cq_desc, completed_index, VNIC_RQ_RETURN_DESC, enic_rq_indicate_buf, opaque); @@ -1609,7 +1609,7 @@ static int enic_poll(struct napi_struct *napi, int budget) 0 /* don't unmask intr */, 0 /* don't reset intr timer */); - err = vnic_rq_fill(&enic->rq[0], enic_rq_alloc_buf); + err = vnic_rq_fill(&enic->rq[0].vrq, enic_rq_alloc_buf); /* Buffer allocation failed. Stay in polling * mode so we can try to fill the ring again. @@ -1621,7 +1621,7 @@ static int enic_poll(struct napi_struct *napi, int budget) /* Call the function which refreshes the intr coalescing timer * value based on the traffic. */ - enic_calc_int_moderation(enic, &enic->rq[0]); + enic_calc_int_moderation(enic, &enic->rq[0].vrq); if ((rq_work_done < budget) && napi_complete_done(napi, rq_work_done)) { @@ -1630,11 +1630,11 @@ static int enic_poll(struct napi_struct *napi, int budget) */ if (enic->rx_coalesce_setting.use_adaptive_rx_coalesce) - enic_set_int_moderation(enic, &enic->rq[0]); + enic_set_int_moderation(enic, &enic->rq[0].vrq); vnic_intr_unmask(&enic->intr[intr]); - enic->rq_stats[0].napi_complete++; + enic->rq[0].stats.napi_complete++; } else { - enic->rq_stats[0].napi_repoll++; + enic->rq[0].stats.napi_repoll++; } return rq_work_done; @@ -1683,7 +1683,7 @@ static int enic_poll_msix_wq(struct napi_struct *napi, int budget) struct net_device *netdev = napi->dev; struct enic *enic = netdev_priv(netdev); unsigned int wq_index = (napi - &enic->napi[0]) - enic->rq_count; - struct vnic_wq *wq = &enic->wq[wq_index]; + struct vnic_wq *wq = &enic->wq[wq_index].vwq; unsigned int cq; unsigned int intr; unsigned int wq_work_to_do = ENIC_WQ_NAPI_BUDGET; @@ -1737,7 +1737,7 @@ static int enic_poll_msix_rq(struct napi_struct *napi, int budget) 0 /* don't unmask intr */, 0 /* don't reset intr timer */); - err = vnic_rq_fill(&enic->rq[rq], enic_rq_alloc_buf); + err = vnic_rq_fill(&enic->rq[rq].vrq, enic_rq_alloc_buf); /* Buffer allocation failed. Stay in polling mode * so we can try to fill the ring again. @@ -1749,7 +1749,7 @@ static int enic_poll_msix_rq(struct napi_struct *napi, int budget) /* Call the function which refreshes the intr coalescing timer * value based on the traffic. */ - enic_calc_int_moderation(enic, &enic->rq[rq]); + enic_calc_int_moderation(enic, &enic->rq[rq].vrq); if ((work_done < budget) && napi_complete_done(napi, work_done)) { @@ -1758,11 +1758,11 @@ static int enic_poll_msix_rq(struct napi_struct *napi, int budget) */ if (enic->rx_coalesce_setting.use_adaptive_rx_coalesce) - enic_set_int_moderation(enic, &enic->rq[rq]); + enic_set_int_moderation(enic, &enic->rq[rq].vrq); vnic_intr_unmask(&enic->intr[intr]); - enic->rq_stats[rq].napi_complete++; + enic->rq[rq].stats.napi_complete++; } else { - enic->rq_stats[rq].napi_repoll++; + enic->rq[rq].stats.napi_repoll++; } return work_done; @@ -1989,10 +1989,10 @@ static int enic_open(struct net_device *netdev) for (i = 0; i < enic->rq_count; i++) { /* enable rq before updating rq desc */ - vnic_rq_enable(&enic->rq[i]); - vnic_rq_fill(&enic->rq[i], enic_rq_alloc_buf); + vnic_rq_enable(&enic->rq[i].vrq); + vnic_rq_fill(&enic->rq[i].vrq, enic_rq_alloc_buf); /* Need at least one buffer on ring to get going */ - if (vnic_rq_desc_used(&enic->rq[i]) == 0) { + if (vnic_rq_desc_used(&enic->rq[i].vrq) == 0) { netdev_err(netdev, "Unable to alloc receive buffers\n"); err = -ENOMEM; goto err_out_free_rq; @@ -2000,7 +2000,7 @@ static int enic_open(struct net_device *netdev) } for (i = 0; i < enic->wq_count; i++) - vnic_wq_enable(&enic->wq[i]); + vnic_wq_enable(&enic->wq[i].vwq); if (!enic_is_dynamic(enic) && !enic_is_sriov_vf(enic)) enic_dev_add_station_addr(enic); @@ -2027,9 +2027,9 @@ static int enic_open(struct net_device *netdev) err_out_free_rq: for (i = 0; i < enic->rq_count; i++) { - ret = vnic_rq_disable(&enic->rq[i]); + ret = vnic_rq_disable(&enic->rq[i].vrq); if (!ret) - vnic_rq_clean(&enic->rq[i], enic_free_rq_buf); + vnic_rq_clean(&enic->rq[i].vrq, enic_free_rq_buf); } enic_dev_notify_unset(enic); err_out_free_intr: @@ -2071,12 +2071,12 @@ static int enic_stop(struct net_device *netdev) enic_dev_del_station_addr(enic); for (i = 0; i < enic->wq_count; i++) { - err = vnic_wq_disable(&enic->wq[i]); + err = vnic_wq_disable(&enic->wq[i].vwq); if (err) return err; } for (i = 0; i < enic->rq_count; i++) { - err = vnic_rq_disable(&enic->rq[i]); + err = vnic_rq_disable(&enic->rq[i].vrq); if (err) return err; } @@ -2086,9 +2086,9 @@ static int enic_stop(struct net_device *netdev) enic_free_intr(enic); for (i = 0; i < enic->wq_count; i++) - vnic_wq_clean(&enic->wq[i], enic_free_wq_buf); + vnic_wq_clean(&enic->wq[i].vwq, enic_free_wq_buf); for (i = 0; i < enic->rq_count; i++) - vnic_rq_clean(&enic->rq[i], enic_free_rq_buf); + vnic_rq_clean(&enic->rq[i].vrq, enic_free_rq_buf); for (i = 0; i < enic->cq_count; i++) vnic_cq_clean(&enic->cq[i]); for (i = 0; i < enic->intr_count; i++) @@ -2576,7 +2576,7 @@ static void enic_get_queue_stats_rx(struct net_device *dev, int idx, struct netdev_queue_stats_rx *rxs) { struct enic *enic = netdev_priv(dev); - struct enic_rq_stats *rqstats = &enic->rq_stats[idx]; + struct enic_rq_stats *rqstats = &enic->rq[idx].stats; rxs->bytes = rqstats->bytes; rxs->packets = rqstats->packets; @@ -2590,7 +2590,7 @@ static void enic_get_queue_stats_tx(struct net_device *dev, int idx, struct netdev_queue_stats_tx *txs) { struct enic *enic = netdev_priv(dev); - struct enic_wq_stats *wqstats = &enic->wq_stats[idx]; + struct enic_wq_stats *wqstats = &enic->wq[idx].stats; txs->bytes = wqstats->bytes; txs->packets = wqstats->packets; @@ -2993,7 +2993,7 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) INIT_WORK(&enic->change_mtu_work, enic_change_mtu_work); for (i = 0; i < enic->wq_count; i++) - spin_lock_init(&enic->wq_lock[i]); + spin_lock_init(&enic->wq[i].lock); /* Register net device */ diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 1c48aebdbab02b88293544dcabda2d90d1a71a70..60be09acb9fd56b642b7cabc77fac01f526b29a2 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -176,9 +176,9 @@ void enic_free_vnic_resources(struct enic *enic) unsigned int i; for (i = 0; i < enic->wq_count; i++) - vnic_wq_free(&enic->wq[i]); + vnic_wq_free(&enic->wq[i].vwq); for (i = 0; i < enic->rq_count; i++) - vnic_rq_free(&enic->rq[i]); + vnic_rq_free(&enic->rq[i].vrq); for (i = 0; i < enic->cq_count; i++) vnic_cq_free(&enic->cq[i]); for (i = 0; i < enic->intr_count; i++) @@ -233,7 +233,7 @@ void enic_init_vnic_resources(struct enic *enic) for (i = 0; i < enic->rq_count; i++) { cq_index = i; - vnic_rq_init(&enic->rq[i], + vnic_rq_init(&enic->rq[i].vrq, cq_index, error_interrupt_enable, error_interrupt_offset); @@ -241,7 +241,7 @@ void enic_init_vnic_resources(struct enic *enic) for (i = 0; i < enic->wq_count; i++) { cq_index = enic->rq_count + i; - vnic_wq_init(&enic->wq[i], + vnic_wq_init(&enic->wq[i].vwq, cq_index, error_interrupt_enable, error_interrupt_offset); @@ -322,7 +322,7 @@ int enic_alloc_vnic_resources(struct enic *enic) */ for (i = 0; i < enic->wq_count; i++) { - err = vnic_wq_alloc(enic->vdev, &enic->wq[i], i, + err = vnic_wq_alloc(enic->vdev, &enic->wq[i].vwq, i, enic->config.wq_desc_count, sizeof(struct wq_enet_desc)); if (err) @@ -330,7 +330,7 @@ int enic_alloc_vnic_resources(struct enic *enic) } for (i = 0; i < enic->rq_count; i++) { - err = vnic_rq_alloc(enic->vdev, &enic->rq[i], i, + err = vnic_rq_alloc(enic->vdev, &enic->rq[i].vrq, i, enic->config.rq_desc_count, sizeof(struct rq_enet_desc)); if (err) From patchwork Fri Oct 25 01:19:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nelson Escobar via B4 Relay X-Patchwork-Id: 13849898 X-Patchwork-Delegate: kuba@kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C77041173F; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729819263; cv=none; b=mG3of/2lUXndr0bFU3KGorRThX+hZlm+eUiliF3OB5ogmFx3laFbN706um7YYPfvtZcwS3hUSt07VKS45552vnh2oQCUjYDab5mBzs+hiOMzZg+zvZJ45bEt3YqlwM8N/esPz5/RCrSg9JiC7dC8kJm3dy+7QlrZYivgoy9SrZQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729819263; c=relaxed/simple; bh=dNyqJYC+zOgdeFmmz+dCYzAPL6UxHm6lnvbddLBYDlw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=rmwmj7x9JkSSvWOEEV+i82nEGrwClQXaP7Lew5bZ4PCMNEdvrZCjHkLHiShvZvaxksIruvAeZtRWAbUzrNqNvcK6oLZZ5Bj4IJjRt278YXvTNOtAuf1Xuly46rEBNYMuiCSCDP+L9vkHU5oEeNJautgSufBlVCnDsxDmDXh8FUE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=YUpl2FHM; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="YUpl2FHM" Received: by smtp.kernel.org (Postfix) with ESMTPS id 828C5C4CEE4; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1729819263; bh=dNyqJYC+zOgdeFmmz+dCYzAPL6UxHm6lnvbddLBYDlw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=YUpl2FHMAsMENCRXfJo7TOxaKwuhavzaR774ABrZ6ftqObKRIkU6PMiXQ5YXHenYs wWkAkkynFwg9mdKu0DFmtjgmFUKCw6nv6Y8pBXyOHOEVjSonq8ssCgxMk2wyPNnSfM ZtlVELyE4jBn78DqK/c9fdoc0xAU113sFEus+iUgCBHG6Ps9uMS1R16qfX8DYXWhoW pfRKwP7QLCdeHqJIzlPSeREyVU3W0sHHg0TRxhMIVmsLSM/RKxJ5lONwGi/deyTPVw 1LtyFzvg9HGnbM7WW0fol6r9MIpCOPZL0hGMHisWYPNZTpM216vFzhirt+QkSbzRC4 JSTj0/2J4s+7g== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71AF1D1039D; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) From: Nelson Escobar via B4 Relay Date: Thu, 24 Oct 2024 18:19:44 -0700 Subject: [PATCH net-next v2 2/5] enic: Make MSI-X I/O interrupts come after the other required ones Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241024-remove_vic_resource_limits-v2-2-039b8cae5fdd@cisco.com> References: <20241024-remove_vic_resource_limits-v2-0-039b8cae5fdd@cisco.com> In-Reply-To: <20241024-remove_vic_resource_limits-v2-0-039b8cae5fdd@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1729819262; l=2857; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=V+fNI/BLR6TzQSd85AV74fwLJ9hCh1MBL/3lYOO6PGg=; b=09v3X3zJZsYBT9WqWNU4gM9VMZedzVna8urS8sZQ5I9GtANe1ghYqHwcv3chMd+eb3wDvYAMc mDszeyDiqowAg4g6IuljhM8knb3ZYvPUlt+9Xq1NJUXf1w/dvaoNCUK X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Endpoint-Received: by B4 Relay for neescoba@cisco.com/20241023 with auth_id=255 X-Original-From: Nelson Escobar Reply-To: neescoba@cisco.com X-Patchwork-Delegate: kuba@kernel.org From: Nelson Escobar The VIC hardware has a constraint that the MSIX interrupt used for errors be specified as a 7 bit number. Before this patch, it was allocated after the I/O interrupts, which would cause a problem if 128 or more I/O interrupts are in use. So make the required interrupts come before the I/O interrupts to guarantee the error interrupt offset never exceeds 7 bits. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic.h | 20 +++++++++++++++----- drivers/net/ethernet/cisco/enic/enic_res.c | 2 +- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index e6edb43515b97feeb21a9b55a1eeaa9b9381183f..ac7236f76a51bf32e7060ee0482b41fe82b60b44 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -280,18 +280,28 @@ static inline unsigned int enic_msix_wq_intr(struct enic *enic, return enic->cq[enic_cq_wq(enic, wq)].interrupt_offset; } -static inline unsigned int enic_msix_err_intr(struct enic *enic) -{ - return enic->rq_count + enic->wq_count; -} +/* MSIX interrupts are organized as the error interrupt, then the notify + * interrupt followed by all the I/O interrupts. The error interrupt needs + * to fit in 7 bits due to hardware constraints + */ +#define ENIC_MSIX_RESERVED_INTR 2 +#define ENIC_MSIX_ERR_INTR 0 +#define ENIC_MSIX_NOTIFY_INTR 1 +#define ENIC_MSIX_IO_INTR_BASE ENIC_MSIX_RESERVED_INTR +#define ENIC_MSIX_MIN_INTR (ENIC_MSIX_RESERVED_INTR + 2) #define ENIC_LEGACY_IO_INTR 0 #define ENIC_LEGACY_ERR_INTR 1 #define ENIC_LEGACY_NOTIFY_INTR 2 +static inline unsigned int enic_msix_err_intr(struct enic *enic) +{ + return ENIC_MSIX_ERR_INTR; +} + static inline unsigned int enic_msix_notify_intr(struct enic *enic) { - return enic->rq_count + enic->wq_count + 1; + return ENIC_MSIX_NOTIFY_INTR; } static inline bool enic_is_err_intr(struct enic *enic, int intr) diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 60be09acb9fd56b642b7cabc77fac01f526b29a2..6910f83185c44797d769434fbe8af3105215b143 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -257,7 +257,7 @@ void enic_init_vnic_resources(struct enic *enic) switch (intr_mode) { case VNIC_DEV_INTR_MODE_MSIX: - interrupt_offset = i; + interrupt_offset = ENIC_MSIX_IO_INTR_BASE + i; break; default: interrupt_offset = 0; From patchwork Fri Oct 25 01:19:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nelson Escobar via B4 Relay X-Patchwork-Id: 13849896 X-Patchwork-Delegate: kuba@kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C765EBA3D; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729819263; cv=none; b=qLm5qjEXXY4FQ1U5+jiWCN7C2UP4c+IHtCAJ7uX2bTotscMf1DkvOKifIVYOQitBm6LcjqS1/E2qDlKerqqBItihCkOgG1Fn5NeVDev1XxfezpaRSR6jWCpckP0/sGxyctp2/XrWBxOSM5nllYKorL6GtwONhBlDp8VJgqRvOWU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729819263; c=relaxed/simple; bh=FGrImqPLytR/vSMPJclojbKkKDRGJI4i5F8XMfmPHoE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=S5SPJyGqyw0uPxmY8KG8dIjHNtH/EMOctw1xcnoS8paGagZ8iOhvgxCVNYzXqJ1xickm/bsKgLyoFoXGaeVmJ+ZhUiDaZsHu3XWsUASaId6P8/CSV6XmEeYZRoZf30R6Sgk4gse2mCLHWVKsa143d17KePIR3WwtlkZ68GvU4WM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gSdFUDXB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gSdFUDXB" Received: by smtp.kernel.org (Postfix) with ESMTPS id 8B665C4CEE7; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1729819263; bh=FGrImqPLytR/vSMPJclojbKkKDRGJI4i5F8XMfmPHoE=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=gSdFUDXB6PipNFxbKQQLOEuOYO71ZH58FPcY3usd74xh0f05G4h0du4vw4UQl2o5l 5kz4POdn/W5yOOxJuucePlDWY25Bm35MOcubyALHQUHRwx7w6Jw9WuEpkia8v0BueY tH9ODrFcef1eYfCSY+DImUTWAabMEd2i7AQ8HmZRxR8sJrOxVSFhRCVBt2W4d+PMOh ZH5lT9l0ZFUWTt5EWyXFpsQEckZSniKjpRRIFXwPZlA49KsS9YOPMBDXts3qAaJdgi o6i+k7IxIvq+1NJkE5tFFOYX/xSd5SxMSiTyrEKlBRbJUCoQjsOVKDcnR1hJEHse5Z KHMvIl1/LMDrQ== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 808E2D1038E; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) From: Nelson Escobar via B4 Relay Date: Thu, 24 Oct 2024 18:19:45 -0700 Subject: [PATCH net-next v2 3/5] enic: Save resource counts we read from HW Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241024-remove_vic_resource_limits-v2-3-039b8cae5fdd@cisco.com> References: <20241024-remove_vic_resource_limits-v2-0-039b8cae5fdd@cisco.com> In-Reply-To: <20241024-remove_vic_resource_limits-v2-0-039b8cae5fdd@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1729819262; l=4213; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=xLHC8wEqUFjN0bkMiGJDjRPC+kG1EB+4ffnh4Hz1IXk=; b=Q+8WYR68oHWq3/IeOPfaM03pP1NuQr62jm6ptrKdoBJJE9adzIhP1A0Rz792+bMhlIBPaa/e/ lPSicyrSJCxDWkLmQ3XkHpPdzb21SM8GHAMNvAHB/yrwXRva13CZh9y X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Endpoint-Received: by B4 Relay for neescoba@cisco.com/20241023 with auth_id=255 X-Original-From: Nelson Escobar Reply-To: neescoba@cisco.com X-Patchwork-Delegate: kuba@kernel.org From: Nelson Escobar Save the resources counts for wq,rq,cq, and interrupts in *_avail variables so that we don't lose the information when adjusting the counts we are actually using. Report the wq_avail and rq_avail as the channel maximums in 'ethtool -l' output. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic.h | 4 ++++ drivers/net/ethernet/cisco/enic/enic_ethtool.c | 4 ++-- drivers/net/ethernet/cisco/enic/enic_res.c | 19 ++++++++++++------- 3 files changed, 18 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index ac7236f76a51bf32e7060ee0482b41fe82b60b44..1f32413a8f7c690060fe385b50f7447943e72596 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -206,23 +206,27 @@ struct enic { /* work queue cache line section */ ____cacheline_aligned struct enic_wq wq[ENIC_WQ_MAX]; + unsigned int wq_avail; unsigned int wq_count; u16 loop_enable; u16 loop_tag; /* receive queue cache line section */ ____cacheline_aligned struct enic_rq rq[ENIC_RQ_MAX]; + unsigned int rq_avail; unsigned int rq_count; struct vxlan_offload vxlan; struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX]; /* interrupt resource cache line section */ ____cacheline_aligned struct vnic_intr intr[ENIC_INTR_MAX]; + unsigned int intr_avail; unsigned int intr_count; u32 __iomem *legacy_pba; /* memory-mapped */ /* completion queue cache line section */ ____cacheline_aligned struct vnic_cq cq[ENIC_CQ_MAX]; + unsigned int cq_avail; unsigned int cq_count; struct enic_rfs_flw_tbl rfs_h; u32 rx_copybreak; diff --git a/drivers/net/ethernet/cisco/enic/enic_ethtool.c b/drivers/net/ethernet/cisco/enic/enic_ethtool.c index 909d6f7000e160cf2e15de4660c1034cad7d51ba..d607b4f0542ceaef09e9528a591ca27177986143 100644 --- a/drivers/net/ethernet/cisco/enic/enic_ethtool.c +++ b/drivers/net/ethernet/cisco/enic/enic_ethtool.c @@ -695,8 +695,8 @@ static void enic_get_channels(struct net_device *netdev, switch (vnic_dev_get_intr_mode(enic->vdev)) { case VNIC_DEV_INTR_MODE_MSIX: - channels->max_rx = ENIC_RQ_MAX; - channels->max_tx = ENIC_WQ_MAX; + channels->max_rx = min(enic->rq_avail, ENIC_RQ_MAX); + channels->max_tx = min(enic->wq_avail, ENIC_WQ_MAX); channels->rx_count = enic->rq_count; channels->tx_count = enic->wq_count; break; diff --git a/drivers/net/ethernet/cisco/enic/enic_res.c b/drivers/net/ethernet/cisco/enic/enic_res.c index 6910f83185c44797d769434fbe8af3105215b143..e26e43dbfa9c928f40433e0b731d0afb556a694c 100644 --- a/drivers/net/ethernet/cisco/enic/enic_res.c +++ b/drivers/net/ethernet/cisco/enic/enic_res.c @@ -187,16 +187,21 @@ void enic_free_vnic_resources(struct enic *enic) void enic_get_res_counts(struct enic *enic) { - enic->wq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_WQ); - enic->rq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_RQ); - enic->cq_count = vnic_dev_get_res_count(enic->vdev, RES_TYPE_CQ); - enic->intr_count = vnic_dev_get_res_count(enic->vdev, - RES_TYPE_INTR_CTRL); + enic->wq_avail = vnic_dev_get_res_count(enic->vdev, RES_TYPE_WQ); + enic->rq_avail = vnic_dev_get_res_count(enic->vdev, RES_TYPE_RQ); + enic->cq_avail = vnic_dev_get_res_count(enic->vdev, RES_TYPE_CQ); + enic->intr_avail = vnic_dev_get_res_count(enic->vdev, + RES_TYPE_INTR_CTRL); + + enic->wq_count = enic->wq_avail; + enic->rq_count = enic->rq_avail; + enic->cq_count = enic->cq_avail; + enic->intr_count = enic->intr_avail; dev_info(enic_get_dev(enic), "vNIC resources avail: wq %d rq %d cq %d intr %d\n", - enic->wq_count, enic->rq_count, - enic->cq_count, enic->intr_count); + enic->wq_avail, enic->rq_avail, + enic->cq_avail, enic->intr_avail); } void enic_init_vnic_resources(struct enic *enic) From patchwork Fri Oct 25 01:19:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nelson Escobar via B4 Relay X-Patchwork-Id: 13849899 X-Patchwork-Delegate: kuba@kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02610374EA; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729819264; cv=none; b=X5wnOGDdZS+l1bD6maV+p/gAgCj2vhAA4IbP53mB9V7ytywH2SM/ybtpGP/vu6ZF8nrr9jFNHZ8HSqA9wFzkeaAHjg3zJjyKeBiiY2d83QwnSWWoGSnWqT5b1AnfcB/U7GkakxwSv4FZKLB1MzYn76hink0lTHbQ7MPM/9I6mo8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729819264; c=relaxed/simple; bh=8pGJyl7/4GEPFa5wTS2h/jPW/fXnWw+Jz710+1/Uxz8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=jI7HPIYll2Fc98M28FzcGL2RFgDE1jWgG63Wyv9i3wdPrJToRLRYiHOzNz1Xv/bZcpSZ0Qdghs5/uIEa6tZ7Ou0kes0gmTJr37vDbyvxjV50dn7ybS2HIykuvv8F1bk+HjuXI9ODl7XpTrZZZd/Z/rklpIn0lQgqgFBFW6OuI/A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=V1+m5X6n; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="V1+m5X6n" Received: by smtp.kernel.org (Postfix) with ESMTPS id 9AC7FC4CEEA; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1729819263; bh=8pGJyl7/4GEPFa5wTS2h/jPW/fXnWw+Jz710+1/Uxz8=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=V1+m5X6nJOUm/bk1jt5sFxQFAF6C4aJaa39SWpyadxUxeTnxIeJ1vnslbcPxQdUbM LyIkGrcOERZK4H2YG3DOe7LaJiWWqUiZgsyLv+by0tnWnreyfOU69THv/mTGqgp/2n wnwo2pktx15YpfZtBxhRAfr8lhFOt6Z0qxGU2T3+WF70K/PU/G9Xk2/n6DWOVfEbOZ DENSJElHUm1AfPeBEcNFZG7ajIDcRbf/yP1twgK50GvKWsIpufLv5nOGd/wE+jrHWj YwkArSUCKlgQOzjoAVAF8jTK1AX9D/traahKuquo7hIM4taaDl3lMt/tApwLVl1Okm EwoKCsbkxy+Eg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92719D1039C; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) From: Nelson Escobar via B4 Relay Date: Thu, 24 Oct 2024 18:19:46 -0700 Subject: [PATCH net-next v2 4/5] enic: Allocate arrays in enic struct based on VIC config Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241024-remove_vic_resource_limits-v2-4-039b8cae5fdd@cisco.com> References: <20241024-remove_vic_resource_limits-v2-0-039b8cae5fdd@cisco.com> In-Reply-To: <20241024-remove_vic_resource_limits-v2-0-039b8cae5fdd@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1729819262; l=7119; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=IWtPdjwW96BGMVFaLSU5zTZVelFtFaglWGXDyqhKmd8=; b=8L2MxBLPmvqSoDVbiEhz+M07/tyRQLkqXqFHa7xcJg9pZ2pBvTQAtdn1XnzAUbnGYxPyo3oXf Ogffm2q02QYB9a7obwVD13Y/fDBIhx+wAwH+nWKKs0OMwAT2cugQ8Fp X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Endpoint-Received: by B4 Relay for neescoba@cisco.com/20241023 with auth_id=255 X-Original-From: Nelson Escobar Reply-To: neescoba@cisco.com X-Patchwork-Delegate: kuba@kernel.org From: Nelson Escobar Allocate wq, rq, cq, intr, and napi arrays based on the number of resources configured in the VIC. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic.h | 24 ++++----- drivers/net/ethernet/cisco/enic/enic_main.c | 84 ++++++++++++++++++++++++++--- 2 files changed, 87 insertions(+), 21 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic.h b/drivers/net/ethernet/cisco/enic/enic.h index 1f32413a8f7c690060fe385b50f7447943e72596..cfb4667953de2c578911aad138a1d392fa9d3bdc 100644 --- a/drivers/net/ethernet/cisco/enic/enic.h +++ b/drivers/net/ethernet/cisco/enic/enic.h @@ -23,10 +23,8 @@ #define ENIC_BARS_MAX 6 -#define ENIC_WQ_MAX 8 -#define ENIC_RQ_MAX 8 -#define ENIC_CQ_MAX (ENIC_WQ_MAX + ENIC_RQ_MAX) -#define ENIC_INTR_MAX (ENIC_CQ_MAX + 2) +#define ENIC_WQ_MAX 256 +#define ENIC_RQ_MAX 256 #define ENIC_WQ_NAPI_BUDGET 256 @@ -184,8 +182,8 @@ struct enic { struct work_struct reset; struct work_struct tx_hang_reset; struct work_struct change_mtu_work; - struct msix_entry msix_entry[ENIC_INTR_MAX]; - struct enic_msix_entry msix[ENIC_INTR_MAX]; + struct msix_entry *msix_entry; + struct enic_msix_entry *msix; u32 msg_enable; spinlock_t devcmd_lock; u8 mac_addr[ETH_ALEN]; @@ -204,28 +202,24 @@ struct enic { bool enic_api_busy; struct enic_port_profile *pp; - /* work queue cache line section */ - ____cacheline_aligned struct enic_wq wq[ENIC_WQ_MAX]; + struct enic_wq *wq; unsigned int wq_avail; unsigned int wq_count; u16 loop_enable; u16 loop_tag; - /* receive queue cache line section */ - ____cacheline_aligned struct enic_rq rq[ENIC_RQ_MAX]; + struct enic_rq *rq; unsigned int rq_avail; unsigned int rq_count; struct vxlan_offload vxlan; - struct napi_struct napi[ENIC_RQ_MAX + ENIC_WQ_MAX]; + struct napi_struct *napi; - /* interrupt resource cache line section */ - ____cacheline_aligned struct vnic_intr intr[ENIC_INTR_MAX]; + struct vnic_intr *intr; unsigned int intr_avail; unsigned int intr_count; u32 __iomem *legacy_pba; /* memory-mapped */ - /* completion queue cache line section */ - ____cacheline_aligned struct vnic_cq cq[ENIC_CQ_MAX]; + struct vnic_cq *cq; unsigned int cq_avail; unsigned int cq_count; struct enic_rfs_flw_tbl rfs_h; diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index eb00058b6c68ec5c1ac433b54b5bc6f3fb613777..564202e81a711a6791bef7e848627f0a439cc6f3 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -940,7 +940,7 @@ static void enic_get_stats(struct net_device *netdev, net_stats->rx_errors = stats->rx.rx_errors; net_stats->multicast = stats->rx.rx_multicast_frames_ok; - for (i = 0; i < ENIC_RQ_MAX; i++) { + for (i = 0; i < enic->rq_count; i++) { struct enic_rq_stats *rqs = &enic->rq[i].stats; if (!enic->rq[i].vrq.ctrl) @@ -1792,7 +1792,7 @@ static void enic_free_intr(struct enic *enic) free_irq(enic->pdev->irq, enic); break; case VNIC_DEV_INTR_MODE_MSIX: - for (i = 0; i < ARRAY_SIZE(enic->msix); i++) + for (i = 0; i < enic->intr_count; i++) if (enic->msix[i].requested) free_irq(enic->msix_entry[i].vector, enic->msix[i].devid); @@ -1859,7 +1859,7 @@ static int enic_request_intr(struct enic *enic) enic->msix[intr].isr = enic_isr_msix_notify; enic->msix[intr].devid = enic; - for (i = 0; i < ARRAY_SIZE(enic->msix); i++) + for (i = 0; i < enic->intr_count; i++) enic->msix[i].requested = 0; for (i = 0; i < enic->intr_count; i++) { @@ -2456,8 +2456,7 @@ static int enic_set_intr_mode(struct enic *enic) * (the last INTR is used for notifications) */ - BUG_ON(ARRAY_SIZE(enic->msix_entry) < n + m + 2); - for (i = 0; i < n + m + 2; i++) + for (i = 0; i < enic->intr_avail; i++) enic->msix_entry[i].entry = i; /* Use multiple RQs if RSS is enabled @@ -2674,6 +2673,71 @@ static const struct netdev_stat_ops enic_netdev_stat_ops = { .get_base_stats = enic_get_base_stats, }; +static void enic_free_enic_resources(struct enic *enic) +{ + kfree(enic->wq); + enic->wq = NULL; + + kfree(enic->rq); + enic->rq = NULL; + + kfree(enic->cq); + enic->cq = NULL; + + kfree(enic->napi); + enic->napi = NULL; + + kfree(enic->msix_entry); + enic->msix_entry = NULL; + + kfree(enic->msix); + enic->msix = NULL; + + kfree(enic->intr); + enic->intr = NULL; +} + +static int enic_alloc_enic_resources(struct enic *enic) +{ + enic->wq = kcalloc(enic->wq_avail, sizeof(struct enic_wq), GFP_KERNEL); + if (!enic->wq) + goto free_queues; + + enic->rq = kcalloc(enic->rq_avail, sizeof(struct enic_rq), GFP_KERNEL); + if (!enic->rq) + goto free_queues; + + enic->cq = kcalloc(enic->cq_avail, sizeof(struct vnic_cq), GFP_KERNEL); + if (!enic->cq) + goto free_queues; + + enic->napi = kcalloc(enic->wq_avail + enic->rq_avail, + sizeof(struct napi_struct), GFP_KERNEL); + if (!enic->napi) + goto free_queues; + + enic->msix_entry = kcalloc(enic->intr_avail, sizeof(struct msix_entry), + GFP_KERNEL); + if (!enic->msix_entry) + goto free_queues; + + enic->msix = kcalloc(enic->intr_avail, sizeof(struct enic_msix_entry), + GFP_KERNEL); + if (!enic->msix) + goto free_queues; + + enic->intr = kcalloc(enic->intr_avail, sizeof(struct vnic_intr), + GFP_KERNEL); + if (!enic->intr) + goto free_queues; + + return 0; + +free_queues: + enic_free_enic_resources(enic); + return -ENOMEM; +} + static void enic_dev_deinit(struct enic *enic) { unsigned int i; @@ -2691,6 +2755,7 @@ static void enic_dev_deinit(struct enic *enic) enic_free_vnic_resources(enic); enic_clear_intr_mode(enic); enic_free_affinity_hint(enic); + enic_free_enic_resources(enic); } static void enic_kdump_kernel_config(struct enic *enic) @@ -2734,6 +2799,12 @@ static int enic_dev_init(struct enic *enic) enic_get_res_counts(enic); + err = enic_alloc_enic_resources(enic); + if (err) { + dev_err(dev, "Failed to allocate enic resources\n"); + return err; + } + /* modify resource count if we are in kdump_kernel */ enic_kdump_kernel_config(enic); @@ -2746,7 +2817,7 @@ static int enic_dev_init(struct enic *enic) if (err) { dev_err(dev, "Failed to set intr mode based on resource " "counts and system capabilities, aborting\n"); - return err; + goto err_out_free_vnic_resources; } /* Allocate and configure vNIC resources @@ -2788,6 +2859,7 @@ static int enic_dev_init(struct enic *enic) enic_free_affinity_hint(enic); enic_clear_intr_mode(enic); enic_free_vnic_resources(enic); + enic_free_enic_resources(enic); return err; } From patchwork Fri Oct 25 01:19:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nelson Escobar via B4 Relay X-Patchwork-Id: 13849900 X-Patchwork-Delegate: kuba@kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0570838DC7; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729819264; cv=none; b=iWhET678/w1le9UzHT2D5QFeCBQS+NJ8EEzbtbtn8EJD3MEJhdr2G3mT5Le4bOOTmFEh7JhpiVwIAuUC3oEgbn4GRdOH9IrOpNN9OkRzccKWscoOIqzwgb956h+Imh3zHi33AYWAstEQsG3f5ekHA4ipK0BexmvGY0JEDIAyhAk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729819264; c=relaxed/simple; bh=yrqbiZFhn51WEomiKGvcXRetuIKBXye22S0xMCc0lk8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=OPOGenGz4FdhEPB4uNFiiU1aegw3e57Z3e/jc7L86k89EsVrS46foT7ggRSLxS+kmetWv2/sj7YZql/cRFKCKetvApcdJGKz/EKKwQVteewP7oqn/M5V5C7wOQDeVfB+ML4ISASSGul+wX1zYGCo3XaUHddPxlEpcU2irqUOkHM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gZWmymXM; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gZWmymXM" Received: by smtp.kernel.org (Postfix) with ESMTPS id AC4EAC4CEEE; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1729819263; bh=yrqbiZFhn51WEomiKGvcXRetuIKBXye22S0xMCc0lk8=; h=From:Date:Subject:References:In-Reply-To:To:Cc:Reply-To:From; b=gZWmymXMSOEezRJ3VCSJ5GujLLHsG8QGBXR2s9v0D36kt3mNNIdsKOiiXajdA45P2 2A+0/7w7/1gLMPjTZvHEQJlK4+WCjc8CaIg1zaoMgcF5he8BLI7tRt9iBSZ6S7Dp4a oRQ/50gYTiy6rF4w36d2rUT6qCrl7Yg1+ydMhkqEd0+xo9B4j/Clqa7V0qbnefXnJM 7xOtBkEAYIPVrEeKdunGDv0CzvrEBl4EgGjXWTCoCo7brVjExZPc4vzK6xfV+vifK9 M9BucIao793uZFeAzNBrblwA4qoCb2mWTWTDV/ZFRklXpKPyY6xBGHA8trOp4SK3km HD06jMv2wfIOg== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3333D10399; Fri, 25 Oct 2024 01:21:03 +0000 (UTC) From: Nelson Escobar via B4 Relay Date: Thu, 24 Oct 2024 18:19:47 -0700 Subject: [PATCH net-next v2 5/5] enic: Adjust used MSI-X wq/rq/cq/interrupt resources in a more robust way Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241024-remove_vic_resource_limits-v2-5-039b8cae5fdd@cisco.com> References: <20241024-remove_vic_resource_limits-v2-0-039b8cae5fdd@cisco.com> In-Reply-To: <20241024-remove_vic_resource_limits-v2-0-039b8cae5fdd@cisco.com> To: John Daley , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Christian Benvenuti , Satish Kharat , Andrew Lunn , "David S. Miller" Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Nelson Escobar , Simon Horman X-Mailer: b4 0.14.2 X-Developer-Signature: v=1; a=ed25519-sha256; t=1729819262; l=8413; i=neescoba@cisco.com; s=20241023; h=from:subject:message-id; bh=GXz0cT0Pl6EsGEqJO9PYLd3Kcxc4wzzJoA1tyJnCrJs=; b=QxcOUUMNugJ/vWYyjm3BtWGcEwdjRnFb0lASNna1zPMDnDrG6xJP9t8l3AmMX2/elHC9Fte40 LTZF12OyF+oCSQIkIRNHiZRrBeAt6rcv7IbFLmwPExbYnIQt6rFOmxg X-Developer-Key: i=neescoba@cisco.com; a=ed25519; pk=bLqWB7VU0KFoVybF4LVB4c2Redvnplt7+5zLHf4KwZM= X-Endpoint-Received: by B4 Relay for neescoba@cisco.com/20241023 with auth_id=255 X-Original-From: Nelson Escobar Reply-To: neescoba@cisco.com X-Patchwork-Delegate: kuba@kernel.org From: Nelson Escobar Instead of erroring out on probe if the resources are not configured exactly right in hardware, try to make due with the resources we do have. To accomplish this do the following: - Make enic_set_intr_mode() only set up interrupt related stuff. - Move resource adjustment out of enic_set_intr_mode() into its own function, and basing the resources used on the most constrained resource. - Move the kdump resources limitations into the new function too. Co-developed-by: John Daley Signed-off-by: John Daley Co-developed-by: Satish Kharat Signed-off-by: Satish Kharat Signed-off-by: Nelson Escobar Reviewed-by: Simon Horman --- drivers/net/ethernet/cisco/enic/enic_main.c | 197 ++++++++++++++-------------- 1 file changed, 96 insertions(+), 101 deletions(-) diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c index 564202e81a711a6791bef7e848627f0a439cc6f3..fb979c19a1785ace633c51e47de391cf4a4640b3 100644 --- a/drivers/net/ethernet/cisco/enic/enic_main.c +++ b/drivers/net/ethernet/cisco/enic/enic_main.c @@ -2440,113 +2440,120 @@ static void enic_tx_hang_reset(struct work_struct *work) rtnl_unlock(); } -static int enic_set_intr_mode(struct enic *enic) +static int enic_adjust_resources(struct enic *enic) { - unsigned int n = min_t(unsigned int, enic->rq_count, ENIC_RQ_MAX); - unsigned int m = min_t(unsigned int, enic->wq_count, ENIC_WQ_MAX); - unsigned int i; + unsigned int max_queues; + unsigned int rq_avail; + unsigned int wq_avail; - /* Set interrupt mode (INTx, MSI, MSI-X) depending - * on system capabilities. - * - * Try MSI-X first - * - * We need n RQs, m WQs, n+m CQs, and n+m+2 INTRs - * (the second to last INTR is used for WQ/RQ errors) - * (the last INTR is used for notifications) - */ + if (enic->rq_avail < 1 || enic->wq_avail < 1 || enic->cq_avail < 2) { + dev_err(enic_get_dev(enic), + "Not enough resources available rq: %d wq: %d cq: %d\n", + enic->rq_avail, enic->wq_avail, + enic->cq_avail); + return -ENOSPC; + } - for (i = 0; i < enic->intr_avail; i++) - enic->msix_entry[i].entry = i; + if (is_kdump_kernel()) { + dev_info(enic_get_dev(enic), "Running from within kdump kernel. Using minimal resources\n"); + enic->rq_avail = 1; + enic->wq_avail = 1; + enic->config.rq_desc_count = ENIC_MIN_RQ_DESCS; + enic->config.wq_desc_count = ENIC_MIN_WQ_DESCS; + enic->config.mtu = min_t(u16, 1500, enic->config.mtu); + } - /* Use multiple RQs if RSS is enabled - */ + /* if RSS isn't set, then we can only use one RQ */ + if (!ENIC_SETTING(enic, RSS)) + enic->rq_avail = 1; - if (ENIC_SETTING(enic, RSS) && - enic->config.intr_mode < 1 && - enic->rq_count >= n && - enic->wq_count >= m && - enic->cq_count >= n + m && - enic->intr_count >= n + m + 2) { + switch (vnic_dev_get_intr_mode(enic->vdev)) { + case VNIC_DEV_INTR_MODE_INTX: + case VNIC_DEV_INTR_MODE_MSI: + enic->rq_count = 1; + enic->wq_count = 1; + enic->cq_count = 2; + enic->intr_count = enic->intr_avail; + break; + case VNIC_DEV_INTR_MODE_MSIX: + /* Adjust the number of wqs/rqs/cqs/interrupts that will be + * used based on which resource is the most constrained + */ + wq_avail = min(enic->wq_avail, ENIC_WQ_MAX); + rq_avail = min(enic->rq_avail, ENIC_RQ_MAX); + max_queues = min(enic->cq_avail, + enic->intr_avail - ENIC_MSIX_RESERVED_INTR); + if (wq_avail + rq_avail <= max_queues) { + /* we have enough cq and interrupt resources to cover + * the number of wqs and rqs + */ + enic->rq_count = rq_avail; + enic->wq_count = wq_avail; + } else { + /* recalculate wq/rq count */ + if (rq_avail < wq_avail) { + enic->rq_count = min(rq_avail, max_queues / 2); + enic->wq_count = max_queues - enic->rq_count; + } else { + enic->wq_count = min(wq_avail, max_queues / 2); + enic->rq_count = max_queues - enic->wq_count; + } + } + enic->cq_count = enic->rq_count + enic->wq_count; + enic->intr_count = enic->cq_count + ENIC_MSIX_RESERVED_INTR; - if (pci_enable_msix_range(enic->pdev, enic->msix_entry, - n + m + 2, n + m + 2) > 0) { + break; + default: + dev_err(enic_get_dev(enic), "Unknown interrupt mode\n"); + return -EINVAL; + } - enic->rq_count = n; - enic->wq_count = m; - enic->cq_count = n + m; - enic->intr_count = n + m + 2; + return 0; +} - vnic_dev_set_intr_mode(enic->vdev, - VNIC_DEV_INTR_MODE_MSIX); +static int enic_set_intr_mode(struct enic *enic) +{ + unsigned int i; + int num_intr; - return 0; - } - } + /* Set interrupt mode (INTx, MSI, MSI-X) depending + * on system capabilities. + * + * Try MSI-X first + */ if (enic->config.intr_mode < 1 && - enic->rq_count >= 1 && - enic->wq_count >= m && - enic->cq_count >= 1 + m && - enic->intr_count >= 1 + m + 2) { - if (pci_enable_msix_range(enic->pdev, enic->msix_entry, - 1 + m + 2, 1 + m + 2) > 0) { - - enic->rq_count = 1; - enic->wq_count = m; - enic->cq_count = 1 + m; - enic->intr_count = 1 + m + 2; - + enic->intr_avail >= ENIC_MSIX_MIN_INTR) { + for (i = 0; i < enic->intr_avail; i++) + enic->msix_entry[i].entry = i; + + num_intr = pci_enable_msix_range(enic->pdev, enic->msix_entry, + ENIC_MSIX_MIN_INTR, + enic->intr_avail); + if (num_intr > 0) { vnic_dev_set_intr_mode(enic->vdev, - VNIC_DEV_INTR_MODE_MSIX); - + VNIC_DEV_INTR_MODE_MSIX); + enic->intr_avail = num_intr; return 0; } } - /* Next try MSI - * - * We need 1 RQ, 1 WQ, 2 CQs, and 1 INTR - */ + /* Next try MSI */ if (enic->config.intr_mode < 2 && - enic->rq_count >= 1 && - enic->wq_count >= 1 && - enic->cq_count >= 2 && - enic->intr_count >= 1 && + enic->intr_avail >= 1 && !pci_enable_msi(enic->pdev)) { - - enic->rq_count = 1; - enic->wq_count = 1; - enic->cq_count = 2; - enic->intr_count = 1; - + enic->intr_avail = 1; vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_MSI); - return 0; } - /* Next try INTx - * - * We need 1 RQ, 1 WQ, 2 CQs, and 3 INTRs - * (the first INTR is used for WQ/RQ) - * (the second INTR is used for WQ/RQ errors) - * (the last INTR is used for notifications) - */ + /* Next try INTx */ if (enic->config.intr_mode < 3 && - enic->rq_count >= 1 && - enic->wq_count >= 1 && - enic->cq_count >= 2 && - enic->intr_count >= 3) { - - enic->rq_count = 1; - enic->wq_count = 1; - enic->cq_count = 2; - enic->intr_count = 3; - + enic->intr_avail >= 3) { + enic->intr_avail = 3; vnic_dev_set_intr_mode(enic->vdev, VNIC_DEV_INTR_MODE_INTX); - return 0; } @@ -2758,18 +2765,6 @@ static void enic_dev_deinit(struct enic *enic) enic_free_enic_resources(enic); } -static void enic_kdump_kernel_config(struct enic *enic) -{ - if (is_kdump_kernel()) { - dev_info(enic_get_dev(enic), "Running from within kdump kernel. Using minimal resources\n"); - enic->rq_count = 1; - enic->wq_count = 1; - enic->config.rq_desc_count = ENIC_MIN_RQ_DESCS; - enic->config.wq_desc_count = ENIC_MIN_WQ_DESCS; - enic->config.mtu = min_t(u16, 1500, enic->config.mtu); - } -} - static int enic_dev_init(struct enic *enic) { struct device *dev = enic_get_dev(enic); @@ -2805,14 +2800,7 @@ static int enic_dev_init(struct enic *enic) return err; } - /* modify resource count if we are in kdump_kernel - */ - enic_kdump_kernel_config(enic); - - /* Set interrupt mode based on resource counts and system - * capabilities - */ - + /* Set interrupt mode based on system capabilities */ err = enic_set_intr_mode(enic); if (err) { dev_err(dev, "Failed to set intr mode based on resource " @@ -2820,6 +2808,13 @@ static int enic_dev_init(struct enic *enic) goto err_out_free_vnic_resources; } + /* Adjust resource counts based on most constrained resources */ + err = enic_adjust_resources(enic); + if (err) { + dev_err(dev, "Failed to adjust resources\n"); + goto err_out_free_vnic_resources; + } + /* Allocate and configure vNIC resources */