From patchwork Sat Jul 13 23:43:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Chan X-Patchwork-Id: 13732558 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9103016D9AC for ; Sat, 13 Jul 2024 23:44:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720914257; cv=none; b=P9EH+3E2R6JKGUiZAIhUd8ghsX+P93Fr4NaITLKOg5YI5j2D5jTz4qdNgarqG6MZPoz4ONlm4niUG2nZXeI8KBlFvwkwuYnGvSExwlPzOgaK7fvR2DtEDEQXzTYxDItKq9LCP+9V4Rpt1w/BuIoX1GfNATZl7RbdU15W2rsNSL0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1720914257; c=relaxed/simple; bh=FQSlL4ckG/Uw7ciNvLyJOMsaN8n1iL+y/TfQiPByeVg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=uDXQHtpnF54un+UdeTNTL34kWCK0a7RpZhoC0K+BykrJ9ginJqAzh7HTtxDdfRLnUiLmxdzY6vyW0fKJ3uq+HESRxvsAs/mR9fdAfSBU9pDKLbpTBn+xOhpQTwRPrnUApHW01GHzk8d5Q0Zvof1K+tuU/JTctoUljrtcl8eyRe0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=g05XVfzp; arc=none smtp.client-ip=209.85.222.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="g05XVfzp" Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-79efd2151d9so213329485a.0 for ; Sat, 13 Jul 2024 16:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1720914254; x=1721519054; darn=vger.kernel.org; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=lsB48AihHIi6XV9k58M2cfdXObYOIPb4O8A08KshAMc=; b=g05XVfzpSDvZ5prLytBCoXobrYrHV/8Fo1jQMaHLB3mL9E/Cm8HkPbjJ9HCbFEn70L 4cf7JIr7GafdiLRdGYBnnzmbc6tMKawXP4VY2FQKnfMkgHnoeKiLLgKXQHR/nMWBcc3k 23szg18K0YTwydDUJkQJhJhXfIKk/XfqJ/nT8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720914254; x=1721519054; h=mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lsB48AihHIi6XV9k58M2cfdXObYOIPb4O8A08KshAMc=; b=Fo+lcRDdiE09i84etFR3X+VICXUbkXcNwHL/2u/Bkr0NundCMiPcipblxEnHIsHoG4 aFz7Ha8oyly8TfJR1rKl5E6cz97Ugnawaxm11xqJ8dg+w4EuIJFK1NRbzsRIUJmeXUuy UX6XyT7uSKEPD3ppX53vwqifUG6DJQySZEfxHSxMkpv0/zUw15jLMpztbfIH58qGED+l 2SPpKh0mPsZgx9wAJWF+0HYKbKFMdcILjI0QX7bTS0VZ7ltWGNtKDg9/uPbVsWWYI4bp YegDtJ3r98yTvFM8PKmHXaF54LhgRy5sG87qCFct6Xlpdo5UvtklqdAYGrtyJPT6mbYf 9kdQ== X-Gm-Message-State: AOJu0YziSCpTOiUYobFpX5MwXtRPVgXcpbbAD4SMgp5xDLP6dbvF9pQ3 WSpBn53re6DDRo8FKyGepADw3LZEt2buR2HlPSI00hwAlPolYIr2h6QHEk/YiQ== X-Google-Smtp-Source: AGHT+IHG/GTGEXJTXCln37EKeljXZ1HWDjO2WcSk2EG8mgGrNkloLd5QyStXcUgBzKlK8j4TUJAW8A== X-Received: by 2002:a05:620a:1a0c:b0:79d:6bba:4a66 with SMTP id af79cd13be357-79f19c08427mr2279038085a.66.1720914254579; Sat, 13 Jul 2024 16:44:14 -0700 (PDT) Received: from lvnvda5233.lvn.broadcom.net ([192.19.161.250]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a160bbe6f7sm78124585a.37.2024.07.13.16.44.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 13 Jul 2024 16:44:14 -0700 (PDT) From: Michael Chan To: davem@davemloft.net Cc: netdev@vger.kernel.org, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, pavan.chebbi@broadcom.com, andrew.gospodarek@broadcom.com, Hongguang Gao , Somnath Kotur Subject: [PATCH net-next 9/9] bnxt_en: Support dynamic MSIX Date: Sat, 13 Jul 2024 16:43:39 -0700 Message-ID: <20240713234339.70293-10-michael.chan@broadcom.com> X-Mailer: git-send-email 2.43.4 In-Reply-To: <20240713234339.70293-1-michael.chan@broadcom.com> References: <20240713234339.70293-1-michael.chan@broadcom.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org A range of MSIX vectors are allocated at initializtion for the number needed for RocE and L2. During run-time, if the user increases or decreases the number of L2 rings, all the MSIX vectors have to be freed and a new range has to be allocated. This is not optimal and causes disruptions to RoCE traffic every time there is a change in L2 MSIX. If the system supports dynamic MSIX allocations, use dynamic allocation to add new L2 MSIX vectors or free unneeded L2 MSIX vectors. RoCE traffic is not affected using this scheme. Reviewed-by: Hongguang Gao Reviewed-by: Somnath Kotur Signed-off-by: Michael Chan Reviewed-by: Simon Horman --- drivers/net/ethernet/broadcom/bnxt/bnxt.c | 57 +++++++++++++++++++++-- 1 file changed, 54 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 7483ea246c9d..c987a9dd969c 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -10597,6 +10597,43 @@ static void bnxt_setup_msix(struct bnxt *bp) static int bnxt_init_int_mode(struct bnxt *bp); +static int bnxt_add_msix(struct bnxt *bp, int total) +{ + int i; + + if (bp->total_irqs >= total) + return total; + + for (i = bp->total_irqs; i < total; i++) { + struct msi_map map; + + map = pci_msix_alloc_irq_at(bp->pdev, i, NULL); + if (map.index < 0) + break; + bp->irq_tbl[i].vector = map.virq; + bp->total_irqs++; + } + return bp->total_irqs; +} + +static int bnxt_trim_msix(struct bnxt *bp, int total) +{ + int i; + + if (bp->total_irqs <= total) + return total; + + for (i = bp->total_irqs; i > total; i--) { + struct msi_map map; + + map.index = i - 1; + map.virq = bp->irq_tbl[i - 1].vector; + pci_msix_free_irq(bp->pdev, map); + bp->total_irqs--; + } + return bp->total_irqs; +} + static int bnxt_setup_int_mode(struct bnxt *bp) { int rc; @@ -10763,6 +10800,7 @@ static void bnxt_clear_int_mode(struct bnxt *bp) int bnxt_reserve_rings(struct bnxt *bp, bool irq_re_init) { bool irq_cleared = false; + bool irq_change = false; int tcs = bp->num_tc; int irqs_required; int rc; @@ -10781,15 +10819,28 @@ int bnxt_reserve_rings(struct bnxt *bp, bool irq_re_init) } if (irq_re_init && BNXT_NEW_RM(bp) && irqs_required != bp->total_irqs) { - bnxt_ulp_irq_stop(bp); - bnxt_clear_int_mode(bp); - irq_cleared = true; + irq_change = true; + if (!pci_msix_can_alloc_dyn(bp->pdev)) { + bnxt_ulp_irq_stop(bp); + bnxt_clear_int_mode(bp); + irq_cleared = true; + } } rc = __bnxt_reserve_rings(bp); if (irq_cleared) { if (!rc) rc = bnxt_init_int_mode(bp); bnxt_ulp_irq_restart(bp, rc); + } else if (irq_change && !rc) { + int total; + + if (irqs_required > bp->total_irqs) + total = bnxt_add_msix(bp, irqs_required); + else + total = bnxt_trim_msix(bp, irqs_required); + + if (total != irqs_required) + rc = -ENOSPC; } if (rc) { netdev_err(bp->dev, "ring reservation/IRQ init failure rc: %d\n", rc);