From patchwork Thu Jun 20 15:23:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Child X-Patchwork-Id: 13705718 X-Patchwork-Delegate: kuba@kernel.org Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3175C1AD405 for ; Thu, 20 Jun 2024 15:25:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718897147; cv=none; b=IFq21ngk7tAvZBxCxuX6AcAt8zZnjt9dkvQS8X+Mv0TBSGKYQSmcBC/4wFdj0bMry1rWFWA+JYVWQ/h3kpQEtJLRMNQF+EU5qJxE5TX3r9ZaF+6pLkaZKTW0QMioK8taHPKfhBMbz2IgHcAB1jXIHztK18r4b8mwUIepQ5X8w2A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718897147; c=relaxed/simple; bh=7EM+evrSP3em0pjD75RU9N96Bg6+EIvYthfmehl3+5M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=i/h9F9B8jbZfeF1ArzPPebv6WBYOJrhQZX6PJ/JOEnSDh/cI5l23yiZxvUY1dpUPXLrgLF1kdDEcNudoSIA+Mm4i2KY+8uxDVkjSb8/kZmUrNu9uw/aoQV0zx6xYs1YlVCtx2iSpfWbYLLeWzPI6g+W2AH3P4xMENFkbAHsVAdg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=COfZLgJo; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="COfZLgJo" Received: from pps.filterd (m0353723.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 45KChqX0028752 for ; Thu, 20 Jun 2024 15:25:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=pp1; bh=GbGDfA8sXQq0b F1q2omxu881VZBeEyumrNz8NBbeRy8=; b=COfZLgJoEhZtCyoM2SWtZQyx0Y44n xaRx/8OarWKTDCntHvXaDdjLLb2j9nGvT9SVJe89B7S652GvIR0lFHkkzzCbJDF+ N5VBPgnaqMEwoENfgZgtQZdC1HZx9CZHBODgFx1MFPKB+xa6eLcTDG7WcqW0eIzG VVaPLAY2YMSW94/DgtGlp8yhYysLU1QS8uv5CCAlEZ6z1aaDP65+AzPHxFdApxWo Jxv3G0HpUr079SUc2I7nGIr9NKCmBIgZe+1s1ZcF81AG+mcgW4+uydWp/ehCm6D9 b7e7ccCuK0JKnuZZwSBEsLnTwzG+hEsqmXwm8Ro+yjTcuYOv5+yU09rRQ== Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3yvg2s95ts-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jun 2024 15:25:41 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 45KEacGh009422 for ; Thu, 20 Jun 2024 15:23:37 GMT Received: from smtprelay03.dal12v.mail.ibm.com ([172.16.1.5]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 3ysqgn72q9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jun 2024 15:23:37 +0000 Received: from smtpav01.wdc07v.mail.ibm.com (smtpav01.wdc07v.mail.ibm.com [10.39.53.228]) by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 45KFNVBr27460128 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 20 Jun 2024 15:23:33 GMT Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 525275806F; Thu, 20 Jun 2024 15:23:31 +0000 (GMT) Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0321F5806E; Thu, 20 Jun 2024 15:23:31 +0000 (GMT) Received: from tinkpad.ibmuc.com (unknown [9.61.39.25]) by smtpav01.wdc07v.mail.ibm.com (Postfix) with ESMTP; Thu, 20 Jun 2024 15:23:30 +0000 (GMT) From: Nick Child To: netdev@vger.kernel.org Cc: nick.child@ibm.com, haren@linux.ibm.com, ricklind@us.ibm.com, Nick Child Subject: [PATCH net 1/2] ibmvnic: Add tx check to prevent skb leak Date: Thu, 20 Jun 2024 10:23:11 -0500 Message-Id: <20240620152312.1032323-2-nnac123@linux.ibm.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240620152312.1032323-1-nnac123@linux.ibm.com> References: <20240620152312.1032323-1-nnac123@linux.ibm.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: LexgGkMGmyJDJYB5m3DOps8dJ5NcNT0z X-Proofpoint-ORIG-GUID: LexgGkMGmyJDJYB5m3DOps8dJ5NcNT0z X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-06-20_07,2024-06-20_04,2024-05-17_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 mlxscore=0 clxscore=1015 impostorscore=0 lowpriorityscore=0 malwarescore=0 adultscore=0 suspectscore=0 bulkscore=0 phishscore=0 priorityscore=1501 mlxlogscore=518 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2405170001 definitions=main-2406200111 X-Patchwork-Delegate: kuba@kernel.org Below is a summary of how the driver stores a reference to an skb during transmit: tx_buff[free_map[consumer_index]]->skb = new_skb; free_map[consumer_index] = IBMVNIC_INVALID_MAP; consumer_index ++; Where variable data looks like this: free_map == [4, IBMVNIC_INVALID_MAP, IBMVNIC_INVALID_MAP, 0, 3] consumer_index^ tx_buff == [skb=null, skb=, skb=, skb=null, skb=null] The driver has checks to ensure that free_map[consumer_index] pointed to a valid index but there was no check to ensure that this index pointed to an unused/null skb address. So, if, by some chance, our free_map and tx_buff lists become out of sync then we were previously risking an skb memory leak. This could then cause tcp congestion control to stop sending packets, eventually leading to ETIMEDOUT. Therefore, add a conditional to ensure that the skb address is null. If not then warn the user (because this is still a bug that should be patched) and free the old pointer to prevent memleak/tcp problems. Signed-off-by: Nick Child --- drivers/net/ethernet/ibm/ibmvnic.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c index 5e9a93bdb518..887d92a88403 100644 --- a/drivers/net/ethernet/ibm/ibmvnic.c +++ b/drivers/net/ethernet/ibm/ibmvnic.c @@ -2482,6 +2482,18 @@ static netdev_tx_t ibmvnic_xmit(struct sk_buff *skb, struct net_device *netdev) (tx_pool->consumer_index + 1) % tx_pool->num_buffers; tx_buff = &tx_pool->tx_buff[bufidx]; + + /* Sanity checks on our free map to make sure it points to an index + * that is not being occupied by another skb. If skb memory is + * not freed then we see congestion control kick in and halt tx. + */ + if (unlikely(tx_buff->skb)) { + dev_warn_ratelimited(dev, "TX free map points to untracked skb (%s %d idx=%d)\n", + skb_is_gso(skb) ? "tso_pool" : "tx_pool", + queue_num, bufidx); + dev_kfree_skb_any(tx_buff->skb); + } + tx_buff->skb = skb; tx_buff->index = bufidx; tx_buff->pool_index = queue_num; From patchwork Thu Jun 20 15:23:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Child X-Patchwork-Id: 13705721 X-Patchwork-Delegate: kuba@kernel.org Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D0821AED46 for ; Thu, 20 Jun 2024 15:27:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718897259; cv=none; b=Z5AFhXry7Xgs0L9/PfBmZZ49GGcw75nj1t+fsPBNeVqxI7gosedB/aI4X+Xu1lOn8riXzmjsJzBpLc2G4nW3GWOKuYIf4je9l6nX74sPPMDncyf+zGSrD+JCUiW06dEli0XvbqqHrIDHbEKJ95y+qfa8fsRipIKqWMNn80yzguw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718897259; c=relaxed/simple; bh=MpFakcow3PkSbqhHhn3NzhQLiCBT20uhaAyMOVuV/S8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Fuu/MHfVsSa1qppbxPwE3kr/EKLZVUOl5n0iKo/0XW+f1E+qJUVg51E8UtG3Fpsd/Em2JYGtebDdsFElzI/obHERhfd40PKgcEgsU8FzrSWQ+GPEbIWyXRaQI1ieVvCLzc0cBAtY4AECYpcZz9qDq6zFygwWPaVNHbAvyk+gFv8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=mVmfqjoF; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="mVmfqjoF" Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 45KFQSke012331 for ; Thu, 20 Jun 2024 15:27:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from :to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=pp1; bh=M3x/fNov4Q5Al NCFcjoPBVtSyft+Tgz0KJ0g5bElVlI=; b=mVmfqjoF1rj0qtNDGNV4Yq6m4zKTy HokM1jLF56aXbEdhFvmEW+Aey9norDWZK5HeTU0fVwMx5iCvCH/WCeiKFVAisJ1i VvotmOcGf63bJQmNX7Aho0Lq3AVbYtUw20bHdkTiK0f3wxHcHt+/fqA6or/z9Yxx tiC0gGtuZ2Ci1F3cqKsxUl9jp+ZDT0LnIeHhfCDHqLiUtvC1xUrj9RffCU6Bu8tw xLxttZmXSh6ANKlmBMy9i4rvvZXug4yFrwip8Vicuh1EuOum3PThst7p6wyZtMeD nJ4MdDJ76zW42aXHDb04gTzxoEXHQC8t35RsgYrhbD2/E5OEa8n8k6IBg== Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3yvpneg30c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jun 2024 15:27:36 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 45KEacGj009422 for ; Thu, 20 Jun 2024 15:23:40 GMT Received: from smtprelay07.dal12v.mail.ibm.com ([172.16.1.9]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 3ysqgn72qt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 20 Jun 2024 15:23:40 +0000 Received: from smtpav01.wdc07v.mail.ibm.com (smtpav01.wdc07v.mail.ibm.com [10.39.53.228]) by smtprelay07.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 45KFNZfk42664534 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 20 Jun 2024 15:23:37 GMT Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 66D9A58063; Thu, 20 Jun 2024 15:23:35 +0000 (GMT) Received: from smtpav01.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D17BF58068; Thu, 20 Jun 2024 15:23:34 +0000 (GMT) Received: from tinkpad.ibmuc.com (unknown [9.61.39.25]) by smtpav01.wdc07v.mail.ibm.com (Postfix) with ESMTP; Thu, 20 Jun 2024 15:23:34 +0000 (GMT) From: Nick Child To: netdev@vger.kernel.org Cc: nick.child@ibm.com, haren@linux.ibm.com, ricklind@us.ibm.com, Nick Child Subject: [PATCH net 2/2] ibmvnic: Free any outstanding tx skbs during scrq reset Date: Thu, 20 Jun 2024 10:23:12 -0500 Message-Id: <20240620152312.1032323-3-nnac123@linux.ibm.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20240620152312.1032323-1-nnac123@linux.ibm.com> References: <20240620152312.1032323-1-nnac123@linux.ibm.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: JDT0LstHQHk4lfDqpuxGePwl_I9MZyQh X-Proofpoint-ORIG-GUID: JDT0LstHQHk4lfDqpuxGePwl_I9MZyQh X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-06-20_07,2024-06-20_04,2024-05-17_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxlogscore=999 spamscore=0 mlxscore=0 phishscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 suspectscore=0 malwarescore=0 bulkscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2405170001 definitions=main-2406200111 X-Patchwork-Delegate: kuba@kernel.org There are 2 types of outstanding tx skb's: Type 1: Packets that are sitting in the drivers ind_buff that are waiting to be batch sent to the NIC. During a device reset, these are freed with a call to ibmvnic_tx_scrq_clean_buffer() Type 2: Packets that have been sent to the NIC and are awaiting a TX completion IRQ. These are free'd during a reset with a call to clean_tx_pools() During any reset which requires us to free the tx irq, ensure that the Type 2 skb references are freed. Since the irq is released, it is impossible for the NIC to inform of any completions. Furthermore, later in the reset process is a call to init_tx_pools() which marks every entry in the tx pool as free (ie not outstanding). So if the driver is to make a call to init_tx_pools(), it must first be sure that the tx pool is empty of skb references. This issue was discovered by observing the following in the logs during EEH testing: TX free map points to untracked skb (tso_pool 0 idx=4) TX free map points to untracked skb (tso_pool 0 idx=5) TX free map points to untracked skb (tso_pool 1 idx=36) Fixes: 65d6470d139a ("ibmvnic: clean pending indirect buffs during reset") Signed-off-by: Nick Child --- drivers/net/ethernet/ibm/ibmvnic.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c index 887d92a88403..23ebeb143987 100644 --- a/drivers/net/ethernet/ibm/ibmvnic.c +++ b/drivers/net/ethernet/ibm/ibmvnic.c @@ -4073,6 +4073,12 @@ static void release_sub_crqs(struct ibmvnic_adapter *adapter, bool do_h_free) adapter->num_active_tx_scrqs = 0; } + /* Clean any remaining outstanding SKBs + * we freed the irq so we won't be hearing + * from them + */ + clean_tx_pools(adapter); + if (adapter->rx_scrq) { for (i = 0; i < adapter->num_active_rx_scrqs; i++) { if (!adapter->rx_scrq[i])