From patchwork Fri Mar 7 03:19:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Hangbin Liu X-Patchwork-Id: 14005706 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24ED219049A; Fri, 7 Mar 2025 03:19:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741317563; cv=none; b=HwheRDVe+rz9UeaV3bX0ZWkJn7+JYqesK1jWwTdLsgboC2t5vvO0PoNBPPYAX1eJD6ST3XElK/FPbpmr8TdtqqE4SEJXbHM7fHWo0v/QG88xHymOVIRNuuY8d8Nv8J4UmXVv+LMjhciAb9P83duLSilJ6kdXuA1z7NlV23PNFcs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741317563; c=relaxed/simple; bh=9jcnBiMIEjCPY7QK6DHqfZ/zJ2duSB/1vtOU+cYGf5w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JpUZSSHcr4P4SmQXdYs+dpT2RK/ofDWOhdRHDhlB2zRnConxNB7mG45SZpBMRqaPWZtnSSCd4CTLBX4HFASe8fwznNzdpG5aRHM+iG9REopP8doevPp+Nnh7bbTnwXpK+fUztRx3rwRuJqAAFOtGNWLiYYl4akeAnmUvEIlj77o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Y31wT/hV; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Y31wT/hV" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2ff80290e44so300109a91.0; Thu, 06 Mar 2025 19:19:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741317561; x=1741922361; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VVXPe/IPw1lxQV8lI9bp/vRnBlqZTCq+2dNe5J1DRJc=; b=Y31wT/hVxxTRblb0OLostbJbf+3VbLhOy1AKJfjHLnFvqqvmAw8bLf6cnqy0p2kX8J NfSje4azOp32vLRWX7Ev0eD99RJRUz6cMvX1gqzZR6w6ThZfHHRjNc9zqGAAsG5VL/48 nNceWhy/EDCLTAGE5HK9gGl/6gw8LVP8SCIAAt9nERmn+rBZw30VBxTjLKv7fd0uv1i0 ZAzpFdO3i6fyCY6N7BkvJ5Obi/OmlprfzpWSVjZ2A7dzePfKPT8f96jt4nyqiCshrylM EFctzo308TkpPG1eA+lRNYQUpYZdg84jBnbauIAL/2Z0UrE9ud2RauOtuvPk7mxv0Y4X 8C1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741317561; x=1741922361; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VVXPe/IPw1lxQV8lI9bp/vRnBlqZTCq+2dNe5J1DRJc=; b=JmW8eIngYlD4kaE5GTrXjop4kULW1urSnSKAd5nYpkL1rONKsq4H7/8gg5XkvyGQVB NS2Z82qiUO3b1JDzCs9bXog1dpnsT9tMFdkOJ+3HTuDXfG+SKMlvimLFTxE1emYgLO6z +eb7dy/r4rewDdFvpz62uoZ7O3y7LabEbc2cP434cJeiSvvZ1t0FD6xzWnYURjDfhmPD 1/6i2kNabDu3ACaPThWREvjBUEdLekanOsNbxkiXwrq3JA6MxzqYbGwZntVDc3JBY02+ 8s83AR6rV6KTO+PnUwUrHbcoJ3ofsM/DMF85y5wCjXuU5Hh3LVwX1J1DblM/+zb6ZKqZ jCGA== X-Forwarded-Encrypted: i=1; AJvYcCUXL6jzI4WhM3q2VcSJYdz+/Ag6CqgqxaV2j+VNtQyxisZPUPCEAA6Nkuvt7nsehvThpTk/9G8wvjLzWFs=@vger.kernel.org, AJvYcCXesuXUFwWEyfuc1I9P0nLnRvoNzNyEpxyB4ksET9wWZl3GdYO3UlkRcRksFDN2QVhu8TAnbHe38UpLzuDBGsxF@vger.kernel.org X-Gm-Message-State: AOJu0YxRQeGAYMpYuyuJlI6J1SqM4y05qNfQDKIGHLnnjy0+vQxhIMHC fP5hucG1s5j8T172wjen5KcH9puPzjh19iANq91dlQx1WO0UYSw/Ooc8KPT1Lj59Yw== X-Gm-Gg: ASbGncuniKX2yWL/QzZFGf9cwAwHF7C0uSanTw/7GMJS+137EHszMRy02dt2AhG/ZmW No4ibFkzaozClbj/KXVExAKV5sVfqqJzPFLzZJCdxNzQz9PNmoigc9J2RMgnnhZPUeBRcDYFLwB 3+J+/5ARarkb9Qf2N+np3DiQjShiYcemuUQcOyuX/F2iRTAZmxv1+n9RfdixJmMy6d+UmsrXzwS bn1x54ANfGnoe6WxCSLrbCvu4yOyY4IJj1vcHRWPihsVkftMFJoOo6mSy2eILnvHUrJ6TtM04uW Kcn+xN0H2kdhIHmPn4axiGpW5q6eJTQI8oyXYWIgiKhu0NGlc+wb87Woc9GExMKs X-Google-Smtp-Source: AGHT+IEyrtYKfMbZllONi71VjxAuC6n5czwLwdDLEZhkPbV4VGajnC13vLGNxA3KYJqfPoUX3vxiNg== X-Received: by 2002:a17:90b:3c8d:b0:2ee:7411:ca99 with SMTP id 98e67ed59e1d1-2ff7ce7b230mr2540773a91.1.1741317560947; Thu, 06 Mar 2025 19:19:20 -0800 (PST) Received: from fedora.dns.podman ([43.228.180.230]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-224109dd5e2sm20013165ad.15.2025.03.06.19.19.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Mar 2025 19:19:20 -0800 (PST) From: Hangbin Liu To: netdev@vger.kernel.org Cc: Jay Vosburgh , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Nikolay Aleksandrov , Simon Horman , Shuah Khan , Tariq Toukan , Jianbo Liu , Jarod Wilson , Steffen Klassert , Cosmin Ratiu , Petr Machata , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Hangbin Liu Subject: [PATCHv5 net 1/3] bonding: fix calling sleeping function in spin lock and some race conditions Date: Fri, 7 Mar 2025 03:19:01 +0000 Message-ID: <20250307031903.223973-2-liuhangbin@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250307031903.223973-1-liuhangbin@gmail.com> References: <20250307031903.223973-1-liuhangbin@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The fixed commit placed mutex_lock() inside spin_lock_bh(), which triggers a warning: BUG: sleeping function called from invalid context at... Fix this by moving the IPsec deletion operation to bond_ipsec_free_sa, which is not held by spin_lock_bh(). Additionally, there are also some race conditions as bond_ipsec_del_sa_all() and __xfrm_state_delete could running in parallel without any lock. e.g. bond_ipsec_del_sa_all() __xfrm_state_delete() - .xdo_dev_state_delete - bond_ipsec_del_sa() - .xdo_dev_state_free - .xdo_dev_state_delete() - bond_ipsec_free_sa() bond active_slave changes - .xdo_dev_state_free() bond_ipsec_add_sa_all() - ipsec->xs->xso.real_dev = real_dev; - xdo_dev_state_add To fix this, let's add xs->lock during bond_ipsec_del_sa_all(), and delete the IPsec list when the XFRM state is DEAD, which could prevent xdo_dev_state_free() from being triggered again in bond_ipsec_free_sa(). In bond_ipsec_add_sa(), if .xdo_dev_state_add() failed, the xso.real_dev is set without clean. Which will cause trouble if __xfrm_state_delete is called at the same time. Reset the xso.real_dev to NULL if state add failed. Despite the above fixes, there are still races in bond_ipsec_add_sa() and bond_ipsec_add_sa_all(). If __xfrm_state_delete() is called immediately after we set the xso.real_dev and before .xdo_dev_state_add() is finished, like ipsec->xs->xso.real_dev = real_dev;                                  __xfrm_state_delete                                  - bond_ipsec_del_sa()                                    - .xdo_dev_state_delete() - bond_ipsec_free_sa()                                    - .xdo_dev_state_free() .xdo_dev_state_add() But there is no good solution yet. So I just added a FIXME note in here and hope we can fix it in future. Fixes: 2aeeef906d5a ("bonding: change ipsec_lock from spin lock to mutex") Reported-by: Jakub Kicinski Closes: https://lore.kernel.org/netdev/20241212062734.182a0164@kernel.org Suggested-by: Cosmin Ratiu Signed-off-by: Hangbin Liu --- drivers/net/bonding/bond_main.c | 69 ++++++++++++++++++++++++--------- 1 file changed, 51 insertions(+), 18 deletions(-) diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c index e45bba240cbc..dd3d0d41d98f 100644 --- a/drivers/net/bonding/bond_main.c +++ b/drivers/net/bonding/bond_main.c @@ -506,6 +506,7 @@ static int bond_ipsec_add_sa(struct xfrm_state *xs, list_add(&ipsec->list, &bond->ipsec_list); mutex_unlock(&bond->ipsec_lock); } else { + xs->xso.real_dev = NULL; kfree(ipsec); } out: @@ -541,7 +542,15 @@ static void bond_ipsec_add_sa_all(struct bonding *bond) if (ipsec->xs->xso.real_dev == real_dev) continue; + /* Skip dead xfrm states, they'll be freed later. */ + if (ipsec->xs->km.state == XFRM_STATE_DEAD) + continue; + ipsec->xs->xso.real_dev = real_dev; + /* FIXME: there is a race that before .xdo_dev_state_add() + * is called, the __xfrm_state_delete() is called in parallel, + * which will call .xdo_dev_state_delete() and xdo_dev_state_free() + */ if (real_dev->xfrmdev_ops->xdo_dev_state_add(ipsec->xs, NULL)) { slave_warn(bond_dev, real_dev, "%s: failed to add SA\n", __func__); ipsec->xs->xso.real_dev = NULL; @@ -560,7 +569,6 @@ static void bond_ipsec_del_sa(struct xfrm_state *xs) struct net_device *bond_dev = xs->xso.dev; struct net_device *real_dev; netdevice_tracker tracker; - struct bond_ipsec *ipsec; struct bonding *bond; struct slave *slave; @@ -592,22 +600,13 @@ static void bond_ipsec_del_sa(struct xfrm_state *xs) real_dev->xfrmdev_ops->xdo_dev_state_delete(xs); out: netdev_put(real_dev, &tracker); - mutex_lock(&bond->ipsec_lock); - list_for_each_entry(ipsec, &bond->ipsec_list, list) { - if (ipsec->xs == xs) { - list_del(&ipsec->list); - kfree(ipsec); - break; - } - } - mutex_unlock(&bond->ipsec_lock); } static void bond_ipsec_del_sa_all(struct bonding *bond) { struct net_device *bond_dev = bond->dev; + struct bond_ipsec *ipsec, *tmp_ipsec; struct net_device *real_dev; - struct bond_ipsec *ipsec; struct slave *slave; slave = rtnl_dereference(bond->curr_active_slave); @@ -616,9 +615,22 @@ static void bond_ipsec_del_sa_all(struct bonding *bond) return; mutex_lock(&bond->ipsec_lock); - list_for_each_entry(ipsec, &bond->ipsec_list, list) { - if (!ipsec->xs->xso.real_dev) + list_for_each_entry_safe(ipsec, tmp_ipsec, &bond->ipsec_list, list) { + spin_lock_bh(&ipsec->xs->lock); + if (!ipsec->xs->xso.real_dev) { + spin_unlock_bh(&ipsec->xs->lock); continue; + } + + if (ipsec->xs->km.state == XFRM_STATE_DEAD) { + list_del(&ipsec->list); + kfree(ipsec); + /* Need to free device here, or the xs->xso.real_dev + * may changed in bond_ipsec_add_sa_all and free + * on old device will never be called. + */ + goto next; + } if (!real_dev->xfrmdev_ops || !real_dev->xfrmdev_ops->xdo_dev_state_delete || @@ -626,11 +638,20 @@ static void bond_ipsec_del_sa_all(struct bonding *bond) slave_warn(bond_dev, real_dev, "%s: no slave xdo_dev_state_delete\n", __func__); - } else { - real_dev->xfrmdev_ops->xdo_dev_state_delete(ipsec->xs); - if (real_dev->xfrmdev_ops->xdo_dev_state_free) - real_dev->xfrmdev_ops->xdo_dev_state_free(ipsec->xs); + spin_unlock_bh(&ipsec->xs->lock); + continue; } + + real_dev->xfrmdev_ops->xdo_dev_state_delete(ipsec->xs); + +next: + /* set real_dev to NULL in case __xfrm_state_delete() is called in parallel */ + ipsec->xs->xso.real_dev = NULL; + + /* Unlock before freeing device state, it could sleep. */ + spin_unlock_bh(&ipsec->xs->lock); + if (real_dev->xfrmdev_ops->xdo_dev_state_free) + real_dev->xfrmdev_ops->xdo_dev_state_free(ipsec->xs); } mutex_unlock(&bond->ipsec_lock); } @@ -638,6 +659,7 @@ static void bond_ipsec_del_sa_all(struct bonding *bond) static void bond_ipsec_free_sa(struct xfrm_state *xs) { struct net_device *bond_dev = xs->xso.dev; + struct bond_ipsec *ipsec, *tmp_ipsec; struct net_device *real_dev; netdevice_tracker tracker; struct bonding *bond; @@ -659,13 +681,24 @@ static void bond_ipsec_free_sa(struct xfrm_state *xs) if (!xs->xso.real_dev) goto out; - WARN_ON(xs->xso.real_dev != real_dev); + if (xs->xso.real_dev != real_dev) + goto out; if (real_dev && real_dev->xfrmdev_ops && real_dev->xfrmdev_ops->xdo_dev_state_free) real_dev->xfrmdev_ops->xdo_dev_state_free(xs); out: netdev_put(real_dev, &tracker); + + mutex_lock(&bond->ipsec_lock); + list_for_each_entry_safe(ipsec, tmp_ipsec, &bond->ipsec_list, list) { + if (ipsec->xs == xs) { + list_del(&ipsec->list); + kfree(ipsec); + break; + } + } + mutex_unlock(&bond->ipsec_lock); } /** From patchwork Fri Mar 7 03:19:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hangbin Liu X-Patchwork-Id: 14005707 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DDF218E02A; Fri, 7 Mar 2025 03:19:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741317569; cv=none; b=TS/k3n1v0MejUex41rVjHoGqWVO7znxj3g2O7T5G9BarXbXlCj0iI5hlBwtaFlPbDbCnaYbvukTU4B2bTHZSo3kytnkWunN9os5/amnPWgjgUt9yB6qwqLdr31uP1jAswnCtcQ1VY+5yvZISDz0zgJMqWLdvfYk2UN7yx0DPSvY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741317569; c=relaxed/simple; bh=Uc2Y4LX7nLo4V+yn1T2U9yZeBAqdGZWqoe3sbEo99As=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sIKwQAcv/VIFsWpQPM+6Z+9w5tM4BEJlf9YcQ6Nyy8KNeiO88n/r8qfrb/ZNxU1M5MGrOzYhCijnzXE/I4YtHMtueXS/NeLUZhvezEdHYcVXpQMFAYJ68fuz8ZWSJSBAqLtoGXj6igUJdnMXbdateMQcJp57vXc/cprYUw82PWY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dc1IqCq1; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dc1IqCq1" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2ff087762bbso2164995a91.3; Thu, 06 Mar 2025 19:19:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741317566; x=1741922366; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IfVhujCifJI6eQhwz0SXL+R1pNFuvqV62APgboYQN3s=; b=dc1IqCq1srMQGuHkkICOM6HVb7wbIYqrfqkZ2J27Hfbkg19HKq9+4qze8nWduOLJye PJYZ21ywMuK5A351rf17d05TE/orwxPHZ4n7ETN3sKEq8ukGWm67FmjjZOwZHkwRsRZJ VqMLj1+8SkbHWBhxUe79J9Mji5pV4FoWxd1jynhK/BzVFn0WfwO/9BL9x/oEKuygK5tW Q/z2d7nMRdeq5cqz9gcbhe3Xu7tG6sT495Vq01CH5BxP52zX0AxMSbysSODUQ/J4Rh7J UNm0kO7YzJK0JanohQW3W5mBAkgAt4+DCJ3v+FD2mWMgABUKY7EVCJ/vZlfjH/tAvH+7 XwWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741317566; x=1741922366; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IfVhujCifJI6eQhwz0SXL+R1pNFuvqV62APgboYQN3s=; b=RBo4i/RoqRBw2pn9ipAA48jAWIdRKNdaYYQiqPUEpfpqUt4DB3DUrgpjpLRcFp5zXb IF1ED82JUaRajpghFJC+i14lVzAUXGdslUlP2CZeu/k4yVoPJdF9bYvuoRJUJQU6bWVY Um93uvL7WFmdd0YUHhJuvvMQaGyKpRwv7nmYT79V9254wWoZvn9BLJecZZ95j5P2l/c8 hGVXEzR7eHxO5jIJkaZvCJCp6RPBejfTnNZ7G9sGilRC05LxnjIKSfkOwl13fyMZnCXp gh8SBGCaeK/1ct3YvVp2fUIbOB7i3QLV8nbKVkhEw5D4UyfMPmtIcCDg/oIu8mIuFT5Y aElw== X-Forwarded-Encrypted: i=1; AJvYcCV+rp2eUhT3vx0j0g8lR3DuBjAFL61kVh9bQYOdSj6C10mHrBwByhEqLokRw5ZBEyK6sqkoaCWik9qgqSaT5K6b@vger.kernel.org, AJvYcCW0HvauTYEorkKglJSVqnYPQiwSZ0ZoyRMrLR6431arNHtZmUbafFolFC5Qqa28qka+IIwDsbPUd7JDh3I=@vger.kernel.org X-Gm-Message-State: AOJu0Yy8JPc09LKILgrOfL6MQNvgminpMRCFQY3WbKeVIkPsCiWm0px+ 4HSCLw+nf9gCUHVg2x+TCYl3BzgXS9oWOJFxD3oqvj9tAzVA6lTyynhL+e2UGdb+7g== X-Gm-Gg: ASbGncuyHmg8x1ENi5mBP9P5DJH44Rj+veunXil9KSnDFgA44BDcxqmPkj8tl3Qjf+0 nXlF60CmSPbDeXMbCsrYdQazrUeGz+LbqTJ/51l3VSzVi8szNJNQIX/6XZmY38t1dmMoQJCIIHE OQzDb6POBJwET8f1nQWW9OYZKf11x6Tf4ffVD3Ho5e06QoDN/DhPdLoyLzP2PGzQ8DXVPePPBVx 0Inxq5ZPX4fkH1AEDMbhXqZX8Kd8BudE7LyQgxm3A50xCVK99P7CVdoN14Mln7yS4K+jzhIGylX bZNzMDCBHg251qnKP2Y9d+9xdhrryMJWb3Gm5Bp8/rIP5mwUPpxeUDofAnQQFHTP X-Google-Smtp-Source: AGHT+IH9BdiBzTzH4qtj3b4pWxNcrckb/AcaufdblDqmhdyDhISkyBvr09Txp5nhkbg0LogOUyHYRg== X-Received: by 2002:a17:90b:17c3:b0:2ff:784b:ffe with SMTP id 98e67ed59e1d1-2ff7ce8361fmr2955637a91.11.1741317566262; Thu, 06 Mar 2025 19:19:26 -0800 (PST) Received: from fedora.dns.podman ([43.228.180.230]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-224109dd5e2sm20013165ad.15.2025.03.06.19.19.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Mar 2025 19:19:25 -0800 (PST) From: Hangbin Liu To: netdev@vger.kernel.org Cc: Jay Vosburgh , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Nikolay Aleksandrov , Simon Horman , Shuah Khan , Tariq Toukan , Jianbo Liu , Jarod Wilson , Steffen Klassert , Cosmin Ratiu , Petr Machata , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Hangbin Liu Subject: [PATCHv5 net 2/3] bonding: fix xfrm offload feature setup on active-backup mode Date: Fri, 7 Mar 2025 03:19:02 +0000 Message-ID: <20250307031903.223973-3-liuhangbin@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250307031903.223973-1-liuhangbin@gmail.com> References: <20250307031903.223973-1-liuhangbin@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The active-backup bonding mode supports XFRM ESP offload. However, when a bond is added using command like `ip link add bond0 type bond mode 1 miimon 100`, the `ethtool -k` command shows that the XFRM ESP offload is disabled. This occurs because, in bond_newlink(), we change bond link first and register bond device later. So the XFRM feature update in bond_option_mode_set() is not called as the bond device is not yet registered, leading to the offload feature not being set successfully. To resolve this issue, we can modify the code order in bond_newlink() to ensure that the bond device is registered first before changing the bond link parameters. This change will allow the XFRM ESP offload feature to be correctly enabled. Fixes: 007ab5345545 ("bonding: fix feature flag setting at init time") Signed-off-by: Hangbin Liu --- drivers/net/bonding/bond_main.c | 2 +- drivers/net/bonding/bond_netlink.c | 16 +++++++++------- include/net/bonding.h | 1 + 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c index dd3d0d41d98f..a060960927e9 100644 --- a/drivers/net/bonding/bond_main.c +++ b/drivers/net/bonding/bond_main.c @@ -4422,7 +4422,7 @@ void bond_work_init_all(struct bonding *bond) INIT_DELAYED_WORK(&bond->slave_arr_work, bond_slave_arr_handler); } -static void bond_work_cancel_all(struct bonding *bond) +void bond_work_cancel_all(struct bonding *bond) { cancel_delayed_work_sync(&bond->mii_work); cancel_delayed_work_sync(&bond->arp_work); diff --git a/drivers/net/bonding/bond_netlink.c b/drivers/net/bonding/bond_netlink.c index 2a6a424806aa..ed16af6db557 100644 --- a/drivers/net/bonding/bond_netlink.c +++ b/drivers/net/bonding/bond_netlink.c @@ -568,18 +568,20 @@ static int bond_newlink(struct net *src_net, struct net_device *bond_dev, struct nlattr *tb[], struct nlattr *data[], struct netlink_ext_ack *extack) { + struct bonding *bond = netdev_priv(bond_dev); int err; - err = bond_changelink(bond_dev, tb, data, extack); - if (err < 0) + err = register_netdevice(bond_dev); + if (err) return err; - err = register_netdevice(bond_dev); - if (!err) { - struct bonding *bond = netdev_priv(bond_dev); + netif_carrier_off(bond_dev); + bond_work_init_all(bond); - netif_carrier_off(bond_dev); - bond_work_init_all(bond); + err = bond_changelink(bond_dev, tb, data, extack); + if (err) { + bond_work_cancel_all(bond); + unregister_netdevice(bond_dev); } return err; diff --git a/include/net/bonding.h b/include/net/bonding.h index 8bb5f016969f..e5e005cd2e17 100644 --- a/include/net/bonding.h +++ b/include/net/bonding.h @@ -707,6 +707,7 @@ struct bond_vlan_tag *bond_verify_device_path(struct net_device *start_dev, int bond_update_slave_arr(struct bonding *bond, struct slave *skipslave); void bond_slave_arr_work_rearm(struct bonding *bond, unsigned long delay); void bond_work_init_all(struct bonding *bond); +void bond_work_cancel_all(struct bonding *bond); #ifdef CONFIG_PROC_FS void bond_create_proc_entry(struct bonding *bond); From patchwork Fri Mar 7 03:19:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hangbin Liu X-Patchwork-Id: 14005708 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE9A6199235; Fri, 7 Mar 2025 03:19:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741317574; cv=none; b=iF7exLOu/zn89SceVBjrM2Gl2YfhoLBzhFte1WRjFHZarKtziGgiUIKwyJ647KwJY5DY5Bm342ZtAUW/T5G1QqG4YSfcyIv6Y1L/KRCdtXt3IDl6CsyP6iSnmcIfbCgzHMv6d3RyKk031xyp2u8nwWDaM7pGwpyc0bv2oYBkgeo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741317574; c=relaxed/simple; bh=QK4lwkBaLpYkzW8OpVr6l0dFgx76+HSEo+fzV171KDI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GR5VlLMbtYZzYZlmT1NvAeg8FWAaMabNn4nc/Gu7nARhkpMJ+KWe0lru+POUYlD74jL/ykgSYE+4LjRnZ/wKJ26WVyE9/Pq1AYxq4U1ipiPvrbolPkuf4kwpfQDthVYQgwbyKQIudRd96sa2Fk7UK0gbWdXtA0t7JSte6iyGbL4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZiffwQrJ; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZiffwQrJ" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-22337bc9ac3so28761285ad.1; Thu, 06 Mar 2025 19:19:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741317572; x=1741922372; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KkYXSSC3tJ8D/xoGrXlhmNN1ijluTdRxzvHht/qe2VM=; b=ZiffwQrJUkip4rROJHzKoE7+rfbFvP5W06rVSxHG11qwbuO2+YrYjQVo0+XmX4CxBg l4i/zrfdFK0S58KNKwJCj9oe9O4ZB8MbWmYtA7D33/b58ZUAm4ESpOBN8hqdhh9+quYg f2/C/y2ossW/vcDrpT7x7S1k0ZyYvUziEYB3nzaKDG17zxaEN/6sGGxfhvovl3uXoP1r L9lxAp6SQlCs705gF0+5kfpKuovI+1ZDZCFMP28MUtTqD75ldt0kvoAbtG0neKwq8Yva 1+Dmlrs34jeD1AtEhxtl0RzYNkrdakNOuVEnjxnfKEH1qFfIdpv9t/1iuSoNJMeBCWEi HO9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741317572; x=1741922372; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KkYXSSC3tJ8D/xoGrXlhmNN1ijluTdRxzvHht/qe2VM=; b=F49VA7AOuvuMYVfwwGS0UyLbC42jUQYIGUXdB+SABO0cy+/FVLCnaGzrDfYpiSP9xT KrTeNjA/O6bnPdb6rxzT9lEF2LSr1tLJna0kwDIpMJuiGcV4hqSY/G+r7epYwfMsK7Ch KKT7dVUbYdegIvVnIMa03qOhq7fR0Ls7zPiI8DjoA3sM1nDVUMMrdcQKLhVDfLJoTYjb V9xN5RLi0clZmkMzm3QvOIN1+rCpu4Q5PZ+s5snK+m8D3+G6P0TB35rYmsIOzzpmx1xU R6AC5mgYgyo9QEJr+bJJFEpBh+rrq1WdlWWMiuNn86c+rB2HO8aQW+8Go+E4vohDRQyU XaSQ== X-Forwarded-Encrypted: i=1; AJvYcCV7QqTuDg6fUaEPSh00xiMq0XlNjRLJgqcbdkGuqFVM6LewfoCwgLVd6umADA5wPbzDp1BawatR06z8nZaIkio6@vger.kernel.org, AJvYcCW+AAiAYeaWIa2zBnqRJmJr7yT2Fy3F4qv2itKwXdT5+dgmHfGUpTc2nDMmrY8DtWmB8U3Fk9eHnjyv4pI=@vger.kernel.org X-Gm-Message-State: AOJu0YwN3FGhCOGsFiy1dq6hVD/atFkqzpazIzgAyLBwyG1DhLEUe94n E58gf55QyU9Qi9TgWuzevq3jalkiLnkRRDq0AsXbP1QeJ93MUiteKC3/u7DyKzd5vQ== X-Gm-Gg: ASbGnctf+uGtjSa0KftlAfkM4G6CHyEwGFxM8xBiFc8PB87msB8M6NTRevpeWS8r+ns 7TX6nVyBmO1kdFvxhIQRETEdr6FuMHGko6NY+5YIhpYGbnGZ00hnCfiJShkk5L70psJPkalUhKD aoPngMjJ6i0s2uUYyaxYgG8PBUf2UWvJM0PvwaQl2cbyxT4ekDKjWmk7Xk8Joxv9DtYNiPBIJb+ VD9m5QI2A0mWXnvw+tyDuziiOMa3XQn3YWQXJMd7kQ/I5LdTfXEmohsv+7uiHR+sOugHzzKDeo+ WQStys8ThTFuhC+fxb7c+PKDPPsUk1lEf/U5Ff16f2lvwTPQJcHj8BUbm9cwBpRU X-Google-Smtp-Source: AGHT+IFZyhyvr2XGalTzNKpQy8s4HxN8N0MwaY/NNXoNcmh6dVchxFXBgPTfwtr0189P+LrVEESDrA== X-Received: by 2002:a17:903:40cb:b0:224:584:6f07 with SMTP id d9443c01a7336-22428ac94camr27553035ad.37.1741317571906; Thu, 06 Mar 2025 19:19:31 -0800 (PST) Received: from fedora.dns.podman ([43.228.180.230]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-224109dd5e2sm20013165ad.15.2025.03.06.19.19.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Mar 2025 19:19:31 -0800 (PST) From: Hangbin Liu To: netdev@vger.kernel.org Cc: Jay Vosburgh , Andrew Lunn , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Nikolay Aleksandrov , Simon Horman , Shuah Khan , Tariq Toukan , Jianbo Liu , Jarod Wilson , Steffen Klassert , Cosmin Ratiu , Petr Machata , linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Hangbin Liu Subject: [PATCHv5 net 3/3] selftests: bonding: add ipsec offload test Date: Fri, 7 Mar 2025 03:19:03 +0000 Message-ID: <20250307031903.223973-4-liuhangbin@gmail.com> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20250307031903.223973-1-liuhangbin@gmail.com> References: <20250307031903.223973-1-liuhangbin@gmail.com> Precedence: bulk X-Mailing-List: linux-kselftest@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This introduces a test for IPSec offload over bonding, utilizing netdevsim for the testing process, as veth interfaces do not support IPSec offload. The test will ensure that the IPSec offload functionality remains operational even after a failover event occurs in the bonding configuration. Here is the test result: TEST: bond_ipsec_offload (active_slave eth0) [ OK ] TEST: bond_ipsec_offload (active_slave eth1) [ OK ] Reviewed-by: Petr Machata Signed-off-by: Hangbin Liu --- .../selftests/drivers/net/bonding/Makefile | 3 +- .../drivers/net/bonding/bond_ipsec_offload.sh | 154 ++++++++++++++++++ .../selftests/drivers/net/bonding/config | 4 + 3 files changed, 160 insertions(+), 1 deletion(-) create mode 100755 tools/testing/selftests/drivers/net/bonding/bond_ipsec_offload.sh diff --git a/tools/testing/selftests/drivers/net/bonding/Makefile b/tools/testing/selftests/drivers/net/bonding/Makefile index 2b10854e4b1e..d5a7de16d33a 100644 --- a/tools/testing/selftests/drivers/net/bonding/Makefile +++ b/tools/testing/selftests/drivers/net/bonding/Makefile @@ -10,7 +10,8 @@ TEST_PROGS := \ mode-2-recovery-updelay.sh \ bond_options.sh \ bond-eth-type-change.sh \ - bond_macvlan_ipvlan.sh + bond_macvlan_ipvlan.sh \ + bond_ipsec_offload.sh TEST_FILES := \ lag_lib.sh \ diff --git a/tools/testing/selftests/drivers/net/bonding/bond_ipsec_offload.sh b/tools/testing/selftests/drivers/net/bonding/bond_ipsec_offload.sh new file mode 100755 index 000000000000..4b19949a4c33 --- /dev/null +++ b/tools/testing/selftests/drivers/net/bonding/bond_ipsec_offload.sh @@ -0,0 +1,154 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +# IPsec over bonding offload test: +# +# +----------------+ +# | bond0 | +# | | | +# | eth0 eth1 | +# +---+-------+----+ +# +# We use netdevsim instead of physical interfaces +#------------------------------------------------------------------- +# Example commands +# ip x s add proto esp src 192.0.2.1 dst 192.0.2.2 \ +# spi 0x07 mode transport reqid 0x07 replay-window 32 \ +# aead 'rfc4106(gcm(aes))' 1234567890123456dcba 128 \ +# sel src 192.0.2.1/24 dst 192.0.2.2/24 +# offload dev bond0 dir out +# ip x p add dir out src 192.0.2.1/24 dst 192.0.2.2/24 \ +# tmpl proto esp src 192.0.2.1 dst 192.0.2.2 \ +# spi 0x07 mode transport reqid 0x07 +# +#------------------------------------------------------------------- + +lib_dir=$(dirname "$0") +source "$lib_dir"/../../../net/lib.sh +algo="aead rfc4106(gcm(aes)) 0x3132333435363738393031323334353664636261 128" +srcip=192.0.2.1 +dstip=192.0.2.2 +ipsec0=/sys/kernel/debug/netdevsim/netdevsim0/ports/0/ipsec +ipsec1=/sys/kernel/debug/netdevsim/netdevsim0/ports/1/ipsec +active_slave="" + +active_slave_changed() +{ + local old_active_slave=$1 + local new_active_slave=$(ip -n ${ns} -d -j link show bond0 | \ + jq -r ".[].linkinfo.info_data.active_slave") + [ "$new_active_slave" != "$old_active_slave" -a "$new_active_slave" != "null" ] +} + +test_offload() +{ + # use ping to exercise the Tx path + ip netns exec $ns ping -I bond0 -c 3 -W 1 -i 0 $dstip >/dev/null + + active_slave=$(ip -n ${ns} -d -j link show bond0 | \ + jq -r ".[].linkinfo.info_data.active_slave") + + if [ $active_slave = $nic0 ]; then + sysfs=$ipsec0 + elif [ $active_slave = $nic1 ]; then + sysfs=$ipsec1 + else + check_err 1 "bond_ipsec_offload invalid active_slave $active_slave" + fi + + # The tx/rx order in sysfs may changed after failover + grep -q "SA count=2 tx=3" $sysfs && grep -q "tx ipaddr=$dstip" $sysfs + check_err $? "incorrect tx count with link ${active_slave}" + + log_test bond_ipsec_offload "active_slave ${active_slave}" +} + +setup_env() +{ + if ! mount | grep -q debugfs; then + mount -t debugfs none /sys/kernel/debug/ &> /dev/null + defer umount /sys/kernel/debug/ + + fi + + # setup netdevsim since dummy/veth dev doesn't have offload support + if [ ! -w /sys/bus/netdevsim/new_device ] ; then + modprobe -q netdevsim + if [ $? -ne 0 ]; then + echo "SKIP: can't load netdevsim for ipsec offload" + exit $ksft_skip + fi + defer modprobe -r netdevsim + fi + + setup_ns ns + defer cleanup_ns $ns +} + +setup_bond() +{ + ip -n $ns link add bond0 type bond mode active-backup miimon 100 + ip -n $ns addr add $srcip/24 dev bond0 + ip -n $ns link set bond0 up + + ifaces=$(ip netns exec $ns bash -c ' + sysfsnet=/sys/bus/netdevsim/devices/netdevsim0/net/ + echo "0 2" > /sys/bus/netdevsim/new_device + while [ ! -d $sysfsnet ] ; do :; done + udevadm settle + ls $sysfsnet + ') + nic0=$(echo $ifaces | cut -f1 -d ' ') + nic1=$(echo $ifaces | cut -f2 -d ' ') + ip -n $ns link set $nic0 master bond0 + ip -n $ns link set $nic1 master bond0 + + # we didn't create a peer, make sure we can Tx by adding a permanent + # neighbour this need to be added after enslave + ip -n $ns neigh add $dstip dev bond0 lladdr 00:11:22:33:44:55 + + # create offloaded SAs, both in and out + ip -n $ns x p add dir out src $srcip/24 dst $dstip/24 \ + tmpl proto esp src $srcip dst $dstip spi 9 \ + mode transport reqid 42 + + ip -n $ns x p add dir in src $dstip/24 dst $srcip/24 \ + tmpl proto esp src $dstip dst $srcip spi 9 \ + mode transport reqid 42 + + ip -n $ns x s add proto esp src $srcip dst $dstip spi 9 \ + mode transport reqid 42 $algo sel src $srcip/24 dst $dstip/24 \ + offload dev bond0 dir out + + ip -n $ns x s add proto esp src $dstip dst $srcip spi 9 \ + mode transport reqid 42 $algo sel src $dstip/24 dst $srcip/24 \ + offload dev bond0 dir in + + # does offload show up in ip output + lines=`ip -n $ns x s list | grep -c "crypto offload parameters: dev bond0 dir"` + if [ $lines -ne 2 ] ; then + check_err 1 "bond_ipsec_offload SA offload missing from list output" + fi +} + +trap defer_scopes_cleanup EXIT +setup_env +setup_bond + +# start Offload testing +test_offload + +# do failover and re-test +ip -n $ns link set $active_slave down +slowwait 5 active_slave_changed $active_slave +test_offload + +# make sure offload get removed from driver +ip -n $ns x s flush +ip -n $ns x p flush +line0=$(grep -c "SA count=0" $ipsec0) +line1=$(grep -c "SA count=0" $ipsec1) +[ $line0 -ne 1 -o $line1 -ne 1 ] +check_fail $? "bond_ipsec_offload SA not removed from driver" + +exit $EXIT_STATUS diff --git a/tools/testing/selftests/drivers/net/bonding/config b/tools/testing/selftests/drivers/net/bonding/config index dad4e5fda4db..054fb772846f 100644 --- a/tools/testing/selftests/drivers/net/bonding/config +++ b/tools/testing/selftests/drivers/net/bonding/config @@ -9,3 +9,7 @@ CONFIG_NET_CLS_FLOWER=y CONFIG_NET_SCH_INGRESS=y CONFIG_NLMON=y CONFIG_VETH=y +CONFIG_INET_ESP=y +CONFIG_INET_ESP_OFFLOAD=y +CONFIG_XFRM_USER=m +CONFIG_NETDEVSIM=m