From patchwork Tue Aug 23 13:31:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 12952316 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04FFAC32774 for ; Tue, 23 Aug 2022 17:02:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343655AbiHWRBw (ORCPT ); Tue, 23 Aug 2022 13:01:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344386AbiHWRBQ (ORCPT ); Tue, 23 Aug 2022 13:01:16 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56A9885AB2 for ; Tue, 23 Aug 2022 06:32:23 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id EA9C0614C4 for ; Tue, 23 Aug 2022 13:32:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CD21EC433D7; Tue, 23 Aug 2022 13:32:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661261542; bh=7CWFJPDYmNTO1tgfwh7HDem3FSnpzBD/Ih5sSJH6BCA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=T4wme9Zz9k5vZR1544YAl2/Gc9JE7fvQsW1wL/Pqiofv8doxUjZ60f49tkMEPQo61 s3Pv52yVZJCA9zEgBCkNirlBFLzp560FTKozfQYI9I1eJjczGteZ+GEBvQuV4Utq4M neJ1uY4Eslk8ps3NuinGmB9NpNTJI8TUgpIMBqP4l+mQdMXIJiXBWNIuizWR1NsW2b q2v1KU24ZaNpgv74PDtQWUOet7vlheknXswQ3NK0C1C9UInLYAYvM69MpvNkx8X5gR YXB0jBEjssplgN1aYypzRmIQUOFRH/asf6p1g2JqlIZ49JANxZeN+STevp2Y7Za2ti 7SMWmpJDLlmmw== From: Leon Romanovsky To: Steffen Klassert Cc: Leon Romanovsky , "David S. Miller" , Eric Dumazet , Herbert Xu , Jakub Kicinski , netdev@vger.kernel.org, Paolo Abeni , Raed Salem , Saeed Mahameed Subject: [PATCH xfrm-next v3 1/6] xfrm: add new full offload flag Date: Tue, 23 Aug 2022 16:31:58 +0300 Message-Id: <2b6fdc02be2619455d98bde3b55ba8b71638ac33.1661260787.git.leonro@nvidia.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Leon Romanovsky In the next patches, the xfrm core code will be extended to support new type of offload - full offload. In that mode, both policy and state should be specially configured in order to perform whole offloaded data path. Full offload takes care of encryption, decryption, encapsulation and other operations with headers. As this mode is new for XFRM policy flow, we can "start fresh" with flag bits and release first and second bit for future use. Reviewed-by: Raed Salem Signed-off-by: Leon Romanovsky --- include/net/xfrm.h | 7 +++++++ include/uapi/linux/xfrm.h | 6 ++++++ net/xfrm/xfrm_device.c | 3 +++ net/xfrm/xfrm_user.c | 2 ++ 4 files changed, 18 insertions(+) diff --git a/include/net/xfrm.h b/include/net/xfrm.h index 6e8fa98f786f..b4d487053dfd 100644 --- a/include/net/xfrm.h +++ b/include/net/xfrm.h @@ -131,12 +131,19 @@ enum { XFRM_DEV_OFFLOAD_OUT, }; +enum { + XFRM_DEV_OFFLOAD_UNSPECIFIED, + XFRM_DEV_OFFLOAD_CRYPTO, + XFRM_DEV_OFFLOAD_FULL, +}; + struct xfrm_dev_offload { struct net_device *dev; netdevice_tracker dev_tracker; struct net_device *real_dev; unsigned long offload_handle; u8 dir : 2; + u8 type : 2; }; struct xfrm_mode { diff --git a/include/uapi/linux/xfrm.h b/include/uapi/linux/xfrm.h index b1f3e6a8f11a..3b084246d610 100644 --- a/include/uapi/linux/xfrm.h +++ b/include/uapi/linux/xfrm.h @@ -519,6 +519,12 @@ struct xfrm_user_offload { */ #define XFRM_OFFLOAD_IPV6 1 #define XFRM_OFFLOAD_INBOUND 2 +/* Two bits above are relevant for state path only, while + * offload is used for both policy and state flows. + * + * In policy offload mode, they are free and can be safely reused. + */ +#define XFRM_OFFLOAD_FULL 4 struct xfrm_userpolicy_default { #define XFRM_USERPOLICY_UNSPEC 0 diff --git a/net/xfrm/xfrm_device.c b/net/xfrm/xfrm_device.c index 637ca8838436..6d1124eb1ec8 100644 --- a/net/xfrm/xfrm_device.c +++ b/net/xfrm/xfrm_device.c @@ -270,12 +270,15 @@ int xfrm_dev_state_add(struct net *net, struct xfrm_state *x, else xso->dir = XFRM_DEV_OFFLOAD_OUT; + xso->type = XFRM_DEV_OFFLOAD_CRYPTO; + err = dev->xfrmdev_ops->xdo_dev_state_add(x); if (err) { xso->dev = NULL; xso->dir = 0; xso->real_dev = NULL; netdev_put(dev, &xso->dev_tracker); + xso->type = XFRM_DEV_OFFLOAD_UNSPECIFIED; if (err != -EOPNOTSUPP) return err; diff --git a/net/xfrm/xfrm_user.c b/net/xfrm/xfrm_user.c index 2ff017117730..9c0aef815730 100644 --- a/net/xfrm/xfrm_user.c +++ b/net/xfrm/xfrm_user.c @@ -854,6 +854,8 @@ static int copy_user_offload(struct xfrm_dev_offload *xso, struct sk_buff *skb) xuo->ifindex = xso->dev->ifindex; if (xso->dir == XFRM_DEV_OFFLOAD_IN) xuo->flags = XFRM_OFFLOAD_INBOUND; + if (xso->type == XFRM_DEV_OFFLOAD_FULL) + xuo->flags |= XFRM_OFFLOAD_FULL; return 0; } From patchwork Tue Aug 23 13:31:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 12952314 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EB39C38142 for ; Tue, 23 Aug 2022 17:01:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245714AbiHWRBp (ORCPT ); Tue, 23 Aug 2022 13:01:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344345AbiHWRBM (ORCPT ); Tue, 23 Aug 2022 13:01:12 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5529D14E112 for ; Tue, 23 Aug 2022 06:32:15 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D9137614C4 for ; Tue, 23 Aug 2022 13:32:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BC0AFC433D6; Tue, 23 Aug 2022 13:32:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661261534; bh=PZL91AXffzFMtihnRFLeRE15m2nJ0tedTboZ/kLRP9w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B/N0/KvZUHOg3KQM0pMd4MhMEaIrDfCBtbR4fQGH8Lb0uEmyttR7LeeBJwsYiur3D k9KLEzpp7pjOmvVHniUYt0hz/mEnLYe0cZQBsWBau6Gj449raMe17QxtIBVbGRVU6l QM3B1aBtRyWNH+heJfDwXBwnfdU/QBOOcYtBYh/YSvj1593+8VssgW48323FRsAukB 50AmwEyHwRsxZx9kx0fshdh61qE3VIqVb/ToxUgBuoi13dvc255MiCoyGXpwPMvrJD ul82tzy+9HispReuS1HTMNgMtrDK2FiMdOh/Hpc8UByLk1xbC75FtKli8GdcHF6CP6 FsSHuiskjdvMg== From: Leon Romanovsky To: Steffen Klassert Cc: Leon Romanovsky , Ayush Sawal , "David S. Miller" , Eric Dumazet , Herbert Xu , intel-wired-lan@lists.osuosl.org, Jakub Kicinski , Jesse Brandeburg , netdev@vger.kernel.org, Paolo Abeni , Raed Salem , Rohit Maheshwari , Saeed Mahameed , Tony Nguyen , Vinay Kumar Yadav Subject: [PATCH xfrm-next v3 2/6] xfrm: allow state full offload mode Date: Tue, 23 Aug 2022 16:31:59 +0300 Message-Id: <0e3b5ac73d11f255c73cf9bb35f2e5cf4433ec5d.1661260787.git.leonro@nvidia.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Leon Romanovsky Allow users to configure xfrm states with full offload mode. The full mode must be requested both for policy and state, and such requires us to do not implement fallback. We explicitly return an error if requested full mode can't be configured. Reviewed-by: Raed Salem Signed-off-by: Leon Romanovsky --- .../inline_crypto/ch_ipsec/chcr_ipsec.c | 4 ++++ .../net/ethernet/intel/ixgbe/ixgbe_ipsec.c | 5 ++++ drivers/net/ethernet/intel/ixgbevf/ipsec.c | 5 ++++ .../mellanox/mlx5/core/en_accel/ipsec.c | 4 ++++ drivers/net/netdevsim/ipsec.c | 5 ++++ net/xfrm/xfrm_device.c | 24 +++++++++++++++---- 6 files changed, 42 insertions(+), 5 deletions(-) diff --git a/drivers/net/ethernet/chelsio/inline_crypto/ch_ipsec/chcr_ipsec.c b/drivers/net/ethernet/chelsio/inline_crypto/ch_ipsec/chcr_ipsec.c index 585590520076..ca21794281d6 100644 --- a/drivers/net/ethernet/chelsio/inline_crypto/ch_ipsec/chcr_ipsec.c +++ b/drivers/net/ethernet/chelsio/inline_crypto/ch_ipsec/chcr_ipsec.c @@ -283,6 +283,10 @@ static int ch_ipsec_xfrm_add_state(struct xfrm_state *x) pr_debug("Cannot offload xfrm states with geniv other than seqiv\n"); return -EINVAL; } + if (x->xso.type != XFRM_DEV_OFFLOAD_CRYPTO) { + pr_debug("Unsupported xfrm offload\n"); + return -EINVAL; + } sa_entry = kzalloc(sizeof(*sa_entry), GFP_KERNEL); if (!sa_entry) { diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c index 774de63dd93a..53a969e34883 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c @@ -585,6 +585,11 @@ static int ixgbe_ipsec_add_sa(struct xfrm_state *xs) return -EINVAL; } + if (xs->xso.type != XFRM_DEV_OFFLOAD_CRYPTO) { + netdev_err(dev, "Unsupported ipsec offload type\n"); + return -EINVAL; + } + if (xs->xso.dir == XFRM_DEV_OFFLOAD_IN) { struct rx_sa rsa; diff --git a/drivers/net/ethernet/intel/ixgbevf/ipsec.c b/drivers/net/ethernet/intel/ixgbevf/ipsec.c index 9984ebc62d78..c1cf540d162a 100644 --- a/drivers/net/ethernet/intel/ixgbevf/ipsec.c +++ b/drivers/net/ethernet/intel/ixgbevf/ipsec.c @@ -280,6 +280,11 @@ static int ixgbevf_ipsec_add_sa(struct xfrm_state *xs) return -EINVAL; } + if (xs->xso.type != XFRM_DEV_OFFLOAD_CRYPTO) { + netdev_err(dev, "Unsupported ipsec offload type\n"); + return -EINVAL; + } + if (xs->xso.dir == XFRM_DEV_OFFLOAD_IN) { struct rx_sa rsa; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c index 2a8fd7020622..c182b640b80d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c @@ -256,6 +256,10 @@ static inline int mlx5e_xfrm_validate_state(struct xfrm_state *x) netdev_info(netdev, "Cannot offload xfrm states with geniv other than seqiv\n"); return -EINVAL; } + if (x->xso.type != XFRM_DEV_OFFLOAD_CRYPTO) { + netdev_info(netdev, "Unsupported xfrm offload type\n"); + return -EINVAL; + } return 0; } diff --git a/drivers/net/netdevsim/ipsec.c b/drivers/net/netdevsim/ipsec.c index 386336a38f34..b93baf5c8bee 100644 --- a/drivers/net/netdevsim/ipsec.c +++ b/drivers/net/netdevsim/ipsec.c @@ -149,6 +149,11 @@ static int nsim_ipsec_add_sa(struct xfrm_state *xs) return -EINVAL; } + if (xs->xso.type != XFRM_DEV_OFFLOAD_CRYPTO) { + netdev_err(dev, "Unsupported ipsec offload type\n"); + return -EINVAL; + } + /* find the first unused index */ ret = nsim_ipsec_find_empty_idx(ipsec); if (ret < 0) { diff --git a/net/xfrm/xfrm_device.c b/net/xfrm/xfrm_device.c index 6d1124eb1ec8..5b04e5cdca64 100644 --- a/net/xfrm/xfrm_device.c +++ b/net/xfrm/xfrm_device.c @@ -215,6 +215,7 @@ int xfrm_dev_state_add(struct net *net, struct xfrm_state *x, struct xfrm_dev_offload *xso = &x->xso; xfrm_address_t *saddr; xfrm_address_t *daddr; + bool is_full_offload; if (!x->type_offload) return -EINVAL; @@ -223,9 +224,11 @@ int xfrm_dev_state_add(struct net *net, struct xfrm_state *x, if (x->encap || x->tfcpad) return -EINVAL; - if (xuo->flags & ~(XFRM_OFFLOAD_IPV6 | XFRM_OFFLOAD_INBOUND)) + if (xuo->flags & + ~(XFRM_OFFLOAD_IPV6 | XFRM_OFFLOAD_INBOUND | XFRM_OFFLOAD_FULL)) return -EINVAL; + is_full_offload = xuo->flags & XFRM_OFFLOAD_FULL; dev = dev_get_by_index(net, xuo->ifindex); if (!dev) { if (!(xuo->flags & XFRM_OFFLOAD_INBOUND)) { @@ -240,7 +243,7 @@ int xfrm_dev_state_add(struct net *net, struct xfrm_state *x, x->props.family, xfrm_smark_get(0, x)); if (IS_ERR(dst)) - return 0; + return (is_full_offload) ? -EINVAL : 0; dev = dst->dev; @@ -251,7 +254,7 @@ int xfrm_dev_state_add(struct net *net, struct xfrm_state *x, if (!dev->xfrmdev_ops || !dev->xfrmdev_ops->xdo_dev_state_add) { xso->dev = NULL; dev_put(dev); - return 0; + return (is_full_offload) ? -EINVAL : 0; } if (x->props.flags & XFRM_STATE_ESN && @@ -270,7 +273,10 @@ int xfrm_dev_state_add(struct net *net, struct xfrm_state *x, else xso->dir = XFRM_DEV_OFFLOAD_OUT; - xso->type = XFRM_DEV_OFFLOAD_CRYPTO; + if (is_full_offload) + xso->type = XFRM_DEV_OFFLOAD_FULL; + else + xso->type = XFRM_DEV_OFFLOAD_CRYPTO; err = dev->xfrmdev_ops->xdo_dev_state_add(x); if (err) { @@ -280,7 +286,15 @@ int xfrm_dev_state_add(struct net *net, struct xfrm_state *x, netdev_put(dev, &xso->dev_tracker); xso->type = XFRM_DEV_OFFLOAD_UNSPECIFIED; - if (err != -EOPNOTSUPP) + /* User explicitly requested full offload mode and configured + * policy in addition to the XFRM state. So be civil to users, + * and return an error instead of taking fallback path. + * + * This WARN_ON() can be seen as a documentation for driver + * authors to do not return -EOPNOTSUPP in full offload mode. + */ + WARN_ON(err == -EOPNOTSUPP && is_full_offload); + if (err != -EOPNOTSUPP || is_full_offload) return err; } From patchwork Tue Aug 23 13:32:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 12952315 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2EA4BC32789 for ; Tue, 23 Aug 2022 17:01:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343532AbiHWRBr (ORCPT ); Tue, 23 Aug 2022 13:01:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344369AbiHWRBO (ORCPT ); Tue, 23 Aug 2022 13:01:14 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 718ED14E13F for ; Tue, 23 Aug 2022 06:32:19 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E8D8360A14 for ; Tue, 23 Aug 2022 13:32:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CF097C433C1; Tue, 23 Aug 2022 13:32:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661261538; bh=Dyx3xwqaYs44wgVqzP1C4xeLckS+wOGno6SIfqEVjgo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=W7rRMkapeY4hMqNayrzPwqkwWNMEAeMU6dqdVQ8r/3mdvXJhOsgy/vQGwR9b7WTLh mg9bGLpqdHa+BAJHyYhPqyWHBr1QTb1nUFGcY4gPkysgtpOxHiN9S7ds7ftMUrStHx bN8RwBrP9EiRSzEgyNdnt5bM/0SIeVXqj2XOX/2zvTwFTiNrjiQt3NSlqO9TU/Prh/ x88HkGKFAO1w5wTXeS9stUYNcx4BJS5JsyDHFrppI3+ZhzT4aVOhgTnHMFixACfEtV hoQKsFsc+pXxFkOkHjVHXWL9fH9kfNtWAT3XawJN3iDxnF/XUwpMCjLnn6cWwSkCvf LUuLCmjscttCQ== From: Leon Romanovsky To: Steffen Klassert Cc: Leon Romanovsky , "David S. Miller" , Eric Dumazet , Herbert Xu , Jakub Kicinski , netdev@vger.kernel.org, Paolo Abeni , Raed Salem , Saeed Mahameed Subject: [PATCH xfrm-next v3 3/6] xfrm: add an interface to offload policy Date: Tue, 23 Aug 2022 16:32:00 +0300 Message-Id: <704ab5e848f07c746e6dbd62c5721af5d50f862f.1661260787.git.leonro@nvidia.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Leon Romanovsky Extend netlink interface to add and delete XFRM policy from the device. This functionality is a first step to implement full IPsec offload solution. Signed-off-by: Raed Salem Signed-off-by: Leon Romanovsky --- include/linux/netdevice.h | 3 ++ include/net/xfrm.h | 42 ++++++++++++++++++++++++ net/xfrm/xfrm_device.c | 61 ++++++++++++++++++++++++++++++++++- net/xfrm/xfrm_policy.c | 68 +++++++++++++++++++++++++++++++++++++++ net/xfrm/xfrm_user.c | 17 ++++++++++ 5 files changed, 190 insertions(+), 1 deletion(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 64e8662632f8..f5a49d2b5992 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1012,6 +1012,9 @@ struct xfrmdev_ops { bool (*xdo_dev_offload_ok) (struct sk_buff *skb, struct xfrm_state *x); void (*xdo_dev_state_advance_esn) (struct xfrm_state *x); + int (*xdo_dev_policy_add) (struct xfrm_policy *x); + void (*xdo_dev_policy_delete) (struct xfrm_policy *x); + void (*xdo_dev_policy_free) (struct xfrm_policy *x); }; #endif diff --git a/include/net/xfrm.h b/include/net/xfrm.h index b4d487053dfd..587697eb1d31 100644 --- a/include/net/xfrm.h +++ b/include/net/xfrm.h @@ -129,6 +129,7 @@ struct xfrm_state_walk { enum { XFRM_DEV_OFFLOAD_IN = 1, XFRM_DEV_OFFLOAD_OUT, + XFRM_DEV_OFFLOAD_FWD, }; enum { @@ -534,6 +535,8 @@ struct xfrm_policy { struct xfrm_tmpl xfrm_vec[XFRM_MAX_DEPTH]; struct hlist_node bydst_inexact_list; struct rcu_head rcu; + + struct xfrm_dev_offload xdo; }; static inline struct net *xp_net(const struct xfrm_policy *xp) @@ -1577,6 +1580,7 @@ struct xfrm_state *xfrm_find_acq_byseq(struct net *net, u32 mark, u32 seq); int xfrm_state_delete(struct xfrm_state *x); int xfrm_state_flush(struct net *net, u8 proto, bool task_valid, bool sync); int xfrm_dev_state_flush(struct net *net, struct net_device *dev, bool task_valid); +int xfrm_dev_policy_flush(struct net *net, struct net_device *dev, bool task_valid); void xfrm_sad_getinfo(struct net *net, struct xfrmk_sadinfo *si); void xfrm_spd_getinfo(struct net *net, struct xfrmk_spdinfo *si); u32 xfrm_replay_seqhi(struct xfrm_state *x, __be32 net_seq); @@ -1887,6 +1891,8 @@ void xfrm_dev_backlog(struct softnet_data *sd); struct sk_buff *validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t features, bool *again); int xfrm_dev_state_add(struct net *net, struct xfrm_state *x, struct xfrm_user_offload *xuo); +int xfrm_dev_policy_add(struct net *net, struct xfrm_policy *xp, + struct xfrm_user_offload *xuo, u8 dir); bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x); static inline void xfrm_dev_state_advance_esn(struct xfrm_state *x) @@ -1935,6 +1941,28 @@ static inline void xfrm_dev_state_free(struct xfrm_state *x) netdev_put(dev, &xso->dev_tracker); } } + +static inline void xfrm_dev_policy_delete(struct xfrm_policy *x) +{ + struct xfrm_dev_offload *xdo = &x->xdo; + struct net_device *dev = xdo->dev; + + if (dev && dev->xfrmdev_ops && dev->xfrmdev_ops->xdo_dev_policy_delete) + dev->xfrmdev_ops->xdo_dev_policy_delete(x); +} + +static inline void xfrm_dev_policy_free(struct xfrm_policy *x) +{ + struct xfrm_dev_offload *xdo = &x->xdo; + struct net_device *dev = xdo->dev; + + if (dev && dev->xfrmdev_ops) { + if (dev->xfrmdev_ops->xdo_dev_policy_free) + dev->xfrmdev_ops->xdo_dev_policy_free(x); + xdo->dev = NULL; + netdev_put(dev, &xdo->dev_tracker); + } +} #else static inline void xfrm_dev_resume(struct sk_buff *skb) { @@ -1962,6 +1990,20 @@ static inline void xfrm_dev_state_free(struct xfrm_state *x) { } +static inline int xfrm_dev_policy_add(struct net *net, struct xfrm_policy *xp, + struct xfrm_user_offload *xuo, u8 dir) +{ + return 0; +} + +static inline void xfrm_dev_policy_delete(struct xfrm_policy *x) +{ +} + +static inline void xfrm_dev_policy_free(struct xfrm_policy *x) +{ +} + static inline bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x) { return false; diff --git a/net/xfrm/xfrm_device.c b/net/xfrm/xfrm_device.c index 5b04e5cdca64..1cc482e9c87d 100644 --- a/net/xfrm/xfrm_device.c +++ b/net/xfrm/xfrm_device.c @@ -302,6 +302,63 @@ int xfrm_dev_state_add(struct net *net, struct xfrm_state *x, } EXPORT_SYMBOL_GPL(xfrm_dev_state_add); +int xfrm_dev_policy_add(struct net *net, struct xfrm_policy *xp, + struct xfrm_user_offload *xuo, u8 dir) +{ + struct xfrm_dev_offload *xdo = &xp->xdo; + struct net_device *dev; + int err; + + if (!xuo->flags || xuo->flags & ~XFRM_OFFLOAD_FULL) + /* We support only Full offload mode and it means + * that user must set XFRM_OFFLOAD_FULL bit. + */ + return -EINVAL; + + dev = dev_get_by_index(net, xuo->ifindex); + if (!dev) + return -EINVAL; + + if (!dev->xfrmdev_ops || !dev->xfrmdev_ops->xdo_dev_policy_add) { + xdo->dev = NULL; + dev_put(dev); + return -EINVAL; + } + + xdo->dev = dev; + netdev_tracker_alloc(dev, &xdo->dev_tracker, GFP_ATOMIC); + xdo->real_dev = dev; + xdo->type = XFRM_DEV_OFFLOAD_FULL; + switch (dir) { + case XFRM_POLICY_IN: + xdo->dir = XFRM_DEV_OFFLOAD_IN; + break; + case XFRM_POLICY_OUT: + xdo->dir = XFRM_DEV_OFFLOAD_OUT; + break; + case XFRM_POLICY_FWD: + xdo->dir = XFRM_DEV_OFFLOAD_FWD; + break; + default: + xdo->dev = NULL; + dev_put(dev); + return -EINVAL; + } + + err = dev->xfrmdev_ops->xdo_dev_policy_add(xp); + if (err) { + xdo->dev = NULL; + xdo->real_dev = NULL; + xdo->type = XFRM_DEV_OFFLOAD_UNSPECIFIED; + xdo->dir = 0; + netdev_put(dev, &xdo->dev_tracker); + return err; + } + + return 0; +} +EXPORT_SYMBOL_GPL(xfrm_dev_policy_add); + bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x) { int mtu; @@ -404,8 +461,10 @@ static int xfrm_api_check(struct net_device *dev) static int xfrm_dev_down(struct net_device *dev) { - if (dev->features & NETIF_F_HW_ESP) + if (dev->features & NETIF_F_HW_ESP) { xfrm_dev_state_flush(dev_net(dev), dev, true); + xfrm_dev_policy_flush(dev_net(dev), dev, true); + } return NOTIFY_DONE; } diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c index f1a0bab920a5..4ee422c367f1 100644 --- a/net/xfrm/xfrm_policy.c +++ b/net/xfrm/xfrm_policy.c @@ -425,6 +425,7 @@ void xfrm_policy_destroy(struct xfrm_policy *policy) if (del_timer(&policy->timer) || del_timer(&policy->polq.hold_timer)) BUG(); + xfrm_dev_policy_free(policy); call_rcu(&policy->rcu, xfrm_policy_destroy_rcu); } EXPORT_SYMBOL(xfrm_policy_destroy); @@ -1769,12 +1770,41 @@ xfrm_policy_flush_secctx_check(struct net *net, u8 type, bool task_valid) } return err; } + +static inline int xfrm_dev_policy_flush_secctx_check(struct net *net, + struct net_device *dev, + bool task_valid) +{ + struct xfrm_policy *pol; + int err = 0; + + list_for_each_entry(pol, &net->xfrm.policy_all, walk.all) { + if (pol->walk.dead || + xfrm_policy_id2dir(pol->index) >= XFRM_POLICY_MAX || + pol->xdo.dev != dev) + continue; + + err = security_xfrm_policy_delete(pol->security); + if (err) { + xfrm_audit_policy_delete(pol, 0, task_valid); + return err; + } + } + return err; +} #else static inline int xfrm_policy_flush_secctx_check(struct net *net, u8 type, bool task_valid) { return 0; } + +static inline int xfrm_dev_policy_flush_secctx_check(struct net *net, + struct net_device *dev, + bool task_valid) +{ + return 0; +} #endif int xfrm_policy_flush(struct net *net, u8 type, bool task_valid) @@ -1814,6 +1844,43 @@ int xfrm_policy_flush(struct net *net, u8 type, bool task_valid) } EXPORT_SYMBOL(xfrm_policy_flush); +int xfrm_dev_policy_flush(struct net *net, struct net_device *dev, bool task_valid) +{ + int dir, err = 0, cnt = 0; + struct xfrm_policy *pol; + + spin_lock_bh(&net->xfrm.xfrm_policy_lock); + + err = xfrm_dev_policy_flush_secctx_check(net, dev, task_valid); + if (err) + goto out; + +again: + list_for_each_entry(pol, &net->xfrm.policy_all, walk.all) { + dir = xfrm_policy_id2dir(pol->index); + if (pol->walk.dead || + dir >= XFRM_POLICY_MAX || + pol->xdo.dev != dev) + continue; + + __xfrm_policy_unlink(pol, dir); + spin_unlock_bh(&net->xfrm.xfrm_policy_lock); + cnt++; + xfrm_audit_policy_delete(pol, 1, task_valid); + xfrm_policy_kill(pol); + spin_lock_bh(&net->xfrm.xfrm_policy_lock); + goto again; + } + if (cnt) + __xfrm_policy_inexact_flush(net); + else + err = -ESRCH; +out: + spin_unlock_bh(&net->xfrm.xfrm_policy_lock); + return err; +} +EXPORT_SYMBOL(xfrm_dev_policy_flush); + int xfrm_policy_walk(struct net *net, struct xfrm_policy_walk *walk, int (*func)(struct xfrm_policy *, int, int, void*), void *data) @@ -2246,6 +2313,7 @@ int xfrm_policy_delete(struct xfrm_policy *pol, int dir) pol = __xfrm_policy_unlink(pol, dir); spin_unlock_bh(&net->xfrm.xfrm_policy_lock); if (pol) { + xfrm_dev_policy_delete(pol); xfrm_policy_kill(pol); return 0; } diff --git a/net/xfrm/xfrm_user.c b/net/xfrm/xfrm_user.c index 9c0aef815730..698ff84da6ba 100644 --- a/net/xfrm/xfrm_user.c +++ b/net/xfrm/xfrm_user.c @@ -1747,6 +1747,14 @@ static struct xfrm_policy *xfrm_policy_construct(struct net *net, struct xfrm_us if (attrs[XFRMA_IF_ID]) xp->if_id = nla_get_u32(attrs[XFRMA_IF_ID]); + /* configure the hardware if offload is requested */ + if (attrs[XFRMA_OFFLOAD_DEV]) { + err = xfrm_dev_policy_add(net, xp, + nla_data(attrs[XFRMA_OFFLOAD_DEV]), p->dir); + if (err) + goto error; + } + return xp; error: *errp = err; @@ -1785,6 +1793,7 @@ static int xfrm_add_policy(struct sk_buff *skb, struct nlmsghdr *nlh, xfrm_audit_policy_add(xp, err ? 0 : 1, true); if (err) { + xfrm_dev_policy_delete(xp); security_xfrm_policy_free(xp->security); kfree(xp); return err; @@ -1897,6 +1906,8 @@ static int dump_one_policy(struct xfrm_policy *xp, int dir, int count, void *ptr err = xfrm_mark_put(skb, &xp->mark); if (!err) err = xfrm_if_id_put(skb, xp->if_id); + if (!err && xp->xdo.dev) + err = copy_user_offload(&xp->xdo, skb); if (err) { nlmsg_cancel(skb, nlh); return err; @@ -3213,6 +3224,8 @@ static int build_acquire(struct sk_buff *skb, struct xfrm_state *x, err = xfrm_mark_put(skb, &xp->mark); if (!err) err = xfrm_if_id_put(skb, xp->if_id); + if (!err && xp->xdo.dev) + err = copy_user_offload(&xp->xdo, skb); if (err) { nlmsg_cancel(skb, nlh); return err; @@ -3331,6 +3344,8 @@ static int build_polexpire(struct sk_buff *skb, struct xfrm_policy *xp, err = xfrm_mark_put(skb, &xp->mark); if (!err) err = xfrm_if_id_put(skb, xp->if_id); + if (!err && xp->xdo.dev) + err = copy_user_offload(&xp->xdo, skb); if (err) { nlmsg_cancel(skb, nlh); return err; @@ -3414,6 +3429,8 @@ static int xfrm_notify_policy(struct xfrm_policy *xp, int dir, const struct km_e err = xfrm_mark_put(skb, &xp->mark); if (!err) err = xfrm_if_id_put(skb, xp->if_id); + if (!err && xp->xdo.dev) + err = copy_user_offload(&xp->xdo, skb); if (err) goto out_free_skb; From patchwork Tue Aug 23 13:32:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 12952320 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D5BCC32789 for ; Tue, 23 Aug 2022 17:03:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245061AbiHWRDm (ORCPT ); Tue, 23 Aug 2022 13:03:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343764AbiHWRBh (ORCPT ); Tue, 23 Aug 2022 13:01:37 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DFF2114EB97 for ; Tue, 23 Aug 2022 06:32:36 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 42A6CB81D62 for ; Tue, 23 Aug 2022 13:32:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 82697C433D6; Tue, 23 Aug 2022 13:32:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661261554; bh=77fIqVq6GyQ2HZOS+UyBZdm7wbS+2TJKlvRiomjkWxs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=faLYxZnPWYDMi5mqRdGmxLqsJyDJ+Gyc9O2OgLsldLVJqJr74BqkVlvVefJiCgQPd SEuY2aQjVKPbEwT0stQUqqAZDklj/pM+SMnnqJ89YSAbVylCFYQFgtHQsr6ddGA1LS tcjvjiCGdhnh4ozqwhCxSY1+JF3xyBydFOTgpNJHU+P1YZ/SlMxA7cbLa/ZbEl3lF+ Ej7trJZk9G0AWPxcxmNHQ+zpUP2w4tYGPJSybpVB1hHU0F4vnnZyzAixpDOTXBD9EG Nr5MXBEOqOPpq4XiCztiVlQuXIOw5mO6RDLvJIbGClWL2L/tsOfm3qnhteaN7VEKOa 8PZViQSXZ7nmg== From: Leon Romanovsky To: Steffen Klassert Cc: Leon Romanovsky , "David S. Miller" , Eric Dumazet , Herbert Xu , Jakub Kicinski , netdev@vger.kernel.org, Paolo Abeni , Raed Salem , Saeed Mahameed Subject: [PATCH xfrm-next v3 4/6] xfrm: add TX datapath support for IPsec full offload mode Date: Tue, 23 Aug 2022 16:32:01 +0300 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Leon Romanovsky In IPsec full mode, the device is going to encrypt and encapsulate packets that are associated with offloaded policy. After successful policy lookup to indicate if packets should be offloaded or not, the stack forwards packets to the device to do the magic. Signed-off-by: Raed Salem Signed-off-by: Huy Nguyen Signed-off-by: Leon Romanovsky --- net/xfrm/xfrm_device.c | 14 ++++++++++++-- net/xfrm/xfrm_output.c | 12 +++++++++++- 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/net/xfrm/xfrm_device.c b/net/xfrm/xfrm_device.c index 1cc482e9c87d..2f0c2c6a1ed6 100644 --- a/net/xfrm/xfrm_device.c +++ b/net/xfrm/xfrm_device.c @@ -120,6 +120,16 @@ struct sk_buff *validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t featur if (xo->flags & XFRM_GRO || x->xso.dir == XFRM_DEV_OFFLOAD_IN) return skb; + /* The packet was sent to HW IPsec full offload engine, + * but to wrong device. Drop the packet, so it won't skip + * XFRM stack. + */ + if (x->xso.type == XFRM_DEV_OFFLOAD_FULL && x->xso.dev != dev) { + kfree_skb(skb); + dev_core_stats_tx_dropped_inc(dev); + return NULL; + } + /* This skb was already validated on the upper/virtual dev */ if ((x->xso.dev != dev) && (x->xso.real_dev == dev)) return skb; @@ -369,8 +379,8 @@ bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x) if (!x->type_offload || x->encap) return false; - if ((!dev || (dev == xfrm_dst_path(dst)->dev)) && - (!xdst->child->xfrm)) { + if (x->xso.type == XFRM_DEV_OFFLOAD_FULL || + (!dev || (dev == xfrm_dst_path(dst)->dev)) && !xdst->child->xfrm) { mtu = xfrm_state_mtu(x, xdst->child_mtu_cached); if (skb->len <= mtu) goto ok; diff --git a/net/xfrm/xfrm_output.c b/net/xfrm/xfrm_output.c index 555ab35cd119..b554a98926bd 100644 --- a/net/xfrm/xfrm_output.c +++ b/net/xfrm/xfrm_output.c @@ -494,7 +494,7 @@ static int xfrm_output_one(struct sk_buff *skb, int err) struct xfrm_state *x = dst->xfrm; struct net *net = xs_net(x); - if (err <= 0) + if (err <= 0 || x->xso.type == XFRM_DEV_OFFLOAD_FULL) goto resume; do { @@ -719,6 +719,16 @@ int xfrm_output(struct sock *sk, struct sk_buff *skb) break; } + if (x->xso.type == XFRM_DEV_OFFLOAD_FULL) { + if (!xfrm_dev_offload_ok(skb, x)) { + XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTERROR); + kfree_skb(skb); + return -EHOSTUNREACH; + } + + return xfrm_output_resume(sk, skb, 0); + } + secpath_reset(skb); if (xfrm_dev_offload_ok(skb, x)) { From patchwork Tue Aug 23 13:32:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 12952317 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA100C32772 for ; Tue, 23 Aug 2022 17:02:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343742AbiHWRC1 (ORCPT ); Tue, 23 Aug 2022 13:02:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35912 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344433AbiHWRBT (ORCPT ); Tue, 23 Aug 2022 13:01:19 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D8628A570D for ; Tue, 23 Aug 2022 06:32:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 96375B81D65 for ; Tue, 23 Aug 2022 13:32:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D9BC7C433D6; Tue, 23 Aug 2022 13:32:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661261546; bh=KiTQ4803ADQd032O3gfZoeapnos5edX29xoFCbOnWRE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JOi5ULHRzEcLtXWaW7J24618vMstFP46qRnPFL4OvpSuq22Obd82cDLPlQZ36VZlA W6lupwbX3XWTc09pws7PxtfFdtM8T0QRrktMIakEjHLmkPTp5NORknyllCkKzcydyB LC3DrD4DZVDByTtbYtXPjdeKrG6FETIxdME3l5pHqEDk05JifoK9SUjMoBxPNfInWw hrWwzVLjzFUo1ZysCgUg8d0A66Z2bfzppCdWFp6hyO2iUOk8yC6PO6Z/ydcJuHw/6z Hc8E9fPsSrizO94Eg7zEC3Cr3+0jMpJZK2kUiaywZLBEYiN0hpa+IJXu8zFcGRtcBU xv5eDQ+aM8acg== From: Leon Romanovsky To: Steffen Klassert Cc: Leon Romanovsky , "David S. Miller" , Eric Dumazet , Herbert Xu , Jakub Kicinski , netdev@vger.kernel.org, Paolo Abeni , Raed Salem , Saeed Mahameed Subject: [PATCH xfrm-next v3 5/6] xfrm: add RX datapath protection for IPsec full offload mode Date: Tue, 23 Aug 2022 16:32:02 +0300 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Leon Romanovsky Traffic received by device with enabled IPsec full offload should be forwarded to the stack only after decryption, packet headers and trailers removed. Such packets are expected to be seen as normal (non-XFRM) ones, while not-supported packets should be dropped by the HW. Reviewed-by: Raed Salem Signed-off-by: Leon Romanovsky --- include/net/xfrm.h | 55 +++++++++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 23 deletions(-) diff --git a/include/net/xfrm.h b/include/net/xfrm.h index 587697eb1d31..38fff78a1421 100644 --- a/include/net/xfrm.h +++ b/include/net/xfrm.h @@ -1094,6 +1094,29 @@ xfrm_state_addr_cmp(const struct xfrm_tmpl *tmpl, const struct xfrm_state *x, un return !0; } +#ifdef CONFIG_XFRM +static inline struct xfrm_state *xfrm_input_state(struct sk_buff *skb) +{ + struct sec_path *sp = skb_sec_path(skb); + + return sp->xvec[sp->len - 1]; +} +#endif + +static inline struct xfrm_offload *xfrm_offload(struct sk_buff *skb) +{ +#ifdef CONFIG_XFRM + struct sec_path *sp = skb_sec_path(skb); + + if (!sp || !sp->olen || sp->len != sp->olen) + return NULL; + + return &sp->ovec[sp->olen - 1]; +#else + return NULL; +#endif +} + #ifdef CONFIG_XFRM int __xfrm_policy_check(struct sock *, int dir, struct sk_buff *skb, unsigned short family); @@ -1125,10 +1148,19 @@ static inline int __xfrm_policy_check2(struct sock *sk, int dir, { struct net *net = dev_net(skb->dev); int ndir = dir | (reverse ? XFRM_POLICY_MASK + 1 : 0); + struct xfrm_offload *xo = xfrm_offload(skb); + struct xfrm_state *x; if (sk && sk->sk_policy[XFRM_POLICY_IN]) return __xfrm_policy_check(sk, ndir, skb, family); + if (xo) { + x = xfrm_input_state(skb); + if (x->xso.type == XFRM_DEV_OFFLOAD_FULL) + return (xo->flags & CRYPTO_DONE) && + (xo->status & CRYPTO_SUCCESS); + } + return __xfrm_check_nopolicy(net, skb, dir) || __xfrm_check_dev_nopolicy(skb, dir, family) || __xfrm_policy_check(sk, ndir, skb, family); @@ -1860,29 +1892,6 @@ static inline void xfrm_states_delete(struct xfrm_state **states, int n) } #endif -#ifdef CONFIG_XFRM -static inline struct xfrm_state *xfrm_input_state(struct sk_buff *skb) -{ - struct sec_path *sp = skb_sec_path(skb); - - return sp->xvec[sp->len - 1]; -} -#endif - -static inline struct xfrm_offload *xfrm_offload(struct sk_buff *skb) -{ -#ifdef CONFIG_XFRM - struct sec_path *sp = skb_sec_path(skb); - - if (!sp || !sp->olen || sp->len != sp->olen) - return NULL; - - return &sp->ovec[sp->olen - 1]; -#else - return NULL; -#endif -} - void __init xfrm_dev_init(void); #ifdef CONFIG_XFRM_OFFLOAD From patchwork Tue Aug 23 13:32:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 12952319 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EDBAC32772 for ; Tue, 23 Aug 2022 17:03:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343881AbiHWRDl (ORCPT ); Tue, 23 Aug 2022 13:03:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343659AbiHWRBg (ORCPT ); Tue, 23 Aug 2022 13:01:36 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AB3226E8 for ; Tue, 23 Aug 2022 06:32:31 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CB72361514 for ; Tue, 23 Aug 2022 13:32:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AF7D5C433D7; Tue, 23 Aug 2022 13:32:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1661261550; bh=iqQOZftE3IP3ZfMWVhZ1sFUZszVyi6JBDfKDwAljrj4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hVBmpuAR9h7D/BRi1ffsg2335wy+4QosB0e/tZuCFPIWkNv+37DoNqWHT/VtyTWUV KsJIHjhPQcT+5mC+QiEb1uUfZ6Sw8437SjUHoQhcIjTWFrTugm8cXOh2FqCn9UMzJh TppNT9cVY9/lJWNynmIHPHCts+xWQQVPDcXtyYJdGdauz3pFRVg+kAkv2n7tguCPSi P4Rtd5ZBI7prdvQk3j79Z4IVk+r9+n65LuTMUchZPbPkDHYv6Md7AB+JIYYzbBg6bk jzpbTcWyNgnSq1VzIYOAL/nflIf5QwMA/JODlF3wDCawpWB4FPgnQTMh0DC9GNZcGi k3cWXWxHj1eUg== From: Leon Romanovsky To: Steffen Klassert Cc: Leon Romanovsky , "David S. Miller" , Eric Dumazet , Herbert Xu , Jakub Kicinski , netdev@vger.kernel.org, Paolo Abeni , Raed Salem , Saeed Mahameed Subject: [PATCH xfrm-next v3 6/6] xfrm: enforce separation between priorities of HW/SW policies Date: Tue, 23 Aug 2022 16:32:03 +0300 Message-Id: <15fe70f6d6702c72b29d50d856e1a5514c56c673.1661260787.git.leonro@nvidia.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Leon Romanovsky Devices that implement IPsec full offload mode offload policies too. In RX path, it causes to the situation that HW can't effectively handle mixed SW and HW priorities unless users make sure that HW offloaded policies have higher priorities. In order to make sure that users have coherent picture, let's require that HW offloaded policies have always (both RX and TX) higher priorities than SW ones. To do not over engineer the code, HW policies are treated as SW ones and don't take into account netdev to allow reuse of same priorities for different devices. Reviewed-by: Raed Salem Signed-off-by: Leon Romanovsky --- include/net/netns/xfrm.h | 8 ++- net/xfrm/xfrm_policy.c | 113 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 120 insertions(+), 1 deletion(-) diff --git a/include/net/netns/xfrm.h b/include/net/netns/xfrm.h index bd7c3be4af5d..f0cfa0faf611 100644 --- a/include/net/netns/xfrm.h +++ b/include/net/netns/xfrm.h @@ -29,6 +29,11 @@ struct xfrm_policy_hthresh { u8 rbits6; }; +struct xfrm_policy_prio { + u32 max_sw_prio; + u32 min_hw_prio; +}; + struct netns_xfrm { struct list_head state_all; /* @@ -52,7 +57,7 @@ struct netns_xfrm { unsigned int policy_idx_hmask; struct hlist_head policy_inexact[XFRM_POLICY_MAX]; struct xfrm_policy_hash policy_bydst[XFRM_POLICY_MAX]; - unsigned int policy_count[XFRM_POLICY_MAX * 2]; + unsigned int policy_count[XFRM_POLICY_MAX * 3]; struct work_struct policy_hash_work; struct xfrm_policy_hthresh policy_hthresh; struct list_head inexact_bins; @@ -67,6 +72,7 @@ struct netns_xfrm { u32 sysctl_acq_expires; u8 policy_default[XFRM_POLICY_MAX]; + struct xfrm_policy_prio policy_prio[XFRM_POLICY_MAX]; #ifdef CONFIG_SYSCTL struct ctl_table_header *sysctl_hdr; diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c index 4ee422c367f1..28018691c6b3 100644 --- a/net/xfrm/xfrm_policy.c +++ b/net/xfrm/xfrm_policy.c @@ -1570,13 +1570,70 @@ static struct xfrm_policy *xfrm_policy_insert_list(struct hlist_head *chain, return delpol; } +static int __xfrm_policy_check_hw_priority(struct net *net, + struct xfrm_policy *policy, int dir) +{ + int left, right; + + lockdep_assert_held(&net->xfrm.xfrm_policy_lock); + + if (!net->xfrm.policy_count[dir]) + /* Adding first policy */ + return 0; + + if (policy->xdo.type != XFRM_DEV_OFFLOAD_FULL) { + /* SW priority */ + if (!net->xfrm.policy_count[2 * XFRM_POLICY_MAX + dir]) + /* Special case to allow reuse maximum priority + * (U32_MAX) for SW policies, when no HW policy exist. + */ + return 0; + + left = policy->priority; + right = net->xfrm.policy_prio[dir].min_hw_prio; + } else { + /* HW priority */ + left = net->xfrm.policy_prio[dir].max_sw_prio; + right = policy->priority; + } + if (left >= right) + return -EINVAL; + + return 0; +} + +static void __xfrm_policy_update_hw_priority(struct net *net, + struct xfrm_policy *policy, + int dir) +{ + u32 *hw_prio, *sw_prio; + + lockdep_assert_held(&net->xfrm.xfrm_policy_lock); + + if (policy->xdo.type != XFRM_DEV_OFFLOAD_FULL) { + sw_prio = &net->xfrm.policy_prio[dir].max_sw_prio; + *sw_prio = max(*sw_prio, policy->priority); + return; + } + + hw_prio = &net->xfrm.policy_prio[dir].min_hw_prio; + *hw_prio = min(*hw_prio, policy->priority); +} + int xfrm_policy_insert(int dir, struct xfrm_policy *policy, int excl) { struct net *net = xp_net(policy); struct xfrm_policy *delpol; struct hlist_head *chain; + int ret; spin_lock_bh(&net->xfrm.xfrm_policy_lock); + ret = __xfrm_policy_check_hw_priority(net, policy, dir); + if (ret) { + spin_unlock_bh(&net->xfrm.xfrm_policy_lock); + return ret; + } + chain = policy_hash_bysel(net, &policy->selector, policy->family, dir); if (chain) delpol = xfrm_policy_insert_list(chain, policy, excl); @@ -1606,6 +1663,7 @@ int xfrm_policy_insert(int dir, struct xfrm_policy *policy, int excl) policy->curlft.use_time = 0; if (!mod_timer(&policy->timer, jiffies + HZ)) xfrm_pol_hold(policy); + __xfrm_policy_update_hw_priority(net, policy, dir); spin_unlock_bh(&net->xfrm.xfrm_policy_lock); if (delpol) @@ -2271,6 +2329,8 @@ static void __xfrm_policy_link(struct xfrm_policy *pol, int dir) list_add(&pol->walk.all, &net->xfrm.policy_all); net->xfrm.policy_count[dir]++; + if (pol->xdo.type == XFRM_DEV_OFFLOAD_FULL) + net->xfrm.policy_count[2 * XFRM_POLICY_MAX + dir]++; xfrm_pol_hold(pol); } @@ -2290,6 +2350,8 @@ static struct xfrm_policy *__xfrm_policy_unlink(struct xfrm_policy *pol, } list_del_init(&pol->walk.all); + if (pol->xdo.type == XFRM_DEV_OFFLOAD_FULL) + net->xfrm.policy_count[2 * XFRM_POLICY_MAX + dir]--; net->xfrm.policy_count[dir]--; return pol; @@ -2305,12 +2367,58 @@ static void xfrm_sk_policy_unlink(struct xfrm_policy *pol, int dir) __xfrm_policy_unlink(pol, XFRM_POLICY_MAX + dir); } +static void __xfrm_policy_delete_prio(struct net *net, + struct xfrm_policy *policy, int dir) +{ + struct xfrm_policy *pol; + u32 sw_prio = 0; + + lockdep_assert_held(&net->xfrm.xfrm_policy_lock); + + if (!net->xfrm.policy_count[dir]) { + net->xfrm.policy_prio[dir].max_sw_prio = sw_prio; + net->xfrm.policy_prio[dir].min_hw_prio = U32_MAX; + return; + } + + if (policy->xdo.type == XFRM_DEV_OFFLOAD_FULL && + !net->xfrm.policy_count[2 * XFRM_POLICY_MAX + dir]) { + net->xfrm.policy_prio[dir].min_hw_prio = U32_MAX; + return; + } + + list_for_each_entry(pol, &net->xfrm.policy_all, walk.all) { + if (pol->walk.dead) + continue; + + if (policy->xdo.type != XFRM_DEV_OFFLOAD_FULL) { + /* SW priority */ + if (pol->xdo.type == XFRM_DEV_OFFLOAD_FULL) { + net->xfrm.policy_prio[dir].max_sw_prio = sw_prio; + return; + } + sw_prio = pol->priority; + continue; + } + /* HW priority */ + if (pol->xdo.type != XFRM_DEV_OFFLOAD_FULL) + continue; + + net->xfrm.policy_prio[dir].min_hw_prio = pol->priority; + return; + } + + net->xfrm.policy_prio[dir].max_sw_prio = sw_prio; +} + int xfrm_policy_delete(struct xfrm_policy *pol, int dir) { struct net *net = xp_net(pol); spin_lock_bh(&net->xfrm.xfrm_policy_lock); pol = __xfrm_policy_unlink(pol, dir); + if (pol) + __xfrm_policy_delete_prio(net, pol, dir); spin_unlock_bh(&net->xfrm.xfrm_policy_lock); if (pol) { xfrm_dev_policy_delete(pol); @@ -4111,6 +4219,7 @@ static int __net_init xfrm_policy_init(struct net *net) net->xfrm.policy_count[dir] = 0; net->xfrm.policy_count[XFRM_POLICY_MAX + dir] = 0; + net->xfrm.policy_count[2 * XFRM_POLICY_MAX + dir] = 0; INIT_HLIST_HEAD(&net->xfrm.policy_inexact[dir]); htab = &net->xfrm.policy_bydst[dir]; @@ -4196,6 +4305,10 @@ static int __net_init xfrm_net_init(struct net *net) net->xfrm.policy_default[XFRM_POLICY_FWD] = XFRM_USERPOLICY_ACCEPT; net->xfrm.policy_default[XFRM_POLICY_OUT] = XFRM_USERPOLICY_ACCEPT; + net->xfrm.policy_prio[XFRM_POLICY_IN].min_hw_prio = U32_MAX; + net->xfrm.policy_prio[XFRM_POLICY_FWD].min_hw_prio = U32_MAX; + net->xfrm.policy_prio[XFRM_POLICY_OUT].min_hw_prio = U32_MAX; + rv = xfrm_statistics_init(net); if (rv < 0) goto out_statistics;