From patchwork Thu Oct 6 08:59:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhu Yanjun X-Patchwork-Id: 12999444 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 593D5C43219 for ; Wed, 5 Oct 2022 16:33:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230117AbiJEQdM (ORCPT ); Wed, 5 Oct 2022 12:33:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50362 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229950AbiJEQdI (ORCPT ); Wed, 5 Oct 2022 12:33:08 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 06A467D7AC; Wed, 5 Oct 2022 09:33:07 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="303182095" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="303182095" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Oct 2022 09:33:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="619518073" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="619518073" Received: from unknown (HELO intel-71.bj.intel.com) ([10.238.154.71]) by orsmga007.jf.intel.com with ESMTP; 05 Oct 2022 09:33:04 -0700 From: yanjun.zhu@linux.dev To: jgg@ziepe.ca, leon@kernel.org, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev, netdev@vger.kernel.org, davem@davemloft.net Subject: [PATCHv2 1/6] RDMA/rxe: Creating listening sock in newlink function Date: Thu, 6 Oct 2022 04:59:16 -0400 Message-Id: <20221006085921.1323148-2-yanjun.zhu@linux.dev> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20221006085921.1323148-1-yanjun.zhu@linux.dev> References: <20221006085921.1323148-1-yanjun.zhu@linux.dev> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Zhu Yanjun Originally when the module rdma_rxe is loaded, the sock listening on udp port 4791 is created. Currently moving the creating listening port to newlink function. So when running "rdma link add" command, the sock listening on udp port 4791 is created. Signed-off-by: Zhu Yanjun --- drivers/infiniband/sw/rxe/rxe.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 51daac5c4feb..a22ff2207b42 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -192,6 +192,10 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) goto err; } + err = rxe_net_init(); + if (err) + return err; + err = rxe_net_add(ibdev_name, ndev); if (err) { pr_err("failed to add %s\n", ndev->name); @@ -210,10 +214,6 @@ static int __init rxe_module_init(void) { int err; - err = rxe_net_init(); - if (err) - return err; - rdma_link_register(&rxe_link_ops); pr_info("loaded\n"); return 0; From patchwork Thu Oct 6 08:59:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhu Yanjun X-Patchwork-Id: 12999445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C219C4167B for ; Wed, 5 Oct 2022 16:33:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230196AbiJEQdP (ORCPT ); Wed, 5 Oct 2022 12:33:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230040AbiJEQdK (ORCPT ); Wed, 5 Oct 2022 12:33:10 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A7987DF7F; Wed, 5 Oct 2022 09:33:09 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="303182104" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="303182104" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Oct 2022 09:33:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="619518092" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="619518092" Received: from unknown (HELO intel-71.bj.intel.com) ([10.238.154.71]) by orsmga007.jf.intel.com with ESMTP; 05 Oct 2022 09:33:06 -0700 From: yanjun.zhu@linux.dev To: jgg@ziepe.ca, leon@kernel.org, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev, netdev@vger.kernel.org, davem@davemloft.net Subject: [PATCHv2 2/6] RDMA/rxe: Support more rdma links in init_net Date: Thu, 6 Oct 2022 04:59:17 -0400 Message-Id: <20221006085921.1323148-3-yanjun.zhu@linux.dev> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20221006085921.1323148-1-yanjun.zhu@linux.dev> References: <20221006085921.1323148-1-yanjun.zhu@linux.dev> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Zhu Yanjun In init_net, when several rdma links are created with the command "rdma link add", newlink will check whether the udp port 4791 is listening or not. If not, creating a sock listening on udp port 4791. If yes, increasing the reference count of the sock. Signed-off-by: Zhu Yanjun --- drivers/infiniband/sw/rxe/rxe.c | 9 ++++- drivers/infiniband/sw/rxe/rxe_net.c | 55 +++++++++++++++++++++-------- drivers/infiniband/sw/rxe/rxe_net.h | 1 + 3 files changed, 49 insertions(+), 16 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index a22ff2207b42..84a07638f8df 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -8,6 +8,7 @@ #include #include "rxe.h" #include "rxe_loc.h" +#include "rxe_net.h" MODULE_AUTHOR("Bob Pearson, Frank Zago, John Groves, Kamal Heib"); MODULE_DESCRIPTION("Soft RDMA transport"); @@ -205,7 +206,7 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) return err; } -static struct rdma_link_ops rxe_link_ops = { +struct rdma_link_ops rxe_link_ops = { .type = "rxe", .newlink = rxe_newlink, }; @@ -215,6 +216,12 @@ static int __init rxe_module_init(void) int err; rdma_link_register(&rxe_link_ops); + err = rxe_register_notifier(); + if (err) { + pr_err("Failed to register netdev notifier\n"); + return -1; + } + pr_info("loaded\n"); return 0; } diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index c53f4529f098..4772ea19c6e2 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -620,13 +620,23 @@ static struct notifier_block rxe_net_notifier = { static int rxe_net_ipv4_init(void) { - recv_sockets.sk4 = rxe_setup_udp_tunnel(&init_net, - htons(ROCE_V2_UDP_DPORT), false); - if (IS_ERR(recv_sockets.sk4)) { - recv_sockets.sk4 = NULL; + struct sock *sk; + struct socket *sock; + + rcu_read_lock(); + sk = udp4_lib_lookup(&init_net, 0, 0, htonl(INADDR_ANY), + htons(ROCE_V2_UDP_DPORT), 0); + rcu_read_unlock(); + if (sk) + return 0; + + sock = rxe_setup_udp_tunnel(&init_net, htons(ROCE_V2_UDP_DPORT), false); + if (IS_ERR(sock)) { pr_err("Failed to create IPv4 UDP tunnel\n"); + recv_sockets.sk4 = NULL; return -1; } + recv_sockets.sk4 = sock; return 0; } @@ -634,24 +644,46 @@ static int rxe_net_ipv4_init(void) static int rxe_net_ipv6_init(void) { #if IS_ENABLED(CONFIG_IPV6) + struct sock *sk; + struct socket *sock; + + rcu_read_lock(); + sk = udp6_lib_lookup(&init_net, NULL, 0, &in6addr_any, + htons(ROCE_V2_UDP_DPORT), 0); + rcu_read_unlock(); + if (sk) + return 0; - recv_sockets.sk6 = rxe_setup_udp_tunnel(&init_net, - htons(ROCE_V2_UDP_DPORT), true); - if (PTR_ERR(recv_sockets.sk6) == -EAFNOSUPPORT) { + sock = rxe_setup_udp_tunnel(&init_net, htons(ROCE_V2_UDP_DPORT), true); + if (PTR_ERR(sock) == -EAFNOSUPPORT) { recv_sockets.sk6 = NULL; pr_warn("IPv6 is not supported, can not create a UDPv6 socket\n"); return 0; } - if (IS_ERR(recv_sockets.sk6)) { + if (IS_ERR(sock)) { recv_sockets.sk6 = NULL; pr_err("Failed to create IPv6 UDP tunnel\n"); return -1; } + recv_sockets.sk6 = sock; #endif return 0; } +int rxe_register_notifier(void) +{ + int err; + + err = register_netdevice_notifier(&rxe_net_notifier); + if (err) { + pr_err("Failed to register netdev notifier\n"); + return -1; + } + + return 0; +} + void rxe_net_exit(void) { rxe_release_udp_tunnel(recv_sockets.sk6); @@ -663,19 +695,12 @@ int rxe_net_init(void) { int err; - recv_sockets.sk6 = NULL; - err = rxe_net_ipv4_init(); if (err) return err; err = rxe_net_ipv6_init(); if (err) goto err_out; - err = register_netdevice_notifier(&rxe_net_notifier); - if (err) { - pr_err("Failed to register netdev notifier\n"); - goto err_out; - } return 0; err_out: rxe_net_exit(); diff --git a/drivers/infiniband/sw/rxe/rxe_net.h b/drivers/infiniband/sw/rxe/rxe_net.h index 45d80d00f86b..a222c3eeae12 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.h +++ b/drivers/infiniband/sw/rxe/rxe_net.h @@ -18,6 +18,7 @@ struct rxe_recv_sockets { int rxe_net_add(const char *ibdev_name, struct net_device *ndev); +int rxe_register_notifier(void); int rxe_net_init(void); void rxe_net_exit(void); From patchwork Thu Oct 6 08:59:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhu Yanjun X-Patchwork-Id: 12999447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1E98C4332F for ; Wed, 5 Oct 2022 16:33:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230216AbiJEQdY (ORCPT ); Wed, 5 Oct 2022 12:33:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229998AbiJEQdM (ORCPT ); Wed, 5 Oct 2022 12:33:12 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1259A7D783; Wed, 5 Oct 2022 09:33:11 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="303182113" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="303182113" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Oct 2022 09:33:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="619518107" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="619518107" Received: from unknown (HELO intel-71.bj.intel.com) ([10.238.154.71]) by orsmga007.jf.intel.com with ESMTP; 05 Oct 2022 09:33:08 -0700 From: yanjun.zhu@linux.dev To: jgg@ziepe.ca, leon@kernel.org, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev, netdev@vger.kernel.org, davem@davemloft.net Subject: [PATCHv2 3/6] RDMA/nldev: Add dellink function pointer Date: Thu, 6 Oct 2022 04:59:18 -0400 Message-Id: <20221006085921.1323148-4-yanjun.zhu@linux.dev> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20221006085921.1323148-1-yanjun.zhu@linux.dev> References: <20221006085921.1323148-1-yanjun.zhu@linux.dev> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Zhu Yanjun The newlink function pointer is added. And the sock listening on port 4791 is added in the newlink function. So the dellink function is needed to remove the sock. Signed-off-by: Zhu Yanjun --- drivers/infiniband/core/nldev.c | 6 ++++++ include/rdma/rdma_netlink.h | 2 ++ 2 files changed, 8 insertions(+) diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c index b92358f606d0..ae0db4aced34 100644 --- a/drivers/infiniband/core/nldev.c +++ b/drivers/infiniband/core/nldev.c @@ -1744,6 +1744,12 @@ static int nldev_dellink(struct sk_buff *skb, struct nlmsghdr *nlh, return -EINVAL; } + if (device->link_ops) { + err = device->link_ops->dellink(device); + if (err) + return err; + } + ib_unregister_device_and_put(device); return 0; } diff --git a/include/rdma/rdma_netlink.h b/include/rdma/rdma_netlink.h index c2a79aeee113..bf9df004061f 100644 --- a/include/rdma/rdma_netlink.h +++ b/include/rdma/rdma_netlink.h @@ -5,6 +5,7 @@ #include #include +#include enum { RDMA_NLDEV_ATTR_EMPTY_STRING = 1, @@ -114,6 +115,7 @@ struct rdma_link_ops { struct list_head list; const char *type; int (*newlink)(const char *ibdev_name, struct net_device *ndev); + int (*dellink)(struct ib_device *dev); }; void rdma_link_register(struct rdma_link_ops *ops); From patchwork Thu Oct 6 08:59:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhu Yanjun X-Patchwork-Id: 12999448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30E57C433FE for ; Wed, 5 Oct 2022 16:33:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230184AbiJEQdg (ORCPT ); Wed, 5 Oct 2022 12:33:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50362 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230176AbiJEQdO (ORCPT ); Wed, 5 Oct 2022 12:33:14 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F74E7E002; Wed, 5 Oct 2022 09:33:13 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="303182118" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="303182118" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Oct 2022 09:33:13 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="619518138" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="619518138" Received: from unknown (HELO intel-71.bj.intel.com) ([10.238.154.71]) by orsmga007.jf.intel.com with ESMTP; 05 Oct 2022 09:33:10 -0700 From: yanjun.zhu@linux.dev To: jgg@ziepe.ca, leon@kernel.org, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev, netdev@vger.kernel.org, davem@davemloft.net Subject: [PATCHv2 4/6] RDMA/rxe: Implement dellink in rxe Date: Thu, 6 Oct 2022 04:59:19 -0400 Message-Id: <20221006085921.1323148-5-yanjun.zhu@linux.dev> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20221006085921.1323148-1-yanjun.zhu@linux.dev> References: <20221006085921.1323148-1-yanjun.zhu@linux.dev> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Zhu Yanjun When running "rdma link del" command, dellink function will be called. If the sock refcnt is greater than the refcnt needed for udp tunnel, the sock refcnt will be decreased by 1. If equal, the last rdma link is deleted. The udp tunnel will be destroyed. Signed-off-by: Zhu Yanjun --- drivers/infiniband/sw/rxe/rxe.c | 12 +++++++++++- drivers/infiniband/sw/rxe/rxe_net.c | 16 ++++++++++++++-- drivers/infiniband/sw/rxe/rxe_net.h | 1 + 3 files changed, 26 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 84a07638f8df..1b8b74ea84e9 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -166,10 +166,12 @@ void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu) /* called by ifc layer to create new rxe device. * The caller should allocate memory for rxe by calling ib_alloc_device. */ +static struct rdma_link_ops rxe_link_ops; int rxe_add(struct rxe_dev *rxe, unsigned int mtu, const char *ibdev_name) { rxe_init(rxe); rxe_set_mtu(rxe, mtu); + rxe->ib_dev.link_ops = &rxe_link_ops; return rxe_register_device(rxe, ibdev_name); } @@ -206,9 +208,17 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) return err; } -struct rdma_link_ops rxe_link_ops = { +static int rxe_dellink(struct ib_device *dev) +{ + rxe_net_del(dev); + + return 0; +} + +static struct rdma_link_ops rxe_link_ops = { .type = "rxe", .newlink = rxe_newlink, + .dellink = rxe_dellink, }; static int __init rxe_module_init(void) diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index 4772ea19c6e2..6e35566e933b 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -527,6 +527,20 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev) return 0; } +#define SK_REF_FOR_TUNNEL 2 +void rxe_net_del(struct ib_device *dev) +{ + if (refcount_read(&recv_sockets.sk6->sk->sk_refcnt) > SK_REF_FOR_TUNNEL) + __sock_put(recv_sockets.sk6->sk); + else + rxe_release_udp_tunnel(recv_sockets.sk6); + + if (refcount_read(&recv_sockets.sk4->sk->sk_refcnt) > SK_REF_FOR_TUNNEL) + __sock_put(recv_sockets.sk4->sk); + else + rxe_release_udp_tunnel(recv_sockets.sk4); +} + static void rxe_port_event(struct rxe_dev *rxe, enum ib_event_type event) { @@ -686,8 +700,6 @@ int rxe_register_notifier(void) void rxe_net_exit(void) { - rxe_release_udp_tunnel(recv_sockets.sk6); - rxe_release_udp_tunnel(recv_sockets.sk4); unregister_netdevice_notifier(&rxe_net_notifier); } diff --git a/drivers/infiniband/sw/rxe/rxe_net.h b/drivers/infiniband/sw/rxe/rxe_net.h index a222c3eeae12..f48f22f3353b 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.h +++ b/drivers/infiniband/sw/rxe/rxe_net.h @@ -17,6 +17,7 @@ struct rxe_recv_sockets { }; int rxe_net_add(const char *ibdev_name, struct net_device *ndev); +void rxe_net_del(struct ib_device *dev); int rxe_register_notifier(void); int rxe_net_init(void); From patchwork Thu Oct 6 08:59:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhu Yanjun X-Patchwork-Id: 12999449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F5CEC433F5 for ; Wed, 5 Oct 2022 16:33:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230346AbiJEQdi (ORCPT ); Wed, 5 Oct 2022 12:33:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50660 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229927AbiJEQdS (ORCPT ); Wed, 5 Oct 2022 12:33:18 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EDDCC7E02C; Wed, 5 Oct 2022 09:33:15 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="303182121" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="303182121" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Oct 2022 09:33:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="619518167" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="619518167" Received: from unknown (HELO intel-71.bj.intel.com) ([10.238.154.71]) by orsmga007.jf.intel.com with ESMTP; 05 Oct 2022 09:33:13 -0700 From: yanjun.zhu@linux.dev To: jgg@ziepe.ca, leon@kernel.org, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev, netdev@vger.kernel.org, davem@davemloft.net Subject: [PATCHv2 5/6] RDMA/rxe: Replace global variable with sock lookup functions Date: Thu, 6 Oct 2022 04:59:20 -0400 Message-Id: <20221006085921.1323148-6-yanjun.zhu@linux.dev> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20221006085921.1323148-1-yanjun.zhu@linux.dev> References: <20221006085921.1323148-1-yanjun.zhu@linux.dev> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Zhu Yanjun Originally a global variable is to keep the sock of udp listening on port 4791. In fact, sock lookup functions can be used to get the sock. Signed-off-by: Zhu Yanjun --- drivers/infiniband/sw/rxe/rxe.c | 1 + drivers/infiniband/sw/rxe/rxe_net.c | 58 ++++++++++++++++++++------- drivers/infiniband/sw/rxe/rxe_net.h | 5 --- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + 4 files changed, 45 insertions(+), 20 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 1b8b74ea84e9..933d8e129c47 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -74,6 +74,7 @@ static void rxe_init_device_param(struct rxe_dev *rxe) rxe->ndev->dev_addr); rxe->max_ucontext = RXE_MAX_UCONTEXT; + rxe->l_sk6 = NULL; } /* initialize port attributes */ diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index 6e35566e933b..64b11faccfb4 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -18,8 +18,6 @@ #include "rxe_net.h" #include "rxe_loc.h" -static struct rxe_recv_sockets recv_sockets; - static struct dst_entry *rxe_find_route4(struct net_device *ndev, struct in_addr *saddr, struct in_addr *daddr) @@ -49,6 +47,23 @@ static struct dst_entry *rxe_find_route6(struct net_device *ndev, { struct dst_entry *ndst; struct flowi6 fl6 = { { 0 } }; + struct rxe_dev *rdev; + + rdev = rxe_get_dev_from_net(ndev); + if (!rdev->l_sk6) { + struct sock *sk; + + rcu_read_lock(); + sk = udp6_lib_lookup(&init_net, NULL, 0, &in6addr_any, htons(ROCE_V2_UDP_DPORT), 0); + rcu_read_unlock(); + if (!sk) { + pr_info("file: %s +%d, error\n", __FILE__, __LINE__); + return (struct dst_entry *)sk; + } + __sock_put(sk); + rdev->l_sk6 = sk->sk_socket; + } + memset(&fl6, 0, sizeof(fl6)); fl6.flowi6_oif = ndev->ifindex; @@ -56,8 +71,8 @@ static struct dst_entry *rxe_find_route6(struct net_device *ndev, memcpy(&fl6.daddr, daddr, sizeof(*daddr)); fl6.flowi6_proto = IPPROTO_UDP; - ndst = ipv6_stub->ipv6_dst_lookup_flow(sock_net(recv_sockets.sk6->sk), - recv_sockets.sk6->sk, &fl6, + ndst = ipv6_stub->ipv6_dst_lookup_flow(dev_net(ndev), + rdev->l_sk6->sk, &fl6, NULL); if (IS_ERR(ndst)) { pr_err_ratelimited("no route to %pI6\n", daddr); @@ -530,15 +545,33 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev) #define SK_REF_FOR_TUNNEL 2 void rxe_net_del(struct ib_device *dev) { - if (refcount_read(&recv_sockets.sk6->sk->sk_refcnt) > SK_REF_FOR_TUNNEL) - __sock_put(recv_sockets.sk6->sk); + struct sock *sk; + + rcu_read_lock(); + sk = udp4_lib_lookup(&init_net, 0, 0, htonl(INADDR_ANY), htons(ROCE_V2_UDP_DPORT), 0); + rcu_read_unlock(); + if (!sk) + return; + + __sock_put(sk); + + if (refcount_read(&sk->sk_refcnt) > SK_REF_FOR_TUNNEL) + __sock_put(sk); else - rxe_release_udp_tunnel(recv_sockets.sk6); + rxe_release_udp_tunnel(sk->sk_socket); + + rcu_read_lock(); + sk = udp6_lib_lookup(&init_net, NULL, 0, &in6addr_any, htons(ROCE_V2_UDP_DPORT), 0); + rcu_read_unlock(); + if (!sk) + return; + + __sock_put(sk); - if (refcount_read(&recv_sockets.sk4->sk->sk_refcnt) > SK_REF_FOR_TUNNEL) - __sock_put(recv_sockets.sk4->sk); + if (refcount_read(&sk->sk_refcnt) > SK_REF_FOR_TUNNEL) + __sock_put(sk); else - rxe_release_udp_tunnel(recv_sockets.sk4); + rxe_release_udp_tunnel(sk->sk_socket); } static void rxe_port_event(struct rxe_dev *rxe, @@ -647,10 +680,8 @@ static int rxe_net_ipv4_init(void) sock = rxe_setup_udp_tunnel(&init_net, htons(ROCE_V2_UDP_DPORT), false); if (IS_ERR(sock)) { pr_err("Failed to create IPv4 UDP tunnel\n"); - recv_sockets.sk4 = NULL; return -1; } - recv_sockets.sk4 = sock; return 0; } @@ -670,17 +701,14 @@ static int rxe_net_ipv6_init(void) sock = rxe_setup_udp_tunnel(&init_net, htons(ROCE_V2_UDP_DPORT), true); if (PTR_ERR(sock) == -EAFNOSUPPORT) { - recv_sockets.sk6 = NULL; pr_warn("IPv6 is not supported, can not create a UDPv6 socket\n"); return 0; } if (IS_ERR(sock)) { - recv_sockets.sk6 = NULL; pr_err("Failed to create IPv6 UDP tunnel\n"); return -1; } - recv_sockets.sk6 = sock; #endif return 0; } diff --git a/drivers/infiniband/sw/rxe/rxe_net.h b/drivers/infiniband/sw/rxe/rxe_net.h index f48f22f3353b..027b20e1bab6 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.h +++ b/drivers/infiniband/sw/rxe/rxe_net.h @@ -11,11 +11,6 @@ #include #include -struct rxe_recv_sockets { - struct socket *sk4; - struct socket *sk6; -}; - int rxe_net_add(const char *ibdev_name, struct net_device *ndev); void rxe_net_del(struct ib_device *dev); diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 96af3e054f4d..13b12f02a52e 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -406,6 +406,7 @@ struct rxe_dev { struct rxe_port port; struct crypto_shash *tfm; + struct socket *l_sk6; }; static inline void rxe_counter_inc(struct rxe_dev *rxe, enum rxe_counters index) From patchwork Thu Oct 6 08:59:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhu Yanjun X-Patchwork-Id: 12999450 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6021FC433F5 for ; Wed, 5 Oct 2022 16:33:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230100AbiJEQdk (ORCPT ); Wed, 5 Oct 2022 12:33:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230081AbiJEQdX (ORCPT ); Wed, 5 Oct 2022 12:33:23 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D7CB7E010; Wed, 5 Oct 2022 09:33:18 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="303182130" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="303182130" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Oct 2022 09:33:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="619518193" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="619518193" Received: from unknown (HELO intel-71.bj.intel.com) ([10.238.154.71]) by orsmga007.jf.intel.com with ESMTP; 05 Oct 2022 09:33:15 -0700 From: yanjun.zhu@linux.dev To: jgg@ziepe.ca, leon@kernel.org, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev, netdev@vger.kernel.org, davem@davemloft.net Subject: [PATCHv2 6/6] RDMA/rxe: add the support of net namespace Date: Thu, 6 Oct 2022 04:59:21 -0400 Message-Id: <20221006085921.1323148-7-yanjun.zhu@linux.dev> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20221006085921.1323148-1-yanjun.zhu@linux.dev> References: <20221006085921.1323148-1-yanjun.zhu@linux.dev> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Zhu Yanjun Originally init_net is used to indicate the current net namespace. Currently more net namespaces are supported. Signed-off-by: Zhu Yanjun --- drivers/infiniband/sw/rxe/rxe.c | 2 +- drivers/infiniband/sw/rxe/rxe_net.c | 32 +++++++++++++++++------------ drivers/infiniband/sw/rxe/rxe_net.h | 2 +- 3 files changed, 21 insertions(+), 15 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 933d8e129c47..927c513ace81 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -196,7 +196,7 @@ static int rxe_newlink(const char *ibdev_name, struct net_device *ndev) goto err; } - err = rxe_net_init(); + err = rxe_net_init(ndev); if (err) return err; diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index 64b11faccfb4..b5955d0c284a 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -31,7 +31,7 @@ static struct dst_entry *rxe_find_route4(struct net_device *ndev, memcpy(&fl.daddr, daddr, sizeof(*daddr)); fl.flowi4_proto = IPPROTO_UDP; - rt = ip_route_output_key(&init_net, &fl); + rt = ip_route_output_key(dev_net(ndev), &fl); if (IS_ERR(rt)) { pr_err_ratelimited("no route to %pI4\n", &daddr->s_addr); return NULL; @@ -54,7 +54,8 @@ static struct dst_entry *rxe_find_route6(struct net_device *ndev, struct sock *sk; rcu_read_lock(); - sk = udp6_lib_lookup(&init_net, NULL, 0, &in6addr_any, htons(ROCE_V2_UDP_DPORT), 0); + sk = udp6_lib_lookup(dev_net(ndev), NULL, 0, &in6addr_any, + htons(ROCE_V2_UDP_DPORT), 0); rcu_read_unlock(); if (!sk) { pr_info("file: %s +%d, error\n", __FILE__, __LINE__); @@ -546,9 +547,13 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev) void rxe_net_del(struct ib_device *dev) { struct sock *sk; + struct rxe_dev *rdev; + + rdev = container_of(dev, struct rxe_dev, ib_dev); rcu_read_lock(); - sk = udp4_lib_lookup(&init_net, 0, 0, htonl(INADDR_ANY), htons(ROCE_V2_UDP_DPORT), 0); + sk = udp4_lib_lookup(dev_net(rdev->ndev), 0, 0, htonl(INADDR_ANY), + htons(ROCE_V2_UDP_DPORT), 0); rcu_read_unlock(); if (!sk) return; @@ -561,7 +566,8 @@ void rxe_net_del(struct ib_device *dev) rxe_release_udp_tunnel(sk->sk_socket); rcu_read_lock(); - sk = udp6_lib_lookup(&init_net, NULL, 0, &in6addr_any, htons(ROCE_V2_UDP_DPORT), 0); + sk = udp6_lib_lookup(dev_net(rdev->ndev), NULL, 0, &in6addr_any, + htons(ROCE_V2_UDP_DPORT), 0); rcu_read_unlock(); if (!sk) return; @@ -665,19 +671,19 @@ static struct notifier_block rxe_net_notifier = { .notifier_call = rxe_notify, }; -static int rxe_net_ipv4_init(void) +static int rxe_net_ipv4_init(struct net_device *ndev) { struct sock *sk; struct socket *sock; rcu_read_lock(); - sk = udp4_lib_lookup(&init_net, 0, 0, htonl(INADDR_ANY), + sk = udp4_lib_lookup(dev_net(ndev), 0, 0, htonl(INADDR_ANY), htons(ROCE_V2_UDP_DPORT), 0); rcu_read_unlock(); if (sk) return 0; - sock = rxe_setup_udp_tunnel(&init_net, htons(ROCE_V2_UDP_DPORT), false); + sock = rxe_setup_udp_tunnel(dev_net(ndev), htons(ROCE_V2_UDP_DPORT), false); if (IS_ERR(sock)) { pr_err("Failed to create IPv4 UDP tunnel\n"); return -1; @@ -686,20 +692,20 @@ static int rxe_net_ipv4_init(void) return 0; } -static int rxe_net_ipv6_init(void) +static int rxe_net_ipv6_init(struct net_device *ndev) { #if IS_ENABLED(CONFIG_IPV6) struct sock *sk; struct socket *sock; rcu_read_lock(); - sk = udp6_lib_lookup(&init_net, NULL, 0, &in6addr_any, + sk = udp6_lib_lookup(dev_net(ndev), NULL, 0, &in6addr_any, htons(ROCE_V2_UDP_DPORT), 0); rcu_read_unlock(); if (sk) return 0; - sock = rxe_setup_udp_tunnel(&init_net, htons(ROCE_V2_UDP_DPORT), true); + sock = rxe_setup_udp_tunnel(dev_net(ndev), htons(ROCE_V2_UDP_DPORT), true); if (PTR_ERR(sock) == -EAFNOSUPPORT) { pr_warn("IPv6 is not supported, can not create a UDPv6 socket\n"); return 0; @@ -731,14 +737,14 @@ void rxe_net_exit(void) unregister_netdevice_notifier(&rxe_net_notifier); } -int rxe_net_init(void) +int rxe_net_init(struct net_device *ndev) { int err; - err = rxe_net_ipv4_init(); + err = rxe_net_ipv4_init(ndev); if (err) return err; - err = rxe_net_ipv6_init(); + err = rxe_net_ipv6_init(ndev); if (err) goto err_out; return 0; diff --git a/drivers/infiniband/sw/rxe/rxe_net.h b/drivers/infiniband/sw/rxe/rxe_net.h index 027b20e1bab6..56249677d692 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.h +++ b/drivers/infiniband/sw/rxe/rxe_net.h @@ -15,7 +15,7 @@ int rxe_net_add(const char *ibdev_name, struct net_device *ndev); void rxe_net_del(struct ib_device *dev); int rxe_register_notifier(void); -int rxe_net_init(void); +int rxe_net_init(struct net_device *ndev); void rxe_net_exit(void); #endif /* RXE_NET_H */