From patchwork Thu Oct 6 08:59:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhu Yanjun X-Patchwork-Id: 12999446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0067C433FE for ; Wed, 5 Oct 2022 16:33:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229973AbiJEQdN (ORCPT ); Wed, 5 Oct 2022 12:33:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50360 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229976AbiJEQdI (ORCPT ); Wed, 5 Oct 2022 12:33:08 -0400 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5C0D7D783; Wed, 5 Oct 2022 09:33:05 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="303182092" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="303182092" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Oct 2022 09:33:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10491"; a="619518072" X-IronPort-AV: E=Sophos;i="5.95,161,1661842800"; d="scan'208";a="619518072" Received: from unknown (HELO intel-71.bj.intel.com) ([10.238.154.71]) by orsmga007.jf.intel.com with ESMTP; 05 Oct 2022 09:33:02 -0700 From: yanjun.zhu@linux.dev To: jgg@ziepe.ca, leon@kernel.org, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev, netdev@vger.kernel.org, davem@davemloft.net Cc: Zhu Yanjun Subject: [PATCHv2 0/6] Fix the problem that rxe can not work in net Date: Thu, 6 Oct 2022 04:59:15 -0400 Message-Id: <20221006085921.1323148-1-yanjun.zhu@linux.dev> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Zhu Yanjun When run "ip link add" command to add a rxe rdma link in a net namespace, normally this rxe rdma link can not work in a net name space. The root cause is that a sock listening on udp port 4791 is created in init_net when the rdma_rxe module is loaded into kernel. That is, the sock listening on udp port 4791 is created in init_net. Other net namespace is difficult to use this sock. The following commits will solve this problem. In the first commit, move the creating sock listening on udp port 4791 from module_init function to rdma link creating functions. That is, after the module rdma_rxe is loaded, the sock will not be created. When run "rdma link add ..." command, the sock will be created. So when creating a rdma link in the net namespace, the sock will be created in this net namespace. In the second commit, the functions udp4_lib_lookup and udp6_lib_lookup will check the sock exists in the net namespace or not. If yes, rdma link will increase the reference count of this sock, then continue other jobs instead of creating a new sock to listen on udp port 4791. Since the network notifier is global, when the module rdma_rxe is loaded, this notifier will be registered. After the rdma link is created, the command "rdma link del" is to delete rdma link at the same time the sock is checked. If the reference count of this sock is greater than the sock reference count needed by udp tunnel, the sock reference count is decreased by one. If equal, it indicates that this rdma link is the last one. As such, the udp tunnel is shut down and the sock is closed. The above work should be implemented in linkdel function. But currently no dellink function in rxe. So the 3rd commit addes dellink function pointer. And the 4th commit implements the dellink function in rxe. To now, it is not necessary to keep a global variable to store the sock listening udp port 4791. This global variable can be replaced by the functions udp4_lib_lookup and udp6_lib_lookup totally. Because the function udp6_lib_lookup is in the fast path, a member variable l_sk6 is added to store the sock. If l_sk6 is NULL, udp6_lib_lookup is called to lookup the sock, then the sock is stored in l_sk6, in the future,it can be used directly. All the above work has been done in init_net. And it can also work in the net namespace. So the init_net is replaced by the individual net namespace. This is what the 6th commit does. Because rxe device is dependent on the net device and the sock listening on udp port 4791, every rxe device is in exclusive mode in the individual net namespace. Other rdma netns operations will be considerred in the future. Test steps: 1) Suppose that 2 NICs are in 2 different net namespaces. # ip netns exec net0 ip link 3: eno2: mtu 1500 qdisc mq state UP link/ether 00:1e:67:a0:22:3f brd ff:ff:ff:ff:ff:ff altname enp5s0 # ip netns exec net1 ip link 4: eno3: mtu 1500 qdisc fq_codel link/ether f8:e4:3b:3b:e4:10 brd ff:ff:ff:ff:ff:ff 2) Add rdma link in the different net namespace net0: ip netns exec net0 rdma link add rxe0 type rxe netdev eno2 net1: ip netns exec net1 rdma link add rxe1 type rxe netdev eno3 3) Run rping test. net0 # ip netns exec net0 rping -s -a 192.168.2.1 -C 1& [1] 1737 # ip netns exec net1 rping -c -a 192.168.2.1 -d -v -C 1 verbose count 1 ... ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr ... 4) Remove the rdma links from the net namespaces. net0: ip netns exec net0 rdma link del rxe0 net1: ip netns exec net1 rdma link del rxe1 --- V1->V2: Add the explicit initialization of sk6. --- Zhu Yanjun (6): RDMA/rxe: Creating listening sock in newlink function RDMA/rxe: Support more rdma links in init_net RDMA/nldev: Add dellink function pointer RDMA/rxe: Implement dellink in rxe RDMA/rxe: Replace global variable with sock lookup functions RDMA/rxe: add the support of net namespace drivers/infiniband/core/nldev.c | 6 ++ drivers/infiniband/sw/rxe/rxe.c | 26 +++++- drivers/infiniband/sw/rxe/rxe_net.c | 129 ++++++++++++++++++++------ drivers/infiniband/sw/rxe/rxe_net.h | 9 +- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + include/rdma/rdma_netlink.h | 2 + 6 files changed, 134 insertions(+), 39 deletions(-)