From patchwork Fri May 31 13:15:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13681678 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 26C74158A2E; Fri, 31 May 2024 13:16:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717161369; cv=none; b=gp9dmMq6l24aHtqlCbsuzreC5O2y1fhDiWNCJai/UXHMeHhj/UPZclXHS3LJy6zUuctsPrdd0k+PwTlOW8gEfqfkMfquawdpyuNQ/Yg+2W2ou39D7OENVdYMM/90KK4Fe2nAkZlRQtVwpF+9TGNciDB8p90TkcM+MYyYbsD6lhc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717161369; c=relaxed/simple; bh=BfQKLh10hQ9RZozJx4TlZCeX7KWH/jIn9tpCCrCLYvk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tp1ksSYHBXnh3cDuXz2DaDL9Orhahj8RwoxJQR8HoxSc7SUU/qCjMlMNXUNjhiT2T06Vv5T5kRVDW+zKbUu7InHP6QKDFBjVx9IEXFxGi7Upx1pg1sLGb4Rgs/9uKl3ui8f9SUY2DcF5l9czPiMMhVwbGsWRwwiICECn63DTKv8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Rgta5CRK; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Rgta5CRK" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6913EC32781; Fri, 31 May 2024 13:16:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717161368; bh=BfQKLh10hQ9RZozJx4TlZCeX7KWH/jIn9tpCCrCLYvk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Rgta5CRK0boN0mCgAJWJOgBOGxQZYP9LFfFWglsLBX6EuO0O/iBhKXey7vRePLaAg RpzcvIQWxJ7AHwVQ0ocWeZ3VmHvr1uNw5L8K3XfYH1ifGHzeH//3x+Bvz1LdTrH5JR yLuGyXkTSb1C4Ouh39nYiiKtdeZ/ygoAjcwXzEhP0IFiScyjB4YWyVv7Zl1CDRetEa T+a0HssujvMC57Bhf+0ir8AVHoWDCltadT46pNpu/4IrGq+PFwPtKMiKklycoDAskA r/m28WY2pwMkh1RzLX/oSeDqP9/aPUTFVJ89NhT8D83Oa+VxQvXvOgkiaITBJIh4uf 4j//Nc/B9Ng9w== From: cel@kernel.org To: , Cc: Chuck Lever Subject: [PATCH 1/2] svcrdma: Refactor the creation of listener CMA ID Date: Fri, 31 May 2024 09:15:52 -0400 Message-ID: <20240531131550.64044-5-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240531131550.64044-4-cel@kernel.org> References: <20240531131550.64044-4-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3487; i=chuck.lever@oracle.com; h=from:subject; bh=2PIo1uAgvbLyhllQAw3g6B2ljsIy2S7JgWBEEMmR0CI=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmWc2PRnHK74MIuy+MLxGc1MwSFDsiR2n4Fl0Hu ckp8yWHtAWJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZlnNjwAKCRAzarMzb2Z/ l2iZD/4lGeH44T39tatJ+xnkqSaBBfez9mdYzFtASScCvp0jCIQ2jWhW2DAq3BVEE3QSJ22eJAx cLPsUnIfGBU2eNN4NHylKZ10XML1h/eeFyr3TKTNI/XNrYyeB+1EmXjU0XfDNOIQ1t6Sxh99LdE CHD5TiukTSl/ZkByLJgtgdRDLrW6iotUjWIZaDA/V1CyLbsh3I6ApsOLzeW2vduRIkkJt4iKx/a 28zkcj7+1wiZ7Iq6waKVt3YM+iTN1O/+EcVogHLtNNRrdMR3+xlXERhEU0XFQO17F0sN2hrsGAK 6pNvfGso/ytfny5tYYYIGbcux6bpzCpRkEZ56tm/FKHNeWJ0Wg76cy6lUQVOt4eMIErsc8YA4+L 2Vm7E/1RAASfEWp0oKRFyc1+mtugWcm7zAF8Vd9eNCTUthEzq5izrjxdp1MPuA5hw3mblgGTfRi QyecgFBMd+d3ASnTwPBMekRW84l0HIg+0PX/wJxupH8hROwhUL+vEFwSwtDQUUBKZ6rJALmdFfc UjlJ73xcDqJk4EtTQhwoF+4eaIu162J6mphR8FuiFnWy70AMM1xanInuQbk6vzDfv/lF2XPW+Kv NK20CApZiv47astamDQg/lZr6mbkcM1PMWkHJnYgF38C0bcwWJUNr3ROLWGp1AfcRTQfsHC9Jbp Yr58fJu0EHdZwrA== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever In a moment, I will add a second consumer of CMA ID creation in svcrdma. Refactor so this code can be reused. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/svc_rdma_transport.c | 67 ++++++++++++++---------- 1 file changed, 40 insertions(+), 27 deletions(-) diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c index 2b1c16b9547d..fa50b7494a0a 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c @@ -65,6 +65,8 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv, struct net *net, int node); +static int svc_rdma_listen_handler(struct rdma_cm_id *cma_id, + struct rdma_cm_event *event); static struct svc_xprt *svc_rdma_create(struct svc_serv *serv, struct net *net, struct sockaddr *sa, int salen, @@ -122,6 +124,41 @@ static void qp_event_handler(struct ib_event *event, void *context) } } +static struct rdma_cm_id * +svc_rdma_create_listen_id(struct net *net, struct sockaddr *sap, + void *context) +{ + struct rdma_cm_id *listen_id; + int ret; + + listen_id = rdma_create_id(net, svc_rdma_listen_handler, context, + RDMA_PS_TCP, IB_QPT_RC); + if (IS_ERR(listen_id)) + return listen_id; + + /* Allow both IPv4 and IPv6 sockets to bind a single port + * at the same time. + */ +#if IS_ENABLED(CONFIG_IPV6) + ret = rdma_set_afonly(listen_id, 1); + if (ret) + goto out_destroy; +#endif + ret = rdma_bind_addr(listen_id, sap); + if (ret) + goto out_destroy; + + ret = rdma_listen(listen_id, RPCRDMA_LISTEN_BACKLOG); + if (ret) + goto out_destroy; + + return listen_id; + +out_destroy: + rdma_destroy_id(listen_id); + return ERR_PTR(ret); +} + static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv, struct net *net, int node) { @@ -307,7 +344,6 @@ static struct svc_xprt *svc_rdma_create(struct svc_serv *serv, { struct rdma_cm_id *listen_id; struct svcxprt_rdma *cma_xprt; - int ret; if (sa->sa_family != AF_INET && sa->sa_family != AF_INET6) return ERR_PTR(-EAFNOSUPPORT); @@ -317,30 +353,13 @@ static struct svc_xprt *svc_rdma_create(struct svc_serv *serv, set_bit(XPT_LISTENER, &cma_xprt->sc_xprt.xpt_flags); strcpy(cma_xprt->sc_xprt.xpt_remotebuf, "listener"); - listen_id = rdma_create_id(net, svc_rdma_listen_handler, cma_xprt, - RDMA_PS_TCP, IB_QPT_RC); + listen_id = svc_rdma_create_listen_id(net, sa, cma_xprt); if (IS_ERR(listen_id)) { - ret = PTR_ERR(listen_id); - goto err0; + kfree(cma_xprt); + return (struct svc_xprt *)listen_id; } - - /* Allow both IPv4 and IPv6 sockets to bind a single port - * at the same time. - */ -#if IS_ENABLED(CONFIG_IPV6) - ret = rdma_set_afonly(listen_id, 1); - if (ret) - goto err1; -#endif - ret = rdma_bind_addr(listen_id, sa); - if (ret) - goto err1; cma_xprt->sc_cm_id = listen_id; - ret = rdma_listen(listen_id, RPCRDMA_LISTEN_BACKLOG); - if (ret) - goto err1; - /* * We need to use the address from the cm_id in case the * caller specified 0 for the port number. @@ -349,12 +368,6 @@ static struct svc_xprt *svc_rdma_create(struct svc_serv *serv, svc_xprt_set_local(&cma_xprt->sc_xprt, sa, salen); return &cma_xprt->sc_xprt; - - err1: - rdma_destroy_id(listen_id); - err0: - kfree(cma_xprt); - return ERR_PTR(ret); } /* From patchwork Fri May 31 13:15:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13681679 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2DADF158D89; Fri, 31 May 2024 13:16:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717161371; cv=none; b=rD9hXwfZhzccJM50cW6iNSHEbXJ7yCLNzClysVb0YTj18+yeIzURgnyi8wRpOnphOOezIEIlOHdNuGLiv2BOwVqA9zZJooHRiLdOlNBTFhGH2ReE96UoNGzW0SAzhgAwhBiuONM6qdAgdYJnOPWGHNedF+3oVcVmfbEBdptokjk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717161371; c=relaxed/simple; bh=zYN86TjmgCguRxsLfye2Hp1ZzKNeRsWNS2gyamV76yw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QWZ16vm8P+8JmmUUa4VHujRyT1HM3xue2IrqcS4dGOadDmkE/u0vsitFd4SWyLNKlrjE7QHsvnU9UNrq5UQimBUbyVWiqoVzAnBlesYuE6IyqT2e4wAS1D3Fv+0zVr7jUlIGo9qFZQ6GtmRVymIAKOpeqKSDp/zJME1ilpriEOs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=l2kBmrc1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="l2kBmrc1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6767AC116B1; Fri, 31 May 2024 13:16:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717161370; bh=zYN86TjmgCguRxsLfye2Hp1ZzKNeRsWNS2gyamV76yw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=l2kBmrc1V9B9XL0756gWkwVm/ryijpRzgbLONKZz+rUgAzt387QUkdyyF5z2az/+L qIFqGRcDK/4DuAsvhoIciXaAc/GF6+TRC8RcM2oAlvptAkRXwhxJU93HjnmTUbNZZG UDD4iY6HOYl2dtuv7c4UPDC4Rf6I0PSunJlfcHoUZDxNTzVQq3tqoLiF9azjiBlZH1 aLSIBuADNchQ8g9XOB7BkDogTtKoc6jZQgXc7OhJ0RLW919Fi0Wh41T8tyHqqC9cy0 9STHKP94htvtKf4EGIUh91Xn+JglSkOqL4jNNi9xlyqcM4v0XjoP72bHyJ59/MF/QJ MlRcbnlViEwUw== From: cel@kernel.org To: , Cc: Chuck Lever , Sagi Grimberg Subject: [PATCH 2/2] svcrdma: Handle ADDR_CHANGE CM event properly Date: Fri, 31 May 2024 09:15:53 -0400 Message-ID: <20240531131550.64044-6-cel@kernel.org> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240531131550.64044-4-cel@kernel.org> References: <20240531131550.64044-4-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1793; i=chuck.lever@oracle.com; h=from:subject; bh=xVkiJogTAjn56JWQPU7jR+XzOmBOEzfU4GRVcOmFucc=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmWc2QjQcPIYC3816G6VAl9NywbY4b74ilpM9Tw WxS3TF72NiJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZlnNkAAKCRAzarMzb2Z/ l2wZEACZbNvT1atflPBeE3HcUMERO6kXXi7JkuXOmfjJvM//WZF9aWxQJwLoRZeQ3aOIdcrwhG6 zI9qWirZDIJtlo4qMY1hWfaCo+DgATWF/JvLNC8hEK10fSfoqOnGxN4RjAsV/HDx5b229LujLHf 1/w4Y4Wp5AobwZ8JU05259ROix8OwORS39l2PrwolANmiOTbWtHxcm+SR77VrWI14qMOi4VWf8g lAcxdlxEnOJg/85pEiPBsWjd0enfDRCUGh4GbgfWo+HxKN9FZh/8/pw9Bjht8oKa/ghGl0yavZ7 hIDQdv/RXLjUGXtLcvaHAV4wngZQYDWHVOnqkdHey6q7g8rBbNV2IpwXnVridvYmWzR1U3gnnBR GMbQwZbkvfmut/yG4kbqGiiG2DCtlH1Iydijr37qKtewPY4BBKHv+VFIWyMPulp50uZ+2ODJkke bxI8ddzrr5NYIY6RC6Qfrok4aaP+atdU3m2ul4Wi3cq8BRf9JFy9LeHG3eYSzm8FyFEf2dIi3ld bRc3gVnVxZzlja18LzRfGrdI6LnIffURqI2nU0BJ/+fhuv5lgsVkldmxukN+wFp4+mEbrPCm7fs d9sb2wa8XOtOYs0yCP5K6Ysgo+1njxSeuS9mS0Ib1/ZkCLMNF+u0CKBUZQWGz9upEJgxC5T3Xh5 X0QOwz/g1BggArQ== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever Sagi tells me that when a bonded device reports an address change, the consumer must destroy its listener IDs and create new ones. See commit a032e4f6d60d ("nvmet-rdma: fix bonding failover possible NULL deref"). Suggested-by: Sagi Grimberg Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/svc_rdma_transport.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c index fa50b7494a0a..a003b62fb7d5 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c @@ -284,17 +284,31 @@ static void handle_connect_req(struct rdma_cm_id *new_cma_id, * * Return values: * %0: Do not destroy @cma_id - * %1: Destroy @cma_id (never returned here) + * %1: Destroy @cma_id * * NB: There is never a DEVICE_REMOVAL event for INADDR_ANY listeners. */ static int svc_rdma_listen_handler(struct rdma_cm_id *cma_id, struct rdma_cm_event *event) { + struct svcxprt_rdma *cma_xprt = cma_id->context; + struct svc_xprt *cma_rdma = &cma_xprt->sc_xprt; + struct sockaddr *sap = (struct sockaddr *)&cma_id->route.addr.src_addr; + struct rdma_cm_id *listen_id; + switch (event->event) { case RDMA_CM_EVENT_CONNECT_REQUEST: handle_connect_req(cma_id, &event->param.conn); break; + case RDMA_CM_EVENT_ADDR_CHANGE: + listen_id = svc_rdma_create_listen_id(cma_rdma->xpt_net, + sap, cma_xprt); + if (IS_ERR(listen_id)) { + pr_err("Listener dead, address change failed for device %s\n", + cma_id->device->name); + } else + cma_xprt->sc_cm_id = listen_id; + return 1; default: break; }