From patchwork Wed Oct 27 08:52:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Lu X-Patchwork-Id: 12586693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CB9DC433EF for ; Wed, 27 Oct 2021 08:52:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3F80A610A5 for ; Wed, 27 Oct 2021 08:52:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241047AbhJ0IzG (ORCPT ); Wed, 27 Oct 2021 04:55:06 -0400 Received: from out30-54.freemail.mail.aliyun.com ([115.124.30.54]:59276 "EHLO out30-54.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235961AbhJ0IzF (ORCPT ); Wed, 27 Oct 2021 04:55:05 -0400 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R501e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e01424;MF=tonylu@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0UtsJwZd_1635324758; Received: from localhost(mailfrom:tonylu@linux.alibaba.com fp:SMTPD_---0UtsJwZd_1635324758) by smtp.aliyun-inc.com(127.0.0.1); Wed, 27 Oct 2021 16:52:38 +0800 From: Tony Lu To: kgraul@linux.ibm.com, davem@davemloft.net, kuba@kernel.org, ubraun@linux.ibm.com Cc: netdev@vger.kernel.org, linux-s390@vger.kernel.org, linux-rdma@vger.kernel.org, jacob.qi@linux.alibaba.com, xuanzhuo@linux.alibaba.com, guwen@linux.alibaba.com, dust.li@linux.alibaba.com Subject: [PATCH net 1/4] Revert "net/smc: don't wait for send buffer space when data was already sent" Date: Wed, 27 Oct 2021 16:52:07 +0800 Message-Id: <20211027085208.16048-2-tonylu@linux.alibaba.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211027085208.16048-1-tonylu@linux.alibaba.com> References: <20211027085208.16048-1-tonylu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Tony Lu This reverts commit 6889b36da78a21a312d8b462c1fa25a03c2ff192. When using SMC to replace TCP, some userspace applications like netperf don't check the return code of send syscall correctly, which means how many bytes are sent. If rc of send() is smaller than expected, it should try to send again, instead of exit directly. It is difficult to change the uncorrect behaviors of userspace applications, so choose to revert it. Cc: Karsten Graul Cc: Ursula Braun Cc: David S. Miller Reported-by: Jacob Qi Signed-off-by: Tony Lu Reviewed-by: Xuan Zhuo Reviewed-by: Wen Gu --- net/smc/smc_tx.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/net/smc/smc_tx.c b/net/smc/smc_tx.c index 738a4a99c827..d401286e9058 100644 --- a/net/smc/smc_tx.c +++ b/net/smc/smc_tx.c @@ -178,11 +178,12 @@ int smc_tx_sendmsg(struct smc_sock *smc, struct msghdr *msg, size_t len) conn->local_tx_ctrl.prod_flags.urg_data_pending = 1; if (!atomic_read(&conn->sndbuf_space) || conn->urg_tx_pend) { - if (send_done) - return send_done; rc = smc_tx_wait(smc, msg->msg_flags); - if (rc) + if (rc) { + if (send_done) + return send_done; goto out_err; + } continue; } From patchwork Wed Oct 27 08:52:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Lu X-Patchwork-Id: 12586695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9354C4332F for ; Wed, 27 Oct 2021 08:52:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B254E610E6 for ; Wed, 27 Oct 2021 08:52:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241051AbhJ0IzJ (ORCPT ); Wed, 27 Oct 2021 04:55:09 -0400 Received: from out30-131.freemail.mail.aliyun.com ([115.124.30.131]:33403 "EHLO out30-131.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235961AbhJ0IzH (ORCPT ); Wed, 27 Oct 2021 04:55:07 -0400 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04394;MF=tonylu@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0Uts5VZA_1635324760; Received: from localhost(mailfrom:tonylu@linux.alibaba.com fp:SMTPD_---0Uts5VZA_1635324760) by smtp.aliyun-inc.com(127.0.0.1); Wed, 27 Oct 2021 16:52:40 +0800 From: Tony Lu To: kgraul@linux.ibm.com, davem@davemloft.net, kuba@kernel.org, ubraun@linux.ibm.com Cc: netdev@vger.kernel.org, linux-s390@vger.kernel.org, linux-rdma@vger.kernel.org, jacob.qi@linux.alibaba.com, xuanzhuo@linux.alibaba.com, guwen@linux.alibaba.com, dust.li@linux.alibaba.com Subject: [PATCH net 2/4] net/smc: Fix smc_link->llc_testlink_time overflow Date: Wed, 27 Oct 2021 16:52:08 +0800 Message-Id: <20211027085208.16048-3-tonylu@linux.alibaba.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211027085208.16048-1-tonylu@linux.alibaba.com> References: <20211027085208.16048-1-tonylu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Tony Lu The value of llc_testlink_time is set to the value stored in net->ipv4.sysctl_tcp_keepalive_time when linkgroup init. The value of sysctl_tcp_keepalive_time is already jiffies, so we don't need to multiply by HZ, which would cause smc_link->llc_testlink_time overflow, and test_link send flood. Fixes: 00a049cfde95 ("net/smc: move llc layer related init and clear into smc_llc.c") Cc: Karsten Graul Signed-off-by: Tony Lu Reviewed-by: Xuan Zhuo Reviewed-by: Wen Gu --- net/smc/smc_llc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/smc/smc_llc.c b/net/smc/smc_llc.c index 72f4b72eb175..f1d323439a2a 100644 --- a/net/smc/smc_llc.c +++ b/net/smc/smc_llc.c @@ -1822,7 +1822,7 @@ void smc_llc_link_active(struct smc_link *link) link->smcibdev->ibdev->name, link->ibport); link->state = SMC_LNK_ACTIVE; if (link->lgr->llc_testlink_time) { - link->llc_testlink_time = link->lgr->llc_testlink_time * HZ; + link->llc_testlink_time = link->lgr->llc_testlink_time; schedule_delayed_work(&link->llc_testlink_wrk, link->llc_testlink_time); } From patchwork Wed Oct 27 08:52:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Lu X-Patchwork-Id: 12586697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7C37C433FE for ; Wed, 27 Oct 2021 08:52:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 94AD6610A5 for ; Wed, 27 Oct 2021 08:52:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241057AbhJ0IzL (ORCPT ); Wed, 27 Oct 2021 04:55:11 -0400 Received: from out30-42.freemail.mail.aliyun.com ([115.124.30.42]:37648 "EHLO out30-42.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241052AbhJ0IzJ (ORCPT ); Wed, 27 Oct 2021 04:55:09 -0400 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R291e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e01424;MF=tonylu@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0Uts5tnL_1635324761; Received: from localhost(mailfrom:tonylu@linux.alibaba.com fp:SMTPD_---0Uts5tnL_1635324761) by smtp.aliyun-inc.com(127.0.0.1); Wed, 27 Oct 2021 16:52:42 +0800 From: Tony Lu To: kgraul@linux.ibm.com, davem@davemloft.net, kuba@kernel.org, ubraun@linux.ibm.com Cc: netdev@vger.kernel.org, linux-s390@vger.kernel.org, linux-rdma@vger.kernel.org, jacob.qi@linux.alibaba.com, xuanzhuo@linux.alibaba.com, guwen@linux.alibaba.com, dust.li@linux.alibaba.com Subject: [PATCH net 3/4] net/smc: Correct spelling mistake to TCPF_SYN_RECV Date: Wed, 27 Oct 2021 16:52:09 +0800 Message-Id: <20211027085208.16048-4-tonylu@linux.alibaba.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211027085208.16048-1-tonylu@linux.alibaba.com> References: <20211027085208.16048-1-tonylu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Wen Gu There should use TCPF_SYN_RECV instead of TCP_SYN_RECV. Fixes: 50717a37db03 ("net/smc: nonblocking connect rework") Cc: Ursula Braun Cc: David S. Miller Signed-off-by: Wen Gu Reviewed-by: Tony Lu --- net/smc/af_smc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index c038efc23ce3..78b663dbfa1f 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -1057,7 +1057,7 @@ static void smc_connect_work(struct work_struct *work) if (smc->clcsock->sk->sk_err) { smc->sk.sk_err = smc->clcsock->sk->sk_err; } else if ((1 << smc->clcsock->sk->sk_state) & - (TCPF_SYN_SENT | TCP_SYN_RECV)) { + (TCPF_SYN_SENT | TCPF_SYN_RECV)) { rc = sk_stream_wait_connect(smc->clcsock->sk, &timeo); if ((rc == -EPIPE) && ((1 << smc->clcsock->sk->sk_state) & From patchwork Wed Oct 27 08:52:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tony Lu X-Patchwork-Id: 12586699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33BEAC4321E for ; Wed, 27 Oct 2021 08:52:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 17EC5610A3 for ; Wed, 27 Oct 2021 08:52:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241052AbhJ0IzL (ORCPT ); Wed, 27 Oct 2021 04:55:11 -0400 Received: from out30-54.freemail.mail.aliyun.com ([115.124.30.54]:35796 "EHLO out30-54.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241060AbhJ0IzK (ORCPT ); Wed, 27 Oct 2021 04:55:10 -0400 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04357;MF=tonylu@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0Uts5tnc_1635324763; Received: from localhost(mailfrom:tonylu@linux.alibaba.com fp:SMTPD_---0Uts5tnc_1635324763) by smtp.aliyun-inc.com(127.0.0.1); Wed, 27 Oct 2021 16:52:43 +0800 From: Tony Lu To: kgraul@linux.ibm.com, davem@davemloft.net, kuba@kernel.org, ubraun@linux.ibm.com Cc: netdev@vger.kernel.org, linux-s390@vger.kernel.org, linux-rdma@vger.kernel.org, jacob.qi@linux.alibaba.com, xuanzhuo@linux.alibaba.com, guwen@linux.alibaba.com, dust.li@linux.alibaba.com Subject: [PATCH net 4/4] net/smc: Fix wq mismatch issue caused by smc fallback Date: Wed, 27 Oct 2021 16:52:10 +0800 Message-Id: <20211027085208.16048-5-tonylu@linux.alibaba.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211027085208.16048-1-tonylu@linux.alibaba.com> References: <20211027085208.16048-1-tonylu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Wen Gu A socket_wq mismatch issue may occur because of fallback. When use SMC to replace TCP, applications add an epoll entry into SMC socket's wq, but kernel uses clcsock's wq instead of SMC socket's wq once fallback occurs, which means the application's epoll fd dosen't work anymore. For example: server: nginx -g 'daemon off;' client: smc_run wrk -c 1 -t 1 -d 5 http://11.200.15.93/index.html Running 5s test @ http://11.200.15.93/index.html 1 threads and 1 connections Thread Stats Avg Stdev Max +/- Stdev Latency 0.00us 0.00us 0.00us -nan% Req/Sec 0.00 0.00 0.00 -nan% 0 requests in 5.00s, 0.00B read Requests/sec: 0.00 Transfer/sec: 0.00B This patch fixes this issue by using clcsock's wq regardless of whether fallback occurs. Reported-by: Jacob Qi Signed-off-by: Wen Gu Reviewed-by: Tony Lu --- net/smc/af_smc.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index 78b663dbfa1f..3b7ec0abff52 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -546,6 +546,10 @@ static void smc_switch_to_fallback(struct smc_sock *smc, int reason_code) { smc->use_fallback = true; smc->fallback_rsn = reason_code; + + /* clcsock's sock uses back to clcsock->wq, see also smc_create() */ + rcu_assign_pointer(smc->clcsock->sk->sk_wq, &smc->clcsock->wq); + smc_stat_fallback(smc); if (smc->sk.sk_socket && smc->sk.sk_socket->file) { smc->clcsock->file = smc->sk.sk_socket->file; @@ -1972,6 +1976,10 @@ static int smc_accept(struct socket *sock, struct socket *new_sock, if (rc) goto out; + /* new smc sock uses clcsock's wq. see also smc_create() */ + if (!smc_sk(nsk)->use_fallback) + rcu_assign_pointer(nsk->sk_wq, &smc_sk(nsk)->clcsock->wq); + if (lsmc->sockopt_defer_accept && !(flags & O_NONBLOCK)) { /* wait till data arrives on the socket */ timeo = msecs_to_jiffies(lsmc->sockopt_defer_accept * @@ -2108,6 +2116,9 @@ static __poll_t smc_poll(struct file *file, struct socket *sock, mask = smc->clcsock->ops->poll(file, smc->clcsock, wait); sk->sk_err = smc->clcsock->sk->sk_err; } else { + /* use clcsock->wq in sock_poll_wait(), see also smc_create() */ + sock = smc->clcsock; + if (sk->sk_state != SMC_CLOSED) sock_poll_wait(file, sock, wait); if (sk->sk_err) @@ -2505,6 +2516,10 @@ static int smc_create(struct net *net, struct socket *sock, int protocol, smc->sk.sk_sndbuf = max(smc->clcsock->sk->sk_sndbuf, SMC_BUF_MIN_SIZE); smc->sk.sk_rcvbuf = max(smc->clcsock->sk->sk_rcvbuf, SMC_BUF_MIN_SIZE); + /* In case smc fallbacks to tcp, smc's sock will use clcsock's wq in advance */ + rcu_assign_pointer(sk->sk_wq, &smc->clcsock->wq); + rcu_assign_pointer(smc->clcsock->sk->sk_wq, &smc->sk.sk_socket->wq); + out: return rc; }