From patchwork Sat May 12 13:54:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhu Yanjun X-Patchwork-Id: 10395903 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E67B860236 for ; Sat, 12 May 2018 13:51:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D703428F33 for ; Sat, 12 May 2018 13:51:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C9BB128F4A; Sat, 12 May 2018 13:51:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5A56A28F33 for ; Sat, 12 May 2018 13:51:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751062AbeELNvP (ORCPT ); Sat, 12 May 2018 09:51:15 -0400 Received: from aserp2120.oracle.com ([141.146.126.78]:40656 "EHLO aserp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751033AbeELNvO (ORCPT ); Sat, 12 May 2018 09:51:14 -0400 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id w4CDos0P014601; Sat, 12 May 2018 13:50:54 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2017-10-26; bh=QQG/bIcH8CzhuKEdD5dwpQDarOhfLa0cEpsCZxDhbx4=; b=sr8XEhwCD611kobsqoCAZSat5JbAQK5FlKQMSwCwZyswXrCIQGWlDkYnqXT1MhV/Z8nl Vpq8A+tH0Tp4aZ+PnhngrWSXUQDkmjONkZjg6VHH8aDZmUSeOYOATnqx/pnIVJraHV9H tCVCxSO20OVebNdg4W5FNIcMo2lvlo/E5iY7K5U7IFfHkQFNtDqPKFWInGLjXeDM8GEn l3q6DvvtuAE27if1eP23AB9TYnoJh7l6FqBCHs6txmw6QkpKrSugoivbmjNZysPPrvsC VODIOgKXNa9+PWBBrvisdMyFicSSIW1y65rpVFjYNA/dda/WLvAORCqcMmPNKbBmTJ99 Ew== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by aserp2120.oracle.com with ESMTP id 2hwr8arqc9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 12 May 2018 13:50:53 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id w4CDoqRe001458 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 12 May 2018 13:50:53 GMT Received: from abhmp0017.oracle.com (abhmp0017.oracle.com [141.146.116.23]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id w4CDoqji020790; Sat, 12 May 2018 13:50:52 GMT Received: from office-bj2017.cn.oracle.com (/10.182.69.78) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sat, 12 May 2018 06:50:52 -0700 From: Zhu Yanjun To: monis@mellanox.com, dledford@redhat.com, jgg@ziepe.ca, linux-rdma@vger.kernel.org Subject: [PATCH 3/4] IB/rxe: use another udp port when error Date: Sat, 12 May 2018 09:54:27 -0400 Message-Id: <1526133268-23926-4-git-send-email-yanjun.zhu@oracle.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1526133268-23926-1-git-send-email-yanjun.zhu@oracle.com> References: <1526133268-23926-1-git-send-email-yanjun.zhu@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=8890 signatures=668698 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1711220000 definitions=main-1805120139 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When rxe can not send packets successfully, rxe will use another udp port to send. At the same time, rx will check the udp ports of the incoming packets. CC: Srinivas Eeda CC: Junxiao Bi Reviewed-by: Yuval Shaia Signed-off-by: Zhu Yanjun --- drivers/infiniband/sw/rxe/rxe.h | 3 +++ drivers/infiniband/sw/rxe/rxe_net.c | 52 ++++++++++++++++++++++++++++++++++--- 2 files changed, 51 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.h b/drivers/infiniband/sw/rxe/rxe.h index d831aeb..5164c50 100644 --- a/drivers/infiniband/sw/rxe/rxe.h +++ b/drivers/infiniband/sw/rxe/rxe.h @@ -69,6 +69,9 @@ #define IB_PHYS_STATE_LINK_DOWN (3) #define RXE_ROCE_V2_SPORT (0xc000) +#ifdef CONFIG_RDMA_RXE_DUAL_PORT_MODULE +#define RXE_ROCE_V2_DUAL_SPORT (0xd000) +#endif static inline u32 rxe_crc32(struct rxe_dev *rxe, u32 crc, void *next, size_t len) diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index fb5c7ff..29f6937 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -47,6 +47,11 @@ static LIST_HEAD(rxe_dev_list); static DEFINE_SPINLOCK(dev_list_lock); /* spinlock for device list */ +#ifdef CONFIG_RDMA_RXE_DUAL_PORT_MODULE +static DEFINE_SPINLOCK(port_lock); /* spinlock for port change */ +static __be16 dst_port = htons(ROCE_V2_UDP_DPORT); +static __be16 src_port = htons(RXE_ROCE_V2_SPORT); +#endif struct rxe_dev *net_to_rxe(struct net_device *ndev) { @@ -249,6 +254,30 @@ static struct dst_entry *rxe_find_route(struct rxe_dev *rxe, return dst; } +#ifdef CONFIG_RDMA_RXE_DUAL_PORT_MODULE +static void prepare_udp_hdr(struct sk_buff *skb, __be16 src_port, + __be16 dst_port); + +static inline void prepare_udp_hdr_spin(struct sk_buff *skb) +{ + __be16 dport, sport; + + spin_lock(&port_lock); + dport = dst_port; + sport = src_port; + spin_unlock(&port_lock); + prepare_udp_hdr(skb, sport, dport); +} + +static inline void prepare_udp_ports_spin(__be16 dport, __be16 sport) +{ + spin_lock(&port_lock); + dst_port = dport; + src_port = sport; + spin_unlock(&port_lock); +} +#endif + static int rxe_udp_encap_recv(struct sock *sk, struct sk_buff *skb) { struct udphdr *udph; @@ -276,6 +305,11 @@ static int rxe_udp_encap_recv(struct sock *sk, struct sk_buff *skb) pkt->mask = RXE_GRH_MASK; pkt->paylen = be16_to_cpu(udph->len) - sizeof(*udph); +#ifdef CONFIG_RDMA_RXE_DUAL_PORT_MODULE + if (unlikely(udph->dest != dst_port)) + prepare_udp_ports_spin(udph->dest, udph->source); +#endif + return rxe_rcv(skb); drop: kfree_skb(skb); @@ -408,10 +442,12 @@ static int prepare4(struct rxe_dev *rxe, struct rxe_pkt_info *pkt, if (!memcmp(saddr, daddr, sizeof(*daddr))) pkt->mask |= RXE_LOOPBACK_MASK; - +#ifdef CONFIG_RDMA_RXE_DUAL_PORT_MODULE + prepare_udp_hdr_spin(skb); +#else prepare_udp_hdr(skb, htons(RXE_ROCE_V2_SPORT), htons(ROCE_V2_UDP_DPORT)); - +#endif prepare_ipv4_hdr(dst, skb, saddr->s_addr, daddr->s_addr, IPPROTO_UDP, av->grh.traffic_class, av->grh.hop_limit, df, xnet); @@ -439,10 +475,12 @@ static int prepare6(struct rxe_dev *rxe, struct rxe_pkt_info *pkt, if (!memcmp(saddr, daddr, sizeof(*daddr))) pkt->mask |= RXE_LOOPBACK_MASK; - +#ifdef CONFIG_RDMA_RXE_DUAL_PORT_MODULE + prepare_udp_hdr_spin(skb); +#else prepare_udp_hdr(skb, htons(RXE_ROCE_V2_SPORT), htons(ROCE_V2_UDP_DPORT)); - +#endif prepare_ipv6_hdr(dst, skb, saddr, daddr, IPPROTO_UDP, av->grh.traffic_class, av->grh.hop_limit); @@ -509,6 +547,12 @@ int rxe_send(struct rxe_pkt_info *pkt, struct sk_buff *skb) return -EINVAL; } +#ifdef CONFIG_RDMA_RXE_DUAL_PORT_MODULE + if (unlikely(err != NET_XMIT_SUCCESS)) + prepare_udp_ports_spin(htons(ROCE_V2_UDP_DUAL_DPORT), + htons(RXE_ROCE_V2_DUAL_SPORT)); +#endif + if (unlikely(net_xmit_eval(err))) { pr_debug("error sending packet: %d\n", err); return -EAGAIN;