From patchwork Thu Jan 24 14:47:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Haakon Bugge X-Patchwork-Id: 10779145 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 09C44139A for ; Thu, 24 Jan 2019 14:48:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E91D230832 for ; Thu, 24 Jan 2019 14:48:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DAA1A30A5A; Thu, 24 Jan 2019 14:48:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DF2E930832 for ; Thu, 24 Jan 2019 14:48:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728030AbfAXOsA (ORCPT ); Thu, 24 Jan 2019 09:48:00 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:39294 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727649AbfAXOsA (ORCPT ); Thu, 24 Jan 2019 09:48:00 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id x0OEckBl023642; Thu, 24 Jan 2019 14:47:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2018-07-02; bh=hdWkoON56rkyYAxz8h/E8mIWO9COJ4IsKb9bnE5CdP8=; b=dIda9j0JP358VGv9SgXcqM62cOPyTSm5OV9N+IeC8WoZ6tTuF8ZIQEbnCBElVomd3Fcz FCv6zijZuEPuWmBmNiviHUtHWoDhgkloVmYxQHP6FE2H0BWNZVjLqvq/bZlsUB+et4kZ LS+Ju9z69urFKWP8+WNrvn8BoVcW6jVtUGBYRemYDIWAWWhkjS5dyYPLJfUYVe2NN5iY mAZkMNy4pr1qqmkXCaIy/rp+Sut478h96+RAUpYyP3oyFo8k+K+vDiOuxr68ur2wBI5A KcVy+731tQ52Aky8XzsM7BPM4ikEe1AM8V02vBoXaargjSF92QaDVtwn1ie7tb7F6M59 Zg== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2120.oracle.com with ESMTP id 2q3vhs034u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Jan 2019 14:47:50 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x0OElo1L023497 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Jan 2019 14:47:50 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x0OEloV6011690; Thu, 24 Jan 2019 14:47:50 GMT Received: from lab02.no.oracle.com (/10.172.144.56) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 24 Jan 2019 06:47:49 -0800 From: =?utf-8?q?H=C3=A5kon_Bugge?= To: jgg@mellanox.com Cc: dledford@redhat.com, hal@dev.mellanox.co.il, sean.hefty@intel.com, leon@kernel.org, linux-rdma@vger.kernel.org, ira.weiny@intel.com, aron.silverton@oracle.com, mark.haywood@oracle.com Subject: [PATCH 1/4] ibacm: Replace ioctl with netlink Date: Thu, 24 Jan 2019 15:47:35 +0100 Message-Id: <20190124144738.7961-2-haakon.bugge@oracle.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190124144738.7961-1-haakon.bugge@oracle.com> References: <20190124144738.7961-1-haakon.bugge@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9145 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=3 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901240104 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP ibacm uses ioctl with SIOCGIFCONF to enumerate the interfaces. This particular ioctl will only return interfaces that are "running". This is a problem if, for example, ibacm is started before the interfaces get plumbed, or when an interface gets resurrected. By using netlink (nl), we avoid using the sysfs interface to read the gid, as it is supplied directly by nl. This commit does not change the existing functionality, but is made to prepare for a fix where ibacm is unable to resurrect an interface. Detailed logging added: acm_if_iter: name: stib0 label: stib0 index: 9 flags: broadcast,multicast,up,running,lowerup addr: 192.168.200.200/24 pkey: 0x88b4 guid: 0x21280001a17c97 Change-Id: I6e428d1d6b2d345ccfbd74d482c69b56f8fe98da Signed-off-by: Håkon Bugge --- ibacm/CMakeLists.txt | 2 + ibacm/src/acm.c | 7 ++ ibacm/src/acm_util.c | 267 ++++++++++++++++++++++++------------------- ibacm/src/acm_util.h | 5 +- 4 files changed, 165 insertions(+), 116 deletions(-) diff --git a/ibacm/CMakeLists.txt b/ibacm/CMakeLists.txt index 3a4e2632..a0ef7035 100644 --- a/ibacm/CMakeLists.txt +++ b/ibacm/CMakeLists.txt @@ -7,6 +7,7 @@ publish_headers(infiniband include_directories("include") include_directories("src") include_directories("linux") +include_directories(${NL_INCLUDE_DIRS}) # NOTE: ibacm exports symbols from its own binary for use by ibacm rdma_sbin_executable(ibacm @@ -16,6 +17,7 @@ rdma_sbin_executable(ibacm target_link_libraries(ibacm LINK_PRIVATE ibverbs ibumad + ${NL_LIBRARIES} ${SYSTEMD_LIBRARIES} ${CMAKE_THREAD_LIBS_INIT} ${CMAKE_DL_LIBS} diff --git a/ibacm/src/acm.c b/ibacm/src/acm.c index 6453c5f0..e8d50a4a 100644 --- a/ibacm/src/acm.c +++ b/ibacm/src/acm.c @@ -3250,6 +3250,12 @@ int main(int argc, char **argv) acm_log(0, "Error: failed to create sa resp rcving thread"); return -1; } + + if (acm_init_if_iter_sys()) { + acm_log(0, "Error: unable to initialize acm_if_iter_sys"); + return -1; + } + acm_activate_devices(); acm_log(1, "starting server\n"); acm_server(systemd); @@ -3260,6 +3266,7 @@ int main(int argc, char **argv) acm_close_providers(); acm_stop_sa_handler(); umad_done(); + acm_fini_if_iter_sys(); fclose(flog); return 0; } diff --git a/ibacm/src/acm_util.c b/ibacm/src/acm_util.c index f4654f12..fc22432c 100644 --- a/ibacm/src/acm_util.c +++ b/ibacm/src/acm_util.c @@ -27,48 +27,28 @@ * SOFTWARE. */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include #include #include -#include #include -#include -#include #include -#include #include #include -#include +#include #include #include "acm_mad.h" #include "acm_util.h" -int acm_if_is_ib(char *ifname) -{ - unsigned type; - char buf[128]; - FILE *f; - int ret; - - snprintf(buf, sizeof buf, "//sys//class//net//%s//type", ifname); - f = fopen(buf, "r"); - if (!f) { - acm_log(0, "failed to open %s\n", buf); - return 0; - } - - if (fgets(buf, sizeof buf, f)) { - type = strtol(buf, NULL, 0); - ret = (type == ARPHRD_INFINIBAND); - } else { - acm_log(0, "failed to read interface type\n"); - ret = 0; - } - - fclose(f); - return ret; -} - int acm_if_get_pkey(char *ifname, uint16_t *pkey) { char buf[128], *end; @@ -122,103 +102,160 @@ int acm_if_get_sgid(char *ifname, union ibv_gid *sgid) return ret; } -int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx) +static struct nl_sock *sk = NULL; +static struct nl_cache *link_cache; +static struct nl_cache *addr_cache; + +int acm_init_if_iter_sys(void) { - struct ifconf *ifc; - struct ifreq *ifr; - char ip_str[INET6_ADDRSTRLEN]; - int family = AF_INET6; - int s, ret, i, len; - uint16_t pkey; - union ibv_gid sgid; - uint8_t addr_type; - uint8_t addr[ACM_MAX_ADDRESS]; - char *alias_sep; - - s = socket(family, SOCK_DGRAM, 0); - if (!s) { - family = AF_INET; - s = socket(family, SOCK_DGRAM, 0); - if (!s) - return -1; + int sts; + + sk = nl_socket_alloc(); + if (!sk) { + acm_log(0, "nl_socket_alloc"); + return -1; } - len = sizeof(*ifc) + sizeof(*ifr) * 64; - ifc = malloc(len); - if (!ifc) { - ret = -1; - goto out1; + sts = nl_connect(sk, NETLINK_ROUTE); + if (sts) { + acm_log(0, "nl_connect failed"); + goto out_connect; } - memset(ifc, 0, len); - ifc->ifc_len = len - sizeof(*ifc); - ifc->ifc_req = (struct ifreq *) (ifc + 1); - -retry_ioctl: - ret = ioctl(s, SIOCGIFCONF, ifc); - if (ret < 0) { - acm_log(0, "ioctl IPv%s ifconf error: %s\n", - (family == AF_INET6) ? "6" : "4", strerror(errno)); - if (family == AF_INET6) { - close(s); - family = AF_INET; - s = socket(family, SOCK_DGRAM, 0); - if (!s) { - free(ifc); - return ret; - } - goto retry_ioctl; - } - goto out2; + sts = rtnl_link_alloc_cache(sk, AF_UNSPEC, &link_cache); + if (sts) { + acm_log(0, "rtnl_link_alloc_cache failed"); + goto out_connect; } - ifr = ifc->ifc_req; - for (i = 0; i < ifc->ifc_len / sizeof(struct ifreq); i++) { - switch (ifr[i].ifr_addr.sa_family) { - case AF_INET: - addr_type = ACM_ADDRESS_IP; - memcpy(&addr, &((struct sockaddr_in *) &ifr[i].ifr_addr)->sin_addr, - sizeof addr); - inet_ntop(ifr[i].ifr_addr.sa_family, - &((struct sockaddr_in *) &ifr[i].ifr_addr)->sin_addr, - ip_str, sizeof ip_str); - break; - case AF_INET6: - addr_type = ACM_ADDRESS_IP6; - memcpy(&addr, &((struct sockaddr_in6 *) &ifr[i].ifr_addr)->sin6_addr, - sizeof addr); - inet_ntop(ifr[i].ifr_addr.sa_family, - &((struct sockaddr_in6 *) &ifr[i].ifr_addr)->sin6_addr, - ip_str, sizeof ip_str); - break; - default: - continue; - } + sts = rtnl_addr_alloc_cache(sk, &addr_cache); + if (sts) { + acm_log(0, "rtnl_addr_alloc_cache"); + goto out_addr; + } - acm_log(2, "%s\n", ifr[i].ifr_name); + return 0; - alias_sep = strchr(ifr[i].ifr_name, ':'); - if (alias_sep) - *alias_sep = '\0'; +out_addr: + nl_cache_free(link_cache); - if (!acm_if_is_ib(ifr[i].ifr_name)) - continue; +out_connect: + nl_close(sk); + return sts; +} - ret = acm_if_get_sgid(ifr[i].ifr_name, &sgid); - if (ret) - continue; +void acm_fini_if_iter_sys(void) +{ + nl_cache_free(link_cache); + nl_cache_free(addr_cache); + nl_close(sk); +} - ret = acm_if_get_pkey(ifr[i].ifr_name, &pkey); - if (ret) - continue; +static inline int af2acm_addr_type(int af) +{ + switch (af) { + case AF_INET: + return ACM_ADDRESS_IP; - cb(ifr[i].ifr_name, &sgid, pkey, addr_type, addr, ip_str, ctx); + case AF_INET6: + return ACM_ADDRESS_IP6; } - ret = 0; -out2: - free(ifc); -out1: - close(s); - return ret; + acm_log(0, "Unnkown address family\n"); + return ACM_ADDRESS_INVALID; +} + +struct ctx_and_cb { + void *ctx; + acm_if_iter_cb cb; +}; + +/* pre-decl to avoid compiler warning */ +void acm_if_iter(struct nl_object *obj, void *_ctx_and_cb); +void acm_if_iter(struct nl_object *obj, void *_ctx_and_cb) +{ + struct ctx_and_cb *ctx_cb = (struct ctx_and_cb *)_ctx_and_cb; + struct rtnl_addr *addr = (struct rtnl_addr *)obj; + uint8_t bin_addr[ACM_MAX_ADDRESS] = {}; + char ip_str[INET6_ADDRSTRLEN]; + struct nl_addr *link_addr; + unsigned long *guid_ptr; + struct rtnl_link *link; + char flags_str[128]; + union ibv_gid sgid; + struct nl_addr *a; + uint16_t pkey; + int addr_len; + char *label; + int flags; + int ret; + int af; + + link = rtnl_link_get(link_cache, rtnl_addr_get_ifindex(addr)); + flags = rtnl_link_get_flags(link); + + if (rtnl_link_get_arptype(link) != ARPHRD_INFINIBAND) + return; + + if (!(flags & IFF_RUNNING)) + return; + + if (!(a = rtnl_addr_get_local(addr))) + return; + + if ((addr_len = nl_addr_get_len(a)) > ACM_MAX_ADDRESS) { + acm_log(0, "address too long (%d)\n", addr_len); + return; + } + + af = nl_addr_get_family(a); + if (af != AF_INET && af != AF_INET6) + return; + + label = rtnl_addr_get_label(addr); + if (!label) + return; + + link_addr = rtnl_link_get_addr(link); + memcpy(sgid.raw, nl_addr_get_binary_addr(link_addr) + 4, sizeof(sgid)); + guid_ptr = (unsigned long *)&sgid.raw; + + ret = acm_if_get_pkey(rtnl_link_get_name(link), &pkey); + if (ret) + return; + + acm_log(2, "name: %5s label: %9s index: %2d flags: %s addr: %s pkey: 0x%04x guid: 0x%lx\n", + rtnl_link_get_name(link), label, + rtnl_addr_get_ifindex(addr), + rtnl_link_flags2str(flags, flags_str, sizeof(flags_str)), + nl_addr2str(a, ip_str, sizeof(ip_str)), pkey, + be64toh(*(guid_ptr + 1))); + + memcpy(&bin_addr, nl_addr_get_binary_addr(a), addr_len); + ctx_cb->cb(rtnl_link_get_name(link), &sgid, pkey, af2acm_addr_type(af), bin_addr, ip_str, ctx_cb->ctx); +} + + +int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx) +{ + struct ctx_and_cb ctx_cb; + int sts; + + sts = nl_cache_refill(sk, link_cache); + if (sts) { + acm_log(0, "nl_cache_refill link_cache"); + return sts; + } + + sts = nl_cache_refill(sk, addr_cache); + if (sts) { + acm_log(0, "nl_cache_refill addr_cache"); + return sts; + } + + ctx_cb.ctx = ctx; + ctx_cb.cb = cb; + nl_cache_foreach(addr_cache, acm_if_iter, (void *)&ctx_cb); + + return 0; } diff --git a/ibacm/src/acm_util.h b/ibacm/src/acm_util.h index cfa6f1a2..83b49edd 100644 --- a/ibacm/src/acm_util.h +++ b/ibacm/src/acm_util.h @@ -33,6 +33,7 @@ #include #include + #ifdef ACME_PRINTS #undef acm_log @@ -47,12 +48,14 @@ int acm_if_is_ib(char *ifname); int acm_if_get_pkey(char *ifname, uint16_t *pkey); int acm_if_get_sgid(char *ifname, union ibv_gid *sgid); - +int acm_init_if_iter_sys(void); +void acm_fini_if_iter_sys(void); typedef void (*acm_if_iter_cb)(char *ifname, union ibv_gid *gid, uint16_t pkey, uint8_t addr_type, uint8_t *addr, char *ip_str, void *ctx); int acm_if_iter_sys(acm_if_iter_cb cb, void *ctx); + char **parse(const char *args, int *count); #endif /* ACM_IF_H */ From patchwork Thu Jan 24 14:47:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Haakon Bugge X-Patchwork-Id: 10779147 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 39399139A for ; Thu, 24 Jan 2019 14:48:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 27CCF30832 for ; Thu, 24 Jan 2019 14:48:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1C1FF30A5A; Thu, 24 Jan 2019 14:48:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BBC6630832 for ; Thu, 24 Jan 2019 14:48:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728164AbfAXOsE (ORCPT ); Thu, 24 Jan 2019 09:48:04 -0500 Received: from aserp2130.oracle.com ([141.146.126.79]:54658 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727649AbfAXOsE (ORCPT ); Thu, 24 Jan 2019 09:48:04 -0500 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id x0OEcloX007371; Thu, 24 Jan 2019 14:47:53 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2018-07-02; bh=Yi/qg45ASOew9JiFmaYY0z8fofkxO866MExLnYfNkwU=; b=2kEQpfTQ60dQ1KexLyre0qPg21GzbgEL4+BEcvIM6kOkGhra+4Ms1hvoHCPBvTg9lMj+ vP1VQ+N0J9oxzT+up6KAjEDljCq2uOsQfz2P6YWxtRGwo1eIV/tePBiniP60zkQnnoVN zpQ3uLfPT9nGmMnIGzTpZ6qEPe5heoO63yPFr/P21U2/vym9YkDblLhIbAzqz55nqM3+ /Jnw+/4bq9cZ3fzqjxp4cEI3d0aW2u26VmhCcJuNpWp0DF4w+5Gy9fg9Xy3ug13P8X3V WETFfBbbGL5eNm1ZLTFtqJCDx/AyTbiU3V2oMnRF4MyXVFgWXrGWPainaUYcDe5MlTtc Yw== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by aserp2130.oracle.com with ESMTP id 2q3sdercap-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Jan 2019 14:47:53 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x0OElqqj016758 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Jan 2019 14:47:52 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x0OElqHs001385; Thu, 24 Jan 2019 14:47:52 GMT Received: from lab02.no.oracle.com (/10.172.144.56) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 24 Jan 2019 06:47:51 -0800 From: =?utf-8?q?H=C3=A5kon_Bugge?= To: jgg@mellanox.com Cc: dledford@redhat.com, hal@dev.mellanox.co.il, sean.hefty@intel.com, leon@kernel.org, linux-rdma@vger.kernel.org, ira.weiny@intel.com, aron.silverton@oracle.com, mark.haywood@oracle.com Subject: [PATCH 2/4] ibacm: Print correct pkey Date: Thu, 24 Jan 2019 15:47:36 +0100 Message-Id: <20190124144738.7961-3-haakon.bugge@oracle.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190124144738.7961-1-haakon.bugge@oracle.com> References: <20190124144738.7961-1-haakon.bugge@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9145 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901240104 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In commit 0161b49e3e1e ("ibacm: Unable to assign EP name for limited pkey"), a fix was introduced enabling an ipoib interface to be associated with its corresponding IB port, when a limited pkey was used. Said commit missed to change the debug print, such that the real pkey was printed. Fixes: 0161b49e3e1e ("ibacm: Unable to assign EP name for limited pkey") Change-Id: I5ec20a4a36d4e822cae0b906dd480dcf681ea0b5 Signed-off-by: Håkon Bugge Reviewed-by: Ira Weiny --- ibacm/src/acm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ibacm/src/acm.c b/ibacm/src/acm.c index e8d50a4a..a05a3cc0 100644 --- a/ibacm/src/acm.c +++ b/ibacm/src/acm.c @@ -2090,7 +2090,7 @@ static void acm_ep_ip_iter_cb(char *ifname, union ibv_gid *gid, uint16_t pkey, (ep->endpoint.pkey | IB_PKEY_FULL_MEMBER) == pkey) { if (!acm_ep_insert_addr(ep, ip_str, addr, addr_type)) { acm_log(0, "Added %s %s %d 0x%x from %s\n", ip_str, - dev->device.verbs->device->name, port_num, pkey, + dev->device.verbs->device->name, port_num, ep->endpoint.pkey, ifname); } } From patchwork Thu Jan 24 14:47:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Haakon Bugge X-Patchwork-Id: 10779149 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8DF9C1515 for ; Thu, 24 Jan 2019 14:48:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6A3E53087A for ; Thu, 24 Jan 2019 14:48:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5E84230A5B; Thu, 24 Jan 2019 14:48:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DAD263087A for ; Thu, 24 Jan 2019 14:48:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728288AbfAXOsJ (ORCPT ); Thu, 24 Jan 2019 09:48:09 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:39490 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727649AbfAXOsJ (ORCPT ); Thu, 24 Jan 2019 09:48:09 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.22/8.16.0.22) with SMTP id x0OEcvvS024378; Thu, 24 Jan 2019 14:48:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2018-07-02; bh=FsdzUqLJ+UV/OnARQOP/Go/lwffo9TPkidtuxEyOwbQ=; b=VKCCIMOm4vxO/ncEi+wXEPLtRMZ+dJdJMbO0wgbTa0z/+8VZx6OAOw2GMg/nMNb1DWt3 RA3FYeGTWemmdLWcEvIYGl+S6AwwKePH1AKldMQ+zSz5RctzQYfeYed8/uR2Ykl3eBlE oTNWUyWoQN2TfcMp/Y1pIiFbnjFn5LToGsZlL9kNnGGpBqmRnAwObi2B967MW5Fpzvxp wrUzZ7pWqWsON8BY6W9glsbi50I+cDNvIvF0eEfdX+6UOyaYIxLRPh9PzVGnegumPSct xZB6d6o9MN1PY8amHZaLVSUKrx83mLQQY2K/ZBME4ZQWx1tgDorXgBqRauKeYxXo5Ycp nA== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2120.oracle.com with ESMTP id 2q3vhs0364-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Jan 2019 14:48:00 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x0OElsmr016899 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Jan 2019 14:47:54 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x0OElsP2011716; Thu, 24 Jan 2019 14:47:54 GMT Received: from lab02.no.oracle.com (/10.172.144.56) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 24 Jan 2019 06:47:53 -0800 From: =?utf-8?q?H=C3=A5kon_Bugge?= To: jgg@mellanox.com Cc: dledford@redhat.com, hal@dev.mellanox.co.il, sean.hefty@intel.com, leon@kernel.org, linux-rdma@vger.kernel.org, ira.weiny@intel.com, aron.silverton@oracle.com, mark.haywood@oracle.com Subject: [PATCH 3/4] ibacm: Unable to resurrect an interface Date: Thu, 24 Jan 2019 15:47:37 +0100 Message-Id: <20190124144738.7961-4-haakon.bugge@oracle.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190124144738.7961-1-haakon.bugge@oracle.com> References: <20190124144738.7961-1-haakon.bugge@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9145 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901240104 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When an IB port has been brought back to Active state, after being down, ibacm gets an event about it. It will then (re) enumerate the devices, and does so by executing an ioctl with SIOCGIFCONF. This particular ioctl will only return interfaces that are "running". There may be a delay after the IB port becomes Active until its address has been provisioned, and becomes "running". If ibacm attempts to associate IPoIB interfaces to the port during this interval, it will not see the interface because it is not "running". Later, when ibacm is asked for a Path Record (PR) using the IP address of the resurrected IPoIB interface, it will not be able to find the associated end-point (EP), and the following is printed in the log: acm_svr_resolve_path: notice - unknown local end point address The bug can be provoked by the following script. We have a single HCA with two ports, the IPoIB interfaces are named stib{0,1}, the IP address of the first interface is 192.168.200.200, and the remote IP address is 192.168.200.202. The LID of the IB switch is 1 and the switch port number connected to port 1 of the HCA is 22. This fix depends on the commit that re-factors the use of ioctl in acm_if_iter(), and instead uses netlink. Now, by reducing the requirements of the state of the interface, the EP is added, and afterwards, when an address is assigned, it is associated with the EP. This commit is a new implementation of https://patchwork.kernel.org/patch/10748357, which was NAKed. Change-Id: Ideb7725c3bf29a1c87a335bdf374efd4ea16c007 Signed-off-by: Håkon Bugge --- ibacm/src/acm_util.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/ibacm/src/acm_util.c b/ibacm/src/acm_util.c index fc22432c..2c050aa0 100644 --- a/ibacm/src/acm_util.c +++ b/ibacm/src/acm_util.c @@ -187,19 +187,13 @@ void acm_if_iter(struct nl_object *obj, void *_ctx_and_cb) uint16_t pkey; int addr_len; char *label; - int flags; - int ret; int af; link = rtnl_link_get(link_cache, rtnl_addr_get_ifindex(addr)); - flags = rtnl_link_get_flags(link); if (rtnl_link_get_arptype(link) != ARPHRD_INFINIBAND) return; - if (!(flags & IFF_RUNNING)) - return; - if (!(a = rtnl_addr_get_local(addr))) return; @@ -213,23 +207,23 @@ void acm_if_iter(struct nl_object *obj, void *_ctx_and_cb) return; label = rtnl_addr_get_label(addr); - if (!label) - return; link_addr = rtnl_link_get_addr(link); + /* gid has a 4 byte offset into the link address */ memcpy(sgid.raw, nl_addr_get_binary_addr(link_addr) + 4, sizeof(sgid)); + /* guid is the least significant 64b of the gid */ guid_ptr = (unsigned long *)&sgid.raw; + ++guid_ptr; - ret = acm_if_get_pkey(rtnl_link_get_name(link), &pkey); - if (ret) + if (acm_if_get_pkey(rtnl_link_get_name(link), &pkey)) return; acm_log(2, "name: %5s label: %9s index: %2d flags: %s addr: %s pkey: 0x%04x guid: 0x%lx\n", rtnl_link_get_name(link), label, rtnl_addr_get_ifindex(addr), - rtnl_link_flags2str(flags, flags_str, sizeof(flags_str)), + rtnl_link_flags2str(rtnl_link_get_flags(link), flags_str, sizeof(flags_str)), nl_addr2str(a, ip_str, sizeof(ip_str)), pkey, - be64toh(*(guid_ptr + 1))); + be64toh(*guid_ptr)); memcpy(&bin_addr, nl_addr_get_binary_addr(a), addr_len); ctx_cb->cb(rtnl_link_get_name(link), &sgid, pkey, af2acm_addr_type(af), bin_addr, ip_str, ctx_cb->ctx); From patchwork Thu Jan 24 14:47:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Haakon Bugge X-Patchwork-Id: 10779151 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9D1101515 for ; Thu, 24 Jan 2019 14:48:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8B73A3087A for ; Thu, 24 Jan 2019 14:48:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7FC4830A5B; Thu, 24 Jan 2019 14:48:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 286023087A for ; Thu, 24 Jan 2019 14:48:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728335AbfAXOsL (ORCPT ); Thu, 24 Jan 2019 09:48:11 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:38914 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728170AbfAXOsK (ORCPT ); Thu, 24 Jan 2019 09:48:10 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id x0OEcUeu059622; Thu, 24 Jan 2019 14:48:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2018-07-02; bh=ZdhqM1mPf72cbCguQmhqHaa2cjIE0mCupe97irkeAOE=; b=B7Wcc3zqi2tqaT3Lb2AliIciRgjf+yKYEtaduS++qnI3C+GN14WNhep8OfLhGBtSpa6A QjtcGVUgYHZl+qXOV1JZXDMJ5flvZGceBMwdpl6kFwKT55z2ZLZnGEyFv6Og2FYsxfs8 0ZkHDOr88xl1nyLchcdXow6JfHoVTiT0sr/nB5EsHdoik9JvKECZosF1BTWTTuoUno/a Q2WZzKQfFSJIgLJJzub4pozAhyKp6XUM9FkludKmYJ2cLWYXO4kJhXYNUyrH52ILK3Rp 8d4L2bDSdMVUEVIETO5DYWJ3W98/aNihBYB9Zuz0BXmKHhxjSXgGWzRnB4w9jIE9oqOX yg== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2130.oracle.com with ESMTP id 2q3uav07h5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Jan 2019 14:48:02 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x0OEluee023796 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 24 Jan 2019 14:47:56 GMT Received: from abhmp0002.oracle.com (abhmp0002.oracle.com [141.146.116.8]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x0OEludf024053; Thu, 24 Jan 2019 14:47:56 GMT Received: from lab02.no.oracle.com (/10.172.144.56) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 24 Jan 2019 06:47:55 -0800 From: =?utf-8?q?H=C3=A5kon_Bugge?= To: jgg@mellanox.com Cc: dledford@redhat.com, hal@dev.mellanox.co.il, sean.hefty@intel.com, leon@kernel.org, linux-rdma@vger.kernel.org, ira.weiny@intel.com, aron.silverton@oracle.com, mark.haywood@oracle.com Subject: [PATCH 4/4] ibacm: Use alias name when possible Date: Thu, 24 Jan 2019 15:47:38 +0100 Message-Id: <20190124144738.7961-5-haakon.bugge@oracle.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190124144738.7961-1-haakon.bugge@oracle.com> References: <20190124144738.7961-1-haakon.bugge@oracle.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9145 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901240104 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When using ipoib child interfaces, the names displayed by the ip utility show: # ip addr show | grep inet' ' | grep ib inet 192.168.200.201/24 scope global stib1 inet 192.168.200.200/24 scope global secondary stib1:P03 However, the logging of ibacm uses the device names (slightly edited for better brevity): # grep acm_ep_ip_iter_cb /usr/local/var/log/ibacm.log Added 192.168.200.201/24 mlx4_0 2 0x84b4 from stib1 Added 192.168.200.200/24 mlx4_0 2 0x84b4 from stib1 This commit changes this by using the alias name when possible, and the above would read: # grep acm_ep_ip_iter_cb /usr/local/var/log/ibacm.log Added 192.168.200.201/24 mlx4_0 2 0x84b4 from stib1 Added 192.168.200.200/24 mlx4_0 2 0x84b4 from stib1:P03 Change-Id: I71e91f94f1e9c9037df96a87db57b9718b28eed3 Signed-off-by: Håkon Bugge Reviewed-by: Ira Weiny --- ibacm/src/acm_util.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/ibacm/src/acm_util.c b/ibacm/src/acm_util.c index 2c050aa0..172dcc63 100644 --- a/ibacm/src/acm_util.c +++ b/ibacm/src/acm_util.c @@ -226,7 +226,8 @@ void acm_if_iter(struct nl_object *obj, void *_ctx_and_cb) be64toh(*guid_ptr)); memcpy(&bin_addr, nl_addr_get_binary_addr(a), addr_len); - ctx_cb->cb(rtnl_link_get_name(link), &sgid, pkey, af2acm_addr_type(af), bin_addr, ip_str, ctx_cb->ctx); + ctx_cb->cb(label ? label : rtnl_link_get_name(link), + &sgid, pkey, af2acm_addr_type(af), bin_addr, ip_str, ctx_cb->ctx); }