From patchwork Thu May 4 18:02:09 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 9712447 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2FDD860362 for ; Thu, 4 May 2017 18:02:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2CCE2286B2 for ; Thu, 4 May 2017 18:02:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2029E286B8; Thu, 4 May 2017 18:02:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 120512844C for ; Thu, 4 May 2017 18:02:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756246AbdEDSC2 (ORCPT ); Thu, 4 May 2017 14:02:28 -0400 Received: from mail.kernel.org ([198.145.29.136]:58388 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755732AbdEDSCQ (ORCPT ); Thu, 4 May 2017 14:02:16 -0400 Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 9555720114; Thu, 4 May 2017 18:02:14 +0000 (UTC) Received: from localhost (unknown [213.57.247.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2EB8B2010B; Thu, 4 May 2017 18:02:11 +0000 (UTC) From: Leon Romanovsky To: Stephen Hemminger , Doug Ledford Cc: Leon Romanovsky , Jiri Pirko , Ariel Almog , Dennis Dalessandro , Ram Amrani , Bart Van Assche , Sagi Grimberg , Jason Gunthorpe , Christoph Hellwig , Or Gerlitz , Linux RDMA , Linux Netdev Subject: [RFC iproute2 1/8] rdma: Add basic infrastructure for RDMA tool Date: Thu, 4 May 2017 21:02:09 +0300 Message-Id: <20170504180216.7665-2-leon@kernel.org> X-Mailer: git-send-email 2.12.2 In-Reply-To: <20170504180216.7665-1-leon@kernel.org> References: <20170504180216.7665-1-leon@kernel.org> X-Virus-Scanned: ClamAV using ClamSMTP Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Leon Romanovsky RDMA devices are cross-functional devices from one side, but very tailored for the specific markets from another. Such diversity caused to spread of RDMA related configuration across various tools, e.g. devlink, ip, ethtool, ib specific and vendor specific solutions. This patch adds ability to fill device and port information by reading sysfs entries (legacy). All future configuration settings will be implemented in netlink format, to be aligned with iproute2 package. Signed-off-by: Leon Romanovsky --- Makefile | 2 +- rdma/.gitignore | 1 + rdma/Makefile | 15 +++ rdma/rdma.c | 96 +++++++++++++++++ rdma/rdma.h | 77 ++++++++++++++ rdma/utils.c | 313 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 503 insertions(+), 1 deletion(-) create mode 100644 rdma/.gitignore create mode 100644 rdma/Makefile create mode 100644 rdma/rdma.c create mode 100644 rdma/rdma.h create mode 100644 rdma/utils.c -- 2.12.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/Makefile b/Makefile index 18de7dcb..c255063b 100644 --- a/Makefile +++ b/Makefile @@ -52,7 +52,7 @@ WFLAGS += -Wmissing-declarations -Wold-style-definition -Wformat=2 CFLAGS := $(WFLAGS) $(CCOPTS) -I../include $(DEFINES) $(CFLAGS) YACCFLAGS = -d -t -v -SUBDIRS=lib ip tc bridge misc netem genl tipc devlink man +SUBDIRS=lib ip tc bridge misc netem genl tipc devlink rdma man LIBNETLINK=../lib/libnetlink.a ../lib/libutil.a LDLIBS += $(LIBNETLINK) diff --git a/rdma/.gitignore b/rdma/.gitignore new file mode 100644 index 00000000..51fb172b --- /dev/null +++ b/rdma/.gitignore @@ -0,0 +1 @@ +rdma diff --git a/rdma/Makefile b/rdma/Makefile new file mode 100644 index 00000000..65248b31 --- /dev/null +++ b/rdma/Makefile @@ -0,0 +1,15 @@ +include ../Config + +RDMA_OBJ = rdma.o utils.o +TARGETS=rdma + +all: $(TARGETS) $(LIBS) + +rdma: $(RDMA_OBJ) + $(QUIET_LINK)$(CC) $^ -o $@ + +install: all + install -m 0755 $(TARGETS) $(DESTDIR)$(SBINDIR) + +clean: + rm -f $(RDMA_OBJ) $(TARGETS) diff --git a/rdma/rdma.c b/rdma/rdma.c new file mode 100644 index 00000000..bc7d1483 --- /dev/null +++ b/rdma/rdma.c @@ -0,0 +1,96 @@ +/* + * rdma.c RDMA tool + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + * Authors: Leon Romanovsky + */ + +#include + +#include "rdma.h" +#include "SNAPSHOT.h" + +static void help(char *name) +{ + pr_out("Usage: %s [ OPTIONS ] OBJECT { COMMAND | help }\n" + "where OBJECT := { }\n" + " OPTIONS := { -V[ersion] }\n", name); +} + +static int obj_help(struct rdma *rd) +{ + help(rd->filename); + return 0; +} + +static int rd_cmd(struct rdma *rd) +{ + const struct rdma_obj objs[] = { + { NULL, obj_help }, + { "help", obj_help }, + { 0 } + }; + + return rdma_exec_cmd(rd, objs, "object"); +} + +static int rd_init(struct rdma *rd, int argc, char **argv, char *filename) +{ + rd->filename = filename; + rd->argc = argc; + rd->argv = argv; + INIT_LIST_HEAD(&rd->dev_map_list); + return 0; +} +static void rd_free(struct rdma *rd) +{ + dev_map_cleanup(rd); +} +int main(int argc, char **argv) +{ + char *filename; + static const struct option long_options[] = { + { "version", no_argument, NULL, 'V' }, + { "help", no_argument, NULL, 'h' }, + { NULL, 0, NULL, 0 } + }; + struct rdma rd; + int opt; + int err; + + filename = basename(argv[0]); + + while ((opt = getopt_long(argc, argv, "Vh", + long_options, NULL)) >= 0) { + + switch (opt) { + case 'V': + printf("%s utility, iproute2-ss%s\n", filename, SNAPSHOT); + return EXIT_SUCCESS; + case 'h': + help(filename); + return EXIT_SUCCESS; + default: + pr_err("Unknown option.\n"); + help(filename); + return EXIT_FAILURE; + } + } + + argc -= optind; + argv += optind; + + err = rd_init(&rd, argc, argv, filename); + if (err) + goto out; + + err = rd_cmd(&rd); + /* Always cleanup */ + rd_free(&rd); + +out: return (err) ? EXIT_FAILURE:EXIT_SUCCESS; +} diff --git a/rdma/rdma.h b/rdma/rdma.h new file mode 100644 index 00000000..156bb74c --- /dev/null +++ b/rdma/rdma.h @@ -0,0 +1,77 @@ +/* + * rdma.c RDMA tool + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + * Authors: Leon Romanovsky + */ +#ifndef _RDMA_TOOL_H_ +#define _RDMA_TOOL_H_ + +#include +#include +#include +#include +#include +#include +#include + +#include "list.h" + +#define pr_err(args...) fprintf(stderr, ##args) +#define pr_out(args...) fprintf(stdout, ##args) + +enum rt_protocols { + RT_PROTOCOL_IB, + RT_PROTOCOL_IWARP, + RT_PROTOCOL_ROCE_V1, + RT_PROTOCOL_ROCE_V2, + RT_PROTOCOL_OPA, + RT_NO_PROTOCOL +}; + +struct port_map { + struct list_head list; + char *ifname; + uint32_t idx; +}; + +struct dev_map { + struct list_head list; + char *dev_name; + uint32_t num_ports; + struct list_head port_map_list; + uint32_t idx; +}; + +struct rdma { + int argc; + char **argv; + char *filename; + struct list_head dev_map_list; +}; + +struct rdma_obj { + const char *cmd; + int (*func)(struct rdma *rd); +}; + +/* + * Parser interface + */ +bool rd_no_arg(struct rdma *rd); +uint32_t get_port_from_argv(struct rdma *rd); + +int rdma_exec_cmd(struct rdma *rd, const struct rdma_obj *o, const char *str); +int rdma_sysfs_read_ib(const char *name, int port, const char *field, char *res); + +/* + * Device manipulation + */ +int dev_map_init(struct rdma *rd); +void dev_map_cleanup(struct rdma *rd); +struct dev_map *dev_map_lookup(struct rdma *rd, bool allow_port_index); +#endif /* _RDMA_TOOL_H_ */ diff --git a/rdma/utils.c b/rdma/utils.c new file mode 100644 index 00000000..568d7c0a --- /dev/null +++ b/rdma/utils.c @@ -0,0 +1,313 @@ +/* + * utils.c RDMA tool + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version + * 2 of the License, or (at your option) any later version. + * + * Authors: Leon Romanovsky + */ + +#include +#include + +#include "rdma.h" + +/* + * This macro will be moved to generic layer, + * after code will be accepted. + * it is placed here to avoid rebases with upstream code. + */ +#define MAX(a, b) ((a) > (b) ? (a) : (b)) + +static int rd_argc(struct rdma *rd) +{ + return rd->argc; +} + +static char *rd_argv(struct rdma *rd) +{ + if (!rd_argc(rd)) + return NULL; + return *rd->argv; +} + +static int strcmpx(const char *str1, const char *str2) +{ + if (strlen(str1) > strlen(str2)) + return -1; + return strncmp(str1, str2, strlen(str1)); +} + +static bool rd_argv_match(struct rdma *rd, const char *pattern) +{ + if (!rd_argc(rd)) + return false; + return strcmpx(rd_argv(rd), pattern) == 0; +} + +static void rd_arg_inc(struct rdma *rd) +{ + if (!rd_argc(rd)) + return; + rd->argc--; + rd->argv++; +} + +bool rd_no_arg(struct rdma *rd) +{ + return rd_argc(rd) == 0; +} + +#define SYSFS_INFINIBAND "/sys/class/infiniband" +#define SYSFS_NET "/sys/class/net" +static int sysfs_read(const char *prefix, const char *name, int port, const char *field, char *res) +{ + char fpath[64]; + FILE *fentry; + int result; + + if (port == 0) + snprintf(fpath, 64, "%s/%s/%s", prefix, name, field); + else + snprintf(fpath, 64, "%s/%s/ports/%d/%s", prefix, name, port, field); + + fentry = fopen(fpath, "r"); + if (!fentry) + return -ENOENT; + + result = fread(res, 1, 4096, fentry); + /* Remove last "\n" */ + res[result-1] = '\0'; + fclose(fentry); + return 0; +} + +int rdma_sysfs_read_ib(const char *name, int port, const char *field, char *res) +{ + return sysfs_read(SYSFS_INFINIBAND, name, port, field, res); +} + +static int sysfs_read_net(const char *name, const char *field, char *res) +{ + return sysfs_read(SYSFS_NET, name, 0, field, res); +} + +static struct dev_map *dev_map_alloc(char *dev_name) +{ + struct dev_map *dev_map; + + dev_map = calloc(1, sizeof(*dev_map)); + if (!dev_map) + return NULL; + dev_map->dev_name = strdup(dev_name); + INIT_LIST_HEAD(&dev_map->port_map_list); + + return dev_map; +} + +static void port_map_free(struct port_map *port_map) +{ + free(port_map->ifname); + free(port_map); +} + +static void dev_map_free(struct dev_map *dev_map) +{ + struct port_map *port_map, *tmp; + + list_for_each_entry_safe(port_map, tmp, + &dev_map->port_map_list, list) { + list_del(&port_map->list); + port_map_free(port_map); + } + + free(dev_map->dev_name); + free(dev_map); +} + +void dev_map_cleanup(struct rdma *rd) +{ + struct dev_map *dev_map, *tmp; + + list_for_each_entry_safe(dev_map, tmp, + &rd->dev_map_list, list) { + list_del(&dev_map->list); + dev_map_free(dev_map); + } +} + +uint32_t get_port_from_argv(struct rdma *rd) +{ + char *slash; + + slash = strchr(rd_argv(rd), '/'); + /* if no port found, return 0 */ + return (slash) ? (atoi(slash + 1)):0; +} + +struct dev_map *dev_map_lookup(struct rdma *rd, bool allow_port_index) +{ + struct dev_map *dev_map; + char *dev_name; + char *slash; + + dev_name = strdup(rd_argv(rd)); + if (allow_port_index) { + slash = strrchr(dev_name, '/'); + if (slash) + *slash = '\0'; + } + + list_for_each_entry(dev_map, &rd->dev_map_list, list) + if (strcmp(dev_name, dev_map->dev_name) == 0) { + free(dev_name); + return dev_map; + } + + free(dev_name); + return NULL; +} + +static bool is_dots(char *name) +{ + if (!strcmp(name, ".") || !strcmp(name, "..")) + return true; + return false; +} + +static char *find_ifname(struct dev_map *dev_map, uint32_t port) +{ + struct dirent *dentry, *drentry; + char data[4096]; + char drpath[64]; + uint32_t net_port; + DIR *dir, *drdir; + char *ifname = NULL; + + dir = opendir(SYSFS_NET); + if (!dir) + return NULL; + + while ((dentry = readdir(dir))) { + if (is_dots(dentry->d_name)) + continue; + + if (sysfs_read_net(dentry->d_name, "dev_id", data)) + continue; + + /* handle up to 9 ports, due to insufficient handling of hex */ + net_port = strtoul(data, NULL, 16); + + snprintf(drpath, 64, "%s/%s/device/infiniband/", SYSFS_NET, dentry->d_name); + + drdir = opendir(drpath); + if (!drdir) + continue; + + while ((drentry = readdir(drdir))) { + if (is_dots(drentry->d_name)) + continue; + if (!strcmp(drentry->d_name, dev_map->dev_name) && + net_port == port - 1) { + ifname = strdup(dentry->d_name); + closedir(drdir); + return ifname; + } + } + + closedir(drdir); + + } + + closedir(dir); + return NULL; +} + +static struct port_map *port_map_alloc(struct dev_map *dev_map, uint32_t port) +{ + struct port_map *port_map; + + port_map = calloc(1, sizeof(*port_map)); + if (!port_map) + return NULL; + + port_map->idx = port; + /* + * Expensive operation in sysfs world, need to rescan all net devices. + * Hopefuly, it is one time operation. + * In netlink, it will be simpler + */ + port_map->ifname = find_ifname(dev_map, port); + return port_map; +} + +int dev_map_init(struct rdma *rd) +{ + struct dev_map *dev_map; + struct port_map *port_map; + struct dirent *dentry, *pentry; + uint32_t i = 1; + uint32_t num_ports = 0; + char ports_name[64]; + DIR *dir, *pdir; + dir = opendir(SYSFS_INFINIBAND); + if (!dir) + return -ENOENT; + + while ((dentry = readdir(dir))) { + num_ports = 0; + if (is_dots(dentry->d_name)) + continue; + + dev_map = dev_map_alloc(dentry->d_name); + if (!dev_map) + /* The main function will cleanup the allocations */ + return -ENOMEM; + list_add_tail(&dev_map->list, &rd->dev_map_list); + dev_map->idx = i; + i++; + + snprintf(ports_name, 64, "%s/%s/ports/", SYSFS_INFINIBAND, dentry->d_name); + pdir = opendir(ports_name); + while ((pentry = readdir(pdir))) { + int port; + if (is_dots(pentry->d_name)) + continue; + + port = atoi(pentry->d_name); + port_map = port_map_alloc(dev_map, port); + if (!port_map) + return -ENOENT; + list_add_tail(&port_map->list, &dev_map->port_map_list); + num_ports = MAX(num_ports, port); + } + closedir(pdir); + dev_map->num_ports = num_ports; + } + closedir(dir); + + /* num_ports == 0 => no devices in infiniband folder */ + return (num_ports) ? 0:(-ENOENT); +} + +int rdma_exec_cmd(struct rdma *rd, const struct rdma_obj *objs, const char *str) +{ + const struct rdma_obj *o; + + /* First argument in objs table is default variant */ + if (rd_no_arg(rd)) + return objs->func(rd); + + for (o = objs + 1; o->cmd; ++o) { + if (rd_argv_match(rd, o->cmd)) { + /* Move to next argument */ + rd_arg_inc(rd); + return o->func(rd); + } + } + + pr_err("Unknown %s '%s'.\n", str, rd_argv(rd)); + return 0; +}