From patchwork Fri Dec 9 09:31:00 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 9467711 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 587F260586 for ; Fri, 9 Dec 2016 09:31:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 45702285DE for ; Fri, 9 Dec 2016 09:31:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3892A28609; Fri, 9 Dec 2016 09:31:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, T_DKIM_INVALID, T_TVD_MIME_EPI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 17161285DE for ; Fri, 9 Dec 2016 09:31:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932700AbcLIJbX (ORCPT ); Fri, 9 Dec 2016 04:31:23 -0500 Received: from mail-oi0-f45.google.com ([209.85.218.45]:33212 "EHLO mail-oi0-f45.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932360AbcLIJbV (ORCPT ); Fri, 9 Dec 2016 04:31:21 -0500 Received: by mail-oi0-f45.google.com with SMTP id w63so13181341oiw.0 for ; Fri, 09 Dec 2016 01:31:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=profitbricks-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=UJx69S2pK4+V31p8MRJo8FCUtpazdFECkeE9bhmT8pQ=; b=I47bQFQmt3PxAw31sUJRKZ5Jj0Wl0rR0UGjaIdMBJAyCdL0LECIEmiDUA7aJhhJUA5 wkC2YqhZT/oRmYiZWEj1OPvw1uIW/BHTmYq9MZ1hURmeBxRJeHQ3D7GZktzegtTd8h9m 0dBeeG97acMe0Jxct1AjIE0ue2tFrM4tM8ngmuSrPKeaAI/cO5Z37Cw6T0zytUrKgJA9 rEQ+vCJGcJf2B81mtACQTzr9FFGDAiE3tZirlyD1JYttulJDO2mbBQwLz3Z8bStO7Xfw DuZz8X0n+rJvya2AhahcRvxlKB3+fDvwYk8lCwjtyDyywMY0hYZQL3T7L8YWYTHGBUPI llmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=UJx69S2pK4+V31p8MRJo8FCUtpazdFECkeE9bhmT8pQ=; b=Fr84z11YxoXtCD/w+MR52h6xIyNuE8+feX6ZOpsVSZXJRjtP9l3EionEx6VODh+CRa hURW/gxmph+YYTQJpgOQYhZxcMxZ78n/ZVY+7NlTG60ElwxX2kDfTapEaE7oXX5fgCZR KHTBQ1i7wdcqkhHW73hoHK+mT1gqH/hU4sZJOw6YWm9VGsx1IufSjhYurqOJ/8OgwGUX wMbn6IRAK74EBqYDtkPl5c2OVvYW+B8AqdeiOYV8ZpcNHsotqaWGM2AWcU05MsnTjExQ 6rroH/tK3SVTrcrT6em1kwFbBUfENChAtqucC5k+8Q1KAaGuaKorNW+3HnVFAIFnk/ZN dOoQ== X-Gm-Message-State: AKaTC015itKpdGupxrmb3O4m4R4xAhsVUVXrTh8VK1nQmmeeAJEBmaSkUd4m2tnmDpvDwDVfLMXVYQujPj8sQpmx X-Received: by 10.157.46.149 with SMTP id w21mr40230426ota.219.1481275880939; Fri, 09 Dec 2016 01:31:20 -0800 (PST) MIME-Version: 1.0 Received: by 10.182.165.135 with HTTP; Fri, 9 Dec 2016 01:31:00 -0800 (PST) In-Reply-To: <1828884A29C6694DAF28B7E6B8A82373AB0BC8BE@ORSMSX109.amr.corp.intel.com> References: <1828884A29C6694DAF28B7E6B8A82373AB0BC8BE@ORSMSX109.amr.corp.intel.com> From: Jinpu Wang Date: Fri, 9 Dec 2016 10:31:00 +0100 Message-ID: Subject: Re: [PATCH] cma: resolve to first active IB port To: "Hefty, Sean" Cc: Doug Ledford , Hal Rosenstock , "linux-rdma@vger.kernel.org" , Michael Wang Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Hefty, On Thu, Dec 8, 2016 at 6:47 PM, Hefty, Sean wrote: >> > diff --git a/drivers/infiniband/core/cma.c >> b/drivers/infiniband/core/cma.c >> > index 2a6fc47..9e46b42 100644 >> > --- a/drivers/infiniband/core/cma.c >> > +++ b/drivers/infiniband/core/cma.c >> > @@ -684,6 +684,8 @@ static int cma_resolve_ib_dev(struct >> > rdma_id_private *id_priv) >> > for (i = 0; !ib_get_cached_gid(cur_dev->device, p, i, >> > &gid, NULL); >> > i++) { >> > + struct ib_port_attr attr; >> > + >> > if (!memcmp(&gid, dgid, sizeof(gid))) { >> > cma_dev = cur_dev; >> > sgid = gid; >> > @@ -692,7 +694,9 @@ static int cma_resolve_ib_dev(struct >> > rdma_id_private *id_priv) >> > } >> > >> > if (!cma_dev && (gid.global.subnet_prefix == >> > - dgid->global.subnet_prefix)) { >> > + dgid->global.subnet_prefix) && >> > + (!ib_query_port(cur_dev->device, p, &attr) && >> > + attr.state == IB_PORT_ACTIVE)) { > > I do have a concern about accessing non-cached port data as part of this call. Can we cache the port state and use that instead? Thanks for review. Sure, I updated the v2 patch as attached, is this what you want? Note: in cma_resolve_loopback, it does the same, I leave it like that for now, I can also update this part if you like. From f890782127b7febb11f90a1733a8220bcc944c47 Mon Sep 17 00:00:00 2001 From: Jack Wang Date: Tue, 6 Dec 2016 09:01:04 +0100 Subject: [PATCH] cma: resolve to first active ib_port When resolve addr if we don't give src addr, cma core will try to resolve to ib device on itself, current logic is only check if it has same subnet_prefix, which is not enough if we use default well known gid, we should also check if port is active. v2: cache port_active state in cma_add_one, also register event_handler to track port events. Signed-off-by: Jack Wang --- drivers/infiniband/core/cma.c | 42 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index 2a6fc47..c7a7689 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -151,10 +151,12 @@ static struct idr *cma_pernet_idr(struct net *net, enum rdma_port_space ps) struct cma_device { struct list_head list; struct ib_device *device; + struct ib_event_handler event_handler; struct completion comp; atomic_t refcount; struct list_head id_list; enum ib_gid_type *default_gid_type; + int *port_active; }; struct rdma_bind_list { @@ -692,7 +694,8 @@ static int cma_resolve_ib_dev(struct rdma_id_private *id_priv) } if (!cma_dev && (gid.global.subnet_prefix == - dgid->global.subnet_prefix)) { + dgid->global.subnet_prefix) && + cur_dev->port_active[p]) { cma_dev = cur_dev; sgid = gid; id_priv->id.port_num = p; @@ -4170,17 +4173,43 @@ static struct notifier_block cma_nb = { .notifier_call = cma_netdev_callback }; +static void cma_event_handler(struct ib_event_handler *handler, + struct ib_event *event) +{ + struct cma_device *cma_dev = + container_of(handler, typeof(*cma_dev), event_handler); + u8 port = event->element.port_num; + + /* we're only interested in port Up/Down events */ + if ( event->event != IB_EVENT_PORT_ACTIVE && + event->event != IB_EVENT_PORT_ERR) + return; + + /* cache the state of the port */ + if (event->event == IB_EVENT_PORT_ACTIVE) + cma_dev->port_active[port] = 1; + else + cma_dev->port_active[port] = 0; +} + static void cma_add_one(struct ib_device *device) { struct cma_device *cma_dev; struct rdma_id_private *id_priv; unsigned int i; unsigned long supported_gids = 0; + struct ib_port_attr port_attr; cma_dev = kmalloc(sizeof *cma_dev, GFP_KERNEL); if (!cma_dev) return; + cma_dev->port_active = kmalloc(sizeof (*cma_dev->port_active) * + (device->phys_port_cnt + 1), GFP_KERNEL); + if (!cma_dev->port_active) { + kfree(cma_dev); + return; + } cma_dev->device = device; cma_dev->default_gid_type = kcalloc(device->phys_port_cnt, sizeof(*cma_dev->default_gid_type), @@ -4194,12 +4223,21 @@ static void cma_add_one(struct ib_device *device) WARN_ON(!supported_gids); cma_dev->default_gid_type[i - rdma_start_port(device)] = find_first_bit(&supported_gids, BITS_PER_LONG); + if (!ib_query_port(cma_dev->device, i, &port_attr)) { + cma_dev->port_active[i] = + port_attr.state == IB_PORT_ACTIVE ? 1 : 0; + } else + cma_dev->port_active[i] = 0; } init_completion(&cma_dev->comp); atomic_set(&cma_dev->refcount, 1); INIT_LIST_HEAD(&cma_dev->id_list); ib_set_client_data(device, &cma_client, cma_dev); + INIT_IB_EVENT_HANDLER(&cma_dev->event_handler, device, + cma_event_handler); + if (ib_register_event_handler(&cma_dev->event_handler)) + pr_warn("fail to register event handler\n"); mutex_lock(&lock); list_add_tail(&cma_dev->list, &dev_list); @@ -4269,12 +4307,14 @@ static void cma_remove_one(struct ib_device *device, void *client_data) if (!cma_dev) return; + ib_unregister_event_handler(&cma_dev->event_handler); mutex_lock(&lock); list_del(&cma_dev->list); mutex_unlock(&lock); cma_process_remove(cma_dev); kfree(cma_dev->default_gid_type); + kfree(cma_dev->port_active); kfree(cma_dev); } -- 2.7.4