From patchwork Tue Jun 3 20:04:08 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andreas Noever X-Patchwork-Id: 4290501 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Original-To: patchwork-linux-pci@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 2E4559F1D6 for ; Tue, 3 Jun 2014 20:06:56 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id C6F7420212 for ; Tue, 3 Jun 2014 20:06:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 64C3D201DD for ; Tue, 3 Jun 2014 20:06:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933624AbaFCUFO (ORCPT ); Tue, 3 Jun 2014 16:05:14 -0400 Received: from mail-wi0-f171.google.com ([209.85.212.171]:59320 "EHLO mail-wi0-f171.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934174AbaFCUFD (ORCPT ); Tue, 3 Jun 2014 16:05:03 -0400 Received: by mail-wi0-f171.google.com with SMTP id cc10so7139147wib.16 for ; Tue, 03 Jun 2014 13:05:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LmX4iGYu2PD03DhvcMuKnbLOcQcGZT4Mr/xdi9FPAdE=; b=U3gRvDZbdWx5Ooe7ncclDQObSK86XFR2pA54DzGUmKV9IoTfOn4yFEDa/2D8w3S8Xp HEd9h15XglmEf2B9ikKh+wKbFZTnV+6uAxATC0QQH6OeFNwr+he5lNJI//dfwFZHfBVw 4N2uD4/BPI0csaYfBvWeiB2nc45PxfCVnm/OJxVU+e5ki/WIKr6pZ05zZwhb7UagS+UK gk1qjn/Ms3+yyfbpQw8G1Htmpjkb7yWaCXVxcHezs8Y2BXwfiGhkAXbP2+0yQGu+Pi1H dsLV+b8OPbkdV2SA2TX7V66UDGc/AH9tdUwLO/LlwMx7wwvcvxfMKjmvfH4DfKMOfcL5 N0GA== X-Received: by 10.180.77.70 with SMTP id q6mr35886740wiw.28.1401825900980; Tue, 03 Jun 2014 13:05:00 -0700 (PDT) Received: from localhost.localdomain (77-58-151-250.dclient.hispeed.ch. [77.58.151.250]) by mx.google.com with ESMTPSA id vm8sm615536wjc.27.2014.06.03.13.05.00 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 03 Jun 2014 13:05:00 -0700 (PDT) From: Andreas Noever To: linux-kernel@vger.kernel.org, Matthew Garrett , Greg KH , Bjorn Helgaas , linux-pci@vger.kernel.org Cc: Andreas Noever Subject: [PATCH v5 11/15] thunderbolt: Add support for simple pci tunnels Date: Tue, 3 Jun 2014 22:04:08 +0200 Message-Id: <1401825852-4745-12-git-send-email-andreas.noever@gmail.com> X-Mailer: git-send-email 2.0.0 In-Reply-To: <1401825852-4745-1-git-send-email-andreas.noever@gmail.com> References: <1401825852-4745-1-git-send-email-andreas.noever@gmail.com> Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, T_DKIM_INVALID, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A pci downstream and pci upstream port can be connected through a tunnel. To establish the tunnel we have to setup two unidirectional paths between the two ports. Right now we only support paths with two hops (i.e. no chaining) and at most one pci device per thunderbolt device. Signed-off-by: Andreas Noever --- drivers/thunderbolt/Makefile | 2 +- drivers/thunderbolt/tb.c | 135 +++++++++++++++++++++++ drivers/thunderbolt/tb.h | 1 + drivers/thunderbolt/tunnel_pci.c | 232 +++++++++++++++++++++++++++++++++++++++ drivers/thunderbolt/tunnel_pci.h | 30 +++++ 5 files changed, 399 insertions(+), 1 deletion(-) create mode 100644 drivers/thunderbolt/tunnel_pci.c create mode 100644 drivers/thunderbolt/tunnel_pci.h diff --git a/drivers/thunderbolt/Makefile b/drivers/thunderbolt/Makefile index 3532f36..0122ca6 100644 --- a/drivers/thunderbolt/Makefile +++ b/drivers/thunderbolt/Makefile @@ -1,3 +1,3 @@ obj-${CONFIG_THUNDERBOLT} := thunderbolt.o -thunderbolt-objs := nhi.o ctl.o tb.o switch.o cap.o path.o +thunderbolt-objs := nhi.o ctl.o tb.o switch.o cap.o path.o tunnel_pci.o diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c index 1efcacc..177f61d 100644 --- a/drivers/thunderbolt/tb.c +++ b/drivers/thunderbolt/tb.c @@ -10,6 +10,7 @@ #include "tb.h" #include "tb_regs.h" +#include "tunnel_pci.h" /* enumeration & hot plug handling */ @@ -51,6 +52,124 @@ static void tb_scan_port(struct tb_port *port) tb_scan_switch(sw); } +/** + * tb_free_invalid_tunnels() - destroy tunnels of devices that have gone away + */ +static void tb_free_invalid_tunnels(struct tb *tb) +{ + struct tb_pci_tunnel *tunnel; + struct tb_pci_tunnel *n; + list_for_each_entry_safe(tunnel, n, &tb->tunnel_list, list) + { + if (tb_pci_is_invalid(tunnel)) { + tb_pci_deactivate(tunnel); + tb_pci_free(tunnel); + } + } +} + +/** + * find_pci_up_port() - return the first PCIe up port on @sw or NULL + */ +static struct tb_port *tb_find_pci_up_port(struct tb_switch *sw) +{ + int i; + for (i = 1; i <= sw->config.max_port_number; i++) + if (sw->ports[i].config.type == TB_TYPE_PCIE_UP) + return &sw->ports[i]; + return NULL; +} + +/** + * find_unused_down_port() - return the first inactive PCIe down port on @sw + */ +static struct tb_port *tb_find_unused_down_port(struct tb_switch *sw) +{ + int i; + int cap; + int res; + int data; + for (i = 1; i <= sw->config.max_port_number; i++) { + if (tb_is_upstream_port(&sw->ports[i])) + continue; + if (sw->ports[i].config.type != TB_TYPE_PCIE_DOWN) + continue; + cap = tb_find_cap(&sw->ports[i], TB_CFG_PORT, TB_CAP_PCIE); + if (cap <= 0) + continue; + res = tb_port_read(&sw->ports[i], &data, TB_CFG_PORT, cap, 1); + if (res < 0) + continue; + if (data & 0x80000000) + continue; + return &sw->ports[i]; + } + return NULL; +} + +/** + * tb_activate_pcie_devices() - scan for and activate PCIe devices + * + * This method is somewhat ad hoc. For now it only supports one device + * per port and only devices at depth 1. + */ +static void tb_activate_pcie_devices(struct tb *tb) +{ + int i; + int cap; + u32 data; + struct tb_switch *sw; + struct tb_port *up_port; + struct tb_port *down_port; + struct tb_pci_tunnel *tunnel; + /* scan for pcie devices at depth 1*/ + for (i = 1; i <= tb->root_switch->config.max_port_number; i++) { + if (tb_is_upstream_port(&tb->root_switch->ports[i])) + continue; + if (tb->root_switch->ports[i].config.type != TB_TYPE_PORT) + continue; + if (!tb->root_switch->ports[i].remote) + continue; + sw = tb->root_switch->ports[i].remote->sw; + up_port = tb_find_pci_up_port(sw); + if (!up_port) { + tb_sw_info(sw, "no PCIe devices found, aborting\n"); + continue; + } + + /* check whether port is already activated */ + cap = tb_find_cap(up_port, TB_CFG_PORT, TB_CAP_PCIE); + if (cap <= 0) + continue; + if (tb_port_read(up_port, &data, TB_CFG_PORT, cap, 1)) + continue; + if (data & 0x80000000) { + tb_port_info(up_port, + "PCIe port already activated, aborting\n"); + continue; + } + + down_port = tb_find_unused_down_port(tb->root_switch); + if (!down_port) { + tb_port_info(up_port, + "All PCIe down ports are occupied, aborting\n"); + continue; + } + tunnel = tb_pci_alloc(tb, up_port, down_port); + if (!tunnel) { + tb_port_info(up_port, + "PCIe tunnel allocation failed, aborting\n"); + continue; + } + + if (tb_pci_activate(tunnel)) { + tb_port_info(up_port, + "PCIe tunnel activation failed, aborting\n"); + tb_pci_free(tunnel); + } + + } +} /* hotplug handling */ @@ -101,6 +220,7 @@ static void tb_handle_hotplug(struct work_struct *work) if (port->remote) { tb_port_info(port, "unplugged\n"); tb_sw_set_unpplugged(port->remote->sw); + tb_free_invalid_tunnels(tb); tb_switch_free(port->remote->sw); port->remote = NULL; } else { @@ -118,6 +238,10 @@ static void tb_handle_hotplug(struct work_struct *work) } else if (port->remote->sw->config.depth > 1) { tb_sw_warn(port->remote->sw, "hotplug: chaining not supported\n"); + } else { + tb_sw_info(port->remote->sw, + "hotplug: activating pcie devices\n"); + tb_activate_pcie_devices(tb); } } out: @@ -154,8 +278,17 @@ static void tb_schedule_hotplug_handler(void *data, u64 route, u8 port, */ void thunderbolt_shutdown_and_free(struct tb *tb) { + struct tb_pci_tunnel *tunnel; + struct tb_pci_tunnel *n; + mutex_lock(&tb->lock); + /* tunnels are only present after everything has been initialized */ + list_for_each_entry_safe(tunnel, n, &tb->tunnel_list, list) { + tb_pci_deactivate(tunnel); + tb_pci_free(tunnel); + } + if (tb->root_switch) tb_switch_free(tb->root_switch); tb->root_switch = NULL; @@ -201,6 +334,7 @@ struct tb *thunderbolt_alloc_and_start(struct tb_nhi *nhi) tb->nhi = nhi; mutex_init(&tb->lock); mutex_lock(&tb->lock); + INIT_LIST_HEAD(&tb->tunnel_list); tb->wq = alloc_ordered_workqueue("thunderbolt", 0); if (!tb->wq) @@ -221,6 +355,7 @@ struct tb *thunderbolt_alloc_and_start(struct tb_nhi *nhi) /* Full scan to discover devices added before the driver was loaded. */ tb_scan_switch(tb->root_switch); + tb_activate_pcie_devices(tb); /* Allow tb_handle_hotplug to progress events */ tb->hotplug_active = true; diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h index 8bbdc2b..508abc4 100644 --- a/drivers/thunderbolt/tb.h +++ b/drivers/thunderbolt/tb.h @@ -100,6 +100,7 @@ struct tb { struct tb_ctl *ctl; struct workqueue_struct *wq; /* ordered workqueue for plug events */ struct tb_switch *root_switch; + struct list_head tunnel_list; /* list of active PCIe tunnels */ bool hotplug_active; /* * tb_handle_hotplug will stop progressing plug * events and exit if this is not set (it needs to diff --git a/drivers/thunderbolt/tunnel_pci.c b/drivers/thunderbolt/tunnel_pci.c new file mode 100644 index 0000000..baf1cd3 --- /dev/null +++ b/drivers/thunderbolt/tunnel_pci.c @@ -0,0 +1,232 @@ +/* + * Thunderbolt Cactus Ridge driver - PCIe tunnel + * + * Copyright (c) 2014 Andreas Noever + */ + +#include +#include + +#include "tunnel_pci.h" +#include "tb.h" + +#define __TB_TUNNEL_PRINT(level, tunnel, fmt, arg...) \ + do { \ + struct tb_pci_tunnel *__tunnel = (tunnel); \ + level(__tunnel->tb, "%llx:%x <-> %llx:%x (PCI): " fmt, \ + tb_route(__tunnel->down_port->sw), \ + __tunnel->down_port->port, \ + tb_route(__tunnel->up_port->sw), \ + __tunnel->up_port->port, \ + ## arg); \ + } while (0) + +#define tb_tunnel_WARN(tunnel, fmt, arg...) \ + __TB_TUNNEL_PRINT(tb_WARN, tunnel, fmt, ##arg) +#define tb_tunnel_warn(tunnel, fmt, arg...) \ + __TB_TUNNEL_PRINT(tb_warn, tunnel, fmt, ##arg) +#define tb_tunnel_info(tunnel, fmt, arg...) \ + __TB_TUNNEL_PRINT(tb_info, tunnel, fmt, ##arg) + +static void tb_pci_init_path(struct tb_path *path) +{ + path->egress_fc_enable = TB_PATH_SOURCE | TB_PATH_INTERNAL; + path->egress_shared_buffer = TB_PATH_NONE; + path->ingress_fc_enable = TB_PATH_ALL; + path->ingress_shared_buffer = TB_PATH_NONE; + path->priority = 3; + path->weight = 1; + path->drop_packages = 0; + path->nfc_credits = 0; +} + +/** + * tb_pci_alloc() - allocate a pci tunnel + * + * Allocate a PCI tunnel. The ports must be of type TB_TYPE_PCIE_UP and + * TB_TYPE_PCIE_DOWN. + * + * Currently only paths consisting of two hops are supported (that is the + * ports must be on "adjacent" switches). + * + * The paths are hard-coded to use hop 8 (the only working hop id available on + * my thunderbolt devices). Therefore at most ONE path per device may be + * activated. + * + * Return: Returns a tb_pci_tunnel on success or NULL on failure. + */ +struct tb_pci_tunnel *tb_pci_alloc(struct tb *tb, struct tb_port *up, + struct tb_port *down) +{ + struct tb_pci_tunnel *tunnel = kzalloc(sizeof(*tunnel), GFP_KERNEL); + if (!tunnel) + goto err; + tunnel->tb = tb; + tunnel->down_port = down; + tunnel->up_port = up; + INIT_LIST_HEAD(&tunnel->list); + tunnel->path_to_up = tb_path_alloc(up->sw->tb, 2); + if (!tunnel->path_to_up) + goto err; + tunnel->path_to_down = tb_path_alloc(up->sw->tb, 2); + if (!tunnel->path_to_down) + goto err; + tb_pci_init_path(tunnel->path_to_up); + tb_pci_init_path(tunnel->path_to_down); + + tunnel->path_to_up->hops[0].in_port = down; + tunnel->path_to_up->hops[0].in_hop_index = 8; + tunnel->path_to_up->hops[0].in_counter_index = -1; + tunnel->path_to_up->hops[0].out_port = tb_upstream_port(up->sw)->remote; + tunnel->path_to_up->hops[0].next_hop_index = 8; + + tunnel->path_to_up->hops[1].in_port = tb_upstream_port(up->sw); + tunnel->path_to_up->hops[1].in_hop_index = 8; + tunnel->path_to_up->hops[1].in_counter_index = -1; + tunnel->path_to_up->hops[1].out_port = up; + tunnel->path_to_up->hops[1].next_hop_index = 8; + + tunnel->path_to_down->hops[0].in_port = up; + tunnel->path_to_down->hops[0].in_hop_index = 8; + tunnel->path_to_down->hops[0].in_counter_index = -1; + tunnel->path_to_down->hops[0].out_port = tb_upstream_port(up->sw); + tunnel->path_to_down->hops[0].next_hop_index = 8; + + tunnel->path_to_down->hops[1].in_port = + tb_upstream_port(up->sw)->remote; + tunnel->path_to_down->hops[1].in_hop_index = 8; + tunnel->path_to_down->hops[1].in_counter_index = -1; + tunnel->path_to_down->hops[1].out_port = down; + tunnel->path_to_down->hops[1].next_hop_index = 8; + return tunnel; + +err: + if (tunnel) { + if (tunnel->path_to_down) + tb_path_free(tunnel->path_to_down); + if (tunnel->path_to_up) + tb_path_free(tunnel->path_to_up); + kfree(tunnel); + } + return NULL; +} + +/** + * tb_pci_free() - free a tunnel + * + * The tunnel must have been deactivated. + */ +void tb_pci_free(struct tb_pci_tunnel *tunnel) +{ + if (tunnel->path_to_up->activated || tunnel->path_to_down->activated) { + tb_tunnel_WARN(tunnel, "trying to free an activated tunnel\n"); + return; + } + tb_path_free(tunnel->path_to_up); + tb_path_free(tunnel->path_to_down); + kfree(tunnel); +} + +/** + * tb_pci_is_invalid - check whether an activated path is still valid + */ +bool tb_pci_is_invalid(struct tb_pci_tunnel *tunnel) +{ + WARN_ON(!tunnel->path_to_up->activated); + WARN_ON(!tunnel->path_to_down->activated); + + return tb_path_is_invalid(tunnel->path_to_up) + || tb_path_is_invalid(tunnel->path_to_down); +} + +/** + * tb_pci_port_active() - activate/deactivate PCI capability + * + * Return: Returns 0 on success or an error code on failure. + */ +static int tb_pci_port_active(struct tb_port *port, bool active) +{ + u32 word = active ? 0x80000000 : 0x0; + int cap = tb_find_cap(port, TB_CFG_PORT, TB_CAP_PCIE); + if (cap <= 0) { + tb_port_warn(port, "TB_CAP_PCIE not found: %d\n", cap); + return cap ? cap : -ENXIO; + } + return tb_port_write(port, &word, TB_CFG_PORT, cap, 1); +} + +/** + * tb_pci_restart() - activate a tunnel after a hardware reset + */ +int tb_pci_restart(struct tb_pci_tunnel *tunnel) +{ + int res; + tunnel->path_to_up->activated = false; + tunnel->path_to_down->activated = false; + + tb_tunnel_info(tunnel, "activating\n"); + + res = tb_path_activate(tunnel->path_to_up); + if (res) + goto err; + res = tb_path_activate(tunnel->path_to_down); + if (res) + goto err; + + res = tb_pci_port_active(tunnel->down_port, true); + if (res) + goto err; + + res = tb_pci_port_active(tunnel->up_port, true); + if (res) + goto err; + return 0; +err: + tb_tunnel_warn(tunnel, "activation failed\n"); + tb_pci_deactivate(tunnel); + return res; +} + +/** + * tb_pci_activate() - activate a tunnel + * + * Return: Returns 0 on success or an error code on failure. + */ +int tb_pci_activate(struct tb_pci_tunnel *tunnel) +{ + int res; + if (tunnel->path_to_up->activated || tunnel->path_to_down->activated) { + tb_tunnel_WARN(tunnel, + "trying to activate an already activated tunnel\n"); + return -EINVAL; + } + + res = tb_pci_restart(tunnel); + if (res) + return res; + + list_add(&tunnel->list, &tunnel->tb->tunnel_list); + return 0; +} + + + +/** + * tb_pci_deactivate() - deactivate a tunnel + */ +void tb_pci_deactivate(struct tb_pci_tunnel *tunnel) +{ + tb_tunnel_info(tunnel, "deactivating\n"); + /* + * TODO: enable reset by writing 0x04000000 to TB_CAP_PCIE + 1 on up + * port. Seems to have no effect? + */ + tb_pci_port_active(tunnel->up_port, false); + tb_pci_port_active(tunnel->down_port, false); + if (tunnel->path_to_down->activated) + tb_path_deactivate(tunnel->path_to_down); + if (tunnel->path_to_up->activated) + tb_path_deactivate(tunnel->path_to_up); + list_del_init(&tunnel->list); +} + diff --git a/drivers/thunderbolt/tunnel_pci.h b/drivers/thunderbolt/tunnel_pci.h new file mode 100644 index 0000000..a67f93c --- /dev/null +++ b/drivers/thunderbolt/tunnel_pci.h @@ -0,0 +1,30 @@ +/* + * Thunderbolt Cactus Ridge driver - PCIe tunnel + * + * Copyright (c) 2014 Andreas Noever + */ + +#ifndef TB_PCI_H_ +#define TB_PCI_H_ + +#include "tb.h" + +struct tb_pci_tunnel { + struct tb *tb; + struct tb_port *up_port; + struct tb_port *down_port; + struct tb_path *path_to_up; + struct tb_path *path_to_down; + struct list_head list; +}; + +struct tb_pci_tunnel *tb_pci_alloc(struct tb *tb, struct tb_port *up, + struct tb_port *down); +void tb_pci_free(struct tb_pci_tunnel *tunnel); +int tb_pci_activate(struct tb_pci_tunnel *tunnel); +int tb_pci_restart(struct tb_pci_tunnel *tunnel); +void tb_pci_deactivate(struct tb_pci_tunnel *tunnel); +bool tb_pci_is_invalid(struct tb_pci_tunnel *tunnel); + +#endif +