From patchwork Wed Apr 5 23:19:00 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andre Przywara X-Patchwork-Id: 9665919 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id AD6DE60352 for ; Wed, 5 Apr 2017 23:23:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9E6552816B for ; Wed, 5 Apr 2017 23:23:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9315228584; Wed, 5 Apr 2017 23:23:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B20282816B for ; Wed, 5 Apr 2017 23:23:27 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cvuEO-0004UQ-6q; Wed, 05 Apr 2017 23:20:52 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cvuEN-0004Sy-8E for xen-devel@lists.xenproject.org; Wed, 05 Apr 2017 23:20:51 +0000 Received: from [85.158.143.35] by server-8.bemta-6.messagelabs.com id 65/6D-03648-2DB75E85; Wed, 05 Apr 2017 23:20:50 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrMLMWRWlGSWpSXmKPExsVysyfVTfdi9dM IgzPtqhbft0xmcmD0OPzhCksAYxRrZl5SfkUCa8bkb+kFy0sqbuzoYm1gvBraxcjFISSwiVHi 5725zBDOXkaJvb/msHYxcnKwCehK7Lj5mhnEFhEIlZjz8xGQzcHBLFAp0b2IHyQsLOAh0TH1L ViYRUBV4tPBepAwr4CbxJo/55hAwhICchJX/iWAhDkF3CVeth5jAbGFgEquTZzKMoGRewEjwy pGjeLUorLUIl0jE72kosz0jJLcxMwcXUMDM73c1OLixPTUnMSkYr3k/NxNjEDPMgDBDsZ9HyM PMUpyMCmJ8ir4PIkQ4kvKT6nMSCzOiC8qzUktPsQow8GhJMHLBwwUIcGi1PTUirTMHGCIwaQl OHiURHi1QdK8xQWJucWZ6RCpU4yKUuK8vCAJAZBERmkeXBssrC8xykoJ8zICHSLEU5BalJtZg ir/ilGcg1FJmHdKFdAUnsy8Erjpr4AWMwEtfnLnIcjikkSElFQDY79Lj0lo4tItN65dapifrC tllVf8LjejSU6rqL0ixu/QnGlc3bM58g/VH9GfZ7Ercrl1wuf/Fw5NWzW7MG6vap7juaO8bo9 /2V/m7Nhx8e1+dtlT9xeK53Km+FzSS1soy7h3VrjBUb5pi/ryiqekB51a7bp6O+9Sj27ueK3P D94oLLRc/O/zbyWW4oxEQy3mouJEAMdy9pNmAgAA X-Env-Sender: andre.przywara@arm.com X-Msg-Ref: server-15.tower-21.messagelabs.com!1491434448!62302869!1 X-Originating-IP: [217.140.101.70] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG X-StarScan-Received: X-StarScan-Version: 9.4.12; banners=-,-,- X-VirusChecked: Checked Received: (qmail 58850 invoked from network); 5 Apr 2017 23:20:49 -0000 Received: from foss.arm.com (HELO foss.arm.com) (217.140.101.70) by server-15.tower-21.messagelabs.com with SMTP; 5 Apr 2017 23:20:49 -0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 72F66B16; Wed, 5 Apr 2017 16:20:48 -0700 (PDT) Received: from slackpad.lan (usa-sjc-mx-foss1.foss.arm.com [217.140.101.70]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D9DB73F4FF; Wed, 5 Apr 2017 16:20:46 -0700 (PDT) From: Andre Przywara To: Stefano Stabellini , Julien Grall Date: Thu, 6 Apr 2017 00:19:00 +0100 Message-Id: <1491434362-30310-9-git-send-email-andre.przywara@arm.com> X-Mailer: git-send-email 2.8.2 In-Reply-To: <1491434362-30310-1-git-send-email-andre.przywara@arm.com> References: <1491434362-30310-1-git-send-email-andre.przywara@arm.com> Cc: xen-devel@lists.xenproject.org, Shanker Donthineni , Vijay Kilari Subject: [Xen-devel] [PATCH v5 08/30] ARM: GICv3 ITS: introduce ITS command handling X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP To be able to easily send commands to the ITS, create the respective wrapper functions, which take care of the ring buffer. The first two commands we implement provide methods to map a collection to a redistributor (aka host core) and to flush the command queue (SYNC). Start using these commands for mapping one collection to each host CPU. As an ITS might choose between *two* ways of addressing a redistributor, we store both the MMIO base address as well as the processor number in a per-CPU variable to give each ITS what it wants. Signed-off-by: Andre Przywara Reviewed-by: Stefano Stabellini Reviewed-by: Julien Grall --- xen/arch/arm/gic-v3-its.c | 199 ++++++++++++++++++++++++++++++++++++++ xen/arch/arm/gic-v3-lpi.c | 28 ++++++ xen/arch/arm/gic-v3.c | 26 ++++- xen/include/asm-arm/gic_v3_defs.h | 2 + xen/include/asm-arm/gic_v3_its.h | 37 +++++++ 5 files changed, 291 insertions(+), 1 deletion(-) diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c index d8e978a..1ecd63b 100644 --- a/xen/arch/arm/gic-v3-its.c +++ b/xen/arch/arm/gic-v3-its.c @@ -19,11 +19,14 @@ */ #include +#include #include #include +#include #include #include #include +#include #define ITS_CMD_QUEUE_SZ SZ_1M @@ -38,6 +41,160 @@ bool gicv3_its_host_has_its(void) return !list_empty(&host_its_list); } +#define BUFPTR_MASK GENMASK_ULL(19, 5) +static int its_send_command(struct host_its *hw_its, const void *its_cmd) +{ + /* + * The command queue should actually never become full, if it does anyway + * and this situation is not resolved quickly, this points to a much + * bigger problem, probably an hardware error. + * So to cover the one-off case where we actually hit a full command + * queue, we introduce a small grace period to not give up too quickly. + * Given the usual multi-hundred MHz frequency the ITS usually runs with, + * one millisecond (for a single command) seem to be more than enough. + * But this value is rather arbitrarily chosen based on theoretical + * considerations. + */ + s_time_t deadline = NOW() + MILLISECS(1); + uint64_t readp, writep; + int ret = -EBUSY; + + /* No ITS commands from an interrupt handler (at the moment). */ + ASSERT(!in_irq()); + + spin_lock(&hw_its->cmd_lock); + + do { + readp = readq_relaxed(hw_its->its_base + GITS_CREADR) & BUFPTR_MASK; + writep = readq_relaxed(hw_its->its_base + GITS_CWRITER) & BUFPTR_MASK; + + if ( ((writep + ITS_CMD_SIZE) % ITS_CMD_QUEUE_SZ) != readp ) + { + ret = 0; + break; + } + + /* + * If the command queue is full, wait for a bit in the hope it drains + * before giving up. + */ + spin_unlock(&hw_its->cmd_lock); + cpu_relax(); + udelay(1); + spin_lock(&hw_its->cmd_lock); + } while ( NOW() <= deadline ); + + if ( ret ) + { + spin_unlock(&hw_its->cmd_lock); + if ( printk_ratelimit() ) + printk(XENLOG_WARNING "host ITS: command queue full.\n"); + return ret; + } + + memcpy(hw_its->cmd_buf + writep, its_cmd, ITS_CMD_SIZE); + if ( hw_its->flags & HOST_ITS_FLUSH_CMD_QUEUE ) + clean_and_invalidate_dcache_va_range(hw_its->cmd_buf + writep, + ITS_CMD_SIZE); + else + dsb(ishst); + + writep = (writep + ITS_CMD_SIZE) % ITS_CMD_QUEUE_SZ; + writeq_relaxed(writep & BUFPTR_MASK, hw_its->its_base + GITS_CWRITER); + + spin_unlock(&hw_its->cmd_lock); + + return 0; +} + +/* Wait for an ITS to finish processing all commands. */ +static int gicv3_its_wait_commands(struct host_its *hw_its) +{ + /* + * As there could be quite a number of commands in a queue, we will + * wait a bit longer than the one millisecond for a single command above. + * Again this value is based on theoretical considerations, actually the + * command queue should drain much faster. + */ + s_time_t deadline = NOW() + MILLISECS(100); + uint64_t readp, writep; + + do { + spin_lock(&hw_its->cmd_lock); + readp = readq_relaxed(hw_its->its_base + GITS_CREADR) & BUFPTR_MASK; + writep = readq_relaxed(hw_its->its_base + GITS_CWRITER) & BUFPTR_MASK; + spin_unlock(&hw_its->cmd_lock); + + if ( readp == writep ) + return 0; + + cpu_relax(); + udelay(1); + } while ( NOW() <= deadline ); + + return -ETIMEDOUT; +} + +static uint64_t encode_rdbase(struct host_its *hw_its, unsigned int cpu, + uint64_t reg) +{ + reg &= ~GENMASK_ULL(51, 16); + + reg |= gicv3_get_redist_address(cpu, hw_its->flags & HOST_ITS_USES_PTA); + + return reg; +} + +static int its_send_cmd_sync(struct host_its *its, unsigned int cpu) +{ + uint64_t cmd[4]; + + cmd[0] = GITS_CMD_SYNC; + cmd[1] = 0x00; + cmd[2] = encode_rdbase(its, cpu, 0x0); + cmd[3] = 0x00; + + return its_send_command(its, cmd); +} + +static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id, + unsigned int cpu) +{ + uint64_t cmd[4]; + + cmd[0] = GITS_CMD_MAPC; + cmd[1] = 0x00; + cmd[2] = encode_rdbase(its, cpu, collection_id); + cmd[2] |= GITS_VALID_BIT; + cmd[3] = 0x00; + + return its_send_command(its, cmd); +} + +/* Set up the (1:1) collection mapping for the given host CPU. */ +int gicv3_its_setup_collection(unsigned int cpu) +{ + struct host_its *its; + int ret; + + list_for_each_entry(its, &host_its_list, entry) + { + ret = its_send_cmd_mapc(its, cpu, cpu); + if ( ret ) + return ret; + + ret = its_send_cmd_sync(its, cpu); + if ( ret ) + return ret; + + ret = gicv3_its_wait_commands(its); + if ( ret ) + return ret; + } + + return 0; +} + #define BASER_ATTR_MASK \ ((0x3UL << GITS_BASER_SHAREABILITY_SHIFT) | \ (0x7UL << GITS_BASER_OUTER_CACHEABILITY_SHIFT) | \ @@ -182,6 +339,41 @@ retry: return -EINVAL; } +/* + * Before an ITS gets initialized, it should be in a quiescent state, where + * all outstanding commands and transactions have finished. + * So if the ITS is already enabled, turn it off and wait for all outstanding + * operations to get processed by polling the QUIESCENT bit. + */ +static int gicv3_disable_its(struct host_its *hw_its) +{ + uint32_t reg; + /* + * As we also need to wait for the command queue to drain, we use the same + * (arbitrary) timeout value as above for gicv3_its_wait_commands(). + */ + s_time_t deadline = NOW() + MILLISECS(100); + + reg = readl_relaxed(hw_its->its_base + GITS_CTLR); + if ( !(reg & GITS_CTLR_ENABLE) && (reg & GITS_CTLR_QUIESCENT) ) + return 0; + + writel_relaxed(reg & ~GITS_CTLR_ENABLE, hw_its->its_base + GITS_CTLR); + + do { + reg = readl_relaxed(hw_its->its_base + GITS_CTLR); + if ( reg & GITS_CTLR_QUIESCENT ) + return 0; + + cpu_relax(); + udelay(1); + } while ( NOW() <= deadline ); + + printk(XENLOG_ERR "ITS@%lx not quiescent.\n", hw_its->addr); + + return -ETIMEDOUT; +} + static int gicv3_its_init_single_its(struct host_its *hw_its) { uint64_t reg; @@ -191,10 +383,17 @@ static int gicv3_its_init_single_its(struct host_its *hw_its) if ( !hw_its->its_base ) return -ENOMEM; + ret = gicv3_disable_its(hw_its); + if ( ret ) + return ret; + reg = readq_relaxed(hw_its->its_base + GITS_TYPER); hw_its->devid_bits = GITS_TYPER_DEVICE_ID_BITS(reg); hw_its->evid_bits = GITS_TYPER_EVENT_ID_BITS(reg); hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg); + if ( reg & GITS_TYPER_PTA ) + hw_its->flags |= HOST_ITS_USES_PTA; + spin_lock_init(&hw_its->cmd_lock); for ( i = 0; i < GITS_BASER_NR_REGS; i++ ) { diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c index 7f99ec6..9d3df7f 100644 --- a/xen/arch/arm/gic-v3-lpi.c +++ b/xen/arch/arm/gic-v3-lpi.c @@ -45,6 +45,8 @@ static struct { } lpi_data; struct lpi_redist_data { + paddr_t redist_addr; + unsigned int redist_id; void *pending_table; }; @@ -52,6 +54,32 @@ static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist); #define MAX_NR_HOST_LPIS (lpi_data.max_host_lpi_ids - LPI_OFFSET) +/* + * An ITS can refer to redistributors in two ways: either by an ID (possibly + * the CPU number) or by its MMIO address. This is a hardware implementation + * choice, so we have to cope with both approaches. The GICv3 code calculates + * both values and calls this function to let the ITS store them when it's + * later required to provide them. This is done in a per-CPU variable. + */ +void gicv3_set_redist_address(paddr_t address, unsigned int redist_id) +{ + this_cpu(lpi_redist).redist_addr = address; + this_cpu(lpi_redist).redist_id = redist_id; +} + +/* + * Returns a redistributor's ID (either as an address or as an ID). + * This must be (and is) called only after it has been setup by the above + * function. + */ +uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta) +{ + if ( use_pta ) + return per_cpu(lpi_redist, cpu).redist_addr & GENMASK_ULL(51, 16); + else + return per_cpu(lpi_redist, cpu).redist_id << 16; +} + static int gicv3_lpi_allocate_pendtable(uint64_t *reg) { uint64_t val; diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c index 54d2235..a559e5e 100644 --- a/xen/arch/arm/gic-v3.c +++ b/xen/arch/arm/gic-v3.c @@ -665,8 +665,25 @@ static int __init gicv3_populate_rdist(void) if ( typer & GICR_TYPER_PLPIS ) { + paddr_t rdist_addr; + unsigned int procnum; int ret; + /* + * The ITS refers to redistributors either by their physical + * address or by their ID. Which one to use is an ITS + * choice. So determine those two values here (which we + * can do only here in GICv3 code) and tell the + * ITS code about it, so it can use them later to be able + * to address those redistributors accordingly. + */ + rdist_addr = gicv3.rdist_regions[i].base; + rdist_addr += ptr - gicv3.rdist_regions[i].map_base; + procnum = (typer & GICR_TYPER_PROC_NUM_MASK); + procnum >>= GICR_TYPER_PROC_NUM_SHIFT; + + gicv3_set_redist_address(rdist_addr, procnum); + ret = gicv3_lpi_init_rdist(ptr); if ( ret && ret != -ENODEV ) { @@ -704,7 +721,7 @@ static int __init gicv3_populate_rdist(void) static int gicv3_cpu_init(void) { - int i; + int i, ret; uint32_t priority; /* Register ourselves with the rest of the world */ @@ -714,6 +731,13 @@ static int gicv3_cpu_init(void) if ( gicv3_enable_redist() ) return -ENODEV; + if ( gicv3_its_host_has_its() ) + { + ret = gicv3_its_setup_collection(smp_processor_id()); + if ( ret ) + return ret; + } + /* Set priority on PPI and SGI interrupts */ priority = (GIC_PRI_IPI << 24 | GIC_PRI_IPI << 16 | GIC_PRI_IPI << 8 | GIC_PRI_IPI); diff --git a/xen/include/asm-arm/gic_v3_defs.h b/xen/include/asm-arm/gic_v3_defs.h index 6f02bb0..b7e5a47 100644 --- a/xen/include/asm-arm/gic_v3_defs.h +++ b/xen/include/asm-arm/gic_v3_defs.h @@ -103,6 +103,8 @@ #define GICR_TYPER_PLPIS (1U << 0) #define GICR_TYPER_VLPIS (1U << 1) #define GICR_TYPER_LAST (1U << 4) +#define GICR_TYPER_PROC_NUM_SHIFT 8 +#define GICR_TYPER_PROC_NUM_MASK (0xffff << GICR_TYPER_PROC_NUM_SHIFT) /* For specifying the inner cacheability type only */ #define GIC_BASER_CACHE_nCnB 0ULL diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h index a68ccf3..7d7ec32 100644 --- a/xen/include/asm-arm/gic_v3_its.h +++ b/xen/include/asm-arm/gic_v3_its.h @@ -42,6 +42,7 @@ #define GITS_CTLR_QUIESCENT BIT(31) #define GITS_CTLR_ENABLE BIT(0) +#define GITS_TYPER_PTA BIT_ULL(19) #define GITS_TYPER_DEVIDS_SHIFT 13 #define GITS_TYPER_DEVIDS_MASK (0x1fUL << GITS_TYPER_DEVIDS_SHIFT) #define GITS_TYPER_DEVICE_ID_BITS(r) (((r & GITS_TYPER_DEVIDS_MASK) >> \ @@ -86,9 +87,26 @@ #define GITS_CBASER_SIZE_MASK 0xff +/* ITS command definitions */ +#define ITS_CMD_SIZE 32 + +#define GITS_CMD_MOVI 0x01 +#define GITS_CMD_INT 0x03 +#define GITS_CMD_CLEAR 0x04 +#define GITS_CMD_SYNC 0x05 +#define GITS_CMD_MAPD 0x08 +#define GITS_CMD_MAPC 0x09 +#define GITS_CMD_MAPTI 0x0a +#define GITS_CMD_MAPI 0x0b +#define GITS_CMD_INV 0x0c +#define GITS_CMD_INVALL 0x0d +#define GITS_CMD_MOVALL 0x0e +#define GITS_CMD_DISCARD 0x0f + #include #define HOST_ITS_FLUSH_CMD_QUEUE (1U << 0) +#define HOST_ITS_USES_PTA (1U << 1) /* data structure for each hardware ITS */ struct host_its { @@ -100,6 +118,7 @@ struct host_its { unsigned int devid_bits; unsigned int evid_bits; unsigned int itte_size; + spinlock_t cmd_lock; void *cmd_buf; unsigned int flags; }; @@ -120,6 +139,13 @@ int gicv3_lpi_init_rdist(void __iomem * rdist_base); int gicv3_lpi_init_host_lpis(unsigned int host_lpi_bits); int gicv3_its_init(void); +/* Store the physical address and ID for each redistributor as read from DT. */ +void gicv3_set_redist_address(paddr_t address, unsigned int redist_id); +uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta); + +/* Map a collection for this host CPU to each host ITS. */ +int gicv3_its_setup_collection(unsigned int cpu); + #else static LIST_HEAD(host_its_list); @@ -148,6 +174,17 @@ static inline int gicv3_its_init(void) return 0; } +static inline void gicv3_set_redist_address(paddr_t address, + unsigned int redist_id) +{ +} + +static inline int gicv3_its_setup_collection(unsigned int cpu) +{ + /* We should never get here without an ITS. */ + BUG(); +} + #endif /* CONFIG_HAS_ITS */ #endif