| Message ID | 1611737738-1493-12-git-send-email-anshuman.khandual@arm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | arm64: coresight: Enable ETE and TRBE | expand |
Hi Anshuman On 1/27/21 8:55 AM, Anshuman Khandual wrote: > Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is > accessible via the system registers. The TRBE supports different addressing > modes including CPU virtual address and buffer modes including the circular > buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), > an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the > access to the trace buffer could be prohibited by a higher exception level > (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU > private interrupt (PPI) on address translation errors and when the buffer > is full. Overall implementation here is inspired from the Arm SPE driver. > > Cc: Mathieu Poirier <mathieu.poirier@linaro.org> > Cc: Mike Leach <mike.leach@linaro.org> > Cc: Suzuki K Poulose <suzuki.poulose@arm.com> > Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> This version looks functionally correct to me. There are some minor issues with the devm_ allocated memory and some driver hardening comments. I ran this on a model and have tested this with various scenarios. > --- > Changes in V3: > > - Added new DT bindings document TRBE.yaml > - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 > - Dropped isb() from trbe_reset_local() > - Dropped gap between (void *) and buf->trbe_base > - Changed 'int' to 'unsigned int' in is_trbe_available() > - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), > set_trbe_enabled() and set_trbe_limit_pointer() > - Changed get_trbe_flag_update(), is_trbe_programmable() and > get_trbe_address_align() to accept TRBIDR value > - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), > is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value > - Dropped snapshot mode condition in arm_trbe_alloc_buffer() > - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled > - Compute trbe_limit before trbe_write to get the updated handle > - Added trbe_stop_and_truncate_event() > - Dropped trbe_handle_fatal() > > Documentation/trace/coresight/coresight-trbe.rst | 39 + > arch/arm64/include/asm/sysreg.h | 1 + > drivers/hwtracing/coresight/Kconfig | 11 + > drivers/hwtracing/coresight/Makefile | 1 + > drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ > drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ > 6 files changed, 1235 insertions(+) > create mode 100644 Documentation/trace/coresight/coresight-trbe.rst > create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c > create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h > > diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst > new file mode 100644 > index 0000000..1cbb819 > --- /dev/null > +++ b/Documentation/trace/coresight/coresight-trbe.rst > @@ -0,0 +1,39 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +============================== > +Trace Buffer Extension (TRBE). > +============================== > + > + :Author: Anshuman Khandual <anshuman.khandual@arm.com> > + :Date: November 2020 > + > +Hardware Description > +-------------------- > + > +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system > +memory, CPU traces generated from a corresponding percpu tracing unit. This > +gets plugged in as a coresight sink device because the corresponding trace > +genarators (ETE), are plugged in as source device. > + > +The TRBE is not compliant to CoreSight architecture specifications, but is > +driven via the CoreSight driver framework to support the ETE (which is > +CoreSight compliant) integration. > + > +Sysfs files and directories > +--------------------------- > + > +The TRBE devices appear on the existing coresight bus alongside the other > +coresight devices:: > + > + >$ ls /sys/bus/coresight/devices > + trbe0 trbe1 trbe2 trbe3 > + > +The ``trbe<N>`` named TRBEs are associated with a CPU.:: > + > + >$ ls /sys/bus/coresight/devices/trbe0/ > + align dbm > + > +*Key file items are:-* > + * ``align``: TRBE write pointer alignment > + * ``dbm``: TRBE updates memory with access and dirty flags > + > diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h > index 85ae4db..9e2e9b7 100644 > --- a/arch/arm64/include/asm/sysreg.h > +++ b/arch/arm64/include/asm/sysreg.h > @@ -97,6 +97,7 @@ > #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) > #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) > #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) > +#define TSB_CSYNC __emit_inst(0xd503225f) > > #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) > #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) > diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig > index f154ae7..aa657ab 100644 > --- a/drivers/hwtracing/coresight/Kconfig > +++ b/drivers/hwtracing/coresight/Kconfig > @@ -164,6 +164,17 @@ config CORESIGHT_CTI > To compile this driver as a module, choose M here: the > module will be called coresight-cti. > > +config CORESIGHT_TRBE > + bool "Trace Buffer Extension (TRBE) driver" > + depends on ARM64 > + help > + This driver provides support for percpu Trace Buffer Extension (TRBE). > + TRBE always needs to be used along with it's corresponding percpu ETE > + component. ETE generates trace data which is then captured with TRBE. > + Unlike traditional sink devices, TRBE is a CPU feature accessible via > + system registers. But it's explicit dependency with trace unit (ETE) > + requires it to be plugged in as a coresight sink device. > + > config CORESIGHT_CTI_INTEGRATION_REGS > bool "Access CTI CoreSight Integration Registers" > depends on CORESIGHT_CTI > diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile > index f20e357..d608165 100644 > --- a/drivers/hwtracing/coresight/Makefile > +++ b/drivers/hwtracing/coresight/Makefile > @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o > obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o > obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o > obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o > +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o > coresight-cti-y := coresight-cti-core.o coresight-cti-platform.o \ > coresight-cti-sysfs.o > diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c > new file mode 100644 > index 0000000..1464d8b > --- /dev/null > +++ b/drivers/hwtracing/coresight/coresight-trbe.c > @@ -0,0 +1,1023 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight > + * sink device could then pair with an appropriate per-cpu coresight source > + * device (ETE) thus generating required trace data. Trace can be enabled > + * via the perf framework. > + * > + * Copyright (C) 2020 ARM Ltd. > + * > + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > + */ > +#define DRVNAME "arm_trbe" > + > +#define pr_fmt(fmt) DRVNAME ": " fmt > + > +#include "coresight-trbe.h" > + > +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) > + > +/* > + * A padding packet that will help the user space tools > + * in skipping relevant sections in the captured trace > + * data which could not be decoded. TRBE doesn't support > + * formatting the trace data, unlike the legacy CoreSight > + * sinks and thus we use ETE trace packets to pad the > + * sections of the buffer. > + */ > +#define ETE_IGNORE_PACKET 0x70 > + > +/* > + * Minimum amount of meaningful trace will contain: > + * A-Sync, Trace Info, Trace On, Address, Atom. > + * This is about 44bytes of ETE trace. To be on > + * the safer side, we assume 64bytes is the minimum > + * space required for a meaningful session, before > + * we hit a "WRAP" event. > + */ > +#define TRBE_TRACE_MIN_BUF_SIZE 64 > + > +enum trbe_fault_action { > + TRBE_FAULT_ACT_WRAP, > + TRBE_FAULT_ACT_SPURIOUS, > + TRBE_FAULT_ACT_FATAL, > +}; > + > +struct trbe_buf { > + unsigned long trbe_base; > + unsigned long trbe_limit; > + unsigned long trbe_write; > + int nr_pages; > + void **pages; > + bool snapshot; > + struct trbe_cpudata *cpudata; > +}; > + > +struct trbe_cpudata { > + bool trbe_dbm; > + u64 trbe_align; > + int cpu; > + enum cs_mode mode; > + struct trbe_buf *buf; > + struct trbe_drvdata *drvdata; > +}; > + > +struct trbe_drvdata { > + struct trbe_cpudata __percpu *cpudata; > + struct perf_output_handle __percpu **handle; > + struct hlist_node hotplug_node; > + int irq; > + cpumask_t supported_cpus; > + enum cpuhp_state trbe_online; > + struct platform_device *pdev; > +}; > + > +static int trbe_alloc_node(struct perf_event *event) > +{ > + if (event->cpu == -1) > + return NUMA_NO_NODE; > + return cpu_to_node(event->cpu); > +} > + > +static void trbe_drain_buffer(void) > +{ > + asm(TSB_CSYNC); > + dsb(nsh); > +} > + > +static void trbe_drain_and_disable_local(void) > +{ > + trbe_drain_buffer(); > + write_sysreg_s(0, SYS_TRBLIMITR_EL1); > + isb(); > +} > + > +static void trbe_reset_local(void) > +{ > + trbe_drain_and_disable_local(); > + write_sysreg_s(0, SYS_TRBPTR_EL1); > + write_sysreg_s(0, SYS_TRBBASER_EL1); > + write_sysreg_s(0, SYS_TRBSR_EL1); > +} > + > +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + /* > + * We cannot proceed with the buffer collection and we > + * do not have any data for the current session. The > + * etm_perf driver expects to close out the aux_buffer > + * at event_stop(). So disable the TRBE here and leave > + * the update_buffer() to return a 0 size. > + */ > + trbe_drain_and_disable_local(); > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > +} > + > +/* > + * TRBE Buffer Management > + * > + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, > + * it starts writing trace data from the write pointer onward till the limit pointer. > + * When the write pointer reaches the address just before the limit pointer, it gets > + * wrapped around again to the base pointer. This is called a TRBE wrap event, which > + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver > + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ > + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and > + * LIMIT pointers. > + * > + * Wrap around with an IRQ > + * ------ < ------ < ------- < ----- < ----- > + * | | > + * ------ > ------ > ------- > ----- > ----- > + * > + * +---------------+-----------------------+ > + * | | | > + * +---------------+-----------------------+ > + * Base Pointer Write Pointer Limit Pointer > + * > + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write > + * pointer can be aligned to the implementation defined TRBE trace buffer alignment > + * as captured in trbe_cpudata->trbe_align. > + * > + * > + * head tail wakeup > + * +---------------------------------------+----- ~ ~ ------ > + * |$$$$$$$|################|$$$$$$$$$$$$$$| | > + * +---------------------------------------+----- ~ ~ ------ > + * Base Pointer Write Pointer Limit Pointer > + * > + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing > + * values which tracks all the driver writes and user reads from the perf auxiliary > + * buffer. Generally [head..tail] is the area where the driver can write into unless > + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and > + * configured depending on the perf_output_handle indices, so that the driver does > + * not override into areas in the perf auxiliary buffer which is being or yet to be > + * consumed from the user space. The enabled TRBE buffer area is a moving subset of > + * the allocated perf auxiliary buffer. > + */ > +static void trbe_pad_buf(struct perf_output_handle *handle, int len) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + u64 head = PERF_IDX2OFF(handle->head, buf); > + > + memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); > + if (!buf->snapshot) > + perf_aux_output_skip(handle, len); > +} > + > +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + /* > + * The ETE trace has alignment synchronization packets allowing > + * the decoder to reset in case of an overflow or corruption. > + * So we can use the entire buffer for the snapshot mode. > + */ > + return buf->nr_pages * PAGE_SIZE; > +} > + > +/* > + * TRBE Limit Calculation > + * > + * The following markers are used to illustrate various TRBE buffer situations. > + * > + * $$$$ - Data area, unconsumed captured trace data, not to be overridden > + * #### - Free area, enabled, trace will be written > + * %%%% - Free area, disabled, trace will not be written > + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped > + */ > +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + struct trbe_cpudata *cpudata = buf->cpudata; > + const u64 bufsize = buf->nr_pages * PAGE_SIZE; > + u64 limit = bufsize; > + u64 head, tail, wakeup; > + > + head = PERF_IDX2OFF(handle->head, buf); > + > + /* > + * head > + * ------->| > + * | > + * head TRBE align tail > + * +----|-------|---------------|-------+ > + * |$$$$|=======|###############|$$$$$$$| > + * +----|-------|---------------|-------+ > + * trbe_base trbe_base + nr_pages > + * > + * Perf aux buffer output head position can be misaligned depending on > + * various factors including user space reads. In case misaligned, head > + * needs to be aligned before TRBE can be configured. Pad the alignment > + * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools > + * and skip this section thus advancing the head. > + */ > + if (!IS_ALIGNED(head, cpudata->trbe_align)) { > + unsigned long delta = roundup(head, cpudata->trbe_align) - head; > + > + delta = min(delta, handle->size); > + trbe_pad_buf(handle, delta); > + head = PERF_IDX2OFF(handle->head, buf); > + } > + > + /* > + * head = tail (size = 0) > + * +----|-------------------------------+ > + * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ | > + * +----|-------------------------------+ > + * trbe_base trbe_base + nr_pages > + * > + * Perf aux buffer does not have any space for the driver to write into. > + * Just communicate trace truncation event to the user space by marking > + * it with PERF_AUX_FLAG_TRUNCATED. > + */ > + if (!handle->size) { > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > + return 0; > + } > + > + /* Compute the tail and wakeup indices now that we've aligned head */ > + tail = PERF_IDX2OFF(handle->head + handle->size, buf); > + wakeup = PERF_IDX2OFF(handle->wakeup, buf); > + > + /* > + * Lets calculate the buffer area which TRBE could write into. There > + * are three possible scenarios here. Limit needs to be aligned with > + * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the > + * unconsumed data. > + * > + * 1) head < tail > + * > + * head tail > + * +----|-----------------------|-------+ > + * |$$$$|#######################|$$$$$$$| > + * +----|-----------------------|-------+ > + * trbe_base limit trbe_base + nr_pages > + * > + * TRBE could write into [head..tail] area. Unless the tail is right at > + * the end of the buffer, neither an wrap around nor an IRQ is expected > + * while being enabled. > + * > + * 2) head == tail > + * > + * head = tail (size > 0) > + * +----|-------------------------------+ > + * |%%%%|###############################| > + * +----|-------------------------------+ > + * trbe_base limit = trbe_base + nr_pages > + * > + * TRBE should just write into [head..base + nr_pages] area even though > + * the entire buffer is empty. Reason being, when the trace reaches the > + * end of the buffer, it will just wrap around with an IRQ giving an > + * opportunity to reconfigure the buffer. > + * > + * 3) tail < head > + * > + * tail head > + * +----|-----------------------|-------+ > + * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| > + * +----|-----------------------|-------+ > + * trbe_base limit = trbe_base + nr_pages > + * > + * TRBE should just write into [head..base + nr_pages] area even though > + * the [trbe_base..tail] is also empty. Reason being, when the trace > + * reaches the end of the buffer, it will just wrap around with an IRQ > + * giving an opportunity to reconfigure the buffer. > + */ > + if (head < tail) > + limit = round_down(tail, PAGE_SIZE); > + > + /* > + * Wakeup may be arbitrarily far into the future. If it's not in the > + * current generation, either we'll wrap before hitting it, or it's > + * in the past and has been handled already. > + * > + * If there's a wakeup before we wrap, arrange to be woken up by the > + * page boundary following it. Keep the tail boundary if that's lower. > + * > + * head wakeup tail > + * +----|---------------|-------|-------+ > + * |$$$$|###############|%%%%%%%|$$$$$$$| > + * +----|---------------|-------|-------+ > + * trbe_base limit trbe_base + nr_pages > + */ > + if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) > + limit = min(limit, round_up(wakeup, PAGE_SIZE)); > + > + /* > + * There are two situation when this can happen i.e limit is before > + * the head and hence TRBE cannot be configured. > + * > + * 1) head < tail (aligned down with PAGE_SIZE) and also they are both > + * within the same PAGE size range. > + * > + * PAGE_SIZE > + * |----------------------| > + * > + * limit head tail > + * +------------|------|--------|-------+ > + * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| > + * +------------|------|--------|-------+ > + * trbe_base trbe_base + nr_pages > + * > + * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both > + * head and wakeup are within same PAGE size range. > + * > + * PAGE_SIZE > + * |----------------------| > + * > + * limit head wakeup tail > + * +----|------|-------|--------|-------+ > + * |$$$$$$$$$$$|=======|========|$$$$$$$| > + * +----|------|-------|--------|-------+ > + * trbe_base trbe_base + nr_pages > + */ > + if (limit > head) > + return limit; > + > + trbe_pad_buf(handle, handle->size); > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > + return 0; > +} > + > +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = perf_get_aux(handle); > + u64 limit = __trbe_normal_offset(handle); > + u64 head = PERF_IDX2OFF(handle->head, buf); > + > + /* > + * If the head is too close to the limit and we don't > + * have space for a meaningful run, we rather pad it > + * and start fresh. > + */ > + if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { > + trbe_pad_buf(handle, limit - head); > + limit = __trbe_normal_offset(handle); > + } > + return limit; > +} > + > +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + unsigned long offset; > + > + if (buf->snapshot) > + offset = trbe_snapshot_offset(handle); > + else > + offset = trbe_normal_offset(handle); > + return buf->trbe_base + offset; > +} > + > +static void clr_trbe_status(void) > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + > + WARN_ON(is_trbe_enabled()); > + trbsr &= ~TRBSR_IRQ; > + trbsr &= ~TRBSR_TRG; > + trbsr &= ~TRBSR_WRAP; > + trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); > + trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); > + trbsr &= ~TRBSR_STOP; > + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > +} > + > +static void set_trbe_limit_pointer_enabled(unsigned long addr) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + > + trblimitr &= ~TRBLIMITR_NVM; > + trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); > + trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); > + trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); > + > + /* > + * Fill trace buffer mode is used here while configuring the > + * TRBE for trace capture. In this particular mode, the trace > + * collection is stopped and a maintenance interrupt is raised > + * when the current write pointer wraps. This pause in trace > + * collection gives the software an opportunity to capture the > + * trace data in the interrupt handler, before reconfiguring > + * the TRBE. > + */ > + trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; > + > + /* > + * Trigger mode is not used here while configuring the TRBE for > + * the trace capture. Hence just keep this in the ignore mode. > + */ > + trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; > + trblimitr |= (addr & PAGE_MASK); > + > + trblimitr |= TRBLIMITR_ENABLE; > + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); Personally, I prefer the isb() here rather than at the caller site, to make sure it is all contained here and more importantly we don't execute any other instruction before the "isb()" as the tracing may be enabled for the kernel. > +} > + > +static void trbe_enable_hw(struct trbe_buf *buf) > +{ > + WARN_ON(buf->trbe_write < buf->trbe_base); > + WARN_ON(buf->trbe_write >= buf->trbe_limit); > + set_trbe_disabled(); > + isb(); > + clr_trbe_status(); > + set_trbe_base_pointer(buf->trbe_base); > + set_trbe_write_pointer(buf->trbe_write); > + > + /* > + * Synchronize all the register updates > + * till now before enabling the TRBE. > + */ > + isb(); > + set_trbe_limit_pointer_enabled(buf->trbe_limit); > + > + /* Synchronize the TRBE enable event */ > + isb(); > +} > + > +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, > + struct perf_event *event, void **pages, > + int nr_pages, bool snapshot) > +{ > + struct trbe_buf *buf; > + struct page **pglist; > + int i; > + > + /* > + * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with > + * just a single page, there is not much room left while writing into > + * a partially filled TRBE buffer. Hence restrict the minimum buffer > + * size as two pages. > + */ > + if (nr_pages < 2) > + return NULL; > + > + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); > + if (IS_ERR(buf)) > + return ERR_PTR(-ENOMEM); > + > + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); > + if (IS_ERR(pglist)) { > + kfree(buf); > + return ERR_PTR(-ENOMEM); > + } > + > + for (i = 0; i < nr_pages; i++) > + pglist[i] = virt_to_page(pages[i]); > + > + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); minor nit: space after casting. > + if (IS_ERR((void *)buf->trbe_base)) { > + kfree(pglist); > + kfree(buf); > + return ERR_PTR(buf->trbe_base); > + } > + buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; > + buf->trbe_write = buf->trbe_base; > + buf->snapshot = snapshot; > + buf->nr_pages = nr_pages; > + buf->pages = pages; > + kfree(pglist); > + return buf; > +} > + > +void arm_trbe_free_buffer(void *config) > +{ > + struct trbe_buf *buf = config; > + > + vunmap((void *)buf->trbe_base); > + kfree(buf); > +} > + > +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, > + struct perf_output_handle *handle, > + void *config) > +{ > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > + struct trbe_buf *buf = config; > + unsigned long size, offset; > + > + WARN_ON(buf->cpudata != cpudata); > + WARN_ON(cpudata->cpu != smp_processor_id()); > + WARN_ON(cpudata->drvdata != drvdata); > + if (cpudata->mode != CS_MODE_PERF) > + return -EINVAL; Please return 0 here. As we are expected to pass an "unsigned" size. > + > + /* > + * If the TRBE was disabled due to lack of space in the AUX buffer or a > + * spurious fault, the driver leaves it disabled, truncating the buffer. > + * Since the etm_perf driver expects to close out the AUX buffer, the > + * driver skips it. Thus, just pass in 0 size here to indicate that the > + * buffer was truncated. > + */ > + if (!is_trbe_enabled()) > + return 0; > + /* > + * perf handle structure needs to be shared with the TRBE IRQ handler for > + * capturing trace data and restarting the handle. There is a probability > + * of an undefined reference based crash when etm event is being stopped > + * while a TRBE IRQ also getting processed. This happens due the release > + * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping > + * the TRBE here will ensure that no IRQ could be generated when the perf > + * handle gets freed in etm_event_stop(). > + */ > + trbe_drain_and_disable_local(); > + offset = get_trbe_write_pointer() - get_trbe_base_pointer(); > + size = offset - PERF_IDX2OFF(handle->head, buf); It may be a good idea to verify that the size computation here doesn't overflow. e.g broken interrupt could potentially leave us with "write == base". I understand this is a hardware issue, but the software can be resilient to not send bogus results. > + if (buf->snapshot) > + handle->head += size; > + return size; > +} > + > +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) > +{ > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > + struct perf_output_handle *handle = data; > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + WARN_ON(cpudata->cpu != smp_processor_id()); > + WARN_ON(cpudata->drvdata != drvdata); > + if (mode != CS_MODE_PERF) > + return -EINVAL; > + > + *this_cpu_ptr(drvdata->handle) = handle; > + cpudata->buf = buf; > + cpudata->mode = mode; > + buf->cpudata = cpudata; > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > + if (buf->trbe_limit == buf->trbe_base) { > + trbe_stop_and_truncate_event(handle); > + return 0; > + } > + trbe_enable_hw(buf); > + return 0; > +} > + > +static int arm_trbe_disable(struct coresight_device *csdev) > +{ > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > + struct trbe_buf *buf = cpudata->buf; > + > + WARN_ON(buf->cpudata != cpudata); > + WARN_ON(cpudata->cpu != smp_processor_id()); > + WARN_ON(cpudata->drvdata != drvdata); > + if (cpudata->mode != CS_MODE_PERF) > + return -EINVAL; > + > + trbe_drain_and_disable_local(); > + buf->cpudata = NULL; > + cpudata->buf = NULL; > + cpudata->mode = CS_MODE_DISABLED; > + return 0; > +} > + > +static void trbe_handle_spurious(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > + if (buf->trbe_limit == buf->trbe_base) { > + trbe_drain_and_disable_local(); > + return; > + } > + trbe_enable_hw(buf); > +} > + > +static void trbe_handle_overflow(struct perf_output_handle *handle) > +{ > + struct perf_event *event = handle->event; > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + unsigned long offset, size; > + struct etm_event_data *event_data; > + > + offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); > + size = offset - PERF_IDX2OFF(handle->head, buf); > + if (buf->snapshot) > + handle->head = offset; > + perf_aux_output_end(handle, size); > + > + event_data = perf_aux_output_begin(handle, event); > + if (!event_data) { We may add a comment here to explain how this would be handled ? e.g, /* * We are unable to restart the trace collection, * thus leave the TRBE disabled. The etm-perf driver * is able to detect this with a disconnnected handle * (handle->event = NULL). */ > + trbe_drain_and_disable_local(); > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > + return; > + } > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > + if (buf->trbe_limit == buf->trbe_base) { > + trbe_stop_and_truncate_event(handle); > + return; > + } > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; > + trbe_enable_hw(buf); > +} > + > +static bool is_perf_trbe(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + struct trbe_cpudata *cpudata = buf->cpudata; > + struct trbe_drvdata *drvdata = cpudata->drvdata; > + int cpu = smp_processor_id(); > + > + WARN_ON(buf->trbe_base != get_trbe_base_pointer()); > + WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); > + > + if (cpudata->mode != CS_MODE_PERF) > + return false; > + > + if (cpudata->cpu != cpu) > + return false; > + > + if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) > + return false; > + > + return true; > +} > + > +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + int ec = get_trbe_ec(trbsr); > + int bsc = get_trbe_bsc(trbsr); > + > + WARN_ON(is_trbe_running(trbsr)); > + if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) > + return TRBE_FAULT_ACT_FATAL; > + > + if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) > + return TRBE_FAULT_ACT_FATAL; > + > + if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { > + if (get_trbe_write_pointer() == get_trbe_base_pointer()) > + return TRBE_FAULT_ACT_WRAP; > + } > + return TRBE_FAULT_ACT_SPURIOUS; > +} > + > +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) > +{ > + struct perf_output_handle **handle_ptr = dev; > + struct perf_output_handle *handle = *handle_ptr; > + enum trbe_fault_action act; > + > + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); > + clr_trbe_irq(); > + > + /* > + * Ensure the trace is visible to the CPUs and > + * any external aborts have been resolved. > + */ > + trbe_drain_buffer(); > + isb(); > + > + if (!perf_get_aux(handle)) We may want to ensure that handle is not NULL, which can only happen if the hardware is not following the software. if (WARN_ON_ONCE(!handle) || !perf_get_aux(handle)) > + return IRQ_NONE; > + > + if (!is_perf_trbe(handle)) > + return IRQ_NONE; > + > + irq_work_run(); > + > + act = trbe_get_fault_act(handle); > + switch (act) { > + case TRBE_FAULT_ACT_WRAP: > + trbe_handle_overflow(handle); > + break; > + case TRBE_FAULT_ACT_SPURIOUS: > + trbe_handle_spurious(handle); > + break; > + case TRBE_FAULT_ACT_FATAL: > + trbe_stop_and_truncate_event(handle); > + break; > + } > + return IRQ_HANDLED; > +} > + > +static const struct coresight_ops_sink arm_trbe_sink_ops = { > + .enable = arm_trbe_enable, > + .disable = arm_trbe_disable, > + .alloc_buffer = arm_trbe_alloc_buffer, > + .free_buffer = arm_trbe_free_buffer, > + .update_buffer = arm_trbe_update_buffer, > +}; > + > +static const struct coresight_ops arm_trbe_cs_ops = { > + .sink_ops = &arm_trbe_sink_ops, > +}; > + > +static ssize_t align_show(struct device *dev, struct device_attribute *attr, char *buf) > +{ > + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > + > + return sprintf(buf, "%llx\n", cpudata->trbe_align); > +} > +static DEVICE_ATTR_RO(align); > + > +static ssize_t dbm_show(struct device *dev, struct device_attribute *attr, char *buf) > +{ > + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > + > + return sprintf(buf, "%d\n", cpudata->trbe_dbm); > +} > +static DEVICE_ATTR_RO(dbm); > + > +static struct attribute *arm_trbe_attrs[] = { > + &dev_attr_align.attr, > + &dev_attr_dbm.attr, > + NULL, > +}; > + > +static const struct attribute_group arm_trbe_group = { > + .attrs = arm_trbe_attrs, > +}; > + > +static const struct attribute_group *arm_trbe_groups[] = { > + &arm_trbe_group, > + NULL, > +}; > + > +static void arm_trbe_probe_coresight_cpu(void *info) > +{ > + struct trbe_drvdata *drvdata = info; > + struct coresight_desc desc = { 0 }; > + int cpu = smp_processor_id(); > + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); This must be done only after the "is_trbe_available()". Otherwise we get an UNDEFINED instruction abort. > + struct device *dev; > + > + if (WARN_ON(!cpudata)) > + goto cpu_clear; > + > + if (trbe_csdev) > + return; > + > + cpudata->cpu = smp_processor_id(); > + cpudata->drvdata = drvdata; > + dev = &cpudata->drvdata->pdev->dev; > + > + if (!is_trbe_available()) { > + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + > + if (!is_trbe_programmable(trbidr)) { > + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); > + if (IS_ERR(desc.name)) > + goto cpu_clear; > + > + desc.type = CORESIGHT_DEV_TYPE_SINK; > + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; > + desc.ops = &arm_trbe_cs_ops; > + desc.pdata = dev_get_platdata(dev); > + desc.groups = arm_trbe_groups; > + desc.dev = dev; > + trbe_csdev = coresight_register(&desc); > + if (IS_ERR(trbe_csdev)) > + goto cpu_clear; > + > + dev_set_drvdata(&trbe_csdev->dev, cpudata); > + cpudata->trbe_dbm = get_trbe_flag_update(trbidr); > + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); > + if (cpudata->trbe_align > SZ_2K) { > + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); > + goto cpu_clear; Should we unregister the coresight device in this case ? There is no point in having the device around if it is not supported. > + } > + per_cpu(csdev_sink, cpu) = trbe_csdev; > + trbe_reset_local(); > + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > + return; > +cpu_clear: > + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); > +} > + > +static void arm_trbe_remove_coresight_cpu(void *info) > +{ > + int cpu = smp_processor_id(); > + struct trbe_drvdata *drvdata = info; > + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > + nit: Is it better to use this_cpu_ptr() every where above/below to make it explicit ? > + if (trbe_csdev) { > + coresight_unregister(trbe_csdev); > + cpudata->drvdata = NULL; > + per_cpu(csdev_sink, cpu) = NULL; > + } > + disable_percpu_irq(drvdata->irq); > + trbe_reset_local(); > +} > + > +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) > +{ > + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); > + if (IS_ERR(drvdata->cpudata)) > + return PTR_ERR(drvdata->cpudata); > + > + arm_trbe_probe_coresight_cpu(drvdata); > + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); > + return 0; > +} > + > +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) > +{ > + arm_trbe_remove_coresight_cpu(drvdata); > + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); > + free_percpu(drvdata->cpudata); > + return 0; > +} > + > +static int arm_trbe_cpu_startup(unsigned int cpu, struct hlist_node *node) > +{ > + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > + > + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { nit: Add a comment ? /* * If this CPU was not probed for TRBE, initialize * it now. */ > + if (!per_cpu(csdev_sink, cpu)) { > + arm_trbe_probe_coresight_cpu(drvdata); > + } else { > + trbe_reset_local(); > + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > + } > + } > + return 0; > +} > + > +static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node) > +{ > + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > + > + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > + disable_percpu_irq(drvdata->irq); > + trbe_reset_local(); > + } > + return 0; > +} > + > +static int arm_trbe_probe_cpuhp(struct trbe_drvdata *drvdata) > +{ > + enum cpuhp_state trbe_online; > + > + trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, > + arm_trbe_cpu_startup, arm_trbe_cpu_teardown); > + if (trbe_online < 0) > + return -EINVAL; > + > + if (cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node)) > + return -EINVAL; > + > + drvdata->trbe_online = trbe_online; > + return 0; > +} > + > +static void arm_trbe_remove_cpuhp(struct trbe_drvdata *drvdata) > +{ > + cpuhp_remove_multi_state(drvdata->trbe_online); > +} > + > +static int arm_trbe_probe_irq(struct platform_device *pdev, > + struct trbe_drvdata *drvdata) > +{ > + drvdata->irq = platform_get_irq(pdev, 0); > + if (!drvdata->irq) { > + pr_err("IRQ not found for the platform device\n"); > + return -ENXIO; > + } > + > + if (!irq_is_percpu(drvdata->irq)) { > + pr_err("IRQ is not a PPI\n"); > + return -EINVAL; > + } > + > + if (irq_get_percpu_devid_partition(drvdata->irq, &drvdata->supported_cpus)) > + return -EINVAL; > + > + drvdata->handle = alloc_percpu(typeof(*drvdata->handle)); > + if (!drvdata->handle) > + return -ENOMEM; > + > + if (request_percpu_irq(drvdata->irq, arm_trbe_irq_handler, DRVNAME, drvdata->handle)) { > + free_percpu(drvdata->handle); > + return -EINVAL; > + } > + return 0; > +} > + > +static void arm_trbe_remove_irq(struct trbe_drvdata *drvdata) > +{ > + free_percpu_irq(drvdata->irq, drvdata->handle); > + free_percpu(drvdata->handle); > +} > + > +static int arm_trbe_device_probe(struct platform_device *pdev) > +{ > + struct coresight_platform_data *pdata; > + struct trbe_drvdata *drvdata; > + struct device *dev = &pdev->dev; > + int ret; > + > + drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL); > + if (IS_ERR(drvdata)) > + return -ENOMEM; > + > + pdata = coresight_get_platform_data(dev); Not sure if this is necessary. We don't have any CoreSight specific data in the bindings. > + if (IS_ERR(pdata)) { > + kfree(drvdata); This is not required. The cleanup should automatically free this up. In fact this will be problematic and will cause double free. > + return -ENOMEM; > + } > + > + dev_set_drvdata(dev, drvdata); > + dev->platform_data = pdata; > + drvdata->pdev = pdev; > + ret = arm_trbe_probe_irq(pdev, drvdata); > + if (ret) > + goto irq_failed; > + > + ret = arm_trbe_probe_coresight(drvdata); > + if (ret) > + goto probe_failed; > + > + ret = arm_trbe_probe_cpuhp(drvdata); > + if (ret) > + goto cpuhp_failed; > + > + return 0; > +cpuhp_failed: > + arm_trbe_remove_coresight(drvdata); > +probe_failed: > + arm_trbe_remove_irq(drvdata); > +irq_failed: > + kfree(pdata); > + kfree(drvdata); None of these "kfree" is needed. It will be automatically freed when the probe fails. > + return ret; > +} > + > +static int arm_trbe_device_remove(struct platform_device *pdev) > +{ > + struct coresight_platform_data *pdata = dev_get_platdata(&pdev->dev); > + struct trbe_drvdata *drvdata = platform_get_drvdata(pdev); > + > + arm_trbe_remove_coresight(drvdata); > + arm_trbe_remove_cpuhp(drvdata); > + arm_trbe_remove_irq(drvdata); > + kfree(pdata); > + kfree(drvdata); Same as above. > + return 0; > +} > + > +static const struct of_device_id arm_trbe_of_match[] = { > + { .compatible = "arm,trace-buffer-extension"}, > + {}, > +}; > +MODULE_DEVICE_TABLE(of, arm_trbe_of_match); > + > +static struct platform_driver arm_trbe_driver = { > + .driver = { > + .name = DRVNAME, > + .of_match_table = of_match_ptr(arm_trbe_of_match), > + .suppress_bind_attrs = true, > + }, > + .probe = arm_trbe_device_probe, > + .remove = arm_trbe_device_remove, > +}; > + > +static int __init arm_trbe_init(void) > +{ > + int ret; > + > + if (arm64_kernel_unmapped_at_el0()) { > + pr_err("TRBE wouldn't work if kernel gets unmapped at EL0\n"); > + return -EOPNOTSUPP; > + } > + > + ret = platform_driver_register(&arm_trbe_driver); > + if (!ret) > + return 0; > + > + pr_err("Error registering %s platform driver\n", DRVNAME); > + return ret; > +} > + > +static void __exit arm_trbe_exit(void) > +{ > + platform_driver_unregister(&arm_trbe_driver); > +} > +module_init(arm_trbe_init); > +module_exit(arm_trbe_exit); > + > +MODULE_AUTHOR("Anshuman Khandual <anshuman.khandual@arm.com>"); > +MODULE_DESCRIPTION("Arm Trace Buffer Extension (TRBE) driver"); > +MODULE_LICENSE("GPL v2"); > diff --git a/drivers/hwtracing/coresight/coresight-trbe.h b/drivers/hwtracing/coresight/coresight-trbe.h > new file mode 100644 > index 0000000..43308bc > --- /dev/null > +++ b/drivers/hwtracing/coresight/coresight-trbe.h > @@ -0,0 +1,160 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * This contains all required hardware related helper functions for > + * Trace Buffer Extension (TRBE) driver in the coresight framework. > + * > + * Copyright (C) 2020 ARM Ltd. > + * > + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > + */ > +#include <linux/coresight.h> > +#include <linux/device.h> > +#include <linux/irq.h> > +#include <linux/kernel.h> > +#include <linux/of.h> > +#include <linux/platform_device.h> > +#include <linux/smp.h> > + > +#include "coresight-etm-perf.h" > + > +DECLARE_PER_CPU(struct coresight_device *, csdev_sink); This belongs to coresight-priv.h. > + > +static inline bool is_trbe_available(void) > +{ > + u64 aa64dfr0 = read_sysreg_s(SYS_ID_AA64DFR0_EL1); > + unsigned int trbe = cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_TRBE_SHIFT); > + > + return trbe >= 0b0001; > +} > + > +static inline bool is_trbe_enabled(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + return trblimitr & TRBLIMITR_ENABLE; > +} > + > +#define TRBE_EC_OTHERS 0 > +#define TRBE_EC_STAGE1_ABORT 36 > +#define TRBE_EC_STAGE2_ABORT 37 > + > +static inline int get_trbe_ec(u64 trbsr) > +{ > + return (trbsr >> TRBSR_EC_SHIFT) & TRBSR_EC_MASK; > +} > + > +#define TRBE_BSC_NOT_STOPPED 0 > +#define TRBE_BSC_FILLED 1 nit: Use space instead of TAB here. > +#define TRBE_BSC_TRIGGERED 2 > + > +static inline int get_trbe_bsc(u64 trbsr) > +{ > + return (trbsr >> TRBSR_BSC_SHIFT) & TRBSR_BSC_MASK; > +} > + > +static inline void clr_trbe_irq(void) > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + > + trbsr &= ~TRBSR_IRQ; > + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > +} > + > +static inline bool is_trbe_irq(u64 trbsr) > +{ > + return trbsr & TRBSR_IRQ; > +} > + > +static inline bool is_trbe_trg(u64 trbsr) > +{ > + return trbsr & TRBSR_TRG; > +} > + > +static inline bool is_trbe_wrap(u64 trbsr) > +{ > + return trbsr & TRBSR_WRAP; > +} > + > +static inline bool is_trbe_abort(u64 trbsr) > +{ > + return trbsr & TRBSR_ABORT; > +} > + > +static inline bool is_trbe_running(u64 trbsr) > +{ > + return !(trbsr & TRBSR_STOP); > +} > + > +#define TRBE_TRIG_MODE_STOP 0 > +#define TRBE_TRIG_MODE_IRQ 1 > +#define TRBE_TRIG_MODE_IGNORE 3 > + > +#define TRBE_FILL_MODE_FILL 0 > +#define TRBE_FILL_MODE_WRAP 1 > +#define TRBE_FILL_MODE_CIRCULAR_BUFFER 3 > + > +static inline void set_trbe_disabled(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + trblimitr &= ~TRBLIMITR_ENABLE; > + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > +} > + > +static inline bool get_trbe_flag_update(u64 trbidr) > +{ > + return trbidr & TRBIDR_FLAG; > +} > + > +static inline bool is_trbe_programmable(u64 trbidr) > +{ > + return !(trbidr & TRBIDR_PROG); > +} > + > +static inline int get_trbe_address_align(u64 trbidr) > +{ > + return (trbidr >> TRBIDR_ALIGN_SHIFT) & TRBIDR_ALIGN_MASK; > +} > + > +static inline unsigned long get_trbe_write_pointer(void) > +{ > + u64 trbptr = read_sysreg_s(SYS_TRBPTR_EL1); > + unsigned long addr = (trbptr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; You don't need this shifting and masking. trbptr_el1 is a whole 64bit register > + > + return addr; > +} > + > +static inline void set_trbe_write_pointer(unsigned long addr) > +{ > + WARN_ON(is_trbe_enabled()); > + addr = (addr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; Same as above. > + write_sysreg_s(addr, SYS_TRBPTR_EL1); > +} > + > +static inline unsigned long get_trbe_limit_pointer(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + unsigned long limit = (trblimitr >> TRBLIMITR_LIMIT_SHIFT) & TRBLIMITR_LIMIT_MASK; > + unsigned long addr = limit << TRBLIMITR_LIMIT_SHIFT; > + > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + return addr; > +} > + > +static inline unsigned long get_trbe_base_pointer(void) > +{ > + u64 trbbaser = read_sysreg_s(SYS_TRBBASER_EL1); > + unsigned long addr = (trbbaser >> TRBBASER_BASE_SHIFT) & TRBBASER_BASE_MASK; > + > + addr = addr << TRBBASER_BASE_SHIFT; nit: Could we instead do : addr = trbbaser & (TRBBASER_BASE_MASK << TRBBASER_BASE_SHIFT); ? > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + return addr; > +} > + > +static inline void set_trbe_base_pointer(unsigned long addr) > +{ > + WARN_ON(is_trbe_enabled()); > + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); TRBBASER_BASE_SHIFT ? > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + write_sysreg_s(addr, SYS_TRBBASER_EL1); > +} > Suzuki
On 1/29/21 3:53 PM, Suzuki K Poulose wrote: > Hi Anshuman > > On 1/27/21 8:55 AM, Anshuman Khandual wrote: >> Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is >> accessible via the system registers. The TRBE supports different addressing >> modes including CPU virtual address and buffer modes including the circular >> buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), >> an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the >> access to the trace buffer could be prohibited by a higher exception level >> (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU >> private interrupt (PPI) on address translation errors and when the buffer >> is full. Overall implementation here is inspired from the Arm SPE driver. >> >> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> >> Cc: Mike Leach <mike.leach@linaro.org> >> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> >> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> > > This version looks functionally correct to me. There are some minor > issues with the devm_ allocated memory and some driver hardening comments. > I ran this on a model and have tested this with various scenarios. Okay. > >> --- >> Changes in V3: >> >> - Added new DT bindings document TRBE.yaml >> - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 >> - Dropped isb() from trbe_reset_local() >> - Dropped gap between (void *) and buf->trbe_base >> - Changed 'int' to 'unsigned int' in is_trbe_available() >> - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), >>   set_trbe_enabled() and set_trbe_limit_pointer() >> - Changed get_trbe_flag_update(), is_trbe_programmable() and >>   get_trbe_address_align() to accept TRBIDR value >> - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), >>   is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value >> - Dropped snapshot mode condition in arm_trbe_alloc_buffer() >> - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled >> - Compute trbe_limit before trbe_write to get the updated handle >> - Added trbe_stop_and_truncate_event() >> - Dropped trbe_handle_fatal() >> >>  Documentation/trace/coresight/coresight-trbe.rst |  39 + >>  arch/arm64/include/asm/sysreg.h                 |   1 + >>  drivers/hwtracing/coresight/Kconfig             |  11 + >>  drivers/hwtracing/coresight/Makefile            |   1 + >>  drivers/hwtracing/coresight/coresight-trbe.c    | 1023 ++++++++++++++++++++++ >>  drivers/hwtracing/coresight/coresight-trbe.h    | 160 ++++ >>  6 files changed, 1235 insertions(+) >>  create mode 100644 Documentation/trace/coresight/coresight-trbe.rst >>  create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c >>  create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h >> >> diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst >> new file mode 100644 >> index 0000000..1cbb819 >> --- /dev/null >> +++ b/Documentation/trace/coresight/coresight-trbe.rst >> @@ -0,0 +1,39 @@ >> +.. SPDX-License-Identifier: GPL-2.0 >> + >> +============================== >> +Trace Buffer Extension (TRBE). >> +============================== >> + >> +   :Author:  Anshuman Khandual <anshuman.khandual@arm.com> >> +   :Date:    November 2020 >> + >> +Hardware Description >> +-------------------- >> + >> +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system >> +memory, CPU traces generated from a corresponding percpu tracing unit. This >> +gets plugged in as a coresight sink device because the corresponding trace >> +genarators (ETE), are plugged in as source device. >> + >> +The TRBE is not compliant to CoreSight architecture specifications, but is >> +driven via the CoreSight driver framework to support the ETE (which is >> +CoreSight compliant) integration. >> + >> +Sysfs files and directories >> +--------------------------- >> + >> +The TRBE devices appear on the existing coresight bus alongside the other >> +coresight devices:: >> + >> +   >$ ls /sys/bus/coresight/devices >> +   trbe0 trbe1 trbe2 trbe3 >> + >> +The ``trbe<N>`` named TRBEs are associated with a CPU.:: >> + >> +   >$ ls /sys/bus/coresight/devices/trbe0/ >> +       align dbm >> + >> +*Key file items are:-* >> +  * ``align``: TRBE write pointer alignment >> +  * ``dbm``: TRBE updates memory with access and dirty flags >> + >> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h >> index 85ae4db..9e2e9b7 100644 >> --- a/arch/arm64/include/asm/sysreg.h >> +++ b/arch/arm64/include/asm/sysreg.h >> @@ -97,6 +97,7 @@ >>  #define SET_PSTATE_UAO(x)       __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) >>  #define SET_PSTATE_SSBS(x)       __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) >>  #define SET_PSTATE_TCO(x)       __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) >> +#define TSB_CSYNC           __emit_inst(0xd503225f) >>   #define set_pstate_pan(x)       asm volatile(SET_PSTATE_PAN(x)) >>  #define set_pstate_uao(x)       asm volatile(SET_PSTATE_UAO(x)) >> diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig >> index f154ae7..aa657ab 100644 >> --- a/drivers/hwtracing/coresight/Kconfig >> +++ b/drivers/hwtracing/coresight/Kconfig >> @@ -164,6 +164,17 @@ config CORESIGHT_CTI >>        To compile this driver as a module, choose M here: the >>        module will be called coresight-cti. >>  +config CORESIGHT_TRBE >> +   bool "Trace Buffer Extension (TRBE) driver" >> +   depends on ARM64 >> +   help >> +     This driver provides support for percpu Trace Buffer Extension (TRBE). >> +     TRBE always needs to be used along with it's corresponding percpu ETE >> +     component. ETE generates trace data which is then captured with TRBE. >> +     Unlike traditional sink devices, TRBE is a CPU feature accessible via >> +     system registers. But it's explicit dependency with trace unit (ETE) >> +     requires it to be plugged in as a coresight sink device. >> + >>  config CORESIGHT_CTI_INTEGRATION_REGS >>      bool "Access CTI CoreSight Integration Registers" >>      depends on CORESIGHT_CTI >> diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile >> index f20e357..d608165 100644 >> --- a/drivers/hwtracing/coresight/Makefile >> +++ b/drivers/hwtracing/coresight/Makefile >> @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o >>  obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o >>  obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o >>  obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o >> +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o >>  coresight-cti-y := coresight-cti-core.o   coresight-cti-platform.o \ >>             coresight-cti-sysfs.o >> diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c >> new file mode 100644 >> index 0000000..1464d8b >> --- /dev/null >> +++ b/drivers/hwtracing/coresight/coresight-trbe.c >> @@ -0,0 +1,1023 @@ >> +// SPDX-License-Identifier: GPL-2.0 >> +/* >> + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight >> + * sink device could then pair with an appropriate per-cpu coresight source >> + * device (ETE) thus generating required trace data. Trace can be enabled >> + * via the perf framework. >> + * >> + * Copyright (C) 2020 ARM Ltd. >> + * >> + * Author: Anshuman Khandual <anshuman.khandual@arm.com> >> + */ >> +#define DRVNAME "arm_trbe" >> + >> +#define pr_fmt(fmt) DRVNAME ": " fmt >> + >> +#include "coresight-trbe.h" >> + >> +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) >> + >> +/* >> + * A padding packet that will help the user space tools >> + * in skipping relevant sections in the captured trace >> + * data which could not be decoded. TRBE doesn't support >> + * formatting the trace data, unlike the legacy CoreSight >> + * sinks and thus we use ETE trace packets to pad the >> + * sections of the buffer. >> + */ >> +#define ETE_IGNORE_PACKET        0x70 >> + >> +/* >> + * Minimum amount of meaningful trace will contain: >> + * A-Sync, Trace Info, Trace On, Address, Atom. >> + * This is about 44bytes of ETE trace. To be on >> + * the safer side, we assume 64bytes is the minimum >> + * space required for a meaningful session, before >> + * we hit a "WRAP" event. >> + */ >> +#define TRBE_TRACE_MIN_BUF_SIZE       64 >> + >> +enum trbe_fault_action { >> +   TRBE_FAULT_ACT_WRAP, >> +   TRBE_FAULT_ACT_SPURIOUS, >> +   TRBE_FAULT_ACT_FATAL, >> +}; >> + >> +struct trbe_buf { >> +   unsigned long trbe_base; >> +   unsigned long trbe_limit; >> +   unsigned long trbe_write; >> +   int nr_pages; >> +   void **pages; >> +   bool snapshot; >> +   struct trbe_cpudata *cpudata; >> +}; >> + >> +struct trbe_cpudata { >> +   bool trbe_dbm; >> +   u64 trbe_align; >> +   int cpu; >> +   enum cs_mode mode; >> +   struct trbe_buf *buf; >> +   struct trbe_drvdata *drvdata; >> +}; >> + >> +struct trbe_drvdata { >> +   struct trbe_cpudata __percpu *cpudata; >> +   struct perf_output_handle __percpu **handle; >> +   struct hlist_node hotplug_node; >> +   int irq; >> +   cpumask_t supported_cpus; >> +   enum cpuhp_state trbe_online; >> +   struct platform_device *pdev; >> +}; >> + >> +static int trbe_alloc_node(struct perf_event *event) >> +{ >> +   if (event->cpu == -1) >> +       return NUMA_NO_NODE; >> +   return cpu_to_node(event->cpu); >> +} >> + >> +static void trbe_drain_buffer(void) >> +{ >> +   asm(TSB_CSYNC); >> +   dsb(nsh); >> +} >> + >> +static void trbe_drain_and_disable_local(void) >> +{ >> +   trbe_drain_buffer(); >> +   write_sysreg_s(0, SYS_TRBLIMITR_EL1); >> +   isb(); >> +} >> + >> +static void trbe_reset_local(void) >> +{ >> +   trbe_drain_and_disable_local(); >> +   write_sysreg_s(0, SYS_TRBPTR_EL1); >> +   write_sysreg_s(0, SYS_TRBBASER_EL1); >> +   write_sysreg_s(0, SYS_TRBSR_EL1); >> +} >> + >> +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) >> +{ >> +   struct trbe_buf *buf = etm_perf_sink_config(handle); >> + >> +   /* >> +    * We cannot proceed with the buffer collection and we >> +    * do not have any data for the current session. The >> +    * etm_perf driver expects to close out the aux_buffer >> +    * at event_stop(). So disable the TRBE here and leave >> +    * the update_buffer() to return a 0 size. >> +    */ >> +   trbe_drain_and_disable_local(); >> +   perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >> +   *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; >> +} >> + >> +/* >> + * TRBE Buffer Management >> + * >> + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, >> + * it starts writing trace data from the write pointer onward till the limit pointer. >> + * When the write pointer reaches the address just before the limit pointer, it gets >> + * wrapped around again to the base pointer. This is called a TRBE wrap event, which >> + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver >> + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ >> + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and >> + * LIMIT pointers. >> + * >> + *   Wrap around with an IRQ >> + *   ------ < ------ < ------- < ----- < ----- >> + *   |                   | >> + *   ------ > ------ > ------- > ----- > ----- >> + * >> + *   +---------------+-----------------------+ >> + *   |       |           | >> + *   +---------------+-----------------------+ >> + *   Base Pointer   Write Pointer       Limit Pointer >> + * >> + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write >> + * pointer can be aligned to the implementation defined TRBE trace buffer alignment >> + * as captured in trbe_cpudata->trbe_align. >> + * >> + * >> + *       head       tail       wakeup >> + *   +---------------------------------------+----- ~ ~ ------ >> + *   |$$$$$$$|################|$$$$$$$$$$$$$$|       | >> + *   +---------------------------------------+----- ~ ~ ------ >> + *   Base Pointer   Write Pointer       Limit Pointer >> + * >> + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing >> + * values which tracks all the driver writes and user reads from the perf auxiliary >> + * buffer. Generally [head..tail] is the area where the driver can write into unless >> + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and >> + * configured depending on the perf_output_handle indices, so that the driver does >> + * not override into areas in the perf auxiliary buffer which is being or yet to be >> + * consumed from the user space. The enabled TRBE buffer area is a moving subset of >> + * the allocated perf auxiliary buffer. >> + */ >> +static void trbe_pad_buf(struct perf_output_handle *handle, int len) >> +{ >> +   struct trbe_buf *buf = etm_perf_sink_config(handle); >> +   u64 head = PERF_IDX2OFF(handle->head, buf); >> + >> +   memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); >> +   if (!buf->snapshot) >> +       perf_aux_output_skip(handle, len); >> +} >> + >> +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) >> +{ >> +   struct trbe_buf *buf = etm_perf_sink_config(handle); >> + >> +   /* >> +    * The ETE trace has alignment synchronization packets allowing >> +    * the decoder to reset in case of an overflow or corruption. >> +    * So we can use the entire buffer for the snapshot mode. >> +    */ >> +   return buf->nr_pages * PAGE_SIZE; >> +} >> + >> +/* >> + * TRBE Limit Calculation >> + * >> + * The following markers are used to illustrate various TRBE buffer situations. >> + * >> + * $$$$ - Data area, unconsumed captured trace data, not to be overridden >> + * #### - Free area, enabled, trace will be written >> + * %%%% - Free area, disabled, trace will not be written >> + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped >> + */ >> +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) >> +{ >> +   struct trbe_buf *buf = etm_perf_sink_config(handle); >> +   struct trbe_cpudata *cpudata = buf->cpudata; >> +   const u64 bufsize = buf->nr_pages * PAGE_SIZE; >> +   u64 limit = bufsize; >> +   u64 head, tail, wakeup; >> + >> +   head = PERF_IDX2OFF(handle->head, buf); >> + >> +   /* >> +    *       head >> +    *   ------->| >> +    *   | >> +    *   head   TRBE align   tail >> +    * +----|-------|---------------|-------+ >> +    * |$$$$|=======|###############|$$$$$$$| >> +    * +----|-------|---------------|-------+ >> +    * trbe_base               trbe_base + nr_pages >> +    * >> +    * Perf aux buffer output head position can be misaligned depending on >> +    * various factors including user space reads. In case misaligned, head >> +    * needs to be aligned before TRBE can be configured. Pad the alignment >> +    * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools >> +    * and skip this section thus advancing the head. >> +    */ >> +   if (!IS_ALIGNED(head, cpudata->trbe_align)) { >> +       unsigned long delta = roundup(head, cpudata->trbe_align) - head; >> + >> +       delta = min(delta, handle->size); >> +       trbe_pad_buf(handle, delta); >> +       head = PERF_IDX2OFF(handle->head, buf); >> +   } >> + >> +   /* >> +    *   head = tail (size = 0) >> +    * +----|-------------------------------+ >> +    * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$   | >> +    * +----|-------------------------------+ >> +    * trbe_base               trbe_base + nr_pages >> +    * >> +    * Perf aux buffer does not have any space for the driver to write into. >> +    * Just communicate trace truncation event to the user space by marking >> +    * it with PERF_AUX_FLAG_TRUNCATED. >> +    */ >> +   if (!handle->size) { >> +       perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >> +       return 0; >> +   } >> + >> +   /* Compute the tail and wakeup indices now that we've aligned head */ >> +   tail = PERF_IDX2OFF(handle->head + handle->size, buf); >> +   wakeup = PERF_IDX2OFF(handle->wakeup, buf); >> + >> +   /* >> +    * Lets calculate the buffer area which TRBE could write into. There >> +    * are three possible scenarios here. Limit needs to be aligned with >> +    * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the >> +    * unconsumed data. >> +    * >> +    * 1) head < tail >> +    * >> +    *   head           tail >> +    * +----|-----------------------|-------+ >> +    * |$$$$|#######################|$$$$$$$| >> +    * +----|-----------------------|-------+ >> +    * trbe_base           limit   trbe_base + nr_pages >> +    * >> +    * TRBE could write into [head..tail] area. Unless the tail is right at >> +    * the end of the buffer, neither an wrap around nor an IRQ is expected >> +    * while being enabled. >> +    * >> +    * 2) head == tail >> +    * >> +    *   head = tail (size > 0) >> +    * +----|-------------------------------+ >> +    * |%%%%|###############################| >> +    * +----|-------------------------------+ >> +    * trbe_base               limit = trbe_base + nr_pages >> +    * >> +    * TRBE should just write into [head..base + nr_pages] area even though >> +    * the entire buffer is empty. Reason being, when the trace reaches the >> +    * end of the buffer, it will just wrap around with an IRQ giving an >> +    * opportunity to reconfigure the buffer. >> +    * >> +    * 3) tail < head >> +    * >> +    *   tail           head >> +    * +----|-----------------------|-------+ >> +    * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| >> +    * +----|-----------------------|-------+ >> +    * trbe_base               limit = trbe_base + nr_pages >> +    * >> +    * TRBE should just write into [head..base + nr_pages] area even though >> +    * the [trbe_base..tail] is also empty. Reason being, when the trace >> +    * reaches the end of the buffer, it will just wrap around with an IRQ >> +    * giving an opportunity to reconfigure the buffer. >> +    */ >> +   if (head < tail) >> +       limit = round_down(tail, PAGE_SIZE); >> + >> +   /* >> +    * Wakeup may be arbitrarily far into the future. If it's not in the >> +    * current generation, either we'll wrap before hitting it, or it's >> +    * in the past and has been handled already. >> +    * >> +    * If there's a wakeup before we wrap, arrange to be woken up by the >> +    * page boundary following it. Keep the tail boundary if that's lower. >> +    * >> +    *   head       wakeup   tail >> +    * +----|---------------|-------|-------+ >> +    * |$$$$|###############|%%%%%%%|$$$$$$$| >> +    * +----|---------------|-------|-------+ >> +    * trbe_base       limit       trbe_base + nr_pages >> +    */ >> +   if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) >> +       limit = min(limit, round_up(wakeup, PAGE_SIZE)); >> + >> +   /* >> +    * There are two situation when this can happen i.e limit is before >> +    * the head and hence TRBE cannot be configured. >> +    * >> +    * 1) head < tail (aligned down with PAGE_SIZE) and also they are both >> +    * within the same PAGE size range. >> +    * >> +    *           PAGE_SIZE >> +    *       |----------------------| >> +    * >> +    *       limit   head   tail >> +    * +------------|------|--------|-------+ >> +    * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| >> +    * +------------|------|--------|-------+ >> +    * trbe_base               trbe_base + nr_pages >> +    * >> +    * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both >> +    * head and wakeup are within same PAGE size range. >> +    * >> +    *       PAGE_SIZE >> +    *   |----------------------| >> +    * >> +    *   limit   head   wakeup tail >> +    * +----|------|-------|--------|-------+ >> +    * |$$$$$$$$$$$|=======|========|$$$$$$$| >> +    * +----|------|-------|--------|-------+ >> +    * trbe_base               trbe_base + nr_pages >> +    */ >> +   if (limit > head) >> +       return limit; >> + >> +   trbe_pad_buf(handle, handle->size); >> +   perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >> +   return 0; >> +} >> + >> +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) >> +{ >> +   struct trbe_buf *buf = perf_get_aux(handle); >> +   u64 limit = __trbe_normal_offset(handle); >> +   u64 head = PERF_IDX2OFF(handle->head, buf); >> + >> +   /* >> +    * If the head is too close to the limit and we don't >> +    * have space for a meaningful run, we rather pad it >> +    * and start fresh. >> +    */ >> +   if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { >> +       trbe_pad_buf(handle, limit - head); >> +       limit = __trbe_normal_offset(handle); >> +   } >> +   return limit; >> +} >> + >> +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) >> +{ >> +   struct trbe_buf *buf = etm_perf_sink_config(handle); >> +   unsigned long offset; >> + >> +   if (buf->snapshot) >> +       offset = trbe_snapshot_offset(handle); >> +   else >> +       offset = trbe_normal_offset(handle); >> +   return buf->trbe_base + offset; >> +} >> + >> +static void clr_trbe_status(void) >> +{ >> +   u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); >> + >> +   WARN_ON(is_trbe_enabled()); >> +   trbsr &= ~TRBSR_IRQ; >> +   trbsr &= ~TRBSR_TRG; >> +   trbsr &= ~TRBSR_WRAP; >> +   trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); >> +   trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); >> +   trbsr &= ~TRBSR_STOP; >> +   write_sysreg_s(trbsr, SYS_TRBSR_EL1); >> +} >> + >> +static void set_trbe_limit_pointer_enabled(unsigned long addr) >> +{ >> +   u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); >> + >> +   WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); >> +   WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); >> + >> +   trblimitr &= ~TRBLIMITR_NVM; >> +   trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); >> +   trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); >> +   trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); >> + >> +   /* >> +    * Fill trace buffer mode is used here while configuring the >> +    * TRBE for trace capture. In this particular mode, the trace >> +    * collection is stopped and a maintenance interrupt is raised >> +    * when the current write pointer wraps. This pause in trace >> +    * collection gives the software an opportunity to capture the >> +    * trace data in the interrupt handler, before reconfiguring >> +    * the TRBE. >> +    */ >> +   trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; >> + >> +   /* >> +    * Trigger mode is not used here while configuring the TRBE for >> +    * the trace capture. Hence just keep this in the ignore mode. >> +    */ >> +   trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; >> +   trblimitr |= (addr & PAGE_MASK); >> + >> +   trblimitr |= TRBLIMITR_ENABLE; >> +   write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > Personally, I prefer the isb() here rather than at the caller site, to make sure > it is all contained here and more importantly we don't execute any other instruction > before the "isb()" as the tracing may be enabled for the kernel. Sure, will move the isb() here along with its comment. > >> +} >> + >> +static void trbe_enable_hw(struct trbe_buf *buf) >> +{ >> +   WARN_ON(buf->trbe_write < buf->trbe_base); > +   WARN_ON(buf->trbe_write >= buf->trbe_limit); >> +   set_trbe_disabled(); >> +   isb(); >> +   clr_trbe_status(); >> +   set_trbe_base_pointer(buf->trbe_base); >> +   set_trbe_write_pointer(buf->trbe_write); >> + >> +   /* >> +    * Synchronize all the register updates >> +    * till now before enabling the TRBE. >> +    */ >> +   isb(); >> +   set_trbe_limit_pointer_enabled(buf->trbe_limit); >> + >> +   /* Synchronize the TRBE enable event */ >> +   isb(); >> +} >> + >> +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, >> +                  struct perf_event *event, void **pages, >> +                  int nr_pages, bool snapshot) >> +{ >> +   struct trbe_buf *buf; >> +   struct page **pglist; >> +   int i; >> + >> +   /* >> +    * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with >> +    * just a single page, there is not much room left while writing into >> +    * a partially filled TRBE buffer. Hence restrict the minimum buffer >> +    * size as two pages. >> +    */ >> +   if (nr_pages < 2) >> +       return NULL; >> + >> +   buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); >> +   if (IS_ERR(buf)) >> +       return ERR_PTR(-ENOMEM); >> + >> +   pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); >> +   if (IS_ERR(pglist)) { >> +       kfree(buf); >> +       return ERR_PTR(-ENOMEM); >> +   } >> + >> +   for (i = 0; i < nr_pages; i++) >> +       pglist[i] = virt_to_page(pages[i]); >> + >> +   buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); > > minor nit: space after casting. Will drop the space. > >> +   if (IS_ERR((void *)buf->trbe_base)) { >> +       kfree(pglist); >> +       kfree(buf); >> +       return ERR_PTR(buf->trbe_base); >> +   } >> +   buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; >> +   buf->trbe_write = buf->trbe_base; >> +   buf->snapshot = snapshot; >> +   buf->nr_pages = nr_pages; >> +   buf->pages = pages; >> +   kfree(pglist); >> +   return buf; >> +} >> + >> +void arm_trbe_free_buffer(void *config) >> +{ >> +   struct trbe_buf *buf = config; >> + >> +   vunmap((void *)buf->trbe_base); >> +   kfree(buf); >> +} >> + >> +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, >> +                       struct perf_output_handle *handle, >> +                       void *config) >> +{ >> +   struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >> +   struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >> +   struct trbe_buf *buf = config; >> +   unsigned long size, offset; >> + >> +   WARN_ON(buf->cpudata != cpudata); >> +   WARN_ON(cpudata->cpu != smp_processor_id()); >> +   WARN_ON(cpudata->drvdata != drvdata); >> +   if (cpudata->mode != CS_MODE_PERF) >> +       return -EINVAL; > > Please return 0 here. As we are expected to pass an "unsigned" size. Sure, will do. > >> + >> +   /* >> +    * If the TRBE was disabled due to lack of space in the AUX buffer or a >> +    * spurious fault, the driver leaves it disabled, truncating the buffer. >> +    * Since the etm_perf driver expects to close out the AUX buffer, the >> +    * driver skips it. Thus, just pass in 0 size here to indicate that the >> +    * buffer was truncated. >> +    */ >> +   if (!is_trbe_enabled()) >> +       return 0; >> +   /* >> +    * perf handle structure needs to be shared with the TRBE IRQ handler for >> +    * capturing trace data and restarting the handle. There is a probability >> +    * of an undefined reference based crash when etm event is being stopped >> +    * while a TRBE IRQ also getting processed. This happens due the release >> +    * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping >> +    * the TRBE here will ensure that no IRQ could be generated when the perf >> +    * handle gets freed in etm_event_stop(). >> +    */ >> +   trbe_drain_and_disable_local(); >> +   offset = get_trbe_write_pointer() - get_trbe_base_pointer(); >> +   size = offset - PERF_IDX2OFF(handle->head, buf); > > It may be a good idea to verify that the size computation here doesn't overflow. > e.g broken interrupt could potentially leave us with "write == base". I understand > this is a hardware issue, but the software can be resilient to not send bogus > results. Okay, will do the necessary. > >> +   if (buf->snapshot) >> +       handle->head += size; >> +   return size; >> +} >> + >> +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) >> +{ >> +   struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >> +   struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >> +   struct perf_output_handle *handle = data; >> +   struct trbe_buf *buf = etm_perf_sink_config(handle); >> + >> +   WARN_ON(cpudata->cpu != smp_processor_id()); >> +   WARN_ON(cpudata->drvdata != drvdata); >> +   if (mode != CS_MODE_PERF) >> +       return -EINVAL; >> + >> +   *this_cpu_ptr(drvdata->handle) = handle; >> +   cpudata->buf = buf; >> +   cpudata->mode = mode; >> +   buf->cpudata = cpudata; >> +   buf->trbe_limit = compute_trbe_buffer_limit(handle); >> +   buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >> +   if (buf->trbe_limit == buf->trbe_base) { >> +       trbe_stop_and_truncate_event(handle); >> +       return 0; >> +   } >> +   trbe_enable_hw(buf); >> +   return 0; >> +} >> + >> +static int arm_trbe_disable(struct coresight_device *csdev) >> +{ >> +   struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >> +   struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >> +   struct trbe_buf *buf = cpudata->buf; >> + >> +   WARN_ON(buf->cpudata != cpudata); >> +   WARN_ON(cpudata->cpu != smp_processor_id()); >> +   WARN_ON(cpudata->drvdata != drvdata); >> +   if (cpudata->mode != CS_MODE_PERF) >> +       return -EINVAL; >> + >> +   trbe_drain_and_disable_local(); >> +   buf->cpudata = NULL; >> +   cpudata->buf = NULL; >> +   cpudata->mode = CS_MODE_DISABLED; >> +   return 0; >> +} >> + >> +static void trbe_handle_spurious(struct perf_output_handle *handle) >> +{ >> +   struct trbe_buf *buf = etm_perf_sink_config(handle); >> + >> +   buf->trbe_limit = compute_trbe_buffer_limit(handle); >> +   buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >> +   if (buf->trbe_limit == buf->trbe_base) { >> +       trbe_drain_and_disable_local(); >> +       return; >> +   } >> +   trbe_enable_hw(buf); >> +} >> + >> +static void trbe_handle_overflow(struct perf_output_handle *handle) >> +{ >> +   struct perf_event *event = handle->event; >> +   struct trbe_buf *buf = etm_perf_sink_config(handle); >> +   unsigned long offset, size; >> +   struct etm_event_data *event_data; >> + >> +   offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); >> +   size = offset - PERF_IDX2OFF(handle->head, buf); >> +   if (buf->snapshot) >> +       handle->head = offset; >> +   perf_aux_output_end(handle, size); >> + >> +   event_data = perf_aux_output_begin(handle, event); >> +   if (!event_data) { > > We may add a comment here to explain how this would be handled ? e.g, > >        /* >         * We are unable to restart the trace collection, >         * thus leave the TRBE disabled. The etm-perf driver >         * is able to detect this with a disconnnected handle >         * (handle->event = NULL). >         */ Sure, will add the above. > >> +       trbe_drain_and_disable_local(); >> +       *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; >> +       return; >> +   } >> +   buf->trbe_limit = compute_trbe_buffer_limit(handle); >> +   buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >> +   if (buf->trbe_limit == buf->trbe_base) { >> +       trbe_stop_and_truncate_event(handle); >> +       return; >> +   } >> +   *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; >> +   trbe_enable_hw(buf); >> +} >> + >> +static bool is_perf_trbe(struct perf_output_handle *handle) >> +{ >> +   struct trbe_buf *buf = etm_perf_sink_config(handle); >> +   struct trbe_cpudata *cpudata = buf->cpudata; >> +   struct trbe_drvdata *drvdata = cpudata->drvdata; >> +   int cpu = smp_processor_id(); >> + >> +   WARN_ON(buf->trbe_base != get_trbe_base_pointer()); >> +   WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); >> + >> +   if (cpudata->mode != CS_MODE_PERF) >> +       return false; >> + >> +   if (cpudata->cpu != cpu) >> +       return false; >> + >> +   if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) >> +       return false; >> + >> +   return true; >> +} >> + >> +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) >> +{ >> +   u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); >> +   int ec = get_trbe_ec(trbsr); >> +   int bsc = get_trbe_bsc(trbsr); >> + >> +   WARN_ON(is_trbe_running(trbsr)); >> +   if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) >> +       return TRBE_FAULT_ACT_FATAL; >> + >> +   if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) >> +       return TRBE_FAULT_ACT_FATAL; >> + >> +   if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { >> +       if (get_trbe_write_pointer() == get_trbe_base_pointer()) >> +           return TRBE_FAULT_ACT_WRAP; >> +   } >> +   return TRBE_FAULT_ACT_SPURIOUS; >> +} >> + >> +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) >> +{ >> +   struct perf_output_handle **handle_ptr = dev; >> +   struct perf_output_handle *handle = *handle_ptr; >> +   enum trbe_fault_action act; >> + >> +   WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); >> +   clr_trbe_irq(); >> + >> +   /* >> +    * Ensure the trace is visible to the CPUs and >> +    * any external aborts have been resolved. >> +    */ >> +   trbe_drain_buffer(); >> +   isb(); >> + >> +   if (!perf_get_aux(handle)) > > We may want to ensure that handle is not NULL, which can only happen if the > hardware is not following the software. > >     if (WARN_ON_ONCE(!handle) || !perf_get_aux(handle)) Okay, will do the change. > >> +       return IRQ_NONE; >> + >> +   if (!is_perf_trbe(handle)) >> +       return IRQ_NONE; >> + >> +   irq_work_run(); >> + >> +   act = trbe_get_fault_act(handle); >> +   switch (act) { >> +   case TRBE_FAULT_ACT_WRAP: >> +       trbe_handle_overflow(handle); >> +       break; >> +   case TRBE_FAULT_ACT_SPURIOUS: >> +       trbe_handle_spurious(handle); >> +       break; >> +   case TRBE_FAULT_ACT_FATAL: >> +       trbe_stop_and_truncate_event(handle); >> +       break; >> +   } >> +   return IRQ_HANDLED; >> +} >> + >> +static const struct coresight_ops_sink arm_trbe_sink_ops = { >> +   .enable       = arm_trbe_enable, >> +   .disable   = arm_trbe_disable, >> +   .alloc_buffer   = arm_trbe_alloc_buffer, >> +   .free_buffer   = arm_trbe_free_buffer, >> +   .update_buffer   = arm_trbe_update_buffer, >> +}; >> + >> +static const struct coresight_ops arm_trbe_cs_ops = { >> +   .sink_ops   = &arm_trbe_sink_ops, >> +}; >> + >> +static ssize_t align_show(struct device *dev, struct device_attribute *attr, char *buf) >> +{ >> +   struct trbe_cpudata *cpudata = dev_get_drvdata(dev); >> + >> +   return sprintf(buf, "%llx\n", cpudata->trbe_align); >> +} >> +static DEVICE_ATTR_RO(align); >> + >> +static ssize_t dbm_show(struct device *dev, struct device_attribute *attr, char *buf) >> +{ >> +   struct trbe_cpudata *cpudata = dev_get_drvdata(dev); >> + >> +   return sprintf(buf, "%d\n", cpudata->trbe_dbm); >> +} >> +static DEVICE_ATTR_RO(dbm); >> + >> +static struct attribute *arm_trbe_attrs[] = { >> +   &dev_attr_align.attr, >> +   &dev_attr_dbm.attr, >> +   NULL, >> +}; >> + >> +static const struct attribute_group arm_trbe_group = { >> +   .attrs = arm_trbe_attrs, >> +}; >> + >> +static const struct attribute_group *arm_trbe_groups[] = { >> +   &arm_trbe_group, >> +   NULL, >> +}; >> + >> +static void arm_trbe_probe_coresight_cpu(void *info) >> +{ >> +   struct trbe_drvdata *drvdata = info; >> +   struct coresight_desc desc = { 0 }; >> +   int cpu = smp_processor_id(); >> +   struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); >> +   struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); >> +   u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); > > This must be done only after the "is_trbe_available()". Otherwise > we get an UNDEFINED instruction abort. Okay, will move read_sysreg_s() after is_trbe_programmable() instead. > >> +   struct device *dev; >> + >> +   if (WARN_ON(!cpudata)) >> +       goto cpu_clear; >> + >> +   if (trbe_csdev) >> +       return; >> + >> +   cpudata->cpu = smp_processor_id(); >> +   cpudata->drvdata = drvdata; >> +   dev = &cpudata->drvdata->pdev->dev; >> + >> +   if (!is_trbe_available()) { >> +       pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); >> +       goto cpu_clear; >> +   } >> + >> +   if (!is_trbe_programmable(trbidr)) { >> +       pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); >> +       goto cpu_clear; >> +   } >> +   desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); >> +   if (IS_ERR(desc.name)) >> +       goto cpu_clear; >> + >> +   desc.type = CORESIGHT_DEV_TYPE_SINK; >> +   desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; >> +   desc.ops = &arm_trbe_cs_ops; >> +   desc.pdata = dev_get_platdata(dev); >> +   desc.groups = arm_trbe_groups; >> +   desc.dev = dev; >> +   trbe_csdev = coresight_register(&desc); >> +   if (IS_ERR(trbe_csdev)) >> +       goto cpu_clear; >> + >> +   dev_set_drvdata(&trbe_csdev->dev, cpudata); >> +   cpudata->trbe_dbm = get_trbe_flag_update(trbidr); >> +   cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); >> +   if (cpudata->trbe_align > SZ_2K) { >> +       pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); >> +       goto cpu_clear; > > Should we unregister the coresight device in this case ? There is no point > in having the device around if it is not supported. Okay, will call coresight_unregister() in that case. > >> +   } >> +   per_cpu(csdev_sink, cpu) = trbe_csdev; >> +   trbe_reset_local(); >> +   enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); >> +   return; >> +cpu_clear: >> +   cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); >> +} >> + >> +static void arm_trbe_remove_coresight_cpu(void *info) >> +{ >> +   int cpu = smp_processor_id(); >> +   struct trbe_drvdata *drvdata = info; >> +   struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); >> +   struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); >> + > > nit: Is it better to use this_cpu_ptr() every where above/below to make it explicit ? csdev_sink is already a percpu pointer. > >> +   if (trbe_csdev) { >> +       coresight_unregister(trbe_csdev); >> +       cpudata->drvdata = NULL; >> +       per_cpu(csdev_sink, cpu) = NULL; >> +   } >> +   disable_percpu_irq(drvdata->irq); >> +   trbe_reset_local(); >> +} >> + >> +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) >> +{ >> +   drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); >> +   if (IS_ERR(drvdata->cpudata)) >> +       return PTR_ERR(drvdata->cpudata); >> + >> +   arm_trbe_probe_coresight_cpu(drvdata); >> +   smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); >> +   return 0; >> +} >> + >> +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) >> +{ >> +   arm_trbe_remove_coresight_cpu(drvdata); >> +   smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); >> +   free_percpu(drvdata->cpudata); >> +   return 0; >> +} >> + >> +static int arm_trbe_cpu_startup(unsigned int cpu, struct hlist_node *node) >> +{ >> +   struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); >> + >> +   if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > > nit: Add a comment ? > >        /* >         * If this CPU was not probed for TRBE, initialize >         * it now. >         */ Sure, will add the above. > >> +       if (!per_cpu(csdev_sink, cpu)) { >> +           arm_trbe_probe_coresight_cpu(drvdata); >> +       } else { >> +           trbe_reset_local(); >> +           enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); >> +       } >> +   } >> +   return 0; >> +} >> + >> +static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node) >> +{ >> +   struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); >> + >> +   if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { >> +       disable_percpu_irq(drvdata->irq); >> +       trbe_reset_local(); >> +   } >> +   return 0; >> +} >> + >> +static int arm_trbe_probe_cpuhp(struct trbe_drvdata *drvdata) >> +{ >> +   enum cpuhp_state trbe_online; >> + >> +   trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, >> +                   arm_trbe_cpu_startup, arm_trbe_cpu_teardown); >> +   if (trbe_online < 0) >> +       return -EINVAL; >> + >> +   if (cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node)) >> +       return -EINVAL; >> + >> +   drvdata->trbe_online = trbe_online; >> +   return 0; >> +} >> + >> +static void arm_trbe_remove_cpuhp(struct trbe_drvdata *drvdata) >> +{ >> +   cpuhp_remove_multi_state(drvdata->trbe_online); >> +} >> + >> +static int arm_trbe_probe_irq(struct platform_device *pdev, >> +                 struct trbe_drvdata *drvdata) >> +{ >> +   drvdata->irq = platform_get_irq(pdev, 0); >> +   if (!drvdata->irq) { >> +       pr_err("IRQ not found for the platform device\n"); >> +       return -ENXIO; >> +   } >> + >> +   if (!irq_is_percpu(drvdata->irq)) { >> +       pr_err("IRQ is not a PPI\n"); >> +       return -EINVAL; >> +   } >> + >> +   if (irq_get_percpu_devid_partition(drvdata->irq, &drvdata->supported_cpus)) >> +       return -EINVAL; >> + >> +   drvdata->handle = alloc_percpu(typeof(*drvdata->handle)); >> +   if (!drvdata->handle) >> +       return -ENOMEM; >> + >> +   if (request_percpu_irq(drvdata->irq, arm_trbe_irq_handler, DRVNAME, drvdata->handle)) { >> +       free_percpu(drvdata->handle); >> +       return -EINVAL; >> +   } >> +   return 0; >> +} >> + >> +static void arm_trbe_remove_irq(struct trbe_drvdata *drvdata) >> +{ >> +   free_percpu_irq(drvdata->irq, drvdata->handle); >> +   free_percpu(drvdata->handle); >> +} >> + >> +static int arm_trbe_device_probe(struct platform_device *pdev) >> +{ >> +   struct coresight_platform_data *pdata; >> +   struct trbe_drvdata *drvdata; >> +   struct device *dev = &pdev->dev; >> +   int ret; >> + >> +   drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL); >> +   if (IS_ERR(drvdata)) >> +       return -ENOMEM; >> + >> +   pdata = coresight_get_platform_data(dev); > > Not sure if this is necessary. We don't have any CoreSight specific > data in the bindings. Without desc.pdata which is dev_get_platdata(dev), coresight_register() just fails. Do you see any other alternative for desc.pdata instead ? [ 1.465221] pc : coresight_register+0x1f0/0x338 [ 1.465383] lr : coresight_register+0x144/0x338 [ 1.465483] sp : ffff80001003ba80 [ 1.465583] x29: ffff80001003ba80 x28: ffff000800286c10 [ 1.465712] x27: 0000000000000002 x26: ffff00087f7d6e88 [ 1.465883] x25: 0000000000000020 x24: ffff000802168b80 [ 1.465983] x23: ffff800011bd99c0 x22: ffff800011e6f000 [ 1.466183] x21: 0000000000000000 x20: ffff80001003bb28 [ 1.466286] x19: ffff00080217b400 x18: ffffffffffffffff [ 1.466483] x17: 00000000000000c0 x16: fffffc0020085e40 [ 1.466613] x15: ffff800011bd9948 x14: ffff000802131a1c [ 1.466783] x13: ffff00080213127d x12: 0000000000000000 [ 1.466883] x11: 0101010101010101 x10: 7f7f7f7f7f7f7f7f [ 1.467083] x9 : 686260717376672e x8 : 7f7f7f7f7f7f7f7f [ 1.467206] x7 : 72716475687162ff x6 : 8000000000000000 [ 1.467383] x5 : ffff000800090000 x4 : ffff000800090000 [ 1.467514] x3 : 0000000000000000 x2 : 0000000000000000 [ 1.467596] x1 : 0000000000000001 x0 : 0000000000000000 [ 1.467783] Call trace: [ 1.467883] coresight_register+0x1f0/0x338 [ 1.468006] arm_trbe_probe_coresight_cpu+0x144/0x238 [ 1.468088] arm_trbe_device_probe+0xd0/0x200 [ 1.468283] platform_probe+0x68/0xe0 [ 1.468383] really_probe+0x118/0x3e0 [ 1.468508] driver_probe_device+0x5c/0xc0 [ 1.468583] device_driver_attach+0x74/0x80 [ 1.468683] __driver_attach+0x8c/0xd8 [ 1.468825] bus_for_each_dev+0x7c/0xd8 [ 1.468907] driver_attach+0x24/0x30 [ 1.469071] bus_add_driver+0x154/0x200 [ 1.469183] driver_register+0x64/0x120 [ 1.469290] __platform_driver_register+0x28/0x38 [ 1.469399] arm_trbe_init+0x58/0x88 [ 1.469562] do_one_initcall+0x60/0x1d8 [ 1.469644] kernel_init_freeable+0x1f4/0x24c [ 1.469783] kernel_init+0x14/0x118 [ 1.469890] ret_from_fork+0x10/0x30 [ 1.469983] Code: 17ffff98 92800173 17ffffef f9400262 (b9400440) [ 1.470183] ---[ end trace 20f096c5ea194114 ]--- [ 1.470300] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00 > >> +   if (IS_ERR(pdata)) { >> +       kfree(drvdata); > > This is not required. The cleanup should automatically free this up. > In fact this will be problematic and will cause double free. Okay. > >> +       return -ENOMEM; >> +   } >> + >> +   dev_set_drvdata(dev, drvdata); >> +   dev->platform_data = pdata; >> +   drvdata->pdev = pdev; >> +   ret = arm_trbe_probe_irq(pdev, drvdata); >> +   if (ret) >> +       goto irq_failed; >> + >> +   ret = arm_trbe_probe_coresight(drvdata); >> +   if (ret) >> +       goto probe_failed; >> + >> +   ret = arm_trbe_probe_cpuhp(drvdata); >> +   if (ret) >> +       goto cpuhp_failed; >> + >> +   return 0; >> +cpuhp_failed: >> +   arm_trbe_remove_coresight(drvdata); >> +probe_failed: >> +   arm_trbe_remove_irq(drvdata); >> +irq_failed: >> +   kfree(pdata); >> +   kfree(drvdata); > > None of these "kfree" is needed. It will be automatically freed > when the probe fails. Will drop these kfree(). > >> +   return ret; >> +} >> + >> +static int arm_trbe_device_remove(struct platform_device *pdev) >> +{ >> +   struct coresight_platform_data *pdata = dev_get_platdata(&pdev->dev); >> +   struct trbe_drvdata *drvdata = platform_get_drvdata(pdev); >> + >> +   arm_trbe_remove_coresight(drvdata); >> +   arm_trbe_remove_cpuhp(drvdata); >> +   arm_trbe_remove_irq(drvdata); > >> +   kfree(pdata); >> +   kfree(drvdata); > > Same as above. Will drop these kfree(). > >> +   return 0; >> +} >> + >> +static const struct of_device_id arm_trbe_of_match[] = { >> +   { .compatible = "arm,trace-buffer-extension"}, >> +   {}, >> +}; >> +MODULE_DEVICE_TABLE(of, arm_trbe_of_match); >> + >> +static struct platform_driver arm_trbe_driver = { >> +   .driver   = { >> +       .name = DRVNAME, >> +       .of_match_table = of_match_ptr(arm_trbe_of_match), >> +       .suppress_bind_attrs = true, >> +   }, >> +   .probe   = arm_trbe_device_probe, >> +   .remove   = arm_trbe_device_remove, >> +}; >> + >> +static int __init arm_trbe_init(void) >> +{ >> +   int ret; >> + >> +   if (arm64_kernel_unmapped_at_el0()) { >> +       pr_err("TRBE wouldn't work if kernel gets unmapped at EL0\n"); >> +       return -EOPNOTSUPP; >> +   } >> + >> +   ret = platform_driver_register(&arm_trbe_driver); >> +   if (!ret) >> +       return 0; >> + >> +   pr_err("Error registering %s platform driver\n", DRVNAME); >> +   return ret; >> +} >> + >> +static void __exit arm_trbe_exit(void) >> +{ >> +   platform_driver_unregister(&arm_trbe_driver); >> +} >> +module_init(arm_trbe_init); >> +module_exit(arm_trbe_exit); >> + >> +MODULE_AUTHOR("Anshuman Khandual <anshuman.khandual@arm.com>"); >> +MODULE_DESCRIPTION("Arm Trace Buffer Extension (TRBE) driver"); >> +MODULE_LICENSE("GPL v2"); >> diff --git a/drivers/hwtracing/coresight/coresight-trbe.h b/drivers/hwtracing/coresight/coresight-trbe.h >> new file mode 100644 >> index 0000000..43308bc >> --- /dev/null >> +++ b/drivers/hwtracing/coresight/coresight-trbe.h >> @@ -0,0 +1,160 @@ >> +/* SPDX-License-Identifier: GPL-2.0 */ >> +/* >> + * This contains all required hardware related helper functions for >> + * Trace Buffer Extension (TRBE) driver in the coresight framework. >> + * >> + * Copyright (C) 2020 ARM Ltd. >> + * >> + * Author: Anshuman Khandual <anshuman.khandual@arm.com> >> + */ >> +#include <linux/coresight.h> >> +#include <linux/device.h> >> +#include <linux/irq.h> >> +#include <linux/kernel.h> >> +#include <linux/of.h> >> +#include <linux/platform_device.h> >> +#include <linux/smp.h> >> + >> +#include "coresight-etm-perf.h" >> + >> +DECLARE_PER_CPU(struct coresight_device *, csdev_sink); > > This belongs to coresight-priv.h. Okay, will move. > >> + >> +static inline bool is_trbe_available(void) >> +{ >> +   u64 aa64dfr0 = read_sysreg_s(SYS_ID_AA64DFR0_EL1); >> +   unsigned int trbe = cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_TRBE_SHIFT); >> + >> +   return trbe >= 0b0001; >> +} >> + >> +static inline bool is_trbe_enabled(void) >> +{ >> +   u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); >> + >> +   return trblimitr & TRBLIMITR_ENABLE; >> +} >> + >> +#define TRBE_EC_OTHERS       0 >> +#define TRBE_EC_STAGE1_ABORT   36 >> +#define TRBE_EC_STAGE2_ABORT   37 >> + >> +static inline int get_trbe_ec(u64 trbsr) >> +{ >> +   return (trbsr >> TRBSR_EC_SHIFT) & TRBSR_EC_MASK; >> +} >> + >> +#define TRBE_BSC_NOT_STOPPED   0 >> +#define   TRBE_BSC_FILLED       1 > > nit: Use space instead of TAB here. Sure, will change. > >> +#define TRBE_BSC_TRIGGERED   2 >> + >> +static inline int get_trbe_bsc(u64 trbsr) >> +{ >> +   return (trbsr >> TRBSR_BSC_SHIFT) & TRBSR_BSC_MASK; >> +} >> + >> +static inline void clr_trbe_irq(void) >> +{ >> +   u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); >> + >> +   trbsr &= ~TRBSR_IRQ; >> +   write_sysreg_s(trbsr, SYS_TRBSR_EL1); >> +} >> + >> +static inline bool is_trbe_irq(u64 trbsr) >> +{ >> +   return trbsr & TRBSR_IRQ; >> +} >> + >> +static inline bool is_trbe_trg(u64 trbsr) >> +{ >> +   return trbsr & TRBSR_TRG; >> +} >> + >> +static inline bool is_trbe_wrap(u64 trbsr) >> +{ >> +   return trbsr & TRBSR_WRAP; >> +} >> + >> +static inline bool is_trbe_abort(u64 trbsr) >> +{ >> +   return trbsr & TRBSR_ABORT; >> +} >> + >> +static inline bool is_trbe_running(u64 trbsr) >> +{ >> +   return !(trbsr & TRBSR_STOP); >> +} >> + >> +#define TRBE_TRIG_MODE_STOP       0 >> +#define TRBE_TRIG_MODE_IRQ       1 >> +#define TRBE_TRIG_MODE_IGNORE       3 >> + >> +#define TRBE_FILL_MODE_FILL       0 >> +#define TRBE_FILL_MODE_WRAP       1 >> +#define TRBE_FILL_MODE_CIRCULAR_BUFFER   3 >> + >> +static inline void set_trbe_disabled(void) >> +{ >> +   u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); >> + >> +   trblimitr &= ~TRBLIMITR_ENABLE; >> +   write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); >> +} >> + >> +static inline bool get_trbe_flag_update(u64 trbidr) >> +{ >> +   return trbidr & TRBIDR_FLAG; >> +} >> + >> +static inline bool is_trbe_programmable(u64 trbidr) >> +{ >> +   return !(trbidr & TRBIDR_PROG); >> +} >> + >> +static inline int get_trbe_address_align(u64 trbidr) >> +{ >> +   return (trbidr >> TRBIDR_ALIGN_SHIFT) & TRBIDR_ALIGN_MASK; >> +} >> + >> +static inline unsigned long get_trbe_write_pointer(void) >> +{ >> +   u64 trbptr = read_sysreg_s(SYS_TRBPTR_EL1); >> +   unsigned long addr = (trbptr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; > > You don't need this shifting and masking. trbptr_el1 is a whole 64bit register Okay, will also drop TRBPTR_PTR_SHIFT and TRBPTR_PTR_MASK from here. > >> + >> +   return addr; >> +} >> + >> +static inline void set_trbe_write_pointer(unsigned long addr) >> +{ >> +   WARN_ON(is_trbe_enabled()); >> +   addr = (addr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; > > Same as above. > >> +   write_sysreg_s(addr, SYS_TRBPTR_EL1); >> +} >> + >> +static inline unsigned long get_trbe_limit_pointer(void) >> +{ >> +   u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); >> +   unsigned long limit = (trblimitr >> TRBLIMITR_LIMIT_SHIFT) & TRBLIMITR_LIMIT_MASK; >> +   unsigned long addr = limit << TRBLIMITR_LIMIT_SHIFT; >> + >> +   WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); >> +   return addr; >> +} >> + >> +static inline unsigned long get_trbe_base_pointer(void) >> +{ >> +   u64 trbbaser = read_sysreg_s(SYS_TRBBASER_EL1); > > >> +   unsigned long addr = (trbbaser >> TRBBASER_BASE_SHIFT) & TRBBASER_BASE_MASK; >> + >> +   addr = addr << TRBBASER_BASE_SHIFT; > > nit: Could we instead do : > >     addr = trbbaser & (TRBBASER_BASE_MASK << TRBBASER_BASE_SHIFT); ? Okay, will change. > >> +   WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); >> +   return addr; >> +} >> + >> +static inline void set_trbe_base_pointer(unsigned long addr) >> +{ >> +   WARN_ON(is_trbe_enabled()); >> +   WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > > TRBBASER_BASE_SHIFT ? Right, will replace. - Anshuman
On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: > Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is > accessible via the system registers. The TRBE supports different addressing > modes including CPU virtual address and buffer modes including the circular > buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), > an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the > access to the trace buffer could be prohibited by a higher exception level > (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU > private interrupt (PPI) on address translation errors and when the buffer > is full. Overall implementation here is inspired from the Arm SPE driver. > I got this message when applying the patch: Applying: coresight: sink: Add TRBE driver .git/rebase-apply/patch:76: new blank line at EOF. + warning: 1 line adds whitespace errors. > Cc: Mathieu Poirier <mathieu.poirier@linaro.org> > Cc: Mike Leach <mike.leach@linaro.org> > Cc: Suzuki K Poulose <suzuki.poulose@arm.com> > Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> > --- > Changes in V3: > > - Added new DT bindings document TRBE.yaml > - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 > - Dropped isb() from trbe_reset_local() > - Dropped gap between (void *) and buf->trbe_base > - Changed 'int' to 'unsigned int' in is_trbe_available() > - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), > set_trbe_enabled() and set_trbe_limit_pointer() > - Changed get_trbe_flag_update(), is_trbe_programmable() and > get_trbe_address_align() to accept TRBIDR value > - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), > is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value > - Dropped snapshot mode condition in arm_trbe_alloc_buffer() > - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled > - Compute trbe_limit before trbe_write to get the updated handle > - Added trbe_stop_and_truncate_event() > - Dropped trbe_handle_fatal() > > Documentation/trace/coresight/coresight-trbe.rst | 39 + > arch/arm64/include/asm/sysreg.h | 1 + > drivers/hwtracing/coresight/Kconfig | 11 + > drivers/hwtracing/coresight/Makefile | 1 + > drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ > drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ > 6 files changed, 1235 insertions(+) > create mode 100644 Documentation/trace/coresight/coresight-trbe.rst > create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c > create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h > > diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst > new file mode 100644 > index 0000000..1cbb819 > --- /dev/null > +++ b/Documentation/trace/coresight/coresight-trbe.rst > @@ -0,0 +1,39 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +============================== > +Trace Buffer Extension (TRBE). > +============================== > + > + :Author: Anshuman Khandual <anshuman.khandual@arm.com> > + :Date: November 2020 > + > +Hardware Description > +-------------------- > + > +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system > +memory, CPU traces generated from a corresponding percpu tracing unit. This > +gets plugged in as a coresight sink device because the corresponding trace > +genarators (ETE), are plugged in as source device. > + > +The TRBE is not compliant to CoreSight architecture specifications, but is > +driven via the CoreSight driver framework to support the ETE (which is > +CoreSight compliant) integration. > + > +Sysfs files and directories > +--------------------------- > + > +The TRBE devices appear on the existing coresight bus alongside the other > +coresight devices:: > + > + >$ ls /sys/bus/coresight/devices > + trbe0 trbe1 trbe2 trbe3 > + > +The ``trbe<N>`` named TRBEs are associated with a CPU.:: > + > + >$ ls /sys/bus/coresight/devices/trbe0/ > + align dbm > + > +*Key file items are:-* > + * ``align``: TRBE write pointer alignment > + * ``dbm``: TRBE updates memory with access and dirty flags > + Please add documentation for these, the same way it was done for all the other CS components [1]. [1]. https://elixir.bootlin.com/linux/latest/source/Documentation/ABI/testing (sysfs-bus-coresight-device-xyz) > diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h > index 85ae4db..9e2e9b7 100644 > --- a/arch/arm64/include/asm/sysreg.h > +++ b/arch/arm64/include/asm/sysreg.h > @@ -97,6 +97,7 @@ > #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) > #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) > #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) > +#define TSB_CSYNC __emit_inst(0xd503225f) > > #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) > #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) > diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig > index f154ae7..aa657ab 100644 > --- a/drivers/hwtracing/coresight/Kconfig > +++ b/drivers/hwtracing/coresight/Kconfig > @@ -164,6 +164,17 @@ config CORESIGHT_CTI > To compile this driver as a module, choose M here: the > module will be called coresight-cti. > > +config CORESIGHT_TRBE > + bool "Trace Buffer Extension (TRBE) driver" Shouldn't that be "tristate"? > + depends on ARM64 I would make it dependent on ETMv4 as well since it can only be used by that component. > + help > + This driver provides support for percpu Trace Buffer Extension (TRBE). > + TRBE always needs to be used along with it's corresponding percpu ETE > + component. ETE generates trace data which is then captured with TRBE. > + Unlike traditional sink devices, TRBE is a CPU feature accessible via > + system registers. But it's explicit dependency with trace unit (ETE) > + requires it to be plugged in as a coresight sink device. Please add: "To compile this driver as a module, choose M here: the module will be called coresight-trbe." I'm out of time for today, I will continue on Monday. Mathieu > + > config CORESIGHT_CTI_INTEGRATION_REGS > bool "Access CTI CoreSight Integration Registers" > depends on CORESIGHT_CTI > diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile > index f20e357..d608165 100644 > --- a/drivers/hwtracing/coresight/Makefile > +++ b/drivers/hwtracing/coresight/Makefile > @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o > obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o > obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o > obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o > +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o > coresight-cti-y := coresight-cti-core.o coresight-cti-platform.o \ > coresight-cti-sysfs.o > diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c > new file mode 100644 > index 0000000..1464d8b > --- /dev/null > +++ b/drivers/hwtracing/coresight/coresight-trbe.c > @@ -0,0 +1,1023 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight > + * sink device could then pair with an appropriate per-cpu coresight source > + * device (ETE) thus generating required trace data. Trace can be enabled > + * via the perf framework. > + * > + * Copyright (C) 2020 ARM Ltd. > + * > + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > + */ > +#define DRVNAME "arm_trbe" > + > +#define pr_fmt(fmt) DRVNAME ": " fmt > + > +#include "coresight-trbe.h" > + > +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) > + > +/* > + * A padding packet that will help the user space tools > + * in skipping relevant sections in the captured trace > + * data which could not be decoded. TRBE doesn't support > + * formatting the trace data, unlike the legacy CoreSight > + * sinks and thus we use ETE trace packets to pad the > + * sections of the buffer. > + */ > +#define ETE_IGNORE_PACKET 0x70 > + > +/* > + * Minimum amount of meaningful trace will contain: > + * A-Sync, Trace Info, Trace On, Address, Atom. > + * This is about 44bytes of ETE trace. To be on > + * the safer side, we assume 64bytes is the minimum > + * space required for a meaningful session, before > + * we hit a "WRAP" event. > + */ > +#define TRBE_TRACE_MIN_BUF_SIZE 64 > + > +enum trbe_fault_action { > + TRBE_FAULT_ACT_WRAP, > + TRBE_FAULT_ACT_SPURIOUS, > + TRBE_FAULT_ACT_FATAL, > +}; > + > +struct trbe_buf { > + unsigned long trbe_base; > + unsigned long trbe_limit; > + unsigned long trbe_write; > + int nr_pages; > + void **pages; > + bool snapshot; > + struct trbe_cpudata *cpudata; > +}; > + > +struct trbe_cpudata { > + bool trbe_dbm; > + u64 trbe_align; > + int cpu; > + enum cs_mode mode; > + struct trbe_buf *buf; > + struct trbe_drvdata *drvdata; > +}; > + > +struct trbe_drvdata { > + struct trbe_cpudata __percpu *cpudata; > + struct perf_output_handle __percpu **handle; > + struct hlist_node hotplug_node; > + int irq; > + cpumask_t supported_cpus; > + enum cpuhp_state trbe_online; > + struct platform_device *pdev; > +}; > + > +static int trbe_alloc_node(struct perf_event *event) > +{ > + if (event->cpu == -1) > + return NUMA_NO_NODE; > + return cpu_to_node(event->cpu); > +} > + > +static void trbe_drain_buffer(void) > +{ > + asm(TSB_CSYNC); > + dsb(nsh); > +} > + > +static void trbe_drain_and_disable_local(void) > +{ > + trbe_drain_buffer(); > + write_sysreg_s(0, SYS_TRBLIMITR_EL1); > + isb(); > +} > + > +static void trbe_reset_local(void) > +{ > + trbe_drain_and_disable_local(); > + write_sysreg_s(0, SYS_TRBPTR_EL1); > + write_sysreg_s(0, SYS_TRBBASER_EL1); > + write_sysreg_s(0, SYS_TRBSR_EL1); > +} > + > +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + /* > + * We cannot proceed with the buffer collection and we > + * do not have any data for the current session. The > + * etm_perf driver expects to close out the aux_buffer > + * at event_stop(). So disable the TRBE here and leave > + * the update_buffer() to return a 0 size. > + */ > + trbe_drain_and_disable_local(); > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > +} > + > +/* > + * TRBE Buffer Management > + * > + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, > + * it starts writing trace data from the write pointer onward till the limit pointer. > + * When the write pointer reaches the address just before the limit pointer, it gets > + * wrapped around again to the base pointer. This is called a TRBE wrap event, which > + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver > + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ > + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and > + * LIMIT pointers. > + * > + * Wrap around with an IRQ > + * ------ < ------ < ------- < ----- < ----- > + * | | > + * ------ > ------ > ------- > ----- > ----- > + * > + * +---------------+-----------------------+ > + * | | | > + * +---------------+-----------------------+ > + * Base Pointer Write Pointer Limit Pointer > + * > + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write > + * pointer can be aligned to the implementation defined TRBE trace buffer alignment > + * as captured in trbe_cpudata->trbe_align. > + * > + * > + * head tail wakeup > + * +---------------------------------------+----- ~ ~ ------ > + * |$$$$$$$|################|$$$$$$$$$$$$$$| | > + * +---------------------------------------+----- ~ ~ ------ > + * Base Pointer Write Pointer Limit Pointer > + * > + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing > + * values which tracks all the driver writes and user reads from the perf auxiliary > + * buffer. Generally [head..tail] is the area where the driver can write into unless > + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and > + * configured depending on the perf_output_handle indices, so that the driver does > + * not override into areas in the perf auxiliary buffer which is being or yet to be > + * consumed from the user space. The enabled TRBE buffer area is a moving subset of > + * the allocated perf auxiliary buffer. > + */ > +static void trbe_pad_buf(struct perf_output_handle *handle, int len) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + u64 head = PERF_IDX2OFF(handle->head, buf); > + > + memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); > + if (!buf->snapshot) > + perf_aux_output_skip(handle, len); > +} > + > +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + /* > + * The ETE trace has alignment synchronization packets allowing > + * the decoder to reset in case of an overflow or corruption. > + * So we can use the entire buffer for the snapshot mode. > + */ > + return buf->nr_pages * PAGE_SIZE; > +} > + > +/* > + * TRBE Limit Calculation > + * > + * The following markers are used to illustrate various TRBE buffer situations. > + * > + * $$$$ - Data area, unconsumed captured trace data, not to be overridden > + * #### - Free area, enabled, trace will be written > + * %%%% - Free area, disabled, trace will not be written > + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped > + */ > +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + struct trbe_cpudata *cpudata = buf->cpudata; > + const u64 bufsize = buf->nr_pages * PAGE_SIZE; > + u64 limit = bufsize; > + u64 head, tail, wakeup; > + > + head = PERF_IDX2OFF(handle->head, buf); > + > + /* > + * head > + * ------->| > + * | > + * head TRBE align tail > + * +----|-------|---------------|-------+ > + * |$$$$|=======|###############|$$$$$$$| > + * +----|-------|---------------|-------+ > + * trbe_base trbe_base + nr_pages > + * > + * Perf aux buffer output head position can be misaligned depending on > + * various factors including user space reads. In case misaligned, head > + * needs to be aligned before TRBE can be configured. Pad the alignment > + * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools > + * and skip this section thus advancing the head. > + */ > + if (!IS_ALIGNED(head, cpudata->trbe_align)) { > + unsigned long delta = roundup(head, cpudata->trbe_align) - head; > + > + delta = min(delta, handle->size); > + trbe_pad_buf(handle, delta); > + head = PERF_IDX2OFF(handle->head, buf); > + } > + > + /* > + * head = tail (size = 0) > + * +----|-------------------------------+ > + * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ | > + * +----|-------------------------------+ > + * trbe_base trbe_base + nr_pages > + * > + * Perf aux buffer does not have any space for the driver to write into. > + * Just communicate trace truncation event to the user space by marking > + * it with PERF_AUX_FLAG_TRUNCATED. > + */ > + if (!handle->size) { > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > + return 0; > + } > + > + /* Compute the tail and wakeup indices now that we've aligned head */ > + tail = PERF_IDX2OFF(handle->head + handle->size, buf); > + wakeup = PERF_IDX2OFF(handle->wakeup, buf); > + > + /* > + * Lets calculate the buffer area which TRBE could write into. There > + * are three possible scenarios here. Limit needs to be aligned with > + * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the > + * unconsumed data. > + * > + * 1) head < tail > + * > + * head tail > + * +----|-----------------------|-------+ > + * |$$$$|#######################|$$$$$$$| > + * +----|-----------------------|-------+ > + * trbe_base limit trbe_base + nr_pages > + * > + * TRBE could write into [head..tail] area. Unless the tail is right at > + * the end of the buffer, neither an wrap around nor an IRQ is expected > + * while being enabled. > + * > + * 2) head == tail > + * > + * head = tail (size > 0) > + * +----|-------------------------------+ > + * |%%%%|###############################| > + * +----|-------------------------------+ > + * trbe_base limit = trbe_base + nr_pages > + * > + * TRBE should just write into [head..base + nr_pages] area even though > + * the entire buffer is empty. Reason being, when the trace reaches the > + * end of the buffer, it will just wrap around with an IRQ giving an > + * opportunity to reconfigure the buffer. > + * > + * 3) tail < head > + * > + * tail head > + * +----|-----------------------|-------+ > + * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| > + * +----|-----------------------|-------+ > + * trbe_base limit = trbe_base + nr_pages > + * > + * TRBE should just write into [head..base + nr_pages] area even though > + * the [trbe_base..tail] is also empty. Reason being, when the trace > + * reaches the end of the buffer, it will just wrap around with an IRQ > + * giving an opportunity to reconfigure the buffer. > + */ > + if (head < tail) > + limit = round_down(tail, PAGE_SIZE); > + > + /* > + * Wakeup may be arbitrarily far into the future. If it's not in the > + * current generation, either we'll wrap before hitting it, or it's > + * in the past and has been handled already. > + * > + * If there's a wakeup before we wrap, arrange to be woken up by the > + * page boundary following it. Keep the tail boundary if that's lower. > + * > + * head wakeup tail > + * +----|---------------|-------|-------+ > + * |$$$$|###############|%%%%%%%|$$$$$$$| > + * +----|---------------|-------|-------+ > + * trbe_base limit trbe_base + nr_pages > + */ > + if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) > + limit = min(limit, round_up(wakeup, PAGE_SIZE)); > + > + /* > + * There are two situation when this can happen i.e limit is before > + * the head and hence TRBE cannot be configured. > + * > + * 1) head < tail (aligned down with PAGE_SIZE) and also they are both > + * within the same PAGE size range. > + * > + * PAGE_SIZE > + * |----------------------| > + * > + * limit head tail > + * +------------|------|--------|-------+ > + * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| > + * +------------|------|--------|-------+ > + * trbe_base trbe_base + nr_pages > + * > + * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both > + * head and wakeup are within same PAGE size range. > + * > + * PAGE_SIZE > + * |----------------------| > + * > + * limit head wakeup tail > + * +----|------|-------|--------|-------+ > + * |$$$$$$$$$$$|=======|========|$$$$$$$| > + * +----|------|-------|--------|-------+ > + * trbe_base trbe_base + nr_pages > + */ > + if (limit > head) > + return limit; > + > + trbe_pad_buf(handle, handle->size); > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > + return 0; > +} > + > +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = perf_get_aux(handle); > + u64 limit = __trbe_normal_offset(handle); > + u64 head = PERF_IDX2OFF(handle->head, buf); > + > + /* > + * If the head is too close to the limit and we don't > + * have space for a meaningful run, we rather pad it > + * and start fresh. > + */ > + if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { > + trbe_pad_buf(handle, limit - head); > + limit = __trbe_normal_offset(handle); > + } > + return limit; > +} > + > +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + unsigned long offset; > + > + if (buf->snapshot) > + offset = trbe_snapshot_offset(handle); > + else > + offset = trbe_normal_offset(handle); > + return buf->trbe_base + offset; > +} > + > +static void clr_trbe_status(void) > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + > + WARN_ON(is_trbe_enabled()); > + trbsr &= ~TRBSR_IRQ; > + trbsr &= ~TRBSR_TRG; > + trbsr &= ~TRBSR_WRAP; > + trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); > + trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); > + trbsr &= ~TRBSR_STOP; > + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > +} > + > +static void set_trbe_limit_pointer_enabled(unsigned long addr) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + > + trblimitr &= ~TRBLIMITR_NVM; > + trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); > + trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); > + trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); > + > + /* > + * Fill trace buffer mode is used here while configuring the > + * TRBE for trace capture. In this particular mode, the trace > + * collection is stopped and a maintenance interrupt is raised > + * when the current write pointer wraps. This pause in trace > + * collection gives the software an opportunity to capture the > + * trace data in the interrupt handler, before reconfiguring > + * the TRBE. > + */ > + trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; > + > + /* > + * Trigger mode is not used here while configuring the TRBE for > + * the trace capture. Hence just keep this in the ignore mode. > + */ > + trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; > + trblimitr |= (addr & PAGE_MASK); > + > + trblimitr |= TRBLIMITR_ENABLE; > + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > +} > + > +static void trbe_enable_hw(struct trbe_buf *buf) > +{ > + WARN_ON(buf->trbe_write < buf->trbe_base); > + WARN_ON(buf->trbe_write >= buf->trbe_limit); > + set_trbe_disabled(); > + isb(); > + clr_trbe_status(); > + set_trbe_base_pointer(buf->trbe_base); > + set_trbe_write_pointer(buf->trbe_write); > + > + /* > + * Synchronize all the register updates > + * till now before enabling the TRBE. > + */ > + isb(); > + set_trbe_limit_pointer_enabled(buf->trbe_limit); > + > + /* Synchronize the TRBE enable event */ > + isb(); > +} > + > +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, > + struct perf_event *event, void **pages, > + int nr_pages, bool snapshot) > +{ > + struct trbe_buf *buf; > + struct page **pglist; > + int i; > + > + /* > + * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with > + * just a single page, there is not much room left while writing into > + * a partially filled TRBE buffer. Hence restrict the minimum buffer > + * size as two pages. > + */ > + if (nr_pages < 2) > + return NULL; > + > + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); > + if (IS_ERR(buf)) > + return ERR_PTR(-ENOMEM); > + > + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); > + if (IS_ERR(pglist)) { > + kfree(buf); > + return ERR_PTR(-ENOMEM); > + } > + > + for (i = 0; i < nr_pages; i++) > + pglist[i] = virt_to_page(pages[i]); > + > + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); > + if (IS_ERR((void *)buf->trbe_base)) { > + kfree(pglist); > + kfree(buf); > + return ERR_PTR(buf->trbe_base); > + } > + buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; > + buf->trbe_write = buf->trbe_base; > + buf->snapshot = snapshot; > + buf->nr_pages = nr_pages; > + buf->pages = pages; > + kfree(pglist); > + return buf; > +} > + > +void arm_trbe_free_buffer(void *config) > +{ > + struct trbe_buf *buf = config; > + > + vunmap((void *)buf->trbe_base); > + kfree(buf); > +} > + > +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, > + struct perf_output_handle *handle, > + void *config) > +{ > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > + struct trbe_buf *buf = config; > + unsigned long size, offset; > + > + WARN_ON(buf->cpudata != cpudata); > + WARN_ON(cpudata->cpu != smp_processor_id()); > + WARN_ON(cpudata->drvdata != drvdata); > + if (cpudata->mode != CS_MODE_PERF) > + return -EINVAL; > + > + /* > + * If the TRBE was disabled due to lack of space in the AUX buffer or a > + * spurious fault, the driver leaves it disabled, truncating the buffer. > + * Since the etm_perf driver expects to close out the AUX buffer, the > + * driver skips it. Thus, just pass in 0 size here to indicate that the > + * buffer was truncated. > + */ > + if (!is_trbe_enabled()) > + return 0; > + /* > + * perf handle structure needs to be shared with the TRBE IRQ handler for > + * capturing trace data and restarting the handle. There is a probability > + * of an undefined reference based crash when etm event is being stopped > + * while a TRBE IRQ also getting processed. This happens due the release > + * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping > + * the TRBE here will ensure that no IRQ could be generated when the perf > + * handle gets freed in etm_event_stop(). > + */ > + trbe_drain_and_disable_local(); > + offset = get_trbe_write_pointer() - get_trbe_base_pointer(); > + size = offset - PERF_IDX2OFF(handle->head, buf); > + if (buf->snapshot) > + handle->head += size; > + return size; > +} > + > +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) > +{ > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > + struct perf_output_handle *handle = data; > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + WARN_ON(cpudata->cpu != smp_processor_id()); > + WARN_ON(cpudata->drvdata != drvdata); > + if (mode != CS_MODE_PERF) > + return -EINVAL; > + > + *this_cpu_ptr(drvdata->handle) = handle; > + cpudata->buf = buf; > + cpudata->mode = mode; > + buf->cpudata = cpudata; > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > + if (buf->trbe_limit == buf->trbe_base) { > + trbe_stop_and_truncate_event(handle); > + return 0; > + } > + trbe_enable_hw(buf); > + return 0; > +} > + > +static int arm_trbe_disable(struct coresight_device *csdev) > +{ > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > + struct trbe_buf *buf = cpudata->buf; > + > + WARN_ON(buf->cpudata != cpudata); > + WARN_ON(cpudata->cpu != smp_processor_id()); > + WARN_ON(cpudata->drvdata != drvdata); > + if (cpudata->mode != CS_MODE_PERF) > + return -EINVAL; > + > + trbe_drain_and_disable_local(); > + buf->cpudata = NULL; > + cpudata->buf = NULL; > + cpudata->mode = CS_MODE_DISABLED; > + return 0; > +} > + > +static void trbe_handle_spurious(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > + if (buf->trbe_limit == buf->trbe_base) { > + trbe_drain_and_disable_local(); > + return; > + } > + trbe_enable_hw(buf); > +} > + > +static void trbe_handle_overflow(struct perf_output_handle *handle) > +{ > + struct perf_event *event = handle->event; > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + unsigned long offset, size; > + struct etm_event_data *event_data; > + > + offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); > + size = offset - PERF_IDX2OFF(handle->head, buf); > + if (buf->snapshot) > + handle->head = offset; > + perf_aux_output_end(handle, size); > + > + event_data = perf_aux_output_begin(handle, event); > + if (!event_data) { > + trbe_drain_and_disable_local(); > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > + return; > + } > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > + if (buf->trbe_limit == buf->trbe_base) { > + trbe_stop_and_truncate_event(handle); > + return; > + } > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; > + trbe_enable_hw(buf); > +} > + > +static bool is_perf_trbe(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + struct trbe_cpudata *cpudata = buf->cpudata; > + struct trbe_drvdata *drvdata = cpudata->drvdata; > + int cpu = smp_processor_id(); > + > + WARN_ON(buf->trbe_base != get_trbe_base_pointer()); > + WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); > + > + if (cpudata->mode != CS_MODE_PERF) > + return false; > + > + if (cpudata->cpu != cpu) > + return false; > + > + if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) > + return false; > + > + return true; > +} > + > +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + int ec = get_trbe_ec(trbsr); > + int bsc = get_trbe_bsc(trbsr); > + > + WARN_ON(is_trbe_running(trbsr)); > + if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) > + return TRBE_FAULT_ACT_FATAL; > + > + if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) > + return TRBE_FAULT_ACT_FATAL; > + > + if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { > + if (get_trbe_write_pointer() == get_trbe_base_pointer()) > + return TRBE_FAULT_ACT_WRAP; > + } > + return TRBE_FAULT_ACT_SPURIOUS; > +} > + > +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) > +{ > + struct perf_output_handle **handle_ptr = dev; > + struct perf_output_handle *handle = *handle_ptr; > + enum trbe_fault_action act; > + > + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); > + clr_trbe_irq(); > + > + /* > + * Ensure the trace is visible to the CPUs and > + * any external aborts have been resolved. > + */ > + trbe_drain_buffer(); > + isb(); > + > + if (!perf_get_aux(handle)) > + return IRQ_NONE; > + > + if (!is_perf_trbe(handle)) > + return IRQ_NONE; > + > + irq_work_run(); > + > + act = trbe_get_fault_act(handle); > + switch (act) { > + case TRBE_FAULT_ACT_WRAP: > + trbe_handle_overflow(handle); > + break; > + case TRBE_FAULT_ACT_SPURIOUS: > + trbe_handle_spurious(handle); > + break; > + case TRBE_FAULT_ACT_FATAL: > + trbe_stop_and_truncate_event(handle); > + break; > + } > + return IRQ_HANDLED; > +} > + > +static const struct coresight_ops_sink arm_trbe_sink_ops = { > + .enable = arm_trbe_enable, > + .disable = arm_trbe_disable, > + .alloc_buffer = arm_trbe_alloc_buffer, > + .free_buffer = arm_trbe_free_buffer, > + .update_buffer = arm_trbe_update_buffer, > +}; > + > +static const struct coresight_ops arm_trbe_cs_ops = { > + .sink_ops = &arm_trbe_sink_ops, > +}; > + > +static ssize_t align_show(struct device *dev, struct device_attribute *attr, char *buf) > +{ > + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > + > + return sprintf(buf, "%llx\n", cpudata->trbe_align); > +} > +static DEVICE_ATTR_RO(align); > + > +static ssize_t dbm_show(struct device *dev, struct device_attribute *attr, char *buf) > +{ > + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > + > + return sprintf(buf, "%d\n", cpudata->trbe_dbm); > +} > +static DEVICE_ATTR_RO(dbm); > + > +static struct attribute *arm_trbe_attrs[] = { > + &dev_attr_align.attr, > + &dev_attr_dbm.attr, > + NULL, > +}; > + > +static const struct attribute_group arm_trbe_group = { > + .attrs = arm_trbe_attrs, > +}; > + > +static const struct attribute_group *arm_trbe_groups[] = { > + &arm_trbe_group, > + NULL, > +}; > + > +static void arm_trbe_probe_coresight_cpu(void *info) > +{ > + struct trbe_drvdata *drvdata = info; > + struct coresight_desc desc = { 0 }; > + int cpu = smp_processor_id(); > + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); > + struct device *dev; > + > + if (WARN_ON(!cpudata)) > + goto cpu_clear; > + > + if (trbe_csdev) > + return; > + > + cpudata->cpu = smp_processor_id(); > + cpudata->drvdata = drvdata; > + dev = &cpudata->drvdata->pdev->dev; > + > + if (!is_trbe_available()) { > + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + > + if (!is_trbe_programmable(trbidr)) { > + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); > + if (IS_ERR(desc.name)) > + goto cpu_clear; > + > + desc.type = CORESIGHT_DEV_TYPE_SINK; > + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; > + desc.ops = &arm_trbe_cs_ops; > + desc.pdata = dev_get_platdata(dev); > + desc.groups = arm_trbe_groups; > + desc.dev = dev; > + trbe_csdev = coresight_register(&desc); > + if (IS_ERR(trbe_csdev)) > + goto cpu_clear; > + > + dev_set_drvdata(&trbe_csdev->dev, cpudata); > + cpudata->trbe_dbm = get_trbe_flag_update(trbidr); > + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); > + if (cpudata->trbe_align > SZ_2K) { > + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + per_cpu(csdev_sink, cpu) = trbe_csdev; > + trbe_reset_local(); > + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > + return; > +cpu_clear: > + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); > +} > + > +static void arm_trbe_remove_coresight_cpu(void *info) > +{ > + int cpu = smp_processor_id(); > + struct trbe_drvdata *drvdata = info; > + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > + > + if (trbe_csdev) { > + coresight_unregister(trbe_csdev); > + cpudata->drvdata = NULL; > + per_cpu(csdev_sink, cpu) = NULL; > + } > + disable_percpu_irq(drvdata->irq); > + trbe_reset_local(); > +} > + > +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) > +{ > + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); > + if (IS_ERR(drvdata->cpudata)) > + return PTR_ERR(drvdata->cpudata); > + > + arm_trbe_probe_coresight_cpu(drvdata); > + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); > + return 0; > +} > + > +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) > +{ > + arm_trbe_remove_coresight_cpu(drvdata); > + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); > + free_percpu(drvdata->cpudata); > + return 0; > +} > + > +static int arm_trbe_cpu_startup(unsigned int cpu, struct hlist_node *node) > +{ > + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > + > + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > + if (!per_cpu(csdev_sink, cpu)) { > + arm_trbe_probe_coresight_cpu(drvdata); > + } else { > + trbe_reset_local(); > + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > + } > + } > + return 0; > +} > + > +static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node) > +{ > + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > + > + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > + disable_percpu_irq(drvdata->irq); > + trbe_reset_local(); > + } > + return 0; > +} > + > +static int arm_trbe_probe_cpuhp(struct trbe_drvdata *drvdata) > +{ > + enum cpuhp_state trbe_online; > + > + trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, > + arm_trbe_cpu_startup, arm_trbe_cpu_teardown); > + if (trbe_online < 0) > + return -EINVAL; > + > + if (cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node)) > + return -EINVAL; > + > + drvdata->trbe_online = trbe_online; > + return 0; > +} > + > +static void arm_trbe_remove_cpuhp(struct trbe_drvdata *drvdata) > +{ > + cpuhp_remove_multi_state(drvdata->trbe_online); > +} > + > +static int arm_trbe_probe_irq(struct platform_device *pdev, > + struct trbe_drvdata *drvdata) > +{ > + drvdata->irq = platform_get_irq(pdev, 0); > + if (!drvdata->irq) { > + pr_err("IRQ not found for the platform device\n"); > + return -ENXIO; > + } > + > + if (!irq_is_percpu(drvdata->irq)) { > + pr_err("IRQ is not a PPI\n"); > + return -EINVAL; > + } > + > + if (irq_get_percpu_devid_partition(drvdata->irq, &drvdata->supported_cpus)) > + return -EINVAL; > + > + drvdata->handle = alloc_percpu(typeof(*drvdata->handle)); > + if (!drvdata->handle) > + return -ENOMEM; > + > + if (request_percpu_irq(drvdata->irq, arm_trbe_irq_handler, DRVNAME, drvdata->handle)) { > + free_percpu(drvdata->handle); > + return -EINVAL; > + } > + return 0; > +} > + > +static void arm_trbe_remove_irq(struct trbe_drvdata *drvdata) > +{ > + free_percpu_irq(drvdata->irq, drvdata->handle); > + free_percpu(drvdata->handle); > +} > + > +static int arm_trbe_device_probe(struct platform_device *pdev) > +{ > + struct coresight_platform_data *pdata; > + struct trbe_drvdata *drvdata; > + struct device *dev = &pdev->dev; > + int ret; > + > + drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL); > + if (IS_ERR(drvdata)) > + return -ENOMEM; > + > + pdata = coresight_get_platform_data(dev); > + if (IS_ERR(pdata)) { > + kfree(drvdata); > + return -ENOMEM; > + } > + > + dev_set_drvdata(dev, drvdata); > + dev->platform_data = pdata; > + drvdata->pdev = pdev; > + ret = arm_trbe_probe_irq(pdev, drvdata); > + if (ret) > + goto irq_failed; > + > + ret = arm_trbe_probe_coresight(drvdata); > + if (ret) > + goto probe_failed; > + > + ret = arm_trbe_probe_cpuhp(drvdata); > + if (ret) > + goto cpuhp_failed; > + > + return 0; > +cpuhp_failed: > + arm_trbe_remove_coresight(drvdata); > +probe_failed: > + arm_trbe_remove_irq(drvdata); > +irq_failed: > + kfree(pdata); > + kfree(drvdata); > + return ret; > +} > + > +static int arm_trbe_device_remove(struct platform_device *pdev) > +{ > + struct coresight_platform_data *pdata = dev_get_platdata(&pdev->dev); > + struct trbe_drvdata *drvdata = platform_get_drvdata(pdev); > + > + arm_trbe_remove_coresight(drvdata); > + arm_trbe_remove_cpuhp(drvdata); > + arm_trbe_remove_irq(drvdata); > + kfree(pdata); > + kfree(drvdata); > + return 0; > +} > + > +static const struct of_device_id arm_trbe_of_match[] = { > + { .compatible = "arm,trace-buffer-extension"}, > + {}, > +}; > +MODULE_DEVICE_TABLE(of, arm_trbe_of_match); > + > +static struct platform_driver arm_trbe_driver = { > + .driver = { > + .name = DRVNAME, > + .of_match_table = of_match_ptr(arm_trbe_of_match), > + .suppress_bind_attrs = true, > + }, > + .probe = arm_trbe_device_probe, > + .remove = arm_trbe_device_remove, > +}; > + > +static int __init arm_trbe_init(void) > +{ > + int ret; > + > + if (arm64_kernel_unmapped_at_el0()) { > + pr_err("TRBE wouldn't work if kernel gets unmapped at EL0\n"); > + return -EOPNOTSUPP; > + } > + > + ret = platform_driver_register(&arm_trbe_driver); > + if (!ret) > + return 0; > + > + pr_err("Error registering %s platform driver\n", DRVNAME); > + return ret; > +} > + > +static void __exit arm_trbe_exit(void) > +{ > + platform_driver_unregister(&arm_trbe_driver); > +} > +module_init(arm_trbe_init); > +module_exit(arm_trbe_exit); > + > +MODULE_AUTHOR("Anshuman Khandual <anshuman.khandual@arm.com>"); > +MODULE_DESCRIPTION("Arm Trace Buffer Extension (TRBE) driver"); > +MODULE_LICENSE("GPL v2"); > diff --git a/drivers/hwtracing/coresight/coresight-trbe.h b/drivers/hwtracing/coresight/coresight-trbe.h > new file mode 100644 > index 0000000..43308bc > --- /dev/null > +++ b/drivers/hwtracing/coresight/coresight-trbe.h > @@ -0,0 +1,160 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * This contains all required hardware related helper functions for > + * Trace Buffer Extension (TRBE) driver in the coresight framework. > + * > + * Copyright (C) 2020 ARM Ltd. > + * > + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > + */ > +#include <linux/coresight.h> > +#include <linux/device.h> > +#include <linux/irq.h> > +#include <linux/kernel.h> > +#include <linux/of.h> > +#include <linux/platform_device.h> > +#include <linux/smp.h> > + > +#include "coresight-etm-perf.h" > + > +DECLARE_PER_CPU(struct coresight_device *, csdev_sink); > + > +static inline bool is_trbe_available(void) > +{ > + u64 aa64dfr0 = read_sysreg_s(SYS_ID_AA64DFR0_EL1); > + unsigned int trbe = cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_TRBE_SHIFT); > + > + return trbe >= 0b0001; > +} > + > +static inline bool is_trbe_enabled(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + return trblimitr & TRBLIMITR_ENABLE; > +} > + > +#define TRBE_EC_OTHERS 0 > +#define TRBE_EC_STAGE1_ABORT 36 > +#define TRBE_EC_STAGE2_ABORT 37 > + > +static inline int get_trbe_ec(u64 trbsr) > +{ > + return (trbsr >> TRBSR_EC_SHIFT) & TRBSR_EC_MASK; > +} > + > +#define TRBE_BSC_NOT_STOPPED 0 > +#define TRBE_BSC_FILLED 1 > +#define TRBE_BSC_TRIGGERED 2 > + > +static inline int get_trbe_bsc(u64 trbsr) > +{ > + return (trbsr >> TRBSR_BSC_SHIFT) & TRBSR_BSC_MASK; > +} > + > +static inline void clr_trbe_irq(void) > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + > + trbsr &= ~TRBSR_IRQ; > + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > +} > + > +static inline bool is_trbe_irq(u64 trbsr) > +{ > + return trbsr & TRBSR_IRQ; > +} > + > +static inline bool is_trbe_trg(u64 trbsr) > +{ > + return trbsr & TRBSR_TRG; > +} > + > +static inline bool is_trbe_wrap(u64 trbsr) > +{ > + return trbsr & TRBSR_WRAP; > +} > + > +static inline bool is_trbe_abort(u64 trbsr) > +{ > + return trbsr & TRBSR_ABORT; > +} > + > +static inline bool is_trbe_running(u64 trbsr) > +{ > + return !(trbsr & TRBSR_STOP); > +} > + > +#define TRBE_TRIG_MODE_STOP 0 > +#define TRBE_TRIG_MODE_IRQ 1 > +#define TRBE_TRIG_MODE_IGNORE 3 > + > +#define TRBE_FILL_MODE_FILL 0 > +#define TRBE_FILL_MODE_WRAP 1 > +#define TRBE_FILL_MODE_CIRCULAR_BUFFER 3 > + > +static inline void set_trbe_disabled(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + trblimitr &= ~TRBLIMITR_ENABLE; > + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > +} > + > +static inline bool get_trbe_flag_update(u64 trbidr) > +{ > + return trbidr & TRBIDR_FLAG; > +} > + > +static inline bool is_trbe_programmable(u64 trbidr) > +{ > + return !(trbidr & TRBIDR_PROG); > +} > + > +static inline int get_trbe_address_align(u64 trbidr) > +{ > + return (trbidr >> TRBIDR_ALIGN_SHIFT) & TRBIDR_ALIGN_MASK; > +} > + > +static inline unsigned long get_trbe_write_pointer(void) > +{ > + u64 trbptr = read_sysreg_s(SYS_TRBPTR_EL1); > + unsigned long addr = (trbptr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; > + > + return addr; > +} > + > +static inline void set_trbe_write_pointer(unsigned long addr) > +{ > + WARN_ON(is_trbe_enabled()); > + addr = (addr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; > + write_sysreg_s(addr, SYS_TRBPTR_EL1); > +} > + > +static inline unsigned long get_trbe_limit_pointer(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + unsigned long limit = (trblimitr >> TRBLIMITR_LIMIT_SHIFT) & TRBLIMITR_LIMIT_MASK; > + unsigned long addr = limit << TRBLIMITR_LIMIT_SHIFT; > + > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + return addr; > +} > + > +static inline unsigned long get_trbe_base_pointer(void) > +{ > + u64 trbbaser = read_sysreg_s(SYS_TRBBASER_EL1); > + unsigned long addr = (trbbaser >> TRBBASER_BASE_SHIFT) & TRBBASER_BASE_MASK; > + > + addr = addr << TRBBASER_BASE_SHIFT; > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + return addr; > +} > + > +static inline void set_trbe_base_pointer(unsigned long addr) > +{ > + WARN_ON(is_trbe_enabled()); > + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + write_sysreg_s(addr, SYS_TRBBASER_EL1); > +} > -- > 2.7.4 >
On 2/5/21 11:23 PM, Mathieu Poirier wrote: > On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: >> Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is >> accessible via the system registers. The TRBE supports different addressing >> modes including CPU virtual address and buffer modes including the circular >> buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), >> an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the >> access to the trace buffer could be prohibited by a higher exception level >> (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU >> private interrupt (PPI) on address translation errors and when the buffer >> is full. Overall implementation here is inspired from the Arm SPE driver. >> > > I got this message when applying the patch: > > Applying: coresight: sink: Add TRBE driver > .git/rebase-apply/patch:76: new blank line at EOF. > + > warning: 1 line adds whitespace errors. It could be the additional blank line at the end of documentation file i.e Documentation/trace/coresight/coresight-trbe.rst, will drop it. > >> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> >> Cc: Mike Leach <mike.leach@linaro.org> >> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> >> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> >> --- >> Changes in V3: >> >> - Added new DT bindings document TRBE.yaml >> - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 >> - Dropped isb() from trbe_reset_local() >> - Dropped gap between (void *) and buf->trbe_base >> - Changed 'int' to 'unsigned int' in is_trbe_available() >> - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), >> set_trbe_enabled() and set_trbe_limit_pointer() >> - Changed get_trbe_flag_update(), is_trbe_programmable() and >> get_trbe_address_align() to accept TRBIDR value >> - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), >> is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value >> - Dropped snapshot mode condition in arm_trbe_alloc_buffer() >> - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled >> - Compute trbe_limit before trbe_write to get the updated handle >> - Added trbe_stop_and_truncate_event() >> - Dropped trbe_handle_fatal() >> >> Documentation/trace/coresight/coresight-trbe.rst | 39 + >> arch/arm64/include/asm/sysreg.h | 1 + >> drivers/hwtracing/coresight/Kconfig | 11 + >> drivers/hwtracing/coresight/Makefile | 1 + >> drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ >> drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ >> 6 files changed, 1235 insertions(+) >> create mode 100644 Documentation/trace/coresight/coresight-trbe.rst >> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c >> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h >> >> diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst >> new file mode 100644 >> index 0000000..1cbb819 >> --- /dev/null >> +++ b/Documentation/trace/coresight/coresight-trbe.rst >> @@ -0,0 +1,39 @@ >> +.. SPDX-License-Identifier: GPL-2.0 >> + >> +============================== >> +Trace Buffer Extension (TRBE). >> +============================== >> + >> + :Author: Anshuman Khandual <anshuman.khandual@arm.com> >> + :Date: November 2020 >> + >> +Hardware Description >> +-------------------- >> + >> +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system >> +memory, CPU traces generated from a corresponding percpu tracing unit. This >> +gets plugged in as a coresight sink device because the corresponding trace >> +genarators (ETE), are plugged in as source device. >> + >> +The TRBE is not compliant to CoreSight architecture specifications, but is >> +driven via the CoreSight driver framework to support the ETE (which is >> +CoreSight compliant) integration. >> + >> +Sysfs files and directories >> +--------------------------- >> + >> +The TRBE devices appear on the existing coresight bus alongside the other >> +coresight devices:: >> + >> + >$ ls /sys/bus/coresight/devices >> + trbe0 trbe1 trbe2 trbe3 >> + >> +The ``trbe<N>`` named TRBEs are associated with a CPU.:: >> + >> + >$ ls /sys/bus/coresight/devices/trbe0/ >> + align dbm >> + >> +*Key file items are:-* >> + * ``align``: TRBE write pointer alignment >> + * ``dbm``: TRBE updates memory with access and dirty flags >> + > > Please add documentation for these, the same way it was done for all the other CS > components [1]. > > [1]. https://elixir.bootlin.com/linux/latest/source/Documentation/ABI/testing > (sysfs-bus-coresight-device-xyz) Sure, will add the following new sysfs doc file in this regard. Marked the KernelVersion as 5.12, will change if required. new file mode 100644 index 0000000..5cb090f --- /dev/null +++ b/Documentation/ABI/testing/sysfs-bus-coresight-devices-trbe @@ -0,0 +1,14 @@ +What: /sys/bus/coresight/devices/trbe<cpu>/align +Date: Feb 2021 +KernelVersion: 5.12 +Contact: Anshuman Khandual <anshuman.khandual@arm.com> +Description: (Read) Shows the TRBE write pointer alignment. This value + is fetched from the TRBIDR register. + +What: /sys/bus/coresight/devices/trbe<cpu>/dbm +Date: Feb 2021 +KernelVersion: 5.12 +Contact: Anshuman Khandual <anshuman.khandual@arm.com> +Description: (Read) Shows if TRBE updates in the memory are with access + and dirty flag updates as well. This value is fetched from + the TRBIDR register. > >> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h >> index 85ae4db..9e2e9b7 100644 >> --- a/arch/arm64/include/asm/sysreg.h >> +++ b/arch/arm64/include/asm/sysreg.h >> @@ -97,6 +97,7 @@ >> #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) >> #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) >> #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) >> +#define TSB_CSYNC __emit_inst(0xd503225f) >> >> #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) >> #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) >> diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig >> index f154ae7..aa657ab 100644 >> --- a/drivers/hwtracing/coresight/Kconfig >> +++ b/drivers/hwtracing/coresight/Kconfig >> @@ -164,6 +164,17 @@ config CORESIGHT_CTI >> To compile this driver as a module, choose M here: the >> module will be called coresight-cti. >> >> +config CORESIGHT_TRBE >> + bool "Trace Buffer Extension (TRBE) driver" > > Shouldn't that be "tristate"? Yes, will change. > >> + depends on ARM64 > > I would make it dependent on ETMv4 as well since it can only be used by that > component. Sure, will also add CORESIGHT_SOURCE_ETM4X as dependency. > >> + help >> + This driver provides support for percpu Trace Buffer Extension (TRBE). >> + TRBE always needs to be used along with it's corresponding percpu ETE >> + component. ETE generates trace data which is then captured with TRBE. >> + Unlike traditional sink devices, TRBE is a CPU feature accessible via >> + system registers. But it's explicit dependency with trace unit (ETE) >> + requires it to be plugged in as a coresight sink device. > > Please add: > > "To compile this driver as a module, choose M here: the > module will be called coresight-trbe." Added. > > I'm out of time for today, I will continue on Monday. Sure.
On Fri, Feb 05, 2021 at 10:53:30AM -0700, Mathieu Poirier wrote: > On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: > > Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is > > accessible via the system registers. The TRBE supports different addressing > > modes including CPU virtual address and buffer modes including the circular > > buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), > > an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the > > access to the trace buffer could be prohibited by a higher exception level > > (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU > > private interrupt (PPI) on address translation errors and when the buffer > > is full. Overall implementation here is inspired from the Arm SPE driver. > > > > I got this message when applying the patch: > > Applying: coresight: sink: Add TRBE driver > .git/rebase-apply/patch:76: new blank line at EOF. > + > warning: 1 line adds whitespace errors. > > > Cc: Mathieu Poirier <mathieu.poirier@linaro.org> > > Cc: Mike Leach <mike.leach@linaro.org> > > Cc: Suzuki K Poulose <suzuki.poulose@arm.com> > > Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> > > --- > > Changes in V3: > > > > - Added new DT bindings document TRBE.yaml > > - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 > > - Dropped isb() from trbe_reset_local() > > - Dropped gap between (void *) and buf->trbe_base > > - Changed 'int' to 'unsigned int' in is_trbe_available() > > - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), > > set_trbe_enabled() and set_trbe_limit_pointer() > > - Changed get_trbe_flag_update(), is_trbe_programmable() and > > get_trbe_address_align() to accept TRBIDR value > > - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), > > is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value > > - Dropped snapshot mode condition in arm_trbe_alloc_buffer() > > - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled > > - Compute trbe_limit before trbe_write to get the updated handle > > - Added trbe_stop_and_truncate_event() > > - Dropped trbe_handle_fatal() > > > > Documentation/trace/coresight/coresight-trbe.rst | 39 + > > arch/arm64/include/asm/sysreg.h | 1 + > > drivers/hwtracing/coresight/Kconfig | 11 + > > drivers/hwtracing/coresight/Makefile | 1 + > > drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ > > drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ > > 6 files changed, 1235 insertions(+) > > create mode 100644 Documentation/trace/coresight/coresight-trbe.rst > > create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c > > create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h > > > > diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst > > new file mode 100644 > > index 0000000..1cbb819 > > --- /dev/null > > +++ b/Documentation/trace/coresight/coresight-trbe.rst > > @@ -0,0 +1,39 @@ > > +.. SPDX-License-Identifier: GPL-2.0 > > + > > +============================== > > +Trace Buffer Extension (TRBE). > > +============================== > > + > > + :Author: Anshuman Khandual <anshuman.khandual@arm.com> > > + :Date: November 2020 > > + > > +Hardware Description > > +-------------------- > > + > > +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system > > +memory, CPU traces generated from a corresponding percpu tracing unit. This > > +gets plugged in as a coresight sink device because the corresponding trace > > +genarators (ETE), are plugged in as source device. > > + > > +The TRBE is not compliant to CoreSight architecture specifications, but is > > +driven via the CoreSight driver framework to support the ETE (which is > > +CoreSight compliant) integration. > > + > > +Sysfs files and directories > > +--------------------------- > > + > > +The TRBE devices appear on the existing coresight bus alongside the other > > +coresight devices:: > > + > > + >$ ls /sys/bus/coresight/devices > > + trbe0 trbe1 trbe2 trbe3 > > + > > +The ``trbe<N>`` named TRBEs are associated with a CPU.:: > > + > > + >$ ls /sys/bus/coresight/devices/trbe0/ > > + align dbm > > + > > +*Key file items are:-* > > + * ``align``: TRBE write pointer alignment > > + * ``dbm``: TRBE updates memory with access and dirty flags > > + > > Please add documentation for these, the same way it was done for all the other CS > components [1]. > > [1]. https://elixir.bootlin.com/linux/latest/source/Documentation/ABI/testing > (sysfs-bus-coresight-device-xyz) > > > diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h > > index 85ae4db..9e2e9b7 100644 > > --- a/arch/arm64/include/asm/sysreg.h > > +++ b/arch/arm64/include/asm/sysreg.h > > @@ -97,6 +97,7 @@ > > #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) > > #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) > > #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) > > +#define TSB_CSYNC __emit_inst(0xd503225f) > > > > #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) > > #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) > > diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig > > index f154ae7..aa657ab 100644 > > --- a/drivers/hwtracing/coresight/Kconfig > > +++ b/drivers/hwtracing/coresight/Kconfig > > @@ -164,6 +164,17 @@ config CORESIGHT_CTI > > To compile this driver as a module, choose M here: the > > module will be called coresight-cti. > > > > +config CORESIGHT_TRBE > > + bool "Trace Buffer Extension (TRBE) driver" > > Shouldn't that be "tristate"? > > > + depends on ARM64 > > I would make it dependent on ETMv4 as well since it can only be used by that > component. > > > + help > > + This driver provides support for percpu Trace Buffer Extension (TRBE). > > + TRBE always needs to be used along with it's corresponding percpu ETE > > + component. ETE generates trace data which is then captured with TRBE. > > + Unlike traditional sink devices, TRBE is a CPU feature accessible via > > + system registers. But it's explicit dependency with trace unit (ETE) > > + requires it to be plugged in as a coresight sink device. > > Please add: > > "To compile this driver as a module, choose M here: the > module will be called coresight-trbe." > > I'm out of time for today, I will continue on Monday. > > Mathieu > > > + > > config CORESIGHT_CTI_INTEGRATION_REGS > > bool "Access CTI CoreSight Integration Registers" > > depends on CORESIGHT_CTI > > diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile > > index f20e357..d608165 100644 > > --- a/drivers/hwtracing/coresight/Makefile > > +++ b/drivers/hwtracing/coresight/Makefile > > @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o > > obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o > > obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o > > obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o > > +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o > > coresight-cti-y := coresight-cti-core.o coresight-cti-platform.o \ > > coresight-cti-sysfs.o > > diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c > > new file mode 100644 > > index 0000000..1464d8b > > --- /dev/null > > +++ b/drivers/hwtracing/coresight/coresight-trbe.c > > @@ -0,0 +1,1023 @@ > > +// SPDX-License-Identifier: GPL-2.0 > > +/* > > + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight > > + * sink device could then pair with an appropriate per-cpu coresight source > > + * device (ETE) thus generating required trace data. Trace can be enabled > > + * via the perf framework. > > + * > > + * Copyright (C) 2020 ARM Ltd. > > + * > > + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > > + */ > > +#define DRVNAME "arm_trbe" > > + > > +#define pr_fmt(fmt) DRVNAME ": " fmt > > + > > +#include "coresight-trbe.h" > > + > > +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) > > + > > +/* > > + * A padding packet that will help the user space tools > > + * in skipping relevant sections in the captured trace > > + * data which could not be decoded. TRBE doesn't support > > + * formatting the trace data, unlike the legacy CoreSight > > + * sinks and thus we use ETE trace packets to pad the > > + * sections of the buffer. > > + */ > > +#define ETE_IGNORE_PACKET 0x70 > > + > > +/* > > + * Minimum amount of meaningful trace will contain: > > + * A-Sync, Trace Info, Trace On, Address, Atom. > > + * This is about 44bytes of ETE trace. To be on > > + * the safer side, we assume 64bytes is the minimum > > + * space required for a meaningful session, before > > + * we hit a "WRAP" event. > > + */ > > +#define TRBE_TRACE_MIN_BUF_SIZE 64 > > + > > +enum trbe_fault_action { > > + TRBE_FAULT_ACT_WRAP, > > + TRBE_FAULT_ACT_SPURIOUS, > > + TRBE_FAULT_ACT_FATAL, > > +}; > > + > > +struct trbe_buf { > > + unsigned long trbe_base; > > + unsigned long trbe_limit; > > + unsigned long trbe_write; > > + int nr_pages; > > + void **pages; > > + bool snapshot; > > + struct trbe_cpudata *cpudata; > > +}; > > + > > +struct trbe_cpudata { > > + bool trbe_dbm; > > + u64 trbe_align; > > + int cpu; > > + enum cs_mode mode; > > + struct trbe_buf *buf; > > + struct trbe_drvdata *drvdata; > > +}; > > + > > +struct trbe_drvdata { > > + struct trbe_cpudata __percpu *cpudata; > > + struct perf_output_handle __percpu **handle; > > + struct hlist_node hotplug_node; > > + int irq; > > + cpumask_t supported_cpus; > > + enum cpuhp_state trbe_online; > > + struct platform_device *pdev; > > +}; > > + > > +static int trbe_alloc_node(struct perf_event *event) > > +{ > > + if (event->cpu == -1) > > + return NUMA_NO_NODE; > > + return cpu_to_node(event->cpu); > > +} > > + > > +static void trbe_drain_buffer(void) > > +{ > > + asm(TSB_CSYNC); > > + dsb(nsh); > > +} > > + > > +static void trbe_drain_and_disable_local(void) > > +{ > > + trbe_drain_buffer(); > > + write_sysreg_s(0, SYS_TRBLIMITR_EL1); > > + isb(); > > +} > > + > > +static void trbe_reset_local(void) > > +{ > > + trbe_drain_and_disable_local(); > > + write_sysreg_s(0, SYS_TRBPTR_EL1); > > + write_sysreg_s(0, SYS_TRBBASER_EL1); > > + write_sysreg_s(0, SYS_TRBSR_EL1); > > +} > > + > > +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) > > +{ > > + struct trbe_buf *buf = etm_perf_sink_config(handle); > > + > > + /* > > + * We cannot proceed with the buffer collection and we > > + * do not have any data for the current session. The > > + * etm_perf driver expects to close out the aux_buffer > > + * at event_stop(). So disable the TRBE here and leave > > + * the update_buffer() to return a 0 size. > > + */ > > + trbe_drain_and_disable_local(); > > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > > +} > > + > > +/* > > + * TRBE Buffer Management > > + * > > + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, > > + * it starts writing trace data from the write pointer onward till the limit pointer. > > + * When the write pointer reaches the address just before the limit pointer, it gets > > + * wrapped around again to the base pointer. This is called a TRBE wrap event, which > > + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver > > + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ > > + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and > > + * LIMIT pointers. > > + * > > + * Wrap around with an IRQ > > + * ------ < ------ < ------- < ----- < ----- > > + * | | > > + * ------ > ------ > ------- > ----- > ----- > > + * > > + * +---------------+-----------------------+ > > + * | | | > > + * +---------------+-----------------------+ > > + * Base Pointer Write Pointer Limit Pointer > > + * > > + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write > > + * pointer can be aligned to the implementation defined TRBE trace buffer alignment > > + * as captured in trbe_cpudata->trbe_align. > > + * > > + * > > + * head tail wakeup > > + * +---------------------------------------+----- ~ ~ ------ > > + * |$$$$$$$|################|$$$$$$$$$$$$$$| | > > + * +---------------------------------------+----- ~ ~ ------ > > + * Base Pointer Write Pointer Limit Pointer > > + * > > + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing > > + * values which tracks all the driver writes and user reads from the perf auxiliary > > + * buffer. Generally [head..tail] is the area where the driver can write into unless > > + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and > > + * configured depending on the perf_output_handle indices, so that the driver does > > + * not override into areas in the perf auxiliary buffer which is being or yet to be > > + * consumed from the user space. The enabled TRBE buffer area is a moving subset of > > + * the allocated perf auxiliary buffer. > > + */ > > +static void trbe_pad_buf(struct perf_output_handle *handle, int len) > > +{ > > + struct trbe_buf *buf = etm_perf_sink_config(handle); > > + u64 head = PERF_IDX2OFF(handle->head, buf); > > + > > + memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); > > + if (!buf->snapshot) > > + perf_aux_output_skip(handle, len); > > +} > > + > > +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) > > +{ > > + struct trbe_buf *buf = etm_perf_sink_config(handle); > > + > > + /* > > + * The ETE trace has alignment synchronization packets allowing > > + * the decoder to reset in case of an overflow or corruption. > > + * So we can use the entire buffer for the snapshot mode. > > + */ > > + return buf->nr_pages * PAGE_SIZE; > > +} > > + > > +/* > > + * TRBE Limit Calculation > > + * > > + * The following markers are used to illustrate various TRBE buffer situations. > > + * > > + * $$$$ - Data area, unconsumed captured trace data, not to be overridden > > + * #### - Free area, enabled, trace will be written > > + * %%%% - Free area, disabled, trace will not be written > > + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped > > + */ > > +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) > > +{ > > + struct trbe_buf *buf = etm_perf_sink_config(handle); > > + struct trbe_cpudata *cpudata = buf->cpudata; > > + const u64 bufsize = buf->nr_pages * PAGE_SIZE; > > + u64 limit = bufsize; > > + u64 head, tail, wakeup; > > + > > + head = PERF_IDX2OFF(handle->head, buf); > > + > > + /* > > + * head > > + * ------->| > > + * | > > + * head TRBE align tail > > + * +----|-------|---------------|-------+ > > + * |$$$$|=======|###############|$$$$$$$| > > + * +----|-------|---------------|-------+ > > + * trbe_base trbe_base + nr_pages > > + * > > + * Perf aux buffer output head position can be misaligned depending on > > + * various factors including user space reads. In case misaligned, head > > + * needs to be aligned before TRBE can be configured. Pad the alignment > > + * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools > > + * and skip this section thus advancing the head. > > + */ > > + if (!IS_ALIGNED(head, cpudata->trbe_align)) { > > + unsigned long delta = roundup(head, cpudata->trbe_align) - head; > > + > > + delta = min(delta, handle->size); > > + trbe_pad_buf(handle, delta); > > + head = PERF_IDX2OFF(handle->head, buf); > > + } > > + > > + /* > > + * head = tail (size = 0) > > + * +----|-------------------------------+ > > + * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ | > > + * +----|-------------------------------+ > > + * trbe_base trbe_base + nr_pages > > + * > > + * Perf aux buffer does not have any space for the driver to write into. > > + * Just communicate trace truncation event to the user space by marking > > + * it with PERF_AUX_FLAG_TRUNCATED. > > + */ > > + if (!handle->size) { > > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > > + return 0; > > + } > > + > > + /* Compute the tail and wakeup indices now that we've aligned head */ > > + tail = PERF_IDX2OFF(handle->head + handle->size, buf); > > + wakeup = PERF_IDX2OFF(handle->wakeup, buf); > > + > > + /* > > + * Lets calculate the buffer area which TRBE could write into. There > > + * are three possible scenarios here. Limit needs to be aligned with > > + * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the > > + * unconsumed data. > > + * > > + * 1) head < tail > > + * > > + * head tail > > + * +----|-----------------------|-------+ > > + * |$$$$|#######################|$$$$$$$| > > + * +----|-----------------------|-------+ > > + * trbe_base limit trbe_base + nr_pages > > + * > > + * TRBE could write into [head..tail] area. Unless the tail is right at > > + * the end of the buffer, neither an wrap around nor an IRQ is expected > > + * while being enabled. > > + * > > + * 2) head == tail > > + * > > + * head = tail (size > 0) > > + * +----|-------------------------------+ > > + * |%%%%|###############################| > > + * +----|-------------------------------+ > > + * trbe_base limit = trbe_base + nr_pages > > + * > > + * TRBE should just write into [head..base + nr_pages] area even though > > + * the entire buffer is empty. Reason being, when the trace reaches the > > + * end of the buffer, it will just wrap around with an IRQ giving an > > + * opportunity to reconfigure the buffer. > > + * > > + * 3) tail < head > > + * > > + * tail head > > + * +----|-----------------------|-------+ > > + * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| > > + * +----|-----------------------|-------+ > > + * trbe_base limit = trbe_base + nr_pages > > + * > > + * TRBE should just write into [head..base + nr_pages] area even though > > + * the [trbe_base..tail] is also empty. Reason being, when the trace > > + * reaches the end of the buffer, it will just wrap around with an IRQ > > + * giving an opportunity to reconfigure the buffer. > > + */ > > + if (head < tail) > > + limit = round_down(tail, PAGE_SIZE); > > + > > + /* > > + * Wakeup may be arbitrarily far into the future. If it's not in the > > + * current generation, either we'll wrap before hitting it, or it's > > + * in the past and has been handled already. > > + * > > + * If there's a wakeup before we wrap, arrange to be woken up by the > > + * page boundary following it. Keep the tail boundary if that's lower. > > + * > > + * head wakeup tail > > + * +----|---------------|-------|-------+ > > + * |$$$$|###############|%%%%%%%|$$$$$$$| > > + * +----|---------------|-------|-------+ > > + * trbe_base limit trbe_base + nr_pages > > + */ > > + if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) > > + limit = min(limit, round_up(wakeup, PAGE_SIZE)); > > + > > + /* > > + * There are two situation when this can happen i.e limit is before > > + * the head and hence TRBE cannot be configured. > > + * > > + * 1) head < tail (aligned down with PAGE_SIZE) and also they are both > > + * within the same PAGE size range. > > + * > > + * PAGE_SIZE > > + * |----------------------| > > + * > > + * limit head tail > > + * +------------|------|--------|-------+ > > + * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| > > + * +------------|------|--------|-------+ > > + * trbe_base trbe_base + nr_pages > > + * > > + * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both > > + * head and wakeup are within same PAGE size range. > > + * > > + * PAGE_SIZE > > + * |----------------------| > > + * > > + * limit head wakeup tail > > + * +----|------|-------|--------|-------+ > > + * |$$$$$$$$$$$|=======|========|$$$$$$$| > > + * +----|------|-------|--------|-------+ > > + * trbe_base trbe_base + nr_pages > > + */ > > + if (limit > head) > > + return limit; > > + > > + trbe_pad_buf(handle, handle->size); > > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > > + return 0; > > +} > > + > > +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) > > +{ > > + struct trbe_buf *buf = perf_get_aux(handle); > > + u64 limit = __trbe_normal_offset(handle); > > + u64 head = PERF_IDX2OFF(handle->head, buf); > > + > > + /* > > + * If the head is too close to the limit and we don't > > + * have space for a meaningful run, we rather pad it > > + * and start fresh. > > + */ > > + if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { > > + trbe_pad_buf(handle, limit - head); > > + limit = __trbe_normal_offset(handle); > > + } > > + return limit; > > +} > > + > > +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) > > +{ > > + struct trbe_buf *buf = etm_perf_sink_config(handle); > > + unsigned long offset; > > + > > + if (buf->snapshot) > > + offset = trbe_snapshot_offset(handle); > > + else > > + offset = trbe_normal_offset(handle); > > + return buf->trbe_base + offset; > > +} > > + > > +static void clr_trbe_status(void) > > +{ > > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > > + > > + WARN_ON(is_trbe_enabled()); > > + trbsr &= ~TRBSR_IRQ; > > + trbsr &= ~TRBSR_TRG; > > + trbsr &= ~TRBSR_WRAP; > > + trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); > > + trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); > > + trbsr &= ~TRBSR_STOP; > > + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > > +} > > + > > +static void set_trbe_limit_pointer_enabled(unsigned long addr) > > +{ > > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > > + > > + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > > + > > + trblimitr &= ~TRBLIMITR_NVM; > > + trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); > > + trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); > > + trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); > > + > > + /* > > + * Fill trace buffer mode is used here while configuring the > > + * TRBE for trace capture. In this particular mode, the trace > > + * collection is stopped and a maintenance interrupt is raised > > + * when the current write pointer wraps. This pause in trace > > + * collection gives the software an opportunity to capture the > > + * trace data in the interrupt handler, before reconfiguring > > + * the TRBE. > > + */ > > + trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; > > + > > + /* > > + * Trigger mode is not used here while configuring the TRBE for > > + * the trace capture. Hence just keep this in the ignore mode. > > + */ > > + trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; > > + trblimitr |= (addr & PAGE_MASK); > > + > > + trblimitr |= TRBLIMITR_ENABLE; > > + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > > +} > > + > > +static void trbe_enable_hw(struct trbe_buf *buf) > > +{ > > + WARN_ON(buf->trbe_write < buf->trbe_base); > > + WARN_ON(buf->trbe_write >= buf->trbe_limit); > > + set_trbe_disabled(); > > + isb(); > > + clr_trbe_status(); > > + set_trbe_base_pointer(buf->trbe_base); > > + set_trbe_write_pointer(buf->trbe_write); > > + > > + /* > > + * Synchronize all the register updates > > + * till now before enabling the TRBE. > > + */ > > + isb(); > > + set_trbe_limit_pointer_enabled(buf->trbe_limit); > > + > > + /* Synchronize the TRBE enable event */ > > + isb(); > > +} > > + > > +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, > > + struct perf_event *event, void **pages, > > + int nr_pages, bool snapshot) > > +{ > > + struct trbe_buf *buf; > > + struct page **pglist; > > + int i; > > + > > + /* > > + * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with > > + * just a single page, there is not much room left while writing into > > + * a partially filled TRBE buffer. Hence restrict the minimum buffer > > + * size as two pages. > > + */ > > + if (nr_pages < 2) > > + return NULL; > > + > > + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); > > + if (IS_ERR(buf)) > > + return ERR_PTR(-ENOMEM); > > + > > + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); > > + if (IS_ERR(pglist)) { > > + kfree(buf); > > + return ERR_PTR(-ENOMEM); > > + } > > + > > + for (i = 0; i < nr_pages; i++) > > + pglist[i] = virt_to_page(pages[i]); > > + > > + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); > > + if (IS_ERR((void *)buf->trbe_base)) { > > + kfree(pglist); > > + kfree(buf); > > + return ERR_PTR(buf->trbe_base); > > + } > > + buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; > > + buf->trbe_write = buf->trbe_base; > > + buf->snapshot = snapshot; > > + buf->nr_pages = nr_pages; > > + buf->pages = pages; > > + kfree(pglist); > > + return buf; > > +} > > + > > +void arm_trbe_free_buffer(void *config) > > +{ > > + struct trbe_buf *buf = config; > > + > > + vunmap((void *)buf->trbe_base); > > + kfree(buf); > > +} > > + > > +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, > > + struct perf_output_handle *handle, > > + void *config) > > +{ > > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > > + struct trbe_buf *buf = config; > > + unsigned long size, offset; > > + > > + WARN_ON(buf->cpudata != cpudata); > > + WARN_ON(cpudata->cpu != smp_processor_id()); > > + WARN_ON(cpudata->drvdata != drvdata); > > + if (cpudata->mode != CS_MODE_PERF) > > + return -EINVAL; > > + > > + /* > > + * If the TRBE was disabled due to lack of space in the AUX buffer or a > > + * spurious fault, the driver leaves it disabled, truncating the buffer. > > + * Since the etm_perf driver expects to close out the AUX buffer, the > > + * driver skips it. Thus, just pass in 0 size here to indicate that the > > + * buffer was truncated. > > + */ > > + if (!is_trbe_enabled()) > > + return 0; > > + /* > > + * perf handle structure needs to be shared with the TRBE IRQ handler for > > + * capturing trace data and restarting the handle. There is a probability > > + * of an undefined reference based crash when etm event is being stopped > > + * while a TRBE IRQ also getting processed. This happens due the release > > + * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping > > + * the TRBE here will ensure that no IRQ could be generated when the perf > > + * handle gets freed in etm_event_stop(). > > + */ > > + trbe_drain_and_disable_local(); > > + offset = get_trbe_write_pointer() - get_trbe_base_pointer(); > > + size = offset - PERF_IDX2OFF(handle->head, buf); > > + if (buf->snapshot) > > + handle->head += size; > > + return size; > > +} > > + > > +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) > > +{ > > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > > + struct perf_output_handle *handle = data; > > + struct trbe_buf *buf = etm_perf_sink_config(handle); > > + > > + WARN_ON(cpudata->cpu != smp_processor_id()); > > + WARN_ON(cpudata->drvdata != drvdata); > > + if (mode != CS_MODE_PERF) > > + return -EINVAL; > > + > > + *this_cpu_ptr(drvdata->handle) = handle; > > + cpudata->buf = buf; > > + cpudata->mode = mode; > > + buf->cpudata = cpudata; > > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > > + if (buf->trbe_limit == buf->trbe_base) { > > + trbe_stop_and_truncate_event(handle); > > + return 0; > > + } > > + trbe_enable_hw(buf); > > + return 0; > > +} > > + > > +static int arm_trbe_disable(struct coresight_device *csdev) > > +{ > > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > > + struct trbe_buf *buf = cpudata->buf; > > + > > + WARN_ON(buf->cpudata != cpudata); > > + WARN_ON(cpudata->cpu != smp_processor_id()); > > + WARN_ON(cpudata->drvdata != drvdata); > > + if (cpudata->mode != CS_MODE_PERF) > > + return -EINVAL; > > + > > + trbe_drain_and_disable_local(); > > + buf->cpudata = NULL; > > + cpudata->buf = NULL; > > + cpudata->mode = CS_MODE_DISABLED; > > + return 0; > > +} > > + > > +static void trbe_handle_spurious(struct perf_output_handle *handle) > > +{ > > + struct trbe_buf *buf = etm_perf_sink_config(handle); > > + > > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > > + if (buf->trbe_limit == buf->trbe_base) { > > + trbe_drain_and_disable_local(); > > + return; > > + } > > + trbe_enable_hw(buf); > > +} > > + > > +static void trbe_handle_overflow(struct perf_output_handle *handle) > > +{ > > + struct perf_event *event = handle->event; > > + struct trbe_buf *buf = etm_perf_sink_config(handle); > > + unsigned long offset, size; > > + struct etm_event_data *event_data; > > + > > + offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); > > + size = offset - PERF_IDX2OFF(handle->head, buf); > > + if (buf->snapshot) > > + handle->head = offset; > > + perf_aux_output_end(handle, size); > > + > > + event_data = perf_aux_output_begin(handle, event); > > + if (!event_data) { > > + trbe_drain_and_disable_local(); > > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > > + return; > > + } > > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > > + if (buf->trbe_limit == buf->trbe_base) { > > + trbe_stop_and_truncate_event(handle); > > + return; > > + } > > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; > > + trbe_enable_hw(buf); > > +} > > + > > +static bool is_perf_trbe(struct perf_output_handle *handle) > > +{ > > + struct trbe_buf *buf = etm_perf_sink_config(handle); > > + struct trbe_cpudata *cpudata = buf->cpudata; > > + struct trbe_drvdata *drvdata = cpudata->drvdata; > > + int cpu = smp_processor_id(); > > + > > + WARN_ON(buf->trbe_base != get_trbe_base_pointer()); > > + WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); > > + > > + if (cpudata->mode != CS_MODE_PERF) > > + return false; > > + > > + if (cpudata->cpu != cpu) > > + return false; > > + > > + if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) > > + return false; > > + > > + return true; > > +} > > + > > +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) > > +{ > > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > > + int ec = get_trbe_ec(trbsr); > > + int bsc = get_trbe_bsc(trbsr); > > + > > + WARN_ON(is_trbe_running(trbsr)); > > + if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) > > + return TRBE_FAULT_ACT_FATAL; > > + > > + if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) > > + return TRBE_FAULT_ACT_FATAL; > > + > > + if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { > > + if (get_trbe_write_pointer() == get_trbe_base_pointer()) > > + return TRBE_FAULT_ACT_WRAP; > > + } > > + return TRBE_FAULT_ACT_SPURIOUS; > > +} > > + > > +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) > > +{ > > + struct perf_output_handle **handle_ptr = dev; > > + struct perf_output_handle *handle = *handle_ptr; > > + enum trbe_fault_action act; > > + > > + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); > > + clr_trbe_irq(); > > + > > + /* > > + * Ensure the trace is visible to the CPUs and > > + * any external aborts have been resolved. > > + */ > > + trbe_drain_buffer(); > > + isb(); > > + > > + if (!perf_get_aux(handle)) > > + return IRQ_NONE; > > + > > + if (!is_perf_trbe(handle)) > > + return IRQ_NONE; > > + > > + irq_work_run(); There is a comment in the SPE driver about this. Since this driver closely follows that implementation it would be nice to have the comments as well. Otherwise the reader has to constantly go back to the original driver. I will come back to this function later. > > + > > + act = trbe_get_fault_act(handle); > > + switch (act) { > > + case TRBE_FAULT_ACT_WRAP: > > + trbe_handle_overflow(handle); > > + break; > > + case TRBE_FAULT_ACT_SPURIOUS: > > + trbe_handle_spurious(handle); > > + break; > > + case TRBE_FAULT_ACT_FATAL: > > + trbe_stop_and_truncate_event(handle); > > + break; > > + } > > + return IRQ_HANDLED; > > +} > > + > > +static const struct coresight_ops_sink arm_trbe_sink_ops = { > > + .enable = arm_trbe_enable, > > + .disable = arm_trbe_disable, > > + .alloc_buffer = arm_trbe_alloc_buffer, > > + .free_buffer = arm_trbe_free_buffer, > > + .update_buffer = arm_trbe_update_buffer, > > +}; > > + > > +static const struct coresight_ops arm_trbe_cs_ops = { > > + .sink_ops = &arm_trbe_sink_ops, > > +}; > > + > > +static ssize_t align_show(struct device *dev, struct device_attribute *attr, char *buf) > > +{ > > + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > > + > > + return sprintf(buf, "%llx\n", cpudata->trbe_align); > > +} > > +static DEVICE_ATTR_RO(align); > > + > > +static ssize_t dbm_show(struct device *dev, struct device_attribute *attr, char *buf) > > +{ > > + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > > + > > + return sprintf(buf, "%d\n", cpudata->trbe_dbm); > > +} > > +static DEVICE_ATTR_RO(dbm); > > + > > +static struct attribute *arm_trbe_attrs[] = { > > + &dev_attr_align.attr, > > + &dev_attr_dbm.attr, > > + NULL, > > +}; > > + > > +static const struct attribute_group arm_trbe_group = { > > + .attrs = arm_trbe_attrs, > > +}; > > + > > +static const struct attribute_group *arm_trbe_groups[] = { > > + &arm_trbe_group, > > + NULL, > > +}; > > + > > +static void arm_trbe_probe_coresight_cpu(void *info) > > +{ > > + struct trbe_drvdata *drvdata = info; > > + struct coresight_desc desc = { 0 }; > > + int cpu = smp_processor_id(); > > + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > > + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > > + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); > > + struct device *dev; > > + > > + if (WARN_ON(!cpudata)) > > + goto cpu_clear; Where was the memory for cpudata allocated? As far as I can tell, at this time it is just a pointer that was not allocated and as such it should be NULL. > > + > > + if (trbe_csdev) > > + return; > > + > > + cpudata->cpu = smp_processor_id(); Why call this again when you already did above? And how is arm_trbe_probe_coresight_cpu() is called for every CPU in the system? > > + cpudata->drvdata = drvdata; > > + dev = &cpudata->drvdata->pdev->dev; > > + > > + if (!is_trbe_available()) { > > + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); > > + goto cpu_clear; > > + } > > + > > + if (!is_trbe_programmable(trbidr)) { > > + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); > > + goto cpu_clear; > > + } > > + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); > > + if (IS_ERR(desc.name)) > > + goto cpu_clear; > > + > > + desc.type = CORESIGHT_DEV_TYPE_SINK; > > + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; > > + desc.ops = &arm_trbe_cs_ops; > > + desc.pdata = dev_get_platdata(dev); > > + desc.groups = arm_trbe_groups; > > + desc.dev = dev; > > + trbe_csdev = coresight_register(&desc); > > + if (IS_ERR(trbe_csdev)) > > + goto cpu_clear; > > + > > + dev_set_drvdata(&trbe_csdev->dev, cpudata); > > + cpudata->trbe_dbm = get_trbe_flag_update(trbidr); > > + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); > > + if (cpudata->trbe_align > SZ_2K) { > > + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); > > + goto cpu_clear; > > + } > > + per_cpu(csdev_sink, cpu) = trbe_csdev; > > + trbe_reset_local(); > > + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > > + return; > > +cpu_clear: > > + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); > > +} > > + > > +static void arm_trbe_remove_coresight_cpu(void *info) > > +{ > > + int cpu = smp_processor_id(); > > + struct trbe_drvdata *drvdata = info; > > + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > > + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > > + > > + if (trbe_csdev) { > > + coresight_unregister(trbe_csdev); > > + cpudata->drvdata = NULL; > > + per_cpu(csdev_sink, cpu) = NULL; > > + } > > + disable_percpu_irq(drvdata->irq); > > + trbe_reset_local(); > > +} > > + > > +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) > > +{ > > + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); > > + if (IS_ERR(drvdata->cpudata)) > > + return PTR_ERR(drvdata->cpudata); > > + > > + arm_trbe_probe_coresight_cpu(drvdata); > > + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); > > + return 0; > > +} > > + > > +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) > > +{ > > + arm_trbe_remove_coresight_cpu(drvdata); > > + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); > > + free_percpu(drvdata->cpudata); > > + return 0; > > +} > > + > > +static int arm_trbe_cpu_startup(unsigned int cpu, struct hlist_node *node) > > +{ > > + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > > + > > + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > > + if (!per_cpu(csdev_sink, cpu)) { > > + arm_trbe_probe_coresight_cpu(drvdata); > > + } else { > > + trbe_reset_local(); > > + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > > + } > > + } > > + return 0; > > +} > > + > > +static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node) > > +{ > > + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > > + > > + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > > + disable_percpu_irq(drvdata->irq); > > + trbe_reset_local(); > > + } > > + return 0; > > +} > > + > > +static int arm_trbe_probe_cpuhp(struct trbe_drvdata *drvdata) > > +{ > > + enum cpuhp_state trbe_online; > > + > > + trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, > > + arm_trbe_cpu_startup, arm_trbe_cpu_teardown); > > + if (trbe_online < 0) > > + return -EINVAL; > > + > > + if (cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node)) > > + return -EINVAL; > > + > > + drvdata->trbe_online = trbe_online; > > + return 0; > > +} > > + > > +static void arm_trbe_remove_cpuhp(struct trbe_drvdata *drvdata) > > +{ > > + cpuhp_remove_multi_state(drvdata->trbe_online); > > +} > > + > > +static int arm_trbe_probe_irq(struct platform_device *pdev, > > + struct trbe_drvdata *drvdata) > > +{ > > + drvdata->irq = platform_get_irq(pdev, 0); > > + if (!drvdata->irq) { Please use function platform_get_irq() properly - there is even an example on how to do so in the documentation section of the function. > > + pr_err("IRQ not found for the platform device\n"); > > + return -ENXIO; Why use a different error code? > > + } > > + > > + if (!irq_is_percpu(drvdata->irq)) { > > + pr_err("IRQ is not a PPI\n"); > > + return -EINVAL; > > + } > > + > > + if (irq_get_percpu_devid_partition(drvdata->irq, &drvdata->supported_cpus)) > > + return -EINVAL; > > + > > + drvdata->handle = alloc_percpu(typeof(*drvdata->handle)); > > + if (!drvdata->handle) > > + return -ENOMEM; > > + > > + if (request_percpu_irq(drvdata->irq, arm_trbe_irq_handler, DRVNAME, drvdata->handle)) { > > + free_percpu(drvdata->handle); > > + return -EINVAL; Here too you need to use the error code from the calling function rather than making your own. Please revise for the entire patch. > > + } > > + return 0; > > +} > > + > > +static void arm_trbe_remove_irq(struct trbe_drvdata *drvdata) > > +{ > > + free_percpu_irq(drvdata->irq, drvdata->handle); > > + free_percpu(drvdata->handle); > > +} > > + > > +static int arm_trbe_device_probe(struct platform_device *pdev) > > +{ > > + struct coresight_platform_data *pdata; > > + struct trbe_drvdata *drvdata; > > + struct device *dev = &pdev->dev; > > + int ret; > > + > > + drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL); > > + if (IS_ERR(drvdata)) > > + return -ENOMEM; if (!drvdata) > > + > > + pdata = coresight_get_platform_data(dev); > > + if (IS_ERR(pdata)) { > > + kfree(drvdata); No need to do this since devm_kzalloc() was used above. > > + return -ENOMEM; Why not using the error from coresight_get_platform_data() instead of masking it? > > + } > > + > > + dev_set_drvdata(dev, drvdata); > > + dev->platform_data = pdata; > > + drvdata->pdev = pdev; > > + ret = arm_trbe_probe_irq(pdev, drvdata); > > + if (ret) > > + goto irq_failed; > > + > > + ret = arm_trbe_probe_coresight(drvdata); > > + if (ret) > > + goto probe_failed; > > + > > + ret = arm_trbe_probe_cpuhp(drvdata); > > + if (ret) > > + goto cpuhp_failed; > > + > > + return 0; > > +cpuhp_failed: > > + arm_trbe_remove_coresight(drvdata); > > +probe_failed: > > + arm_trbe_remove_irq(drvdata); > > +irq_failed: > > + kfree(pdata); > > + kfree(drvdata); Same here - both @pdata and @drvdata have been allocated by devm_kzalloc(). devm_kzalloc(). > > + return ret; > > +} > > + > > +static int arm_trbe_device_remove(struct platform_device *pdev) > > +{ > > + struct coresight_platform_data *pdata = dev_get_platdata(&pdev->dev); > > + struct trbe_drvdata *drvdata = platform_get_drvdata(pdev); > > + > > + arm_trbe_remove_coresight(drvdata); > > + arm_trbe_remove_cpuhp(drvdata); > > + arm_trbe_remove_irq(drvdata); > > + kfree(pdata); > > + kfree(drvdata); > > + return 0; > > +} > > + > > +static const struct of_device_id arm_trbe_of_match[] = { > > + { .compatible = "arm,trace-buffer-extension"}, > > + {}, > > +}; > > +MODULE_DEVICE_TABLE(of, arm_trbe_of_match); > > + > > +static struct platform_driver arm_trbe_driver = { > > + .driver = { > > + .name = DRVNAME, > > + .of_match_table = of_match_ptr(arm_trbe_of_match), > > + .suppress_bind_attrs = true, > > + }, > > + .probe = arm_trbe_device_probe, > > + .remove = arm_trbe_device_remove, > > +}; > > + > > +static int __init arm_trbe_init(void) > > +{ > > + int ret; > > + > > + if (arm64_kernel_unmapped_at_el0()) { > > + pr_err("TRBE wouldn't work if kernel gets unmapped at EL0\n"); > > + return -EOPNOTSUPP; > > + } > > + > > + ret = platform_driver_register(&arm_trbe_driver); > > + if (!ret) > > + return 0; > > + > > + pr_err("Error registering %s platform driver\n", DRVNAME); > > + return ret; > > +} > > + > > +static void __exit arm_trbe_exit(void) > > +{ > > + platform_driver_unregister(&arm_trbe_driver); > > +} > > +module_init(arm_trbe_init); > > +module_exit(arm_trbe_exit); > > + > > +MODULE_AUTHOR("Anshuman Khandual <anshuman.khandual@arm.com>"); > > +MODULE_DESCRIPTION("Arm Trace Buffer Extension (TRBE) driver"); > > +MODULE_LICENSE("GPL v2"); > > diff --git a/drivers/hwtracing/coresight/coresight-trbe.h b/drivers/hwtracing/coresight/coresight-trbe.h > > new file mode 100644 > > index 0000000..43308bc > > --- /dev/null > > +++ b/drivers/hwtracing/coresight/coresight-trbe.h > > @@ -0,0 +1,160 @@ > > +/* SPDX-License-Identifier: GPL-2.0 */ > > +/* > > + * This contains all required hardware related helper functions for > > + * Trace Buffer Extension (TRBE) driver in the coresight framework. > > + * > > + * Copyright (C) 2020 ARM Ltd. > > + * > > + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > > + */ > > +#include <linux/coresight.h> > > +#include <linux/device.h> > > +#include <linux/irq.h> > > +#include <linux/kernel.h> > > +#include <linux/of.h> > > +#include <linux/platform_device.h> > > +#include <linux/smp.h> > > + > > +#include "coresight-etm-perf.h" > > + > > +DECLARE_PER_CPU(struct coresight_device *, csdev_sink); > > + > > +static inline bool is_trbe_available(void) > > +{ > > + u64 aa64dfr0 = read_sysreg_s(SYS_ID_AA64DFR0_EL1); > > + unsigned int trbe = cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_TRBE_SHIFT); > > + > > + return trbe >= 0b0001; > > +} > > + > > +static inline bool is_trbe_enabled(void) > > +{ > > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > > + > > + return trblimitr & TRBLIMITR_ENABLE; > > +} > > + > > +#define TRBE_EC_OTHERS 0 > > +#define TRBE_EC_STAGE1_ABORT 36 > > +#define TRBE_EC_STAGE2_ABORT 37 > > + > > +static inline int get_trbe_ec(u64 trbsr) > > +{ > > + return (trbsr >> TRBSR_EC_SHIFT) & TRBSR_EC_MASK; > > +} > > + > > +#define TRBE_BSC_NOT_STOPPED 0 > > +#define TRBE_BSC_FILLED 1 > > +#define TRBE_BSC_TRIGGERED 2 > > + > > +static inline int get_trbe_bsc(u64 trbsr) > > +{ > > + return (trbsr >> TRBSR_BSC_SHIFT) & TRBSR_BSC_MASK; > > +} > > + > > +static inline void clr_trbe_irq(void) > > +{ > > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > > + > > + trbsr &= ~TRBSR_IRQ; > > + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > > +} > > + > > +static inline bool is_trbe_irq(u64 trbsr) > > +{ > > + return trbsr & TRBSR_IRQ; > > +} > > + > > +static inline bool is_trbe_trg(u64 trbsr) > > +{ > > + return trbsr & TRBSR_TRG; > > +} > > + > > +static inline bool is_trbe_wrap(u64 trbsr) > > +{ > > + return trbsr & TRBSR_WRAP; > > +} > > + > > +static inline bool is_trbe_abort(u64 trbsr) > > +{ > > + return trbsr & TRBSR_ABORT; > > +} > > + > > +static inline bool is_trbe_running(u64 trbsr) > > +{ > > + return !(trbsr & TRBSR_STOP); > > +} > > + > > +#define TRBE_TRIG_MODE_STOP 0 > > +#define TRBE_TRIG_MODE_IRQ 1 > > +#define TRBE_TRIG_MODE_IGNORE 3 > > + > > +#define TRBE_FILL_MODE_FILL 0 > > +#define TRBE_FILL_MODE_WRAP 1 > > +#define TRBE_FILL_MODE_CIRCULAR_BUFFER 3 > > + > > +static inline void set_trbe_disabled(void) > > +{ > > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > > + > > + trblimitr &= ~TRBLIMITR_ENABLE; > > + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > > +} > > + > > +static inline bool get_trbe_flag_update(u64 trbidr) > > +{ > > + return trbidr & TRBIDR_FLAG; > > +} > > + > > +static inline bool is_trbe_programmable(u64 trbidr) > > +{ > > + return !(trbidr & TRBIDR_PROG); > > +} > > + > > +static inline int get_trbe_address_align(u64 trbidr) > > +{ > > + return (trbidr >> TRBIDR_ALIGN_SHIFT) & TRBIDR_ALIGN_MASK; > > +} > > + > > +static inline unsigned long get_trbe_write_pointer(void) > > +{ > > + u64 trbptr = read_sysreg_s(SYS_TRBPTR_EL1); > > + unsigned long addr = (trbptr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; > > + > > + return addr; > > +} > > + > > +static inline void set_trbe_write_pointer(unsigned long addr) > > +{ > > + WARN_ON(is_trbe_enabled()); > > + addr = (addr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; > > + write_sysreg_s(addr, SYS_TRBPTR_EL1); > > +} > > + > > +static inline unsigned long get_trbe_limit_pointer(void) > > +{ > > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > > + unsigned long limit = (trblimitr >> TRBLIMITR_LIMIT_SHIFT) & TRBLIMITR_LIMIT_MASK; > > + unsigned long addr = limit << TRBLIMITR_LIMIT_SHIFT; > > + > > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > > + return addr; > > +} > > + > > +static inline unsigned long get_trbe_base_pointer(void) > > +{ > > + u64 trbbaser = read_sysreg_s(SYS_TRBBASER_EL1); > > + unsigned long addr = (trbbaser >> TRBBASER_BASE_SHIFT) & TRBBASER_BASE_MASK; > > + > > + addr = addr << TRBBASER_BASE_SHIFT; > > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > > + return addr; > > +} > > + > > +static inline void set_trbe_base_pointer(unsigned long addr) > > +{ > > + WARN_ON(is_trbe_enabled()); > > + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > > + write_sysreg_s(addr, SYS_TRBBASER_EL1); > > +} > > -- > > 2.7.4 > >
On 2/9/21 11:09 PM, Mathieu Poirier wrote: > On Fri, Feb 05, 2021 at 10:53:30AM -0700, Mathieu Poirier wrote: >> On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: >>> Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is >>> accessible via the system registers. The TRBE supports different addressing >>> modes including CPU virtual address and buffer modes including the circular >>> buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), >>> an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the >>> access to the trace buffer could be prohibited by a higher exception level >>> (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU >>> private interrupt (PPI) on address translation errors and when the buffer >>> is full. Overall implementation here is inspired from the Arm SPE driver. >>> >> >> I got this message when applying the patch: >> >> Applying: coresight: sink: Add TRBE driver >> .git/rebase-apply/patch:76: new blank line at EOF. >> + >> warning: 1 line adds whitespace errors. >> >>> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> >>> Cc: Mike Leach <mike.leach@linaro.org> >>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> >>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> >>> --- >>> Changes in V3: >>> >>> - Added new DT bindings document TRBE.yaml >>> - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 >>> - Dropped isb() from trbe_reset_local() >>> - Dropped gap between (void *) and buf->trbe_base >>> - Changed 'int' to 'unsigned int' in is_trbe_available() >>> - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), >>> set_trbe_enabled() and set_trbe_limit_pointer() >>> - Changed get_trbe_flag_update(), is_trbe_programmable() and >>> get_trbe_address_align() to accept TRBIDR value >>> - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), >>> is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value >>> - Dropped snapshot mode condition in arm_trbe_alloc_buffer() >>> - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled >>> - Compute trbe_limit before trbe_write to get the updated handle >>> - Added trbe_stop_and_truncate_event() >>> - Dropped trbe_handle_fatal() >>> >>> Documentation/trace/coresight/coresight-trbe.rst | 39 + >>> arch/arm64/include/asm/sysreg.h | 1 + >>> drivers/hwtracing/coresight/Kconfig | 11 + >>> drivers/hwtracing/coresight/Makefile | 1 + >>> drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ >>> drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ >>> 6 files changed, 1235 insertions(+) >>> create mode 100644 Documentation/trace/coresight/coresight-trbe.rst >>> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c >>> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h >>> >>> diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst >>> new file mode 100644 >>> index 0000000..1cbb819 >>> --- /dev/null >>> +++ b/Documentation/trace/coresight/coresight-trbe.rst >>> @@ -0,0 +1,39 @@ >>> +.. SPDX-License-Identifier: GPL-2.0 >>> + >>> +============================== >>> +Trace Buffer Extension (TRBE). >>> +============================== >>> + >>> + :Author: Anshuman Khandual <anshuman.khandual@arm.com> >>> + :Date: November 2020 >>> + >>> +Hardware Description >>> +-------------------- >>> + >>> +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system >>> +memory, CPU traces generated from a corresponding percpu tracing unit. This >>> +gets plugged in as a coresight sink device because the corresponding trace >>> +genarators (ETE), are plugged in as source device. >>> + >>> +The TRBE is not compliant to CoreSight architecture specifications, but is >>> +driven via the CoreSight driver framework to support the ETE (which is >>> +CoreSight compliant) integration. >>> + >>> +Sysfs files and directories >>> +--------------------------- >>> + >>> +The TRBE devices appear on the existing coresight bus alongside the other >>> +coresight devices:: >>> + >>> + >$ ls /sys/bus/coresight/devices >>> + trbe0 trbe1 trbe2 trbe3 >>> + >>> +The ``trbe<N>`` named TRBEs are associated with a CPU.:: >>> + >>> + >$ ls /sys/bus/coresight/devices/trbe0/ >>> + align dbm >>> + >>> +*Key file items are:-* >>> + * ``align``: TRBE write pointer alignment >>> + * ``dbm``: TRBE updates memory with access and dirty flags >>> + >> >> Please add documentation for these, the same way it was done for all the other CS >> components [1]. >> >> [1]. https://elixir.bootlin.com/linux/latest/source/Documentation/ABI/testing >> (sysfs-bus-coresight-device-xyz) >> >>> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h >>> index 85ae4db..9e2e9b7 100644 >>> --- a/arch/arm64/include/asm/sysreg.h >>> +++ b/arch/arm64/include/asm/sysreg.h >>> @@ -97,6 +97,7 @@ >>> #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) >>> #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) >>> #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) >>> +#define TSB_CSYNC __emit_inst(0xd503225f) >>> >>> #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) >>> #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) >>> diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig >>> index f154ae7..aa657ab 100644 >>> --- a/drivers/hwtracing/coresight/Kconfig >>> +++ b/drivers/hwtracing/coresight/Kconfig >>> @@ -164,6 +164,17 @@ config CORESIGHT_CTI >>> To compile this driver as a module, choose M here: the >>> module will be called coresight-cti. >>> >>> +config CORESIGHT_TRBE >>> + bool "Trace Buffer Extension (TRBE) driver" >> >> Shouldn't that be "tristate"? >> >>> + depends on ARM64 >> >> I would make it dependent on ETMv4 as well since it can only be used by that >> component. >> >>> + help >>> + This driver provides support for percpu Trace Buffer Extension (TRBE). >>> + TRBE always needs to be used along with it's corresponding percpu ETE >>> + component. ETE generates trace data which is then captured with TRBE. >>> + Unlike traditional sink devices, TRBE is a CPU feature accessible via >>> + system registers. But it's explicit dependency with trace unit (ETE) >>> + requires it to be plugged in as a coresight sink device. >> >> Please add: >> >> "To compile this driver as a module, choose M here: the >> module will be called coresight-trbe." >> >> I'm out of time for today, I will continue on Monday. >> >> Mathieu >> >>> + >>> config CORESIGHT_CTI_INTEGRATION_REGS >>> bool "Access CTI CoreSight Integration Registers" >>> depends on CORESIGHT_CTI >>> diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile >>> index f20e357..d608165 100644 >>> --- a/drivers/hwtracing/coresight/Makefile >>> +++ b/drivers/hwtracing/coresight/Makefile >>> @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o >>> obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o >>> obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o >>> obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o >>> +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o >>> coresight-cti-y := coresight-cti-core.o coresight-cti-platform.o \ >>> coresight-cti-sysfs.o >>> diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c >>> new file mode 100644 >>> index 0000000..1464d8b >>> --- /dev/null >>> +++ b/drivers/hwtracing/coresight/coresight-trbe.c >>> @@ -0,0 +1,1023 @@ >>> +// SPDX-License-Identifier: GPL-2.0 >>> +/* >>> + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight >>> + * sink device could then pair with an appropriate per-cpu coresight source >>> + * device (ETE) thus generating required trace data. Trace can be enabled >>> + * via the perf framework. >>> + * >>> + * Copyright (C) 2020 ARM Ltd. >>> + * >>> + * Author: Anshuman Khandual <anshuman.khandual@arm.com> >>> + */ >>> +#define DRVNAME "arm_trbe" >>> + >>> +#define pr_fmt(fmt) DRVNAME ": " fmt >>> + >>> +#include "coresight-trbe.h" >>> + >>> +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) >>> + >>> +/* >>> + * A padding packet that will help the user space tools >>> + * in skipping relevant sections in the captured trace >>> + * data which could not be decoded. TRBE doesn't support >>> + * formatting the trace data, unlike the legacy CoreSight >>> + * sinks and thus we use ETE trace packets to pad the >>> + * sections of the buffer. >>> + */ >>> +#define ETE_IGNORE_PACKET 0x70 >>> + >>> +/* >>> + * Minimum amount of meaningful trace will contain: >>> + * A-Sync, Trace Info, Trace On, Address, Atom. >>> + * This is about 44bytes of ETE trace. To be on >>> + * the safer side, we assume 64bytes is the minimum >>> + * space required for a meaningful session, before >>> + * we hit a "WRAP" event. >>> + */ >>> +#define TRBE_TRACE_MIN_BUF_SIZE 64 >>> + >>> +enum trbe_fault_action { >>> + TRBE_FAULT_ACT_WRAP, >>> + TRBE_FAULT_ACT_SPURIOUS, >>> + TRBE_FAULT_ACT_FATAL, >>> +}; >>> + >>> +struct trbe_buf { >>> + unsigned long trbe_base; >>> + unsigned long trbe_limit; >>> + unsigned long trbe_write; >>> + int nr_pages; >>> + void **pages; >>> + bool snapshot; >>> + struct trbe_cpudata *cpudata; >>> +}; >>> + >>> +struct trbe_cpudata { >>> + bool trbe_dbm; >>> + u64 trbe_align; >>> + int cpu; >>> + enum cs_mode mode; >>> + struct trbe_buf *buf; >>> + struct trbe_drvdata *drvdata; >>> +}; >>> + >>> +struct trbe_drvdata { >>> + struct trbe_cpudata __percpu *cpudata; >>> + struct perf_output_handle __percpu **handle; >>> + struct hlist_node hotplug_node; >>> + int irq; >>> + cpumask_t supported_cpus; >>> + enum cpuhp_state trbe_online; >>> + struct platform_device *pdev; >>> +}; >>> + >>> +static int trbe_alloc_node(struct perf_event *event) >>> +{ >>> + if (event->cpu == -1) >>> + return NUMA_NO_NODE; >>> + return cpu_to_node(event->cpu); >>> +} >>> + >>> +static void trbe_drain_buffer(void) >>> +{ >>> + asm(TSB_CSYNC); >>> + dsb(nsh); >>> +} >>> + >>> +static void trbe_drain_and_disable_local(void) >>> +{ >>> + trbe_drain_buffer(); >>> + write_sysreg_s(0, SYS_TRBLIMITR_EL1); >>> + isb(); >>> +} >>> + >>> +static void trbe_reset_local(void) >>> +{ >>> + trbe_drain_and_disable_local(); >>> + write_sysreg_s(0, SYS_TRBPTR_EL1); >>> + write_sysreg_s(0, SYS_TRBBASER_EL1); >>> + write_sysreg_s(0, SYS_TRBSR_EL1); >>> +} >>> + >>> +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) >>> +{ >>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>> + >>> + /* >>> + * We cannot proceed with the buffer collection and we >>> + * do not have any data for the current session. The >>> + * etm_perf driver expects to close out the aux_buffer >>> + * at event_stop(). So disable the TRBE here and leave >>> + * the update_buffer() to return a 0 size. >>> + */ >>> + trbe_drain_and_disable_local(); >>> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >>> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; >>> +} >>> + >>> +/* >>> + * TRBE Buffer Management >>> + * >>> + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, >>> + * it starts writing trace data from the write pointer onward till the limit pointer. >>> + * When the write pointer reaches the address just before the limit pointer, it gets >>> + * wrapped around again to the base pointer. This is called a TRBE wrap event, which >>> + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver >>> + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ >>> + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and >>> + * LIMIT pointers. >>> + * >>> + * Wrap around with an IRQ >>> + * ------ < ------ < ------- < ----- < ----- >>> + * | | >>> + * ------ > ------ > ------- > ----- > ----- >>> + * >>> + * +---------------+-----------------------+ >>> + * | | | >>> + * +---------------+-----------------------+ >>> + * Base Pointer Write Pointer Limit Pointer >>> + * >>> + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write >>> + * pointer can be aligned to the implementation defined TRBE trace buffer alignment >>> + * as captured in trbe_cpudata->trbe_align. >>> + * >>> + * >>> + * head tail wakeup >>> + * +---------------------------------------+----- ~ ~ ------ >>> + * |$$$$$$$|################|$$$$$$$$$$$$$$| | >>> + * +---------------------------------------+----- ~ ~ ------ >>> + * Base Pointer Write Pointer Limit Pointer >>> + * >>> + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing >>> + * values which tracks all the driver writes and user reads from the perf auxiliary >>> + * buffer. Generally [head..tail] is the area where the driver can write into unless >>> + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and >>> + * configured depending on the perf_output_handle indices, so that the driver does >>> + * not override into areas in the perf auxiliary buffer which is being or yet to be >>> + * consumed from the user space. The enabled TRBE buffer area is a moving subset of >>> + * the allocated perf auxiliary buffer. >>> + */ >>> +static void trbe_pad_buf(struct perf_output_handle *handle, int len) >>> +{ >>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>> + u64 head = PERF_IDX2OFF(handle->head, buf); >>> + >>> + memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); >>> + if (!buf->snapshot) >>> + perf_aux_output_skip(handle, len); >>> +} >>> + >>> +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) >>> +{ >>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>> + >>> + /* >>> + * The ETE trace has alignment synchronization packets allowing >>> + * the decoder to reset in case of an overflow or corruption. >>> + * So we can use the entire buffer for the snapshot mode. >>> + */ >>> + return buf->nr_pages * PAGE_SIZE; >>> +} >>> + >>> +/* >>> + * TRBE Limit Calculation >>> + * >>> + * The following markers are used to illustrate various TRBE buffer situations. >>> + * >>> + * $$$$ - Data area, unconsumed captured trace data, not to be overridden >>> + * #### - Free area, enabled, trace will be written >>> + * %%%% - Free area, disabled, trace will not be written >>> + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped >>> + */ >>> +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) >>> +{ >>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>> + struct trbe_cpudata *cpudata = buf->cpudata; >>> + const u64 bufsize = buf->nr_pages * PAGE_SIZE; >>> + u64 limit = bufsize; >>> + u64 head, tail, wakeup; >>> + >>> + head = PERF_IDX2OFF(handle->head, buf); >>> + >>> + /* >>> + * head >>> + * ------->| >>> + * | >>> + * head TRBE align tail >>> + * +----|-------|---------------|-------+ >>> + * |$$$$|=======|###############|$$$$$$$| >>> + * +----|-------|---------------|-------+ >>> + * trbe_base trbe_base + nr_pages >>> + * >>> + * Perf aux buffer output head position can be misaligned depending on >>> + * various factors including user space reads. In case misaligned, head >>> + * needs to be aligned before TRBE can be configured. Pad the alignment >>> + * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools >>> + * and skip this section thus advancing the head. >>> + */ >>> + if (!IS_ALIGNED(head, cpudata->trbe_align)) { >>> + unsigned long delta = roundup(head, cpudata->trbe_align) - head; >>> + >>> + delta = min(delta, handle->size); >>> + trbe_pad_buf(handle, delta); >>> + head = PERF_IDX2OFF(handle->head, buf); >>> + } >>> + >>> + /* >>> + * head = tail (size = 0) >>> + * +----|-------------------------------+ >>> + * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ | >>> + * +----|-------------------------------+ >>> + * trbe_base trbe_base + nr_pages >>> + * >>> + * Perf aux buffer does not have any space for the driver to write into. >>> + * Just communicate trace truncation event to the user space by marking >>> + * it with PERF_AUX_FLAG_TRUNCATED. >>> + */ >>> + if (!handle->size) { >>> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >>> + return 0; >>> + } >>> + >>> + /* Compute the tail and wakeup indices now that we've aligned head */ >>> + tail = PERF_IDX2OFF(handle->head + handle->size, buf); >>> + wakeup = PERF_IDX2OFF(handle->wakeup, buf); >>> + >>> + /* >>> + * Lets calculate the buffer area which TRBE could write into. There >>> + * are three possible scenarios here. Limit needs to be aligned with >>> + * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the >>> + * unconsumed data. >>> + * >>> + * 1) head < tail >>> + * >>> + * head tail >>> + * +----|-----------------------|-------+ >>> + * |$$$$|#######################|$$$$$$$| >>> + * +----|-----------------------|-------+ >>> + * trbe_base limit trbe_base + nr_pages >>> + * >>> + * TRBE could write into [head..tail] area. Unless the tail is right at >>> + * the end of the buffer, neither an wrap around nor an IRQ is expected >>> + * while being enabled. >>> + * >>> + * 2) head == tail >>> + * >>> + * head = tail (size > 0) >>> + * +----|-------------------------------+ >>> + * |%%%%|###############################| >>> + * +----|-------------------------------+ >>> + * trbe_base limit = trbe_base + nr_pages >>> + * >>> + * TRBE should just write into [head..base + nr_pages] area even though >>> + * the entire buffer is empty. Reason being, when the trace reaches the >>> + * end of the buffer, it will just wrap around with an IRQ giving an >>> + * opportunity to reconfigure the buffer. >>> + * >>> + * 3) tail < head >>> + * >>> + * tail head >>> + * +----|-----------------------|-------+ >>> + * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| >>> + * +----|-----------------------|-------+ >>> + * trbe_base limit = trbe_base + nr_pages >>> + * >>> + * TRBE should just write into [head..base + nr_pages] area even though >>> + * the [trbe_base..tail] is also empty. Reason being, when the trace >>> + * reaches the end of the buffer, it will just wrap around with an IRQ >>> + * giving an opportunity to reconfigure the buffer. >>> + */ >>> + if (head < tail) >>> + limit = round_down(tail, PAGE_SIZE); >>> + >>> + /* >>> + * Wakeup may be arbitrarily far into the future. If it's not in the >>> + * current generation, either we'll wrap before hitting it, or it's >>> + * in the past and has been handled already. >>> + * >>> + * If there's a wakeup before we wrap, arrange to be woken up by the >>> + * page boundary following it. Keep the tail boundary if that's lower. >>> + * >>> + * head wakeup tail >>> + * +----|---------------|-------|-------+ >>> + * |$$$$|###############|%%%%%%%|$$$$$$$| >>> + * +----|---------------|-------|-------+ >>> + * trbe_base limit trbe_base + nr_pages >>> + */ >>> + if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) >>> + limit = min(limit, round_up(wakeup, PAGE_SIZE)); >>> + >>> + /* >>> + * There are two situation when this can happen i.e limit is before >>> + * the head and hence TRBE cannot be configured. >>> + * >>> + * 1) head < tail (aligned down with PAGE_SIZE) and also they are both >>> + * within the same PAGE size range. >>> + * >>> + * PAGE_SIZE >>> + * |----------------------| >>> + * >>> + * limit head tail >>> + * +------------|------|--------|-------+ >>> + * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| >>> + * +------------|------|--------|-------+ >>> + * trbe_base trbe_base + nr_pages >>> + * >>> + * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both >>> + * head and wakeup are within same PAGE size range. >>> + * >>> + * PAGE_SIZE >>> + * |----------------------| >>> + * >>> + * limit head wakeup tail >>> + * +----|------|-------|--------|-------+ >>> + * |$$$$$$$$$$$|=======|========|$$$$$$$| >>> + * +----|------|-------|--------|-------+ >>> + * trbe_base trbe_base + nr_pages >>> + */ >>> + if (limit > head) >>> + return limit; >>> + >>> + trbe_pad_buf(handle, handle->size); >>> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >>> + return 0; >>> +} >>> + >>> +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) >>> +{ >>> + struct trbe_buf *buf = perf_get_aux(handle); >>> + u64 limit = __trbe_normal_offset(handle); >>> + u64 head = PERF_IDX2OFF(handle->head, buf); >>> + >>> + /* >>> + * If the head is too close to the limit and we don't >>> + * have space for a meaningful run, we rather pad it >>> + * and start fresh. >>> + */ >>> + if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { >>> + trbe_pad_buf(handle, limit - head); >>> + limit = __trbe_normal_offset(handle); >>> + } >>> + return limit; >>> +} >>> + >>> +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) >>> +{ >>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>> + unsigned long offset; >>> + >>> + if (buf->snapshot) >>> + offset = trbe_snapshot_offset(handle); >>> + else >>> + offset = trbe_normal_offset(handle); >>> + return buf->trbe_base + offset; >>> +} >>> + >>> +static void clr_trbe_status(void) >>> +{ >>> + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); >>> + >>> + WARN_ON(is_trbe_enabled()); >>> + trbsr &= ~TRBSR_IRQ; >>> + trbsr &= ~TRBSR_TRG; >>> + trbsr &= ~TRBSR_WRAP; >>> + trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); >>> + trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); >>> + trbsr &= ~TRBSR_STOP; >>> + write_sysreg_s(trbsr, SYS_TRBSR_EL1); >>> +} >>> + >>> +static void set_trbe_limit_pointer_enabled(unsigned long addr) >>> +{ >>> + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); >>> + >>> + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); >>> + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); >>> + >>> + trblimitr &= ~TRBLIMITR_NVM; >>> + trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); >>> + trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); >>> + trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); >>> + >>> + /* >>> + * Fill trace buffer mode is used here while configuring the >>> + * TRBE for trace capture. In this particular mode, the trace >>> + * collection is stopped and a maintenance interrupt is raised >>> + * when the current write pointer wraps. This pause in trace >>> + * collection gives the software an opportunity to capture the >>> + * trace data in the interrupt handler, before reconfiguring >>> + * the TRBE. >>> + */ >>> + trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; >>> + >>> + /* >>> + * Trigger mode is not used here while configuring the TRBE for >>> + * the trace capture. Hence just keep this in the ignore mode. >>> + */ >>> + trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; >>> + trblimitr |= (addr & PAGE_MASK); >>> + >>> + trblimitr |= TRBLIMITR_ENABLE; >>> + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); >>> +} >>> + >>> +static void trbe_enable_hw(struct trbe_buf *buf) >>> +{ >>> + WARN_ON(buf->trbe_write < buf->trbe_base); >>> + WARN_ON(buf->trbe_write >= buf->trbe_limit); >>> + set_trbe_disabled(); >>> + isb(); >>> + clr_trbe_status(); >>> + set_trbe_base_pointer(buf->trbe_base); >>> + set_trbe_write_pointer(buf->trbe_write); >>> + >>> + /* >>> + * Synchronize all the register updates >>> + * till now before enabling the TRBE. >>> + */ >>> + isb(); >>> + set_trbe_limit_pointer_enabled(buf->trbe_limit); >>> + >>> + /* Synchronize the TRBE enable event */ >>> + isb(); >>> +} >>> + >>> +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, >>> + struct perf_event *event, void **pages, >>> + int nr_pages, bool snapshot) >>> +{ >>> + struct trbe_buf *buf; >>> + struct page **pglist; >>> + int i; >>> + >>> + /* >>> + * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with >>> + * just a single page, there is not much room left while writing into >>> + * a partially filled TRBE buffer. Hence restrict the minimum buffer >>> + * size as two pages. >>> + */ >>> + if (nr_pages < 2) >>> + return NULL; >>> + >>> + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); >>> + if (IS_ERR(buf)) >>> + return ERR_PTR(-ENOMEM); >>> + >>> + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); >>> + if (IS_ERR(pglist)) { >>> + kfree(buf); >>> + return ERR_PTR(-ENOMEM); >>> + } >>> + >>> + for (i = 0; i < nr_pages; i++) >>> + pglist[i] = virt_to_page(pages[i]); >>> + >>> + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); >>> + if (IS_ERR((void *)buf->trbe_base)) { >>> + kfree(pglist); >>> + kfree(buf); >>> + return ERR_PTR(buf->trbe_base); >>> + } >>> + buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; >>> + buf->trbe_write = buf->trbe_base; >>> + buf->snapshot = snapshot; >>> + buf->nr_pages = nr_pages; >>> + buf->pages = pages; >>> + kfree(pglist); >>> + return buf; >>> +} >>> + >>> +void arm_trbe_free_buffer(void *config) >>> +{ >>> + struct trbe_buf *buf = config; >>> + >>> + vunmap((void *)buf->trbe_base); >>> + kfree(buf); >>> +} >>> + >>> +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, >>> + struct perf_output_handle *handle, >>> + void *config) >>> +{ >>> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >>> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >>> + struct trbe_buf *buf = config; >>> + unsigned long size, offset; >>> + >>> + WARN_ON(buf->cpudata != cpudata); >>> + WARN_ON(cpudata->cpu != smp_processor_id()); >>> + WARN_ON(cpudata->drvdata != drvdata); >>> + if (cpudata->mode != CS_MODE_PERF) >>> + return -EINVAL; >>> + >>> + /* >>> + * If the TRBE was disabled due to lack of space in the AUX buffer or a >>> + * spurious fault, the driver leaves it disabled, truncating the buffer. >>> + * Since the etm_perf driver expects to close out the AUX buffer, the >>> + * driver skips it. Thus, just pass in 0 size here to indicate that the >>> + * buffer was truncated. >>> + */ >>> + if (!is_trbe_enabled()) >>> + return 0; >>> + /* >>> + * perf handle structure needs to be shared with the TRBE IRQ handler for >>> + * capturing trace data and restarting the handle. There is a probability >>> + * of an undefined reference based crash when etm event is being stopped >>> + * while a TRBE IRQ also getting processed. This happens due the release >>> + * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping >>> + * the TRBE here will ensure that no IRQ could be generated when the perf >>> + * handle gets freed in etm_event_stop(). >>> + */ >>> + trbe_drain_and_disable_local(); >>> + offset = get_trbe_write_pointer() - get_trbe_base_pointer(); >>> + size = offset - PERF_IDX2OFF(handle->head, buf); >>> + if (buf->snapshot) >>> + handle->head += size; >>> + return size; >>> +} >>> + >>> +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) >>> +{ >>> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >>> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >>> + struct perf_output_handle *handle = data; >>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>> + >>> + WARN_ON(cpudata->cpu != smp_processor_id()); >>> + WARN_ON(cpudata->drvdata != drvdata); >>> + if (mode != CS_MODE_PERF) >>> + return -EINVAL; >>> + >>> + *this_cpu_ptr(drvdata->handle) = handle; >>> + cpudata->buf = buf; >>> + cpudata->mode = mode; >>> + buf->cpudata = cpudata; >>> + buf->trbe_limit = compute_trbe_buffer_limit(handle); >>> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >>> + if (buf->trbe_limit == buf->trbe_base) { >>> + trbe_stop_and_truncate_event(handle); >>> + return 0; >>> + } >>> + trbe_enable_hw(buf); >>> + return 0; >>> +} >>> + >>> +static int arm_trbe_disable(struct coresight_device *csdev) >>> +{ >>> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >>> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >>> + struct trbe_buf *buf = cpudata->buf; >>> + >>> + WARN_ON(buf->cpudata != cpudata); >>> + WARN_ON(cpudata->cpu != smp_processor_id()); >>> + WARN_ON(cpudata->drvdata != drvdata); >>> + if (cpudata->mode != CS_MODE_PERF) >>> + return -EINVAL; >>> + >>> + trbe_drain_and_disable_local(); >>> + buf->cpudata = NULL; >>> + cpudata->buf = NULL; >>> + cpudata->mode = CS_MODE_DISABLED; >>> + return 0; >>> +} >>> + >>> +static void trbe_handle_spurious(struct perf_output_handle *handle) >>> +{ >>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>> + >>> + buf->trbe_limit = compute_trbe_buffer_limit(handle); >>> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >>> + if (buf->trbe_limit == buf->trbe_base) { >>> + trbe_drain_and_disable_local(); >>> + return; >>> + } >>> + trbe_enable_hw(buf); >>> +} >>> + >>> +static void trbe_handle_overflow(struct perf_output_handle *handle) >>> +{ >>> + struct perf_event *event = handle->event; >>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>> + unsigned long offset, size; >>> + struct etm_event_data *event_data; >>> + >>> + offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); >>> + size = offset - PERF_IDX2OFF(handle->head, buf); >>> + if (buf->snapshot) >>> + handle->head = offset; >>> + perf_aux_output_end(handle, size); >>> + >>> + event_data = perf_aux_output_begin(handle, event); >>> + if (!event_data) { >>> + trbe_drain_and_disable_local(); >>> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; >>> + return; >>> + } >>> + buf->trbe_limit = compute_trbe_buffer_limit(handle); >>> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >>> + if (buf->trbe_limit == buf->trbe_base) { >>> + trbe_stop_and_truncate_event(handle); >>> + return; >>> + } >>> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; >>> + trbe_enable_hw(buf); >>> +} >>> + >>> +static bool is_perf_trbe(struct perf_output_handle *handle) >>> +{ >>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>> + struct trbe_cpudata *cpudata = buf->cpudata; >>> + struct trbe_drvdata *drvdata = cpudata->drvdata; >>> + int cpu = smp_processor_id(); >>> + >>> + WARN_ON(buf->trbe_base != get_trbe_base_pointer()); >>> + WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); >>> + >>> + if (cpudata->mode != CS_MODE_PERF) >>> + return false; >>> + >>> + if (cpudata->cpu != cpu) >>> + return false; >>> + >>> + if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) >>> + return false; >>> + >>> + return true; >>> +} >>> + >>> +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) >>> +{ >>> + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); >>> + int ec = get_trbe_ec(trbsr); >>> + int bsc = get_trbe_bsc(trbsr); >>> + >>> + WARN_ON(is_trbe_running(trbsr)); >>> + if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) >>> + return TRBE_FAULT_ACT_FATAL; >>> + >>> + if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) >>> + return TRBE_FAULT_ACT_FATAL; >>> + >>> + if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { >>> + if (get_trbe_write_pointer() == get_trbe_base_pointer()) >>> + return TRBE_FAULT_ACT_WRAP; >>> + } >>> + return TRBE_FAULT_ACT_SPURIOUS; >>> +} >>> + >>> +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) >>> +{ >>> + struct perf_output_handle **handle_ptr = dev; >>> + struct perf_output_handle *handle = *handle_ptr; >>> + enum trbe_fault_action act; >>> + >>> + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); >>> + clr_trbe_irq(); >>> + >>> + /* >>> + * Ensure the trace is visible to the CPUs and >>> + * any external aborts have been resolved. >>> + */ >>> + trbe_drain_buffer(); >>> + isb(); >>> + >>> + if (!perf_get_aux(handle)) >>> + return IRQ_NONE; >>> + >>> + if (!is_perf_trbe(handle)) >>> + return IRQ_NONE; >>> + >>> + irq_work_run(); > > There is a comment in the SPE driver about this. Since this driver closely > follows that implementation it would be nice to have the comments as well. > Otherwise the reader has to constantly go back to the original driver. Sure, will add the following comment before irq_work_run(). /* * Ensure perf callbacks have completed, which may disable the * profiling buffer in response to a TRUNCATION flag. */ > > I will come back to this function later. Okay. > >>> + >>> + act = trbe_get_fault_act(handle); >>> + switch (act) { >>> + case TRBE_FAULT_ACT_WRAP: >>> + trbe_handle_overflow(handle); >>> + break; >>> + case TRBE_FAULT_ACT_SPURIOUS: >>> + trbe_handle_spurious(handle); >>> + break; >>> + case TRBE_FAULT_ACT_FATAL: >>> + trbe_stop_and_truncate_event(handle); >>> + break; >>> + } >>> + return IRQ_HANDLED; >>> +} >>> + >>> +static const struct coresight_ops_sink arm_trbe_sink_ops = { >>> + .enable = arm_trbe_enable, >>> + .disable = arm_trbe_disable, >>> + .alloc_buffer = arm_trbe_alloc_buffer, >>> + .free_buffer = arm_trbe_free_buffer, >>> + .update_buffer = arm_trbe_update_buffer, >>> +}; >>> + >>> +static const struct coresight_ops arm_trbe_cs_ops = { >>> + .sink_ops = &arm_trbe_sink_ops, >>> +}; >>> + >>> +static ssize_t align_show(struct device *dev, struct device_attribute *attr, char *buf) >>> +{ >>> + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); >>> + >>> + return sprintf(buf, "%llx\n", cpudata->trbe_align); >>> +} >>> +static DEVICE_ATTR_RO(align); >>> + >>> +static ssize_t dbm_show(struct device *dev, struct device_attribute *attr, char *buf) >>> +{ >>> + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); >>> + >>> + return sprintf(buf, "%d\n", cpudata->trbe_dbm); >>> +} >>> +static DEVICE_ATTR_RO(dbm); >>> + >>> +static struct attribute *arm_trbe_attrs[] = { >>> + &dev_attr_align.attr, >>> + &dev_attr_dbm.attr, >>> + NULL, >>> +}; >>> + >>> +static const struct attribute_group arm_trbe_group = { >>> + .attrs = arm_trbe_attrs, >>> +}; >>> + >>> +static const struct attribute_group *arm_trbe_groups[] = { >>> + &arm_trbe_group, >>> + NULL, >>> +}; >>> + >>> +static void arm_trbe_probe_coresight_cpu(void *info) >>> +{ >>> + struct trbe_drvdata *drvdata = info; >>> + struct coresight_desc desc = { 0 }; >>> + int cpu = smp_processor_id(); >>> + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); >>> + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); >>> + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); >>> + struct device *dev; >>> + >>> + if (WARN_ON(!cpudata)) >>> + goto cpu_clear; > > Where was the memory for cpudata allocated? As far as I can tell, at this time > it is just a pointer that was not allocated and as such it should be NULL. cpudata gets allocated in arm_trbe_probe_coresight() just before calling individual CPU based probes i.e arm_trbe_probe_coresight_cpu() directly and via smp_call_function_many(). arm_trbe_device_probe() arm_trbe_probe_coresight() arm_trbe_probe_coresight_cpu() > >>> + >>> + if (trbe_csdev) >>> + return; >>> + >>> + cpudata->cpu = smp_processor_id(); > > Why call this again when you already did above? And how is Right, this is redundant. Will just assign it as cpu which has already been computed. > arm_trbe_probe_coresight_cpu() is called for every CPU in the system? During boot in arm_trbe_probe_coresight(), it is called once directly on the executing cpu and on all other via smp_call_function_many(). > >>> + cpudata->drvdata = drvdata; >>> + dev = &cpudata->drvdata->pdev->dev; >>> + >>> + if (!is_trbe_available()) { >>> + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); >>> + goto cpu_clear; >>> + } >>> + >>> + if (!is_trbe_programmable(trbidr)) { >>> + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); >>> + goto cpu_clear; >>> + } >>> + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); >>> + if (IS_ERR(desc.name)) >>> + goto cpu_clear; >>> + >>> + desc.type = CORESIGHT_DEV_TYPE_SINK; >>> + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; >>> + desc.ops = &arm_trbe_cs_ops; >>> + desc.pdata = dev_get_platdata(dev); >>> + desc.groups = arm_trbe_groups; >>> + desc.dev = dev; >>> + trbe_csdev = coresight_register(&desc); >>> + if (IS_ERR(trbe_csdev)) >>> + goto cpu_clear; >>> + >>> + dev_set_drvdata(&trbe_csdev->dev, cpudata); >>> + cpudata->trbe_dbm = get_trbe_flag_update(trbidr); >>> + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); >>> + if (cpudata->trbe_align > SZ_2K) { >>> + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); >>> + goto cpu_clear; >>> + } >>> + per_cpu(csdev_sink, cpu) = trbe_csdev; >>> + trbe_reset_local(); >>> + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); >>> + return; >>> +cpu_clear: >>> + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); >>> +} >>> + >>> +static void arm_trbe_remove_coresight_cpu(void *info) >>> +{ >>> + int cpu = smp_processor_id(); >>> + struct trbe_drvdata *drvdata = info; >>> + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); >>> + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); >>> + >>> + if (trbe_csdev) { >>> + coresight_unregister(trbe_csdev); >>> + cpudata->drvdata = NULL; >>> + per_cpu(csdev_sink, cpu) = NULL; >>> + } >>> + disable_percpu_irq(drvdata->irq); >>> + trbe_reset_local(); >>> +} >>> + >>> +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) >>> +{ >>> + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); >>> + if (IS_ERR(drvdata->cpudata)) >>> + return PTR_ERR(drvdata->cpudata); >>> + >>> + arm_trbe_probe_coresight_cpu(drvdata); >>> + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); >>> + return 0; >>> +} >>> + >>> +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) >>> +{ >>> + arm_trbe_remove_coresight_cpu(drvdata); >>> + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); >>> + free_percpu(drvdata->cpudata); >>> + return 0; >>> +} >>> + >>> +static int arm_trbe_cpu_startup(unsigned int cpu, struct hlist_node *node) >>> +{ >>> + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); >>> + >>> + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { >>> + if (!per_cpu(csdev_sink, cpu)) { >>> + arm_trbe_probe_coresight_cpu(drvdata); >>> + } else { >>> + trbe_reset_local(); >>> + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); >>> + } >>> + } >>> + return 0; >>> +} >>> + >>> +static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node) >>> +{ >>> + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); >>> + >>> + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { >>> + disable_percpu_irq(drvdata->irq); >>> + trbe_reset_local(); >>> + } >>> + return 0; >>> +} >>> + >>> +static int arm_trbe_probe_cpuhp(struct trbe_drvdata *drvdata) >>> +{ >>> + enum cpuhp_state trbe_online; >>> + >>> + trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, >>> + arm_trbe_cpu_startup, arm_trbe_cpu_teardown); >>> + if (trbe_online < 0) >>> + return -EINVAL; >>> + >>> + if (cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node)) >>> + return -EINVAL; >>> + >>> + drvdata->trbe_online = trbe_online; >>> + return 0; >>> +} >>> + >>> +static void arm_trbe_remove_cpuhp(struct trbe_drvdata *drvdata) >>> +{ >>> + cpuhp_remove_multi_state(drvdata->trbe_online); >>> +} >>> + >>> +static int arm_trbe_probe_irq(struct platform_device *pdev, >>> + struct trbe_drvdata *drvdata) >>> +{ >>> + drvdata->irq = platform_get_irq(pdev, 0); >>> + if (!drvdata->irq) { > > Please use function platform_get_irq() properly - there is even an example on > how to do so in the documentation section of the function. The documentation says, the format should be. int irq = platform_get_irq(pdev, 0); if (irq < 0) return irq; Will change the conditional check above. > >>> + pr_err("IRQ not found for the platform device\n"); >>> + return -ENXIO; > > Why use a different error code? We could return the irq (which is < 0) but followed the SPE driver which returns ENXIO here. Happy to change either way. > >>> + } >>> + >>> + if (!irq_is_percpu(drvdata->irq)) { >>> + pr_err("IRQ is not a PPI\n"); >>> + return -EINVAL; >>> + } >>> + >>> + if (irq_get_percpu_devid_partition(drvdata->irq, &drvdata->supported_cpus)) >>> + return -EINVAL; >>> + >>> + drvdata->handle = alloc_percpu(typeof(*drvdata->handle)); >>> + if (!drvdata->handle) >>> + return -ENOMEM; >>> + >>> + if (request_percpu_irq(drvdata->irq, arm_trbe_irq_handler, DRVNAME, drvdata->handle)) { >>> + free_percpu(drvdata->handle); >>> + return -EINVAL; > > Here too you need to use the error code from the calling function rather than > making your own. Please revise for the entire patch. Okay, will capture the return value from request_percpu_irq() and return the same when it is an error case i.e being positive. > >>> + } >>> + return 0; >>> +} >>> + >>> +static void arm_trbe_remove_irq(struct trbe_drvdata *drvdata) >>> +{ >>> + free_percpu_irq(drvdata->irq, drvdata->handle); >>> + free_percpu(drvdata->handle); >>> +} >>> + >>> +static int arm_trbe_device_probe(struct platform_device *pdev) >>> +{ >>> + struct coresight_platform_data *pdata; >>> + struct trbe_drvdata *drvdata; >>> + struct device *dev = &pdev->dev; >>> + int ret; >>> + >>> + drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL); >>> + if (IS_ERR(drvdata)) >>> + return -ENOMEM; > > if (!drvdata) Changed. > >>> + >>> + pdata = coresight_get_platform_data(dev); >>> + if (IS_ERR(pdata)) { >>> + kfree(drvdata); > > No need to do this since devm_kzalloc() was used above. Suzuki had pointed out these issues, have already incorporated them i.e dropped kfree() here. > >>> + return -ENOMEM; > > Why not using the error from coresight_get_platform_data() instead of > masking it? Okay, will return PTR_ERR(pdata) instead. > >>> + } >>> + >>> + dev_set_drvdata(dev, drvdata); >>> + dev->platform_data = pdata; >>> + drvdata->pdev = pdev; >>> + ret = arm_trbe_probe_irq(pdev, drvdata); >>> + if (ret) >>> + goto irq_failed; >>> + >>> + ret = arm_trbe_probe_coresight(drvdata); >>> + if (ret) >>> + goto probe_failed; >>> + >>> + ret = arm_trbe_probe_cpuhp(drvdata); >>> + if (ret) >>> + goto cpuhp_failed; >>> + >>> + return 0; >>> +cpuhp_failed: >>> + arm_trbe_remove_coresight(drvdata); >>> +probe_failed: >>> + arm_trbe_remove_irq(drvdata); >>> +irq_failed: >>> + kfree(pdata); >>> + kfree(drvdata); > > Same here - both @pdata and @drvdata have been allocated by devm_kzalloc(). > devm_kzalloc(). Dropped these kfree() statements. - Anshuman
On Wed, Feb 10, 2021 at 09:42:29AM +0530, Anshuman Khandual wrote: > > > On 2/9/21 11:09 PM, Mathieu Poirier wrote: > > On Fri, Feb 05, 2021 at 10:53:30AM -0700, Mathieu Poirier wrote: > >> On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: > >>> Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is > >>> accessible via the system registers. The TRBE supports different addressing > >>> modes including CPU virtual address and buffer modes including the circular > >>> buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), > >>> an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the > >>> access to the trace buffer could be prohibited by a higher exception level > >>> (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU > >>> private interrupt (PPI) on address translation errors and when the buffer > >>> is full. Overall implementation here is inspired from the Arm SPE driver. > >>> > >> > >> I got this message when applying the patch: > >> > >> Applying: coresight: sink: Add TRBE driver > >> .git/rebase-apply/patch:76: new blank line at EOF. > >> + > >> warning: 1 line adds whitespace errors. > >> > >>> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> > >>> Cc: Mike Leach <mike.leach@linaro.org> > >>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> > >>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> > >>> --- > >>> Changes in V3: > >>> > >>> - Added new DT bindings document TRBE.yaml > >>> - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 > >>> - Dropped isb() from trbe_reset_local() > >>> - Dropped gap between (void *) and buf->trbe_base > >>> - Changed 'int' to 'unsigned int' in is_trbe_available() > >>> - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), > >>> set_trbe_enabled() and set_trbe_limit_pointer() > >>> - Changed get_trbe_flag_update(), is_trbe_programmable() and > >>> get_trbe_address_align() to accept TRBIDR value > >>> - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), > >>> is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value > >>> - Dropped snapshot mode condition in arm_trbe_alloc_buffer() > >>> - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled > >>> - Compute trbe_limit before trbe_write to get the updated handle > >>> - Added trbe_stop_and_truncate_event() > >>> - Dropped trbe_handle_fatal() > >>> > >>> Documentation/trace/coresight/coresight-trbe.rst | 39 + > >>> arch/arm64/include/asm/sysreg.h | 1 + > >>> drivers/hwtracing/coresight/Kconfig | 11 + > >>> drivers/hwtracing/coresight/Makefile | 1 + > >>> drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ > >>> drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ > >>> 6 files changed, 1235 insertions(+) > >>> create mode 100644 Documentation/trace/coresight/coresight-trbe.rst > >>> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c > >>> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h > >>> [...] > >>> + > >>> +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) > >>> +{ > >>> + struct perf_output_handle **handle_ptr = dev; > >>> + struct perf_output_handle *handle = *handle_ptr; > >>> + enum trbe_fault_action act; > >>> + > >>> + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); > >>> + clr_trbe_irq(); > >>> + > >>> + /* > >>> + * Ensure the trace is visible to the CPUs and > >>> + * any external aborts have been resolved. > >>> + */ > >>> + trbe_drain_buffer(); > >>> + isb(); > >>> + > >>> + if (!perf_get_aux(handle)) > >>> + return IRQ_NONE; > >>> + > >>> + if (!is_perf_trbe(handle)) > >>> + return IRQ_NONE; > >>> + > >>> + irq_work_run(); > > > > There is a comment in the SPE driver about this. Since this driver closely > > follows that implementation it would be nice to have the comments as well. > > Otherwise the reader has to constantly go back to the original driver. > > Sure, will add the following comment before irq_work_run(). > > /* > * Ensure perf callbacks have completed, which may disable the > * profiling buffer in response to a TRUNCATION flag. > */ > > > > > I will come back to this function later. > > Okay. > > > > >>> + > >>> + act = trbe_get_fault_act(handle); > >>> + switch (act) { > >>> + case TRBE_FAULT_ACT_WRAP: > >>> + trbe_handle_overflow(handle); > >>> + break; > >>> + case TRBE_FAULT_ACT_SPURIOUS: > >>> + trbe_handle_spurious(handle); > >>> + break; > >>> + case TRBE_FAULT_ACT_FATAL: > >>> + trbe_stop_and_truncate_event(handle); > >>> + break; > >>> + } > >>> + return IRQ_HANDLED; > >>> +} > >>> + > >>> +static const struct coresight_ops_sink arm_trbe_sink_ops = { > >>> + .enable = arm_trbe_enable, > >>> + .disable = arm_trbe_disable, > >>> + .alloc_buffer = arm_trbe_alloc_buffer, > >>> + .free_buffer = arm_trbe_free_buffer, > >>> + .update_buffer = arm_trbe_update_buffer, > >>> +}; > >>> + > >>> +static const struct coresight_ops arm_trbe_cs_ops = { > >>> + .sink_ops = &arm_trbe_sink_ops, > >>> +}; > >>> + > >>> +static ssize_t align_show(struct device *dev, struct device_attribute *attr, char *buf) > >>> +{ > >>> + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > >>> + > >>> + return sprintf(buf, "%llx\n", cpudata->trbe_align); > >>> +} > >>> +static DEVICE_ATTR_RO(align); > >>> + > >>> +static ssize_t dbm_show(struct device *dev, struct device_attribute *attr, char *buf) > >>> +{ > >>> + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > >>> + > >>> + return sprintf(buf, "%d\n", cpudata->trbe_dbm); > >>> +} > >>> +static DEVICE_ATTR_RO(dbm); > >>> + > >>> +static struct attribute *arm_trbe_attrs[] = { > >>> + &dev_attr_align.attr, > >>> + &dev_attr_dbm.attr, > >>> + NULL, > >>> +}; > >>> + > >>> +static const struct attribute_group arm_trbe_group = { > >>> + .attrs = arm_trbe_attrs, > >>> +}; > >>> + > >>> +static const struct attribute_group *arm_trbe_groups[] = { > >>> + &arm_trbe_group, > >>> + NULL, > >>> +}; > >>> + > >>> +static void arm_trbe_probe_coresight_cpu(void *info) > >>> +{ > >>> + struct trbe_drvdata *drvdata = info; > >>> + struct coresight_desc desc = { 0 }; > >>> + int cpu = smp_processor_id(); > >>> + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > >>> + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > >>> + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); > >>> + struct device *dev; > >>> + > >>> + if (WARN_ON(!cpudata)) > >>> + goto cpu_clear; > > > > Where was the memory for cpudata allocated? As far as I can tell, at this time > > it is just a pointer that was not allocated and as such it should be NULL. > > cpudata gets allocated in arm_trbe_probe_coresight() just before calling > individual CPU based probes i.e arm_trbe_probe_coresight_cpu() directly > and via smp_call_function_many(). > > arm_trbe_device_probe() > arm_trbe_probe_coresight() > arm_trbe_probe_coresight_cpu() Ah yes, my apologies here. Looking at the code I realised I skipped arm_trbe_probe_coresight() and went straight to arm_trbe_probe_coresight_cpu(). No wonder things didn't make sense. I will take another look at this function. > > > > >>> + > >>> + if (trbe_csdev) > >>> + return; > >>> + > >>> + cpudata->cpu = smp_processor_id(); > > > > Why call this again when you already did above? And how is > > Right, this is redundant. Will just assign it as cpu which has already > been computed. > > > arm_trbe_probe_coresight_cpu() is called for every CPU in the system? > > During boot in arm_trbe_probe_coresight(), it is called once directly on > the executing cpu and on all other via smp_call_function_many(). > > > > >>> + cpudata->drvdata = drvdata; > >>> + dev = &cpudata->drvdata->pdev->dev; > >>> + > >>> + if (!is_trbe_available()) { > >>> + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); > >>> + goto cpu_clear; > >>> + } > >>> + > >>> + if (!is_trbe_programmable(trbidr)) { > >>> + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); > >>> + goto cpu_clear; > >>> + } > >>> + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); > >>> + if (IS_ERR(desc.name)) > >>> + goto cpu_clear; > >>> + > >>> + desc.type = CORESIGHT_DEV_TYPE_SINK; > >>> + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; > >>> + desc.ops = &arm_trbe_cs_ops; > >>> + desc.pdata = dev_get_platdata(dev); > >>> + desc.groups = arm_trbe_groups; > >>> + desc.dev = dev; > >>> + trbe_csdev = coresight_register(&desc); > >>> + if (IS_ERR(trbe_csdev)) > >>> + goto cpu_clear; > >>> + > >>> + dev_set_drvdata(&trbe_csdev->dev, cpudata); > >>> + cpudata->trbe_dbm = get_trbe_flag_update(trbidr); > >>> + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); > >>> + if (cpudata->trbe_align > SZ_2K) { > >>> + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); > >>> + goto cpu_clear; > >>> + } > >>> + per_cpu(csdev_sink, cpu) = trbe_csdev; > >>> + trbe_reset_local(); > >>> + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > >>> + return; > >>> +cpu_clear: > >>> + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); > >>> +} > >>> + > >>> +static void arm_trbe_remove_coresight_cpu(void *info) > >>> +{ > >>> + int cpu = smp_processor_id(); > >>> + struct trbe_drvdata *drvdata = info; > >>> + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > >>> + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > >>> + > >>> + if (trbe_csdev) { > >>> + coresight_unregister(trbe_csdev); > >>> + cpudata->drvdata = NULL; > >>> + per_cpu(csdev_sink, cpu) = NULL; > >>> + } > >>> + disable_percpu_irq(drvdata->irq); > >>> + trbe_reset_local(); > >>> +} > >>> + > >>> +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) > >>> +{ > >>> + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); > >>> + if (IS_ERR(drvdata->cpudata)) > >>> + return PTR_ERR(drvdata->cpudata); > >>> + > >>> + arm_trbe_probe_coresight_cpu(drvdata); > >>> + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); > >>> + return 0; > >>> +} > >>> + > >>> +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) > >>> +{ > >>> + arm_trbe_remove_coresight_cpu(drvdata); > >>> + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); > >>> + free_percpu(drvdata->cpudata); > >>> + return 0; > >>> +} > >>> + > >>> +static int arm_trbe_cpu_startup(unsigned int cpu, struct hlist_node *node) > >>> +{ > >>> + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > >>> + > >>> + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > >>> + if (!per_cpu(csdev_sink, cpu)) { > >>> + arm_trbe_probe_coresight_cpu(drvdata); > >>> + } else { > >>> + trbe_reset_local(); > >>> + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > >>> + } > >>> + } > >>> + return 0; > >>> +} > >>> + > >>> +static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node) > >>> +{ > >>> + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > >>> + > >>> + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > >>> + disable_percpu_irq(drvdata->irq); > >>> + trbe_reset_local(); > >>> + } > >>> + return 0; > >>> +} > >>> + > >>> +static int arm_trbe_probe_cpuhp(struct trbe_drvdata *drvdata) > >>> +{ > >>> + enum cpuhp_state trbe_online; > >>> + > >>> + trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, > >>> + arm_trbe_cpu_startup, arm_trbe_cpu_teardown); > >>> + if (trbe_online < 0) > >>> + return -EINVAL; > >>> + > >>> + if (cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node)) > >>> + return -EINVAL; > >>> + > >>> + drvdata->trbe_online = trbe_online; > >>> + return 0; > >>> +} > >>> + > >>> +static void arm_trbe_remove_cpuhp(struct trbe_drvdata *drvdata) > >>> +{ > >>> + cpuhp_remove_multi_state(drvdata->trbe_online); > >>> +} > >>> + > >>> +static int arm_trbe_probe_irq(struct platform_device *pdev, > >>> + struct trbe_drvdata *drvdata) > >>> +{ > >>> + drvdata->irq = platform_get_irq(pdev, 0); > >>> + if (!drvdata->irq) { > > > > Please use function platform_get_irq() properly - there is even an example on > > how to do so in the documentation section of the function. > > The documentation says, the format should be. > > int irq = platform_get_irq(pdev, 0); > if (irq < 0) > return irq; > > Will change the conditional check above. > > > > >>> + pr_err("IRQ not found for the platform device\n"); > >>> + return -ENXIO; > > > > Why use a different error code? > > We could return the irq (which is < 0) but followed the SPE > driver which returns ENXIO here. Happy to change either way. Please use the right error code. > > > > >>> + } > >>> + > >>> + if (!irq_is_percpu(drvdata->irq)) { > >>> + pr_err("IRQ is not a PPI\n"); > >>> + return -EINVAL; > >>> + } > >>> + > >>> + if (irq_get_percpu_devid_partition(drvdata->irq, &drvdata->supported_cpus)) > >>> + return -EINVAL; > >>> + > >>> + drvdata->handle = alloc_percpu(typeof(*drvdata->handle)); > >>> + if (!drvdata->handle) > >>> + return -ENOMEM; > >>> + > >>> + if (request_percpu_irq(drvdata->irq, arm_trbe_irq_handler, DRVNAME, drvdata->handle)) { > >>> + free_percpu(drvdata->handle); > >>> + return -EINVAL; > > > > Here too you need to use the error code from the calling function rather than > > making your own. Please revise for the entire patch. > > Okay, will capture the return value from request_percpu_irq() and > return the same when it is an error case i.e being positive. > > > > >>> + } > >>> + return 0; > >>> +} > >>> + > >>> +static void arm_trbe_remove_irq(struct trbe_drvdata *drvdata) > >>> +{ > >>> + free_percpu_irq(drvdata->irq, drvdata->handle); > >>> + free_percpu(drvdata->handle); > >>> +} > >>> + > >>> +static int arm_trbe_device_probe(struct platform_device *pdev) > >>> +{ > >>> + struct coresight_platform_data *pdata; > >>> + struct trbe_drvdata *drvdata; > >>> + struct device *dev = &pdev->dev; > >>> + int ret; > >>> + > >>> + drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL); > >>> + if (IS_ERR(drvdata)) > >>> + return -ENOMEM; > > > > if (!drvdata) > > Changed. > > > > >>> + > >>> + pdata = coresight_get_platform_data(dev); > >>> + if (IS_ERR(pdata)) { > >>> + kfree(drvdata); > > > > No need to do this since devm_kzalloc() was used above. > > Suzuki had pointed out these issues, have already incorporated them i.e > dropped kfree() here. > To avoid getting tunel vision I don't look at other comments before reviewing a patchset. As such it is possible to get redundant comments. More to come shortly. > > > >>> + return -ENOMEM; > > > > Why not using the error from coresight_get_platform_data() instead of > > masking it? > > Okay, will return PTR_ERR(pdata) instead. > > > > >>> + } > >>> + > >>> + dev_set_drvdata(dev, drvdata); > >>> + dev->platform_data = pdata; > >>> + drvdata->pdev = pdev; > >>> + ret = arm_trbe_probe_irq(pdev, drvdata); > >>> + if (ret) > >>> + goto irq_failed; > >>> + > >>> + ret = arm_trbe_probe_coresight(drvdata); > >>> + if (ret) > >>> + goto probe_failed; > >>> + > >>> + ret = arm_trbe_probe_cpuhp(drvdata); > >>> + if (ret) > >>> + goto cpuhp_failed; > >>> + > >>> + return 0; > >>> +cpuhp_failed: > >>> + arm_trbe_remove_coresight(drvdata); > >>> +probe_failed: > >>> + arm_trbe_remove_irq(drvdata); > >>> +irq_failed: > >>> + kfree(pdata); > >>> + kfree(drvdata); > > > > Same here - both @pdata and @drvdata have been allocated by devm_kzalloc(). > > devm_kzalloc(). > > Dropped these kfree() statements. > > - Anshuman
On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: > Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is > accessible via the system registers. The TRBE supports different addressing > modes including CPU virtual address and buffer modes including the circular > buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), > an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the > access to the trace buffer could be prohibited by a higher exception level > (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU > private interrupt (PPI) on address translation errors and when the buffer > is full. Overall implementation here is inspired from the Arm SPE driver. > > Cc: Mathieu Poirier <mathieu.poirier@linaro.org> > Cc: Mike Leach <mike.leach@linaro.org> > Cc: Suzuki K Poulose <suzuki.poulose@arm.com> > Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> > --- > Changes in V3: > > - Added new DT bindings document TRBE.yaml > - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 > - Dropped isb() from trbe_reset_local() > - Dropped gap between (void *) and buf->trbe_base > - Changed 'int' to 'unsigned int' in is_trbe_available() > - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), > set_trbe_enabled() and set_trbe_limit_pointer() > - Changed get_trbe_flag_update(), is_trbe_programmable() and > get_trbe_address_align() to accept TRBIDR value > - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), > is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value > - Dropped snapshot mode condition in arm_trbe_alloc_buffer() > - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled > - Compute trbe_limit before trbe_write to get the updated handle > - Added trbe_stop_and_truncate_event() > - Dropped trbe_handle_fatal() > > Documentation/trace/coresight/coresight-trbe.rst | 39 + > arch/arm64/include/asm/sysreg.h | 1 + > drivers/hwtracing/coresight/Kconfig | 11 + > drivers/hwtracing/coresight/Makefile | 1 + > drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ > drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ > 6 files changed, 1235 insertions(+) > create mode 100644 Documentation/trace/coresight/coresight-trbe.rst > create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c > create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h > + [...] > +static void arm_trbe_probe_coresight_cpu(void *info) > +{ > + struct trbe_drvdata *drvdata = info; > + struct coresight_desc desc = { 0 }; > + int cpu = smp_processor_id(); > + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); > + struct device *dev; > + > + if (WARN_ON(!cpudata)) > + goto cpu_clear; There is already a check for this in arm_trbe_probe_coresight(), we couldn't be here if there was a problem with the allocation. > + > + if (trbe_csdev) > + return; Now that's a reason to have a WARN_ON(). If we are probing and a sink is already present in this cpu's slot, something went seriously wrong and we should be clear about it. > + > + cpudata->cpu = smp_processor_id(); > + cpudata->drvdata = drvdata; > + dev = &cpudata->drvdata->pdev->dev; > + > + if (!is_trbe_available()) { > + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + > + if (!is_trbe_programmable(trbidr)) { > + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); We will end up with "arm_trbe0", "arm_trbe1" and so on in sysfs... Is the "arm_" part absolutely needed? I think this should be like what we do for etmv3 and etmv4 where only "etmX" shows up in sysfs. > + if (IS_ERR(desc.name)) > + goto cpu_clear; > + > + desc.type = CORESIGHT_DEV_TYPE_SINK; > + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; > + desc.ops = &arm_trbe_cs_ops; > + desc.pdata = dev_get_platdata(dev); > + desc.groups = arm_trbe_groups; > + desc.dev = dev; > + trbe_csdev = coresight_register(&desc); > + if (IS_ERR(trbe_csdev)) > + goto cpu_clear; > + > + dev_set_drvdata(&trbe_csdev->dev, cpudata); > + cpudata->trbe_dbm = get_trbe_flag_update(trbidr); > + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); > + if (cpudata->trbe_align > SZ_2K) { > + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); > + goto cpu_clear; Here coresight_unregister() should be called. The other option is to call coresight_register() when everything else is known to be fine, which is the favoured approach. > + } > + per_cpu(csdev_sink, cpu) = trbe_csdev; > + trbe_reset_local(); > + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > + return; > +cpu_clear: > + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); > +} > + > +static void arm_trbe_remove_coresight_cpu(void *info) > +{ > + int cpu = smp_processor_id(); > + struct trbe_drvdata *drvdata = info; > + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > + > + if (trbe_csdev) { In what scenario do you see not having a trbe_csdev and still needing to disable IRQs for the HW? If there is a such a case then a few lines of comment is needed. > + coresight_unregister(trbe_csdev); > + cpudata->drvdata = NULL; > + per_cpu(csdev_sink, cpu) = NULL; > + } > + disable_percpu_irq(drvdata->irq); > + trbe_reset_local(); Theoretically this code shouldn't run when the TRBE is enabled, because the CS core will prevent that from happening. As sush disabling interrupts after coresight_unregister() has been called and setting cpudata->drvdata to NULL should be fine. But from an outsider's point of view it will look very bizarre. Either write a comment to explain all that or call the above two before doing the cleanup. > +} > + > +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) > +{ > + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); > + if (IS_ERR(drvdata->cpudata)) > + return PTR_ERR(drvdata->cpudata); As far as I can tell alloc_percpu() returns NULL on failure and nothing else. > + > + arm_trbe_probe_coresight_cpu(drvdata); > + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); The above two calls look racy to me. The executing process could be moved to another CPU between the call to arm_trbe_probe_coresight_cpu() and smp_call_function_many(), which would prevent the initialisation of the TRBE on the new CPU to be done. I suggest using a for_each_cpu() loop where smp_call_function_single() would be used. That way we are guaranteed all the TRBEs will be initialised. > + return 0; > +} > + > +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) > +{ > + arm_trbe_remove_coresight_cpu(drvdata); > + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); Same as above. I'm out of time for today, more to come tomorrow. Mathieu > + free_percpu(drvdata->cpudata); > + return 0; > +} > + > +static int arm_trbe_cpu_startup(unsigned int cpu, struct hlist_node *node) > +{ > + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > + > + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > + if (!per_cpu(csdev_sink, cpu)) { > + arm_trbe_probe_coresight_cpu(drvdata); > + } else { > + trbe_reset_local(); > + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > + } > + } > + return 0; > +} > + > +static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node) > +{ > + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > + > + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > + disable_percpu_irq(drvdata->irq); > + trbe_reset_local(); > + } > + return 0; > +} > + > +static int arm_trbe_probe_cpuhp(struct trbe_drvdata *drvdata) > +{ > + enum cpuhp_state trbe_online; > + > + trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, > + arm_trbe_cpu_startup, arm_trbe_cpu_teardown); > + if (trbe_online < 0) > + return -EINVAL; > + > + if (cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node)) > + return -EINVAL; > + > + drvdata->trbe_online = trbe_online; > + return 0; > +} > + > +static void arm_trbe_remove_cpuhp(struct trbe_drvdata *drvdata) > +{ > + cpuhp_remove_multi_state(drvdata->trbe_online); > +} > + > +static int arm_trbe_probe_irq(struct platform_device *pdev, > + struct trbe_drvdata *drvdata) > +{ > + drvdata->irq = platform_get_irq(pdev, 0); > + if (!drvdata->irq) { > + pr_err("IRQ not found for the platform device\n"); > + return -ENXIO; > + } > + > + if (!irq_is_percpu(drvdata->irq)) { > + pr_err("IRQ is not a PPI\n"); > + return -EINVAL; > + } > + > + if (irq_get_percpu_devid_partition(drvdata->irq, &drvdata->supported_cpus)) > + return -EINVAL; > + > + drvdata->handle = alloc_percpu(typeof(*drvdata->handle)); > + if (!drvdata->handle) > + return -ENOMEM; > + > + if (request_percpu_irq(drvdata->irq, arm_trbe_irq_handler, DRVNAME, drvdata->handle)) { > + free_percpu(drvdata->handle); > + return -EINVAL; > + } > + return 0; > +} > + > +static void arm_trbe_remove_irq(struct trbe_drvdata *drvdata) > +{ > + free_percpu_irq(drvdata->irq, drvdata->handle); > + free_percpu(drvdata->handle); > +} > + > +static int arm_trbe_device_probe(struct platform_device *pdev) > +{ > + struct coresight_platform_data *pdata; > + struct trbe_drvdata *drvdata; > + struct device *dev = &pdev->dev; > + int ret; > + > + drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL); > + if (IS_ERR(drvdata)) > + return -ENOMEM; > + > + pdata = coresight_get_platform_data(dev); > + if (IS_ERR(pdata)) { > + kfree(drvdata); > + return -ENOMEM; > + } > + > + dev_set_drvdata(dev, drvdata); > + dev->platform_data = pdata; > + drvdata->pdev = pdev; > + ret = arm_trbe_probe_irq(pdev, drvdata); > + if (ret) > + goto irq_failed; > + > + ret = arm_trbe_probe_coresight(drvdata); > + if (ret) > + goto probe_failed; > + > + ret = arm_trbe_probe_cpuhp(drvdata); > + if (ret) > + goto cpuhp_failed; > + > + return 0; > +cpuhp_failed: > + arm_trbe_remove_coresight(drvdata); > +probe_failed: > + arm_trbe_remove_irq(drvdata); > +irq_failed: > + kfree(pdata); > + kfree(drvdata); > + return ret; > +} > + > +static int arm_trbe_device_remove(struct platform_device *pdev) > +{ > + struct coresight_platform_data *pdata = dev_get_platdata(&pdev->dev); > + struct trbe_drvdata *drvdata = platform_get_drvdata(pdev); > + > + arm_trbe_remove_coresight(drvdata); > + arm_trbe_remove_cpuhp(drvdata); > + arm_trbe_remove_irq(drvdata); > + kfree(pdata); > + kfree(drvdata); > + return 0; > +} > + > +static const struct of_device_id arm_trbe_of_match[] = { > + { .compatible = "arm,trace-buffer-extension"}, > + {}, > +}; > +MODULE_DEVICE_TABLE(of, arm_trbe_of_match); > + > +static struct platform_driver arm_trbe_driver = { > + .driver = { > + .name = DRVNAME, > + .of_match_table = of_match_ptr(arm_trbe_of_match), > + .suppress_bind_attrs = true, > + }, > + .probe = arm_trbe_device_probe, > + .remove = arm_trbe_device_remove, > +}; > + > +static int __init arm_trbe_init(void) > +{ > + int ret; > + > + if (arm64_kernel_unmapped_at_el0()) { > + pr_err("TRBE wouldn't work if kernel gets unmapped at EL0\n"); > + return -EOPNOTSUPP; > + } > + > + ret = platform_driver_register(&arm_trbe_driver); > + if (!ret) > + return 0; > + > + pr_err("Error registering %s platform driver\n", DRVNAME); > + return ret; > +} > + > +static void __exit arm_trbe_exit(void) > +{ > + platform_driver_unregister(&arm_trbe_driver); > +} > +module_init(arm_trbe_init); > +module_exit(arm_trbe_exit); > + > +MODULE_AUTHOR("Anshuman Khandual <anshuman.khandual@arm.com>"); > +MODULE_DESCRIPTION("Arm Trace Buffer Extension (TRBE) driver"); > +MODULE_LICENSE("GPL v2"); > diff --git a/drivers/hwtracing/coresight/coresight-trbe.h b/drivers/hwtracing/coresight/coresight-trbe.h > new file mode 100644 > index 0000000..43308bc > --- /dev/null > +++ b/drivers/hwtracing/coresight/coresight-trbe.h > @@ -0,0 +1,160 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * This contains all required hardware related helper functions for > + * Trace Buffer Extension (TRBE) driver in the coresight framework. > + * > + * Copyright (C) 2020 ARM Ltd. > + * > + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > + */ > +#include <linux/coresight.h> > +#include <linux/device.h> > +#include <linux/irq.h> > +#include <linux/kernel.h> > +#include <linux/of.h> > +#include <linux/platform_device.h> > +#include <linux/smp.h> > + > +#include "coresight-etm-perf.h" > + > +DECLARE_PER_CPU(struct coresight_device *, csdev_sink); > + > +static inline bool is_trbe_available(void) > +{ > + u64 aa64dfr0 = read_sysreg_s(SYS_ID_AA64DFR0_EL1); > + unsigned int trbe = cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_TRBE_SHIFT); > + > + return trbe >= 0b0001; > +} > + > +static inline bool is_trbe_enabled(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + return trblimitr & TRBLIMITR_ENABLE; > +} > + > +#define TRBE_EC_OTHERS 0 > +#define TRBE_EC_STAGE1_ABORT 36 > +#define TRBE_EC_STAGE2_ABORT 37 > + > +static inline int get_trbe_ec(u64 trbsr) > +{ > + return (trbsr >> TRBSR_EC_SHIFT) & TRBSR_EC_MASK; > +} > + > +#define TRBE_BSC_NOT_STOPPED 0 > +#define TRBE_BSC_FILLED 1 > +#define TRBE_BSC_TRIGGERED 2 > + > +static inline int get_trbe_bsc(u64 trbsr) > +{ > + return (trbsr >> TRBSR_BSC_SHIFT) & TRBSR_BSC_MASK; > +} > + > +static inline void clr_trbe_irq(void) > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + > + trbsr &= ~TRBSR_IRQ; > + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > +} > + > +static inline bool is_trbe_irq(u64 trbsr) > +{ > + return trbsr & TRBSR_IRQ; > +} > + > +static inline bool is_trbe_trg(u64 trbsr) > +{ > + return trbsr & TRBSR_TRG; > +} > + > +static inline bool is_trbe_wrap(u64 trbsr) > +{ > + return trbsr & TRBSR_WRAP; > +} > + > +static inline bool is_trbe_abort(u64 trbsr) > +{ > + return trbsr & TRBSR_ABORT; > +} > + > +static inline bool is_trbe_running(u64 trbsr) > +{ > + return !(trbsr & TRBSR_STOP); > +} > + > +#define TRBE_TRIG_MODE_STOP 0 > +#define TRBE_TRIG_MODE_IRQ 1 > +#define TRBE_TRIG_MODE_IGNORE 3 > + > +#define TRBE_FILL_MODE_FILL 0 > +#define TRBE_FILL_MODE_WRAP 1 > +#define TRBE_FILL_MODE_CIRCULAR_BUFFER 3 > + > +static inline void set_trbe_disabled(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + trblimitr &= ~TRBLIMITR_ENABLE; > + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > +} > + > +static inline bool get_trbe_flag_update(u64 trbidr) > +{ > + return trbidr & TRBIDR_FLAG; > +} > + > +static inline bool is_trbe_programmable(u64 trbidr) > +{ > + return !(trbidr & TRBIDR_PROG); > +} > + > +static inline int get_trbe_address_align(u64 trbidr) > +{ > + return (trbidr >> TRBIDR_ALIGN_SHIFT) & TRBIDR_ALIGN_MASK; > +} > + > +static inline unsigned long get_trbe_write_pointer(void) > +{ > + u64 trbptr = read_sysreg_s(SYS_TRBPTR_EL1); > + unsigned long addr = (trbptr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; > + > + return addr; > +} > + > +static inline void set_trbe_write_pointer(unsigned long addr) > +{ > + WARN_ON(is_trbe_enabled()); > + addr = (addr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; > + write_sysreg_s(addr, SYS_TRBPTR_EL1); > +} > + > +static inline unsigned long get_trbe_limit_pointer(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + unsigned long limit = (trblimitr >> TRBLIMITR_LIMIT_SHIFT) & TRBLIMITR_LIMIT_MASK; > + unsigned long addr = limit << TRBLIMITR_LIMIT_SHIFT; > + > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + return addr; > +} > + > +static inline unsigned long get_trbe_base_pointer(void) > +{ > + u64 trbbaser = read_sysreg_s(SYS_TRBBASER_EL1); > + unsigned long addr = (trbbaser >> TRBBASER_BASE_SHIFT) & TRBBASER_BASE_MASK; > + > + addr = addr << TRBBASER_BASE_SHIFT; > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + return addr; > +} > + > +static inline void set_trbe_base_pointer(unsigned long addr) > +{ > + WARN_ON(is_trbe_enabled()); > + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + write_sysreg_s(addr, SYS_TRBBASER_EL1); > +} > -- > 2.7.4 >
On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: > Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is > accessible via the system registers. The TRBE supports different addressing > modes including CPU virtual address and buffer modes including the circular > buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), > an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the > access to the trace buffer could be prohibited by a higher exception level > (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU > private interrupt (PPI) on address translation errors and when the buffer > is full. Overall implementation here is inspired from the Arm SPE driver. > > Cc: Mathieu Poirier <mathieu.poirier@linaro.org> > Cc: Mike Leach <mike.leach@linaro.org> > Cc: Suzuki K Poulose <suzuki.poulose@arm.com> > Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> > --- > Changes in V3: > > - Added new DT bindings document TRBE.yaml > - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 > - Dropped isb() from trbe_reset_local() > - Dropped gap between (void *) and buf->trbe_base > - Changed 'int' to 'unsigned int' in is_trbe_available() > - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), > set_trbe_enabled() and set_trbe_limit_pointer() > - Changed get_trbe_flag_update(), is_trbe_programmable() and > get_trbe_address_align() to accept TRBIDR value > - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), > is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value > - Dropped snapshot mode condition in arm_trbe_alloc_buffer() > - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled > - Compute trbe_limit before trbe_write to get the updated handle > - Added trbe_stop_and_truncate_event() > - Dropped trbe_handle_fatal() > > Documentation/trace/coresight/coresight-trbe.rst | 39 + > arch/arm64/include/asm/sysreg.h | 1 + > drivers/hwtracing/coresight/Kconfig | 11 + > drivers/hwtracing/coresight/Makefile | 1 + > drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ > drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ > 6 files changed, 1235 insertions(+) > create mode 100644 Documentation/trace/coresight/coresight-trbe.rst > create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c > create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h > > diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst > new file mode 100644 > index 0000000..1cbb819 > --- /dev/null > +++ b/Documentation/trace/coresight/coresight-trbe.rst > @@ -0,0 +1,39 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +============================== > +Trace Buffer Extension (TRBE). > +============================== > + > + :Author: Anshuman Khandual <anshuman.khandual@arm.com> > + :Date: November 2020 > + > +Hardware Description > +-------------------- > + > +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system > +memory, CPU traces generated from a corresponding percpu tracing unit. This > +gets plugged in as a coresight sink device because the corresponding trace > +genarators (ETE), are plugged in as source device. > + > +The TRBE is not compliant to CoreSight architecture specifications, but is > +driven via the CoreSight driver framework to support the ETE (which is > +CoreSight compliant) integration. > + > +Sysfs files and directories > +--------------------------- > + > +The TRBE devices appear on the existing coresight bus alongside the other > +coresight devices:: > + > + >$ ls /sys/bus/coresight/devices > + trbe0 trbe1 trbe2 trbe3 > + > +The ``trbe<N>`` named TRBEs are associated with a CPU.:: > + > + >$ ls /sys/bus/coresight/devices/trbe0/ > + align dbm > + > +*Key file items are:-* > + * ``align``: TRBE write pointer alignment > + * ``dbm``: TRBE updates memory with access and dirty flags > + > diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h > index 85ae4db..9e2e9b7 100644 > --- a/arch/arm64/include/asm/sysreg.h > +++ b/arch/arm64/include/asm/sysreg.h > @@ -97,6 +97,7 @@ > #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) > #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) > #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) > +#define TSB_CSYNC __emit_inst(0xd503225f) > > #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) > #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) > diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig > index f154ae7..aa657ab 100644 > --- a/drivers/hwtracing/coresight/Kconfig > +++ b/drivers/hwtracing/coresight/Kconfig > @@ -164,6 +164,17 @@ config CORESIGHT_CTI > To compile this driver as a module, choose M here: the > module will be called coresight-cti. > > +config CORESIGHT_TRBE > + bool "Trace Buffer Extension (TRBE) driver" > + depends on ARM64 > + help > + This driver provides support for percpu Trace Buffer Extension (TRBE). > + TRBE always needs to be used along with it's corresponding percpu ETE > + component. ETE generates trace data which is then captured with TRBE. > + Unlike traditional sink devices, TRBE is a CPU feature accessible via > + system registers. But it's explicit dependency with trace unit (ETE) > + requires it to be plugged in as a coresight sink device. > + > config CORESIGHT_CTI_INTEGRATION_REGS > bool "Access CTI CoreSight Integration Registers" > depends on CORESIGHT_CTI > diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile > index f20e357..d608165 100644 > --- a/drivers/hwtracing/coresight/Makefile > +++ b/drivers/hwtracing/coresight/Makefile > @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o > obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o > obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o > obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o > +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o > coresight-cti-y := coresight-cti-core.o coresight-cti-platform.o \ > coresight-cti-sysfs.o > diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c > new file mode 100644 > index 0000000..1464d8b > --- /dev/null > +++ b/drivers/hwtracing/coresight/coresight-trbe.c > @@ -0,0 +1,1023 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight > + * sink device could then pair with an appropriate per-cpu coresight source > + * device (ETE) thus generating required trace data. Trace can be enabled > + * via the perf framework. > + * > + * Copyright (C) 2020 ARM Ltd. > + * > + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > + */ > +#define DRVNAME "arm_trbe" > + > +#define pr_fmt(fmt) DRVNAME ": " fmt > + > +#include "coresight-trbe.h" > + > +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) > + > +/* > + * A padding packet that will help the user space tools > + * in skipping relevant sections in the captured trace > + * data which could not be decoded. TRBE doesn't support > + * formatting the trace data, unlike the legacy CoreSight > + * sinks and thus we use ETE trace packets to pad the > + * sections of the buffer. > + */ > +#define ETE_IGNORE_PACKET 0x70 > + > +/* > + * Minimum amount of meaningful trace will contain: > + * A-Sync, Trace Info, Trace On, Address, Atom. > + * This is about 44bytes of ETE trace. To be on > + * the safer side, we assume 64bytes is the minimum > + * space required for a meaningful session, before > + * we hit a "WRAP" event. > + */ > +#define TRBE_TRACE_MIN_BUF_SIZE 64 > + > +enum trbe_fault_action { > + TRBE_FAULT_ACT_WRAP, > + TRBE_FAULT_ACT_SPURIOUS, > + TRBE_FAULT_ACT_FATAL, > +}; > + > +struct trbe_buf { > + unsigned long trbe_base; > + unsigned long trbe_limit; > + unsigned long trbe_write; > + int nr_pages; > + void **pages; > + bool snapshot; > + struct trbe_cpudata *cpudata; > +}; > + > +struct trbe_cpudata { > + bool trbe_dbm; > + u64 trbe_align; > + int cpu; > + enum cs_mode mode; > + struct trbe_buf *buf; > + struct trbe_drvdata *drvdata; > +}; > + > +struct trbe_drvdata { > + struct trbe_cpudata __percpu *cpudata; > + struct perf_output_handle __percpu **handle; > + struct hlist_node hotplug_node; > + int irq; > + cpumask_t supported_cpus; > + enum cpuhp_state trbe_online; > + struct platform_device *pdev; > +}; > + > +static int trbe_alloc_node(struct perf_event *event) > +{ > + if (event->cpu == -1) > + return NUMA_NO_NODE; > + return cpu_to_node(event->cpu); > +} > + > +static void trbe_drain_buffer(void) > +{ > + asm(TSB_CSYNC); > + dsb(nsh); > +} > + > +static void trbe_drain_and_disable_local(void) > +{ > + trbe_drain_buffer(); > + write_sysreg_s(0, SYS_TRBLIMITR_EL1); > + isb(); > +} > + > +static void trbe_reset_local(void) > +{ > + trbe_drain_and_disable_local(); > + write_sysreg_s(0, SYS_TRBPTR_EL1); > + write_sysreg_s(0, SYS_TRBBASER_EL1); > + write_sysreg_s(0, SYS_TRBSR_EL1); > +} > + > +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + /* > + * We cannot proceed with the buffer collection and we > + * do not have any data for the current session. The > + * etm_perf driver expects to close out the aux_buffer > + * at event_stop(). So disable the TRBE here and leave > + * the update_buffer() to return a 0 size. > + */ > + trbe_drain_and_disable_local(); > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > +} > + > +/* > + * TRBE Buffer Management > + * > + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, > + * it starts writing trace data from the write pointer onward till the limit pointer. > + * When the write pointer reaches the address just before the limit pointer, it gets > + * wrapped around again to the base pointer. This is called a TRBE wrap event, which > + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver > + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ > + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and > + * LIMIT pointers. > + * > + * Wrap around with an IRQ > + * ------ < ------ < ------- < ----- < ----- > + * | | > + * ------ > ------ > ------- > ----- > ----- > + * > + * +---------------+-----------------------+ > + * | | | > + * +---------------+-----------------------+ > + * Base Pointer Write Pointer Limit Pointer > + * > + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write > + * pointer can be aligned to the implementation defined TRBE trace buffer alignment > + * as captured in trbe_cpudata->trbe_align. > + * > + * > + * head tail wakeup > + * +---------------------------------------+----- ~ ~ ------ > + * |$$$$$$$|################|$$$$$$$$$$$$$$| | > + * +---------------------------------------+----- ~ ~ ------ > + * Base Pointer Write Pointer Limit Pointer > + * > + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing > + * values which tracks all the driver writes and user reads from the perf auxiliary > + * buffer. Generally [head..tail] is the area where the driver can write into unless > + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and > + * configured depending on the perf_output_handle indices, so that the driver does > + * not override into areas in the perf auxiliary buffer which is being or yet to be > + * consumed from the user space. The enabled TRBE buffer area is a moving subset of > + * the allocated perf auxiliary buffer. > + */ > +static void trbe_pad_buf(struct perf_output_handle *handle, int len) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + u64 head = PERF_IDX2OFF(handle->head, buf); > + > + memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); > + if (!buf->snapshot) > + perf_aux_output_skip(handle, len); > +} > + > +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + /* > + * The ETE trace has alignment synchronization packets allowing > + * the decoder to reset in case of an overflow or corruption. > + * So we can use the entire buffer for the snapshot mode. > + */ > + return buf->nr_pages * PAGE_SIZE; > +} > + > +/* > + * TRBE Limit Calculation > + * > + * The following markers are used to illustrate various TRBE buffer situations. > + * > + * $$$$ - Data area, unconsumed captured trace data, not to be overridden > + * #### - Free area, enabled, trace will be written > + * %%%% - Free area, disabled, trace will not be written > + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped > + */ > +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + struct trbe_cpudata *cpudata = buf->cpudata; > + const u64 bufsize = buf->nr_pages * PAGE_SIZE; > + u64 limit = bufsize; > + u64 head, tail, wakeup; > + > + head = PERF_IDX2OFF(handle->head, buf); > + > + /* > + * head > + * ------->| > + * | > + * head TRBE align tail > + * +----|-------|---------------|-------+ > + * |$$$$|=======|###############|$$$$$$$| > + * +----|-------|---------------|-------+ > + * trbe_base trbe_base + nr_pages > + * > + * Perf aux buffer output head position can be misaligned depending on > + * various factors including user space reads. In case misaligned, head > + * needs to be aligned before TRBE can be configured. Pad the alignment > + * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools > + * and skip this section thus advancing the head. > + */ > + if (!IS_ALIGNED(head, cpudata->trbe_align)) { > + unsigned long delta = roundup(head, cpudata->trbe_align) - head; > + > + delta = min(delta, handle->size); > + trbe_pad_buf(handle, delta); > + head = PERF_IDX2OFF(handle->head, buf); > + } > + > + /* > + * head = tail (size = 0) > + * +----|-------------------------------+ > + * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ | > + * +----|-------------------------------+ > + * trbe_base trbe_base + nr_pages > + * > + * Perf aux buffer does not have any space for the driver to write into. > + * Just communicate trace truncation event to the user space by marking > + * it with PERF_AUX_FLAG_TRUNCATED. > + */ > + if (!handle->size) { > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > + return 0; > + } > + > + /* Compute the tail and wakeup indices now that we've aligned head */ > + tail = PERF_IDX2OFF(handle->head + handle->size, buf); > + wakeup = PERF_IDX2OFF(handle->wakeup, buf); > + > + /* > + * Lets calculate the buffer area which TRBE could write into. There > + * are three possible scenarios here. Limit needs to be aligned with > + * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the > + * unconsumed data. > + * > + * 1) head < tail > + * > + * head tail > + * +----|-----------------------|-------+ > + * |$$$$|#######################|$$$$$$$| > + * +----|-----------------------|-------+ > + * trbe_base limit trbe_base + nr_pages > + * > + * TRBE could write into [head..tail] area. Unless the tail is right at > + * the end of the buffer, neither an wrap around nor an IRQ is expected > + * while being enabled. > + * > + * 2) head == tail > + * > + * head = tail (size > 0) > + * +----|-------------------------------+ > + * |%%%%|###############################| > + * +----|-------------------------------+ > + * trbe_base limit = trbe_base + nr_pages > + * > + * TRBE should just write into [head..base + nr_pages] area even though > + * the entire buffer is empty. Reason being, when the trace reaches the > + * end of the buffer, it will just wrap around with an IRQ giving an > + * opportunity to reconfigure the buffer. > + * > + * 3) tail < head > + * > + * tail head > + * +----|-----------------------|-------+ > + * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| > + * +----|-----------------------|-------+ > + * trbe_base limit = trbe_base + nr_pages > + * > + * TRBE should just write into [head..base + nr_pages] area even though > + * the [trbe_base..tail] is also empty. Reason being, when the trace > + * reaches the end of the buffer, it will just wrap around with an IRQ > + * giving an opportunity to reconfigure the buffer. > + */ > + if (head < tail) > + limit = round_down(tail, PAGE_SIZE); > + > + /* > + * Wakeup may be arbitrarily far into the future. If it's not in the > + * current generation, either we'll wrap before hitting it, or it's > + * in the past and has been handled already. > + * > + * If there's a wakeup before we wrap, arrange to be woken up by the > + * page boundary following it. Keep the tail boundary if that's lower. > + * > + * head wakeup tail > + * +----|---------------|-------|-------+ > + * |$$$$|###############|%%%%%%%|$$$$$$$| > + * +----|---------------|-------|-------+ > + * trbe_base limit trbe_base + nr_pages > + */ > + if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) > + limit = min(limit, round_up(wakeup, PAGE_SIZE)); > + > + /* > + * There are two situation when this can happen i.e limit is before > + * the head and hence TRBE cannot be configured. > + * > + * 1) head < tail (aligned down with PAGE_SIZE) and also they are both > + * within the same PAGE size range. > + * > + * PAGE_SIZE > + * |----------------------| > + * > + * limit head tail > + * +------------|------|--------|-------+ > + * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| > + * +------------|------|--------|-------+ > + * trbe_base trbe_base + nr_pages > + * > + * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both > + * head and wakeup are within same PAGE size range. > + * > + * PAGE_SIZE > + * |----------------------| > + * > + * limit head wakeup tail > + * +----|------|-------|--------|-------+ > + * |$$$$$$$$$$$|=======|========|$$$$$$$| > + * +----|------|-------|--------|-------+ > + * trbe_base trbe_base + nr_pages > + */ > + if (limit > head) > + return limit; > + > + trbe_pad_buf(handle, handle->size); > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > + return 0; > +} > + > +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = perf_get_aux(handle); > + u64 limit = __trbe_normal_offset(handle); > + u64 head = PERF_IDX2OFF(handle->head, buf); > + > + /* > + * If the head is too close to the limit and we don't > + * have space for a meaningful run, we rather pad it > + * and start fresh. > + */ > + if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { > + trbe_pad_buf(handle, limit - head); > + limit = __trbe_normal_offset(handle); > + } > + return limit; > +} > + > +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + unsigned long offset; > + > + if (buf->snapshot) > + offset = trbe_snapshot_offset(handle); > + else > + offset = trbe_normal_offset(handle); > + return buf->trbe_base + offset; > +} > + > +static void clr_trbe_status(void) > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + > + WARN_ON(is_trbe_enabled()); > + trbsr &= ~TRBSR_IRQ; > + trbsr &= ~TRBSR_TRG; > + trbsr &= ~TRBSR_WRAP; > + trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); > + trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); > + trbsr &= ~TRBSR_STOP; > + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > +} > + > +static void set_trbe_limit_pointer_enabled(unsigned long addr) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + > + trblimitr &= ~TRBLIMITR_NVM; > + trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); > + trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); > + trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); > + > + /* > + * Fill trace buffer mode is used here while configuring the > + * TRBE for trace capture. In this particular mode, the trace > + * collection is stopped and a maintenance interrupt is raised > + * when the current write pointer wraps. This pause in trace > + * collection gives the software an opportunity to capture the > + * trace data in the interrupt handler, before reconfiguring > + * the TRBE. > + */ > + trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; > + > + /* > + * Trigger mode is not used here while configuring the TRBE for > + * the trace capture. Hence just keep this in the ignore mode. > + */ > + trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; > + trblimitr |= (addr & PAGE_MASK); > + > + trblimitr |= TRBLIMITR_ENABLE; > + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > +} > + > +static void trbe_enable_hw(struct trbe_buf *buf) > +{ > + WARN_ON(buf->trbe_write < buf->trbe_base); > + WARN_ON(buf->trbe_write >= buf->trbe_limit); > + set_trbe_disabled(); > + isb(); > + clr_trbe_status(); > + set_trbe_base_pointer(buf->trbe_base); > + set_trbe_write_pointer(buf->trbe_write); > + > + /* > + * Synchronize all the register updates > + * till now before enabling the TRBE. > + */ > + isb(); > + set_trbe_limit_pointer_enabled(buf->trbe_limit); > + > + /* Synchronize the TRBE enable event */ > + isb(); > +} > + > +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, > + struct perf_event *event, void **pages, > + int nr_pages, bool snapshot) > +{ > + struct trbe_buf *buf; > + struct page **pglist; > + int i; > + > + /* > + * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with > + * just a single page, there is not much room left while writing into > + * a partially filled TRBE buffer. Hence restrict the minimum buffer > + * size as two pages. > + */ I read this three times and event then I'm not sure to fully get it. I suggest re-working this paragraph. > + if (nr_pages < 2) > + return NULL; > + > + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); > + if (IS_ERR(buf)) > + return ERR_PTR(-ENOMEM); > + > + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); > + if (IS_ERR(pglist)) { > + kfree(buf); > + return ERR_PTR(-ENOMEM); > + } > + > + for (i = 0; i < nr_pages; i++) > + pglist[i] = virt_to_page(pages[i]); > + > + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); > + if (IS_ERR((void *)buf->trbe_base)) { Why not simply make buf->trbe_base a void * instead of having to do all this casting? And IS_ERR() doesn't work with vmap(). > + kfree(pglist); > + kfree(buf); > + return ERR_PTR(buf->trbe_base); > + } > + buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; > + buf->trbe_write = buf->trbe_base; > + buf->snapshot = snapshot; > + buf->nr_pages = nr_pages; > + buf->pages = pages; > + kfree(pglist); > + return buf; > +} > + > +void arm_trbe_free_buffer(void *config) > +{ > + struct trbe_buf *buf = config; > + > + vunmap((void *)buf->trbe_base); > + kfree(buf); > +} > + > +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, > + struct perf_output_handle *handle, > + void *config) > +{ > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > + struct trbe_buf *buf = config; > + unsigned long size, offset; > + > + WARN_ON(buf->cpudata != cpudata); > + WARN_ON(cpudata->cpu != smp_processor_id()); > + WARN_ON(cpudata->drvdata != drvdata); > + if (cpudata->mode != CS_MODE_PERF) > + return -EINVAL; > + > + /* > + * If the TRBE was disabled due to lack of space in the AUX buffer or a > + * spurious fault, the driver leaves it disabled, truncating the buffer. > + * Since the etm_perf driver expects to close out the AUX buffer, the > + * driver skips it. Thus, just pass in 0 size here to indicate that the > + * buffer was truncated. > + */ > + if (!is_trbe_enabled()) > + return 0; > + /* > + * perf handle structure needs to be shared with the TRBE IRQ handler for > + * capturing trace data and restarting the handle. There is a probability > + * of an undefined reference based crash when etm event is being stopped > + * while a TRBE IRQ also getting processed. This happens due the release > + * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping > + * the TRBE here will ensure that no IRQ could be generated when the perf > + * handle gets freed in etm_event_stop(). > + */ > + trbe_drain_and_disable_local(); > + offset = get_trbe_write_pointer() - get_trbe_base_pointer(); > + size = offset - PERF_IDX2OFF(handle->head, buf); > + if (buf->snapshot) > + handle->head += size; > + return size; > +} > + > +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) > +{ > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > + struct perf_output_handle *handle = data; > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + WARN_ON(cpudata->cpu != smp_processor_id()); > + WARN_ON(cpudata->drvdata != drvdata); > + if (mode != CS_MODE_PERF) > + return -EINVAL; > + > + *this_cpu_ptr(drvdata->handle) = handle; > + cpudata->buf = buf; > + cpudata->mode = mode; > + buf->cpudata = cpudata; > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > + if (buf->trbe_limit == buf->trbe_base) { > + trbe_stop_and_truncate_event(handle); > + return 0; > + } > + trbe_enable_hw(buf); > + return 0; > +} > + > +static int arm_trbe_disable(struct coresight_device *csdev) > +{ > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > + struct trbe_buf *buf = cpudata->buf; > + > + WARN_ON(buf->cpudata != cpudata); > + WARN_ON(cpudata->cpu != smp_processor_id()); > + WARN_ON(cpudata->drvdata != drvdata); > + if (cpudata->mode != CS_MODE_PERF) > + return -EINVAL; > + > + trbe_drain_and_disable_local(); > + buf->cpudata = NULL; > + cpudata->buf = NULL; > + cpudata->mode = CS_MODE_DISABLED; > + return 0; > +} > + > +static void trbe_handle_spurious(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > + if (buf->trbe_limit == buf->trbe_base) { > + trbe_drain_and_disable_local(); > + return; > + } > + trbe_enable_hw(buf); > +} > + > +static void trbe_handle_overflow(struct perf_output_handle *handle) > +{ > + struct perf_event *event = handle->event; > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + unsigned long offset, size; > + struct etm_event_data *event_data; > + > + offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); > + size = offset - PERF_IDX2OFF(handle->head, buf); > + if (buf->snapshot) > + handle->head = offset; > + perf_aux_output_end(handle, size); > + > + event_data = perf_aux_output_begin(handle, event); > + if (!event_data) { > + trbe_drain_and_disable_local(); > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > + return; > + } > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > + if (buf->trbe_limit == buf->trbe_base) { > + trbe_stop_and_truncate_event(handle); > + return; > + } > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; > + trbe_enable_hw(buf); > +} > + > +static bool is_perf_trbe(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + struct trbe_cpudata *cpudata = buf->cpudata; > + struct trbe_drvdata *drvdata = cpudata->drvdata; > + int cpu = smp_processor_id(); > + > + WARN_ON(buf->trbe_base != get_trbe_base_pointer()); > + WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); > + > + if (cpudata->mode != CS_MODE_PERF) > + return false; > + > + if (cpudata->cpu != cpu) > + return false; > + > + if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) > + return false; > + > + return true; > +} > + > +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + int ec = get_trbe_ec(trbsr); > + int bsc = get_trbe_bsc(trbsr); > + > + WARN_ON(is_trbe_running(trbsr)); > + if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) > + return TRBE_FAULT_ACT_FATAL; > + > + if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) > + return TRBE_FAULT_ACT_FATAL; > + > + if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { > + if (get_trbe_write_pointer() == get_trbe_base_pointer()) > + return TRBE_FAULT_ACT_WRAP; > + } > + return TRBE_FAULT_ACT_SPURIOUS; > +} > + > +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) > +{ > + struct perf_output_handle **handle_ptr = dev; > + struct perf_output_handle *handle = *handle_ptr; > + enum trbe_fault_action act; > + > + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); > + clr_trbe_irq(); > + > + /* > + * Ensure the trace is visible to the CPUs and > + * any external aborts have been resolved. > + */ > + trbe_drain_buffer(); > + isb(); > + > + if (!perf_get_aux(handle)) > + return IRQ_NONE; > + > + if (!is_perf_trbe(handle)) > + return IRQ_NONE; > + > + irq_work_run(); > + > + act = trbe_get_fault_act(handle); > + switch (act) { > + case TRBE_FAULT_ACT_WRAP: > + trbe_handle_overflow(handle); > + break; > + case TRBE_FAULT_ACT_SPURIOUS: > + trbe_handle_spurious(handle); > + break; > + case TRBE_FAULT_ACT_FATAL: > + trbe_stop_and_truncate_event(handle); > + break; > + } > + return IRQ_HANDLED; > +} > + > +static const struct coresight_ops_sink arm_trbe_sink_ops = { > + .enable = arm_trbe_enable, > + .disable = arm_trbe_disable, > + .alloc_buffer = arm_trbe_alloc_buffer, > + .free_buffer = arm_trbe_free_buffer, > + .update_buffer = arm_trbe_update_buffer, > +}; > + > +static const struct coresight_ops arm_trbe_cs_ops = { > + .sink_ops = &arm_trbe_sink_ops, > +}; > + > +static ssize_t align_show(struct device *dev, struct device_attribute *attr, char *buf) > +{ > + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > + > + return sprintf(buf, "%llx\n", cpudata->trbe_align); > +} > +static DEVICE_ATTR_RO(align); > + > +static ssize_t dbm_show(struct device *dev, struct device_attribute *attr, char *buf) > +{ > + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > + > + return sprintf(buf, "%d\n", cpudata->trbe_dbm); > +} > +static DEVICE_ATTR_RO(dbm); What does "dbm" stand for? Looking at the documentation for TRBIDR_EL1.F, I don't see what "dbm" relates to. > + > +static struct attribute *arm_trbe_attrs[] = { > + &dev_attr_align.attr, > + &dev_attr_dbm.attr, > + NULL, > +}; > + > +static const struct attribute_group arm_trbe_group = { > + .attrs = arm_trbe_attrs, > +}; > + > +static const struct attribute_group *arm_trbe_groups[] = { > + &arm_trbe_group, > + NULL, > +}; > + > +static void arm_trbe_probe_coresight_cpu(void *info) > +{ > + struct trbe_drvdata *drvdata = info; > + struct coresight_desc desc = { 0 }; > + int cpu = smp_processor_id(); > + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); > + struct device *dev; > + > + if (WARN_ON(!cpudata)) > + goto cpu_clear; > + > + if (trbe_csdev) > + return; > + > + cpudata->cpu = smp_processor_id(); > + cpudata->drvdata = drvdata; > + dev = &cpudata->drvdata->pdev->dev; > + > + if (!is_trbe_available()) { > + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + > + if (!is_trbe_programmable(trbidr)) { > + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); > + if (IS_ERR(desc.name)) > + goto cpu_clear; > + > + desc.type = CORESIGHT_DEV_TYPE_SINK; > + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; > + desc.ops = &arm_trbe_cs_ops; > + desc.pdata = dev_get_platdata(dev); > + desc.groups = arm_trbe_groups; > + desc.dev = dev; > + trbe_csdev = coresight_register(&desc); > + if (IS_ERR(trbe_csdev)) > + goto cpu_clear; > + > + dev_set_drvdata(&trbe_csdev->dev, cpudata); > + cpudata->trbe_dbm = get_trbe_flag_update(trbidr); > + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); > + if (cpudata->trbe_align > SZ_2K) { > + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + per_cpu(csdev_sink, cpu) = trbe_csdev; > + trbe_reset_local(); > + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > + return; > +cpu_clear: > + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); > +} > + > +static void arm_trbe_remove_coresight_cpu(void *info) > +{ > + int cpu = smp_processor_id(); > + struct trbe_drvdata *drvdata = info; > + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > + > + if (trbe_csdev) { > + coresight_unregister(trbe_csdev); > + cpudata->drvdata = NULL; > + per_cpu(csdev_sink, cpu) = NULL; > + } > + disable_percpu_irq(drvdata->irq); > + trbe_reset_local(); > +} > + > +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) > +{ > + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); > + if (IS_ERR(drvdata->cpudata)) > + return PTR_ERR(drvdata->cpudata); > + > + arm_trbe_probe_coresight_cpu(drvdata); > + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); > + return 0; > +} > + > +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) > +{ > + arm_trbe_remove_coresight_cpu(drvdata); > + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); > + free_percpu(drvdata->cpudata); > + return 0; > +} > + > +static int arm_trbe_cpu_startup(unsigned int cpu, struct hlist_node *node) > +{ > + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > + > + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > + if (!per_cpu(csdev_sink, cpu)) { > + arm_trbe_probe_coresight_cpu(drvdata); > + } else { > + trbe_reset_local(); > + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > + } > + } > + return 0; > +} > + > +static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node) > +{ > + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > + > + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > + disable_percpu_irq(drvdata->irq); > + trbe_reset_local(); > + } > + return 0; > +} > + > +static int arm_trbe_probe_cpuhp(struct trbe_drvdata *drvdata) > +{ > + enum cpuhp_state trbe_online; > + > + trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, > + arm_trbe_cpu_startup, arm_trbe_cpu_teardown); Indentation trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, arm_trbe_cpu_startup, arm_trbe_cpu_teardown); > + if (trbe_online < 0) > + return -EINVAL Wrong error code > + > + if (cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node)) > + return -EINVAL; Wrong error code. Even if the callbacks aren't called it might be a good idea to properly cleanup the with cpuhp_remove_multi_state(). More to come tomorrow. Thanks, Mathieu > + > + drvdata->trbe_online = trbe_online; > + return 0; > +} > + > +static void arm_trbe_remove_cpuhp(struct trbe_drvdata *drvdata) > +{ > + cpuhp_remove_multi_state(drvdata->trbe_online); > +} > + > +static int arm_trbe_probe_irq(struct platform_device *pdev, > + struct trbe_drvdata *drvdata) > +{ > + drvdata->irq = platform_get_irq(pdev, 0); > + if (!drvdata->irq) { > + pr_err("IRQ not found for the platform device\n"); > + return -ENXIO; > + } > + > + if (!irq_is_percpu(drvdata->irq)) { > + pr_err("IRQ is not a PPI\n"); > + return -EINVAL; > + } > + > + if (irq_get_percpu_devid_partition(drvdata->irq, &drvdata->supported_cpus)) > + return -EINVAL; > + > + drvdata->handle = alloc_percpu(typeof(*drvdata->handle)); > + if (!drvdata->handle) > + return -ENOMEM; > + > + if (request_percpu_irq(drvdata->irq, arm_trbe_irq_handler, DRVNAME, drvdata->handle)) { > + free_percpu(drvdata->handle); > + return -EINVAL; > + } > + return 0; > +} > + > +static void arm_trbe_remove_irq(struct trbe_drvdata *drvdata) > +{ > + free_percpu_irq(drvdata->irq, drvdata->handle); > + free_percpu(drvdata->handle); > +} > + > +static int arm_trbe_device_probe(struct platform_device *pdev) > +{ > + struct coresight_platform_data *pdata; > + struct trbe_drvdata *drvdata; > + struct device *dev = &pdev->dev; > + int ret; > + > + drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL); > + if (IS_ERR(drvdata)) > + return -ENOMEM; > + > + pdata = coresight_get_platform_data(dev); > + if (IS_ERR(pdata)) { > + kfree(drvdata); > + return -ENOMEM; > + } > + > + dev_set_drvdata(dev, drvdata); > + dev->platform_data = pdata; > + drvdata->pdev = pdev; > + ret = arm_trbe_probe_irq(pdev, drvdata); > + if (ret) > + goto irq_failed; > + > + ret = arm_trbe_probe_coresight(drvdata); > + if (ret) > + goto probe_failed; > + > + ret = arm_trbe_probe_cpuhp(drvdata); > + if (ret) > + goto cpuhp_failed; > + > + return 0; > +cpuhp_failed: > + arm_trbe_remove_coresight(drvdata); > +probe_failed: > + arm_trbe_remove_irq(drvdata); > +irq_failed: > + kfree(pdata); > + kfree(drvdata); > + return ret; > +} > + > +static int arm_trbe_device_remove(struct platform_device *pdev) > +{ > + struct coresight_platform_data *pdata = dev_get_platdata(&pdev->dev); > + struct trbe_drvdata *drvdata = platform_get_drvdata(pdev); > + > + arm_trbe_remove_coresight(drvdata); > + arm_trbe_remove_cpuhp(drvdata); Shouldn't this be called before arm_trbe_remove_coresight()? > + arm_trbe_remove_irq(drvdata); > + kfree(pdata); > + kfree(drvdata); > + return 0; > +} > + > +static const struct of_device_id arm_trbe_of_match[] = { > + { .compatible = "arm,trace-buffer-extension"}, > + {}, > +}; > +MODULE_DEVICE_TABLE(of, arm_trbe_of_match); > + > +static struct platform_driver arm_trbe_driver = { > + .driver = { > + .name = DRVNAME, > + .of_match_table = of_match_ptr(arm_trbe_of_match), > + .suppress_bind_attrs = true, > + }, > + .probe = arm_trbe_device_probe, > + .remove = arm_trbe_device_remove, > +}; > + > +static int __init arm_trbe_init(void) > +{ > + int ret; > + > + if (arm64_kernel_unmapped_at_el0()) { > + pr_err("TRBE wouldn't work if kernel gets unmapped at EL0\n"); > + return -EOPNOTSUPP; > + } > + > + ret = platform_driver_register(&arm_trbe_driver); > + if (!ret) > + return 0; > + > + pr_err("Error registering %s platform driver\n", DRVNAME); > + return ret; > +} > + > +static void __exit arm_trbe_exit(void) > +{ > + platform_driver_unregister(&arm_trbe_driver); > +} > +module_init(arm_trbe_init); > +module_exit(arm_trbe_exit); > + > +MODULE_AUTHOR("Anshuman Khandual <anshuman.khandual@arm.com>"); > +MODULE_DESCRIPTION("Arm Trace Buffer Extension (TRBE) driver"); > +MODULE_LICENSE("GPL v2"); > diff --git a/drivers/hwtracing/coresight/coresight-trbe.h b/drivers/hwtracing/coresight/coresight-trbe.h > new file mode 100644 > index 0000000..43308bc > --- /dev/null > +++ b/drivers/hwtracing/coresight/coresight-trbe.h > @@ -0,0 +1,160 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * This contains all required hardware related helper functions for > + * Trace Buffer Extension (TRBE) driver in the coresight framework. > + * > + * Copyright (C) 2020 ARM Ltd. > + * > + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > + */ > +#include <linux/coresight.h> > +#include <linux/device.h> > +#include <linux/irq.h> > +#include <linux/kernel.h> > +#include <linux/of.h> > +#include <linux/platform_device.h> > +#include <linux/smp.h> > + > +#include "coresight-etm-perf.h" > + > +DECLARE_PER_CPU(struct coresight_device *, csdev_sink); > + > +static inline bool is_trbe_available(void) > +{ > + u64 aa64dfr0 = read_sysreg_s(SYS_ID_AA64DFR0_EL1); > + unsigned int trbe = cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_TRBE_SHIFT); > + > + return trbe >= 0b0001; > +} > + > +static inline bool is_trbe_enabled(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + return trblimitr & TRBLIMITR_ENABLE; > +} > + > +#define TRBE_EC_OTHERS 0 > +#define TRBE_EC_STAGE1_ABORT 36 > +#define TRBE_EC_STAGE2_ABORT 37 > + > +static inline int get_trbe_ec(u64 trbsr) > +{ > + return (trbsr >> TRBSR_EC_SHIFT) & TRBSR_EC_MASK; > +} > + > +#define TRBE_BSC_NOT_STOPPED 0 > +#define TRBE_BSC_FILLED 1 > +#define TRBE_BSC_TRIGGERED 2 > + > +static inline int get_trbe_bsc(u64 trbsr) > +{ > + return (trbsr >> TRBSR_BSC_SHIFT) & TRBSR_BSC_MASK; > +} > + > +static inline void clr_trbe_irq(void) > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + > + trbsr &= ~TRBSR_IRQ; > + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > +} > + > +static inline bool is_trbe_irq(u64 trbsr) > +{ > + return trbsr & TRBSR_IRQ; > +} > + > +static inline bool is_trbe_trg(u64 trbsr) > +{ > + return trbsr & TRBSR_TRG; > +} > + > +static inline bool is_trbe_wrap(u64 trbsr) > +{ > + return trbsr & TRBSR_WRAP; > +} > + > +static inline bool is_trbe_abort(u64 trbsr) > +{ > + return trbsr & TRBSR_ABORT; > +} > + > +static inline bool is_trbe_running(u64 trbsr) > +{ > + return !(trbsr & TRBSR_STOP); > +} > + > +#define TRBE_TRIG_MODE_STOP 0 > +#define TRBE_TRIG_MODE_IRQ 1 > +#define TRBE_TRIG_MODE_IGNORE 3 > + > +#define TRBE_FILL_MODE_FILL 0 > +#define TRBE_FILL_MODE_WRAP 1 > +#define TRBE_FILL_MODE_CIRCULAR_BUFFER 3 > + > +static inline void set_trbe_disabled(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + trblimitr &= ~TRBLIMITR_ENABLE; > + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > +} > + > +static inline bool get_trbe_flag_update(u64 trbidr) > +{ > + return trbidr & TRBIDR_FLAG; > +} > + > +static inline bool is_trbe_programmable(u64 trbidr) > +{ > + return !(trbidr & TRBIDR_PROG); > +} > + > +static inline int get_trbe_address_align(u64 trbidr) > +{ > + return (trbidr >> TRBIDR_ALIGN_SHIFT) & TRBIDR_ALIGN_MASK; > +} > + > +static inline unsigned long get_trbe_write_pointer(void) > +{ > + u64 trbptr = read_sysreg_s(SYS_TRBPTR_EL1); > + unsigned long addr = (trbptr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; > + > + return addr; > +} > + > +static inline void set_trbe_write_pointer(unsigned long addr) > +{ > + WARN_ON(is_trbe_enabled()); > + addr = (addr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; > + write_sysreg_s(addr, SYS_TRBPTR_EL1); > +} > + > +static inline unsigned long get_trbe_limit_pointer(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + unsigned long limit = (trblimitr >> TRBLIMITR_LIMIT_SHIFT) & TRBLIMITR_LIMIT_MASK; > + unsigned long addr = limit << TRBLIMITR_LIMIT_SHIFT; > + > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + return addr; > +} > + > +static inline unsigned long get_trbe_base_pointer(void) > +{ > + u64 trbbaser = read_sysreg_s(SYS_TRBBASER_EL1); > + unsigned long addr = (trbbaser >> TRBBASER_BASE_SHIFT) & TRBBASER_BASE_MASK; > + > + addr = addr << TRBBASER_BASE_SHIFT; > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + return addr; > +} > + > +static inline void set_trbe_base_pointer(unsigned long addr) > +{ > + WARN_ON(is_trbe_enabled()); > + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + write_sysreg_s(addr, SYS_TRBBASER_EL1); > +} > -- > 2.7.4 >
On 2/12/21 12:30 AM, Mathieu Poirier wrote: > On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: >> Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is >> accessible via the system registers. The TRBE supports different addressing >> modes including CPU virtual address and buffer modes including the circular >> buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), >> an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the >> access to the trace buffer could be prohibited by a higher exception level >> (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU >> private interrupt (PPI) on address translation errors and when the buffer >> is full. Overall implementation here is inspired from the Arm SPE driver. >> >> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> >> Cc: Mike Leach <mike.leach@linaro.org> >> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> >> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> >> --- >> Changes in V3: >> >> - Added new DT bindings document TRBE.yaml >> - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 >> - Dropped isb() from trbe_reset_local() >> - Dropped gap between (void *) and buf->trbe_base >> - Changed 'int' to 'unsigned int' in is_trbe_available() >> - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), >> set_trbe_enabled() and set_trbe_limit_pointer() >> - Changed get_trbe_flag_update(), is_trbe_programmable() and >> get_trbe_address_align() to accept TRBIDR value >> - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), >> is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value >> - Dropped snapshot mode condition in arm_trbe_alloc_buffer() >> - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled >> - Compute trbe_limit before trbe_write to get the updated handle >> - Added trbe_stop_and_truncate_event() >> - Dropped trbe_handle_fatal() >> >> Documentation/trace/coresight/coresight-trbe.rst | 39 + >> arch/arm64/include/asm/sysreg.h | 1 + >> drivers/hwtracing/coresight/Kconfig | 11 + >> drivers/hwtracing/coresight/Makefile | 1 + >> drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ >> drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ >> 6 files changed, 1235 insertions(+) >> create mode 100644 Documentation/trace/coresight/coresight-trbe.rst >> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c >> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h >> >> diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst >> new file mode 100644 >> index 0000000..1cbb819 >> --- /dev/null >> +++ b/Documentation/trace/coresight/coresight-trbe.rst >> @@ -0,0 +1,39 @@ >> +.. SPDX-License-Identifier: GPL-2.0 >> + >> +============================== >> +Trace Buffer Extension (TRBE). >> +============================== >> + >> + :Author: Anshuman Khandual <anshuman.khandual@arm.com> >> + :Date: November 2020 >> + >> +Hardware Description >> +-------------------- >> + >> +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system >> +memory, CPU traces generated from a corresponding percpu tracing unit. This >> +gets plugged in as a coresight sink device because the corresponding trace >> +genarators (ETE), are plugged in as source device. >> + >> +The TRBE is not compliant to CoreSight architecture specifications, but is >> +driven via the CoreSight driver framework to support the ETE (which is >> +CoreSight compliant) integration. >> + >> +Sysfs files and directories >> +--------------------------- >> + >> +The TRBE devices appear on the existing coresight bus alongside the other >> +coresight devices:: >> + >> + >$ ls /sys/bus/coresight/devices >> + trbe0 trbe1 trbe2 trbe3 >> + >> +The ``trbe<N>`` named TRBEs are associated with a CPU.:: >> + >> + >$ ls /sys/bus/coresight/devices/trbe0/ >> + align dbm >> + >> +*Key file items are:-* >> + * ``align``: TRBE write pointer alignment >> + * ``dbm``: TRBE updates memory with access and dirty flags >> + >> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h >> index 85ae4db..9e2e9b7 100644 >> --- a/arch/arm64/include/asm/sysreg.h >> +++ b/arch/arm64/include/asm/sysreg.h >> @@ -97,6 +97,7 @@ >> #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) >> #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) >> #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) >> +#define TSB_CSYNC __emit_inst(0xd503225f) >> >> #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) >> #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) >> diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig >> index f154ae7..aa657ab 100644 >> --- a/drivers/hwtracing/coresight/Kconfig >> +++ b/drivers/hwtracing/coresight/Kconfig >> @@ -164,6 +164,17 @@ config CORESIGHT_CTI >> To compile this driver as a module, choose M here: the >> module will be called coresight-cti. >> >> +config CORESIGHT_TRBE >> + bool "Trace Buffer Extension (TRBE) driver" >> + depends on ARM64 >> + help >> + This driver provides support for percpu Trace Buffer Extension (TRBE). >> + TRBE always needs to be used along with it's corresponding percpu ETE >> + component. ETE generates trace data which is then captured with TRBE. >> + Unlike traditional sink devices, TRBE is a CPU feature accessible via >> + system registers. But it's explicit dependency with trace unit (ETE) >> + requires it to be plugged in as a coresight sink device. >> + >> config CORESIGHT_CTI_INTEGRATION_REGS >> bool "Access CTI CoreSight Integration Registers" >> depends on CORESIGHT_CTI >> diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile >> index f20e357..d608165 100644 >> --- a/drivers/hwtracing/coresight/Makefile >> +++ b/drivers/hwtracing/coresight/Makefile >> @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o >> obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o >> obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o >> obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o >> +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o >> coresight-cti-y := coresight-cti-core.o coresight-cti-platform.o \ >> coresight-cti-sysfs.o >> diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c >> new file mode 100644 >> index 0000000..1464d8b >> --- /dev/null >> +++ b/drivers/hwtracing/coresight/coresight-trbe.c >> @@ -0,0 +1,1023 @@ >> +// SPDX-License-Identifier: GPL-2.0 >> +/* >> + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight >> + * sink device could then pair with an appropriate per-cpu coresight source >> + * device (ETE) thus generating required trace data. Trace can be enabled >> + * via the perf framework. >> + * >> + * Copyright (C) 2020 ARM Ltd. >> + * >> + * Author: Anshuman Khandual <anshuman.khandual@arm.com> >> + */ >> +#define DRVNAME "arm_trbe" >> + >> +#define pr_fmt(fmt) DRVNAME ": " fmt >> + >> +#include "coresight-trbe.h" >> + >> +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) >> + >> +/* >> + * A padding packet that will help the user space tools >> + * in skipping relevant sections in the captured trace >> + * data which could not be decoded. TRBE doesn't support >> + * formatting the trace data, unlike the legacy CoreSight >> + * sinks and thus we use ETE trace packets to pad the >> + * sections of the buffer. >> + */ >> +#define ETE_IGNORE_PACKET 0x70 >> + >> +/* >> + * Minimum amount of meaningful trace will contain: >> + * A-Sync, Trace Info, Trace On, Address, Atom. >> + * This is about 44bytes of ETE trace. To be on >> + * the safer side, we assume 64bytes is the minimum >> + * space required for a meaningful session, before >> + * we hit a "WRAP" event. >> + */ >> +#define TRBE_TRACE_MIN_BUF_SIZE 64 >> + >> +enum trbe_fault_action { >> + TRBE_FAULT_ACT_WRAP, >> + TRBE_FAULT_ACT_SPURIOUS, >> + TRBE_FAULT_ACT_FATAL, >> +}; >> + >> +struct trbe_buf { >> + unsigned long trbe_base; >> + unsigned long trbe_limit; >> + unsigned long trbe_write; >> + int nr_pages; >> + void **pages; >> + bool snapshot; >> + struct trbe_cpudata *cpudata; >> +}; >> + >> +struct trbe_cpudata { >> + bool trbe_dbm; >> + u64 trbe_align; >> + int cpu; >> + enum cs_mode mode; >> + struct trbe_buf *buf; >> + struct trbe_drvdata *drvdata; >> +}; >> + >> +struct trbe_drvdata { >> + struct trbe_cpudata __percpu *cpudata; >> + struct perf_output_handle __percpu **handle; >> + struct hlist_node hotplug_node; >> + int irq; >> + cpumask_t supported_cpus; >> + enum cpuhp_state trbe_online; >> + struct platform_device *pdev; >> +}; >> + >> +static int trbe_alloc_node(struct perf_event *event) >> +{ >> + if (event->cpu == -1) >> + return NUMA_NO_NODE; >> + return cpu_to_node(event->cpu); >> +} >> + >> +static void trbe_drain_buffer(void) >> +{ >> + asm(TSB_CSYNC); >> + dsb(nsh); >> +} >> + >> +static void trbe_drain_and_disable_local(void) >> +{ >> + trbe_drain_buffer(); >> + write_sysreg_s(0, SYS_TRBLIMITR_EL1); >> + isb(); >> +} >> + >> +static void trbe_reset_local(void) >> +{ >> + trbe_drain_and_disable_local(); >> + write_sysreg_s(0, SYS_TRBPTR_EL1); >> + write_sysreg_s(0, SYS_TRBBASER_EL1); >> + write_sysreg_s(0, SYS_TRBSR_EL1); >> +} >> + >> +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + >> + /* >> + * We cannot proceed with the buffer collection and we >> + * do not have any data for the current session. The >> + * etm_perf driver expects to close out the aux_buffer >> + * at event_stop(). So disable the TRBE here and leave >> + * the update_buffer() to return a 0 size. >> + */ >> + trbe_drain_and_disable_local(); >> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; >> +} >> + >> +/* >> + * TRBE Buffer Management >> + * >> + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, >> + * it starts writing trace data from the write pointer onward till the limit pointer. >> + * When the write pointer reaches the address just before the limit pointer, it gets >> + * wrapped around again to the base pointer. This is called a TRBE wrap event, which >> + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver >> + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ >> + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and >> + * LIMIT pointers. >> + * >> + * Wrap around with an IRQ >> + * ------ < ------ < ------- < ----- < ----- >> + * | | >> + * ------ > ------ > ------- > ----- > ----- >> + * >> + * +---------------+-----------------------+ >> + * | | | >> + * +---------------+-----------------------+ >> + * Base Pointer Write Pointer Limit Pointer >> + * >> + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write >> + * pointer can be aligned to the implementation defined TRBE trace buffer alignment >> + * as captured in trbe_cpudata->trbe_align. >> + * >> + * >> + * head tail wakeup >> + * +---------------------------------------+----- ~ ~ ------ >> + * |$$$$$$$|################|$$$$$$$$$$$$$$| | >> + * +---------------------------------------+----- ~ ~ ------ >> + * Base Pointer Write Pointer Limit Pointer >> + * >> + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing >> + * values which tracks all the driver writes and user reads from the perf auxiliary >> + * buffer. Generally [head..tail] is the area where the driver can write into unless >> + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and >> + * configured depending on the perf_output_handle indices, so that the driver does >> + * not override into areas in the perf auxiliary buffer which is being or yet to be >> + * consumed from the user space. The enabled TRBE buffer area is a moving subset of >> + * the allocated perf auxiliary buffer. >> + */ >> +static void trbe_pad_buf(struct perf_output_handle *handle, int len) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + u64 head = PERF_IDX2OFF(handle->head, buf); >> + >> + memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); >> + if (!buf->snapshot) >> + perf_aux_output_skip(handle, len); >> +} >> + >> +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + >> + /* >> + * The ETE trace has alignment synchronization packets allowing >> + * the decoder to reset in case of an overflow or corruption. >> + * So we can use the entire buffer for the snapshot mode. >> + */ >> + return buf->nr_pages * PAGE_SIZE; >> +} >> + >> +/* >> + * TRBE Limit Calculation >> + * >> + * The following markers are used to illustrate various TRBE buffer situations. >> + * >> + * $$$$ - Data area, unconsumed captured trace data, not to be overridden >> + * #### - Free area, enabled, trace will be written >> + * %%%% - Free area, disabled, trace will not be written >> + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped >> + */ >> +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + struct trbe_cpudata *cpudata = buf->cpudata; >> + const u64 bufsize = buf->nr_pages * PAGE_SIZE; >> + u64 limit = bufsize; >> + u64 head, tail, wakeup; >> + >> + head = PERF_IDX2OFF(handle->head, buf); >> + >> + /* >> + * head >> + * ------->| >> + * | >> + * head TRBE align tail >> + * +----|-------|---------------|-------+ >> + * |$$$$|=======|###############|$$$$$$$| >> + * +----|-------|---------------|-------+ >> + * trbe_base trbe_base + nr_pages >> + * >> + * Perf aux buffer output head position can be misaligned depending on >> + * various factors including user space reads. In case misaligned, head >> + * needs to be aligned before TRBE can be configured. Pad the alignment >> + * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools >> + * and skip this section thus advancing the head. >> + */ >> + if (!IS_ALIGNED(head, cpudata->trbe_align)) { >> + unsigned long delta = roundup(head, cpudata->trbe_align) - head; >> + >> + delta = min(delta, handle->size); >> + trbe_pad_buf(handle, delta); >> + head = PERF_IDX2OFF(handle->head, buf); >> + } >> + >> + /* >> + * head = tail (size = 0) >> + * +----|-------------------------------+ >> + * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ | >> + * +----|-------------------------------+ >> + * trbe_base trbe_base + nr_pages >> + * >> + * Perf aux buffer does not have any space for the driver to write into. >> + * Just communicate trace truncation event to the user space by marking >> + * it with PERF_AUX_FLAG_TRUNCATED. >> + */ >> + if (!handle->size) { >> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >> + return 0; >> + } >> + >> + /* Compute the tail and wakeup indices now that we've aligned head */ >> + tail = PERF_IDX2OFF(handle->head + handle->size, buf); >> + wakeup = PERF_IDX2OFF(handle->wakeup, buf); >> + >> + /* >> + * Lets calculate the buffer area which TRBE could write into. There >> + * are three possible scenarios here. Limit needs to be aligned with >> + * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the >> + * unconsumed data. >> + * >> + * 1) head < tail >> + * >> + * head tail >> + * +----|-----------------------|-------+ >> + * |$$$$|#######################|$$$$$$$| >> + * +----|-----------------------|-------+ >> + * trbe_base limit trbe_base + nr_pages >> + * >> + * TRBE could write into [head..tail] area. Unless the tail is right at >> + * the end of the buffer, neither an wrap around nor an IRQ is expected >> + * while being enabled. >> + * >> + * 2) head == tail >> + * >> + * head = tail (size > 0) >> + * +----|-------------------------------+ >> + * |%%%%|###############################| >> + * +----|-------------------------------+ >> + * trbe_base limit = trbe_base + nr_pages >> + * >> + * TRBE should just write into [head..base + nr_pages] area even though >> + * the entire buffer is empty. Reason being, when the trace reaches the >> + * end of the buffer, it will just wrap around with an IRQ giving an >> + * opportunity to reconfigure the buffer. >> + * >> + * 3) tail < head >> + * >> + * tail head >> + * +----|-----------------------|-------+ >> + * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| >> + * +----|-----------------------|-------+ >> + * trbe_base limit = trbe_base + nr_pages >> + * >> + * TRBE should just write into [head..base + nr_pages] area even though >> + * the [trbe_base..tail] is also empty. Reason being, when the trace >> + * reaches the end of the buffer, it will just wrap around with an IRQ >> + * giving an opportunity to reconfigure the buffer. >> + */ >> + if (head < tail) >> + limit = round_down(tail, PAGE_SIZE); >> + >> + /* >> + * Wakeup may be arbitrarily far into the future. If it's not in the >> + * current generation, either we'll wrap before hitting it, or it's >> + * in the past and has been handled already. >> + * >> + * If there's a wakeup before we wrap, arrange to be woken up by the >> + * page boundary following it. Keep the tail boundary if that's lower. >> + * >> + * head wakeup tail >> + * +----|---------------|-------|-------+ >> + * |$$$$|###############|%%%%%%%|$$$$$$$| >> + * +----|---------------|-------|-------+ >> + * trbe_base limit trbe_base + nr_pages >> + */ >> + if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) >> + limit = min(limit, round_up(wakeup, PAGE_SIZE)); >> + >> + /* >> + * There are two situation when this can happen i.e limit is before >> + * the head and hence TRBE cannot be configured. >> + * >> + * 1) head < tail (aligned down with PAGE_SIZE) and also they are both >> + * within the same PAGE size range. >> + * >> + * PAGE_SIZE >> + * |----------------------| >> + * >> + * limit head tail >> + * +------------|------|--------|-------+ >> + * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| >> + * +------------|------|--------|-------+ >> + * trbe_base trbe_base + nr_pages >> + * >> + * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both >> + * head and wakeup are within same PAGE size range. >> + * >> + * PAGE_SIZE >> + * |----------------------| >> + * >> + * limit head wakeup tail >> + * +----|------|-------|--------|-------+ >> + * |$$$$$$$$$$$|=======|========|$$$$$$$| >> + * +----|------|-------|--------|-------+ >> + * trbe_base trbe_base + nr_pages >> + */ >> + if (limit > head) >> + return limit; >> + >> + trbe_pad_buf(handle, handle->size); >> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >> + return 0; >> +} >> + >> +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = perf_get_aux(handle); >> + u64 limit = __trbe_normal_offset(handle); >> + u64 head = PERF_IDX2OFF(handle->head, buf); >> + >> + /* >> + * If the head is too close to the limit and we don't >> + * have space for a meaningful run, we rather pad it >> + * and start fresh. >> + */ >> + if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { >> + trbe_pad_buf(handle, limit - head); >> + limit = __trbe_normal_offset(handle); >> + } >> + return limit; >> +} >> + >> +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + unsigned long offset; >> + >> + if (buf->snapshot) >> + offset = trbe_snapshot_offset(handle); >> + else >> + offset = trbe_normal_offset(handle); >> + return buf->trbe_base + offset; >> +} >> + >> +static void clr_trbe_status(void) >> +{ >> + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); >> + >> + WARN_ON(is_trbe_enabled()); >> + trbsr &= ~TRBSR_IRQ; >> + trbsr &= ~TRBSR_TRG; >> + trbsr &= ~TRBSR_WRAP; >> + trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); >> + trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); >> + trbsr &= ~TRBSR_STOP; >> + write_sysreg_s(trbsr, SYS_TRBSR_EL1); >> +} >> + >> +static void set_trbe_limit_pointer_enabled(unsigned long addr) >> +{ >> + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); >> + >> + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); >> + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); >> + >> + trblimitr &= ~TRBLIMITR_NVM; >> + trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); >> + trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); >> + trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); >> + >> + /* >> + * Fill trace buffer mode is used here while configuring the >> + * TRBE for trace capture. In this particular mode, the trace >> + * collection is stopped and a maintenance interrupt is raised >> + * when the current write pointer wraps. This pause in trace >> + * collection gives the software an opportunity to capture the >> + * trace data in the interrupt handler, before reconfiguring >> + * the TRBE. >> + */ >> + trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; >> + >> + /* >> + * Trigger mode is not used here while configuring the TRBE for >> + * the trace capture. Hence just keep this in the ignore mode. >> + */ >> + trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; >> + trblimitr |= (addr & PAGE_MASK); >> + >> + trblimitr |= TRBLIMITR_ENABLE; >> + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); >> +} >> + >> +static void trbe_enable_hw(struct trbe_buf *buf) >> +{ >> + WARN_ON(buf->trbe_write < buf->trbe_base); >> + WARN_ON(buf->trbe_write >= buf->trbe_limit); >> + set_trbe_disabled(); >> + isb(); >> + clr_trbe_status(); >> + set_trbe_base_pointer(buf->trbe_base); >> + set_trbe_write_pointer(buf->trbe_write); >> + >> + /* >> + * Synchronize all the register updates >> + * till now before enabling the TRBE. >> + */ >> + isb(); >> + set_trbe_limit_pointer_enabled(buf->trbe_limit); >> + >> + /* Synchronize the TRBE enable event */ >> + isb(); >> +} >> + >> +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, >> + struct perf_event *event, void **pages, >> + int nr_pages, bool snapshot) >> +{ >> + struct trbe_buf *buf; >> + struct page **pglist; >> + int i; >> + >> + /* >> + * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with >> + * just a single page, there is not much room left while writing into >> + * a partially filled TRBE buffer. Hence restrict the minimum buffer >> + * size as two pages. >> + */ > > I read this three times and event then I'm not sure to fully get it. I > suggest re-working this paragraph. I guess it should be "no room left" instead of "not much room left". If there is only a single page and the buffer is partially filled, there wont be space left to write the traces after the WRITE/LIMIT pointer's PAGE_SIZE alignment. /* * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with * just a single page, there would not be any room left while writing * into a partially filled TRBE buffer after the page size alignment. * Hence restrict the minimum buffer size as two pages. */ > > >> + if (nr_pages < 2) >> + return NULL; >> + >> + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); >> + if (IS_ERR(buf)) >> + return ERR_PTR(-ENOMEM); >> + >> + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); >> + if (IS_ERR(pglist)) { >> + kfree(buf); >> + return ERR_PTR(-ENOMEM); >> + } >> + >> + for (i = 0; i < nr_pages; i++) >> + pglist[i] = virt_to_page(pages[i]); >> + >> + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); >> + if (IS_ERR((void *)buf->trbe_base)) { > > Why not simply make buf->trbe_base a void * instead of having to do all this There are many arithmetic and comparison operations involving trbe_base element. Hence it might be better to keep it as unsigned long, also to keeps it consistent with other pointers i.e trbe_write, trbe_limit. Snippet from $cat drivers/hwtracing/coresight/coresight-trbe.c | grep "trbe_base" There are just two places type casting trbe_base back to (void *). memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); return buf->trbe_base + offset; WARN_ON(buf->trbe_write < buf->trbe_base); set_trbe_base_pointer(buf->trbe_base); buf->trbe_base = (unsigned long)vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); if (IS_ERR((void *)buf->trbe_base)) { return ERR_PTR(buf->trbe_base); buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; buf->trbe_write = buf->trbe_base; vunmap((void *)buf->trbe_base); base = get_trbe_base_pointer(); buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); if (buf->trbe_limit == buf->trbe_base) { buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); if (buf->trbe_limit == buf->trbe_base) { offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); if (buf->trbe_limit == buf->trbe_base) { WARN_ON(buf->trbe_base != get_trbe_base_pointer()); if (get_trbe_write_pointer() == get_trbe_base_pointer()) > casting? And IS_ERR() doesn't work with vmap(). Sure, will drop IS_ERR() here. > >> + kfree(pglist); >> + kfree(buf); >> + return ERR_PTR(buf->trbe_base); >> + } >> + buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; >> + buf->trbe_write = buf->trbe_base; >> + buf->snapshot = snapshot; >> + buf->nr_pages = nr_pages; >> + buf->pages = pages; >> + kfree(pglist); >> + return buf; >> +} >> + >> +void arm_trbe_free_buffer(void *config) >> +{ >> + struct trbe_buf *buf = config; >> + >> + vunmap((void *)buf->trbe_base); >> + kfree(buf); >> +} >> + >> +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, >> + struct perf_output_handle *handle, >> + void *config) >> +{ >> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >> + struct trbe_buf *buf = config; >> + unsigned long size, offset; >> + >> + WARN_ON(buf->cpudata != cpudata); >> + WARN_ON(cpudata->cpu != smp_processor_id()); >> + WARN_ON(cpudata->drvdata != drvdata); >> + if (cpudata->mode != CS_MODE_PERF) >> + return -EINVAL; >> + >> + /* >> + * If the TRBE was disabled due to lack of space in the AUX buffer or a >> + * spurious fault, the driver leaves it disabled, truncating the buffer. >> + * Since the etm_perf driver expects to close out the AUX buffer, the >> + * driver skips it. Thus, just pass in 0 size here to indicate that the >> + * buffer was truncated. >> + */ >> + if (!is_trbe_enabled()) >> + return 0; >> + /* >> + * perf handle structure needs to be shared with the TRBE IRQ handler for >> + * capturing trace data and restarting the handle. There is a probability >> + * of an undefined reference based crash when etm event is being stopped >> + * while a TRBE IRQ also getting processed. This happens due the release >> + * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping >> + * the TRBE here will ensure that no IRQ could be generated when the perf >> + * handle gets freed in etm_event_stop(). >> + */ >> + trbe_drain_and_disable_local(); >> + offset = get_trbe_write_pointer() - get_trbe_base_pointer(); >> + size = offset - PERF_IDX2OFF(handle->head, buf); >> + if (buf->snapshot) >> + handle->head += size; >> + return size; >> +} >> + >> +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) >> +{ >> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >> + struct perf_output_handle *handle = data; >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + >> + WARN_ON(cpudata->cpu != smp_processor_id()); >> + WARN_ON(cpudata->drvdata != drvdata); >> + if (mode != CS_MODE_PERF) >> + return -EINVAL; >> + >> + *this_cpu_ptr(drvdata->handle) = handle; >> + cpudata->buf = buf; >> + cpudata->mode = mode; >> + buf->cpudata = cpudata; >> + buf->trbe_limit = compute_trbe_buffer_limit(handle); >> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >> + if (buf->trbe_limit == buf->trbe_base) { >> + trbe_stop_and_truncate_event(handle); >> + return 0; >> + } >> + trbe_enable_hw(buf); >> + return 0; >> +} >> + >> +static int arm_trbe_disable(struct coresight_device *csdev) >> +{ >> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >> + struct trbe_buf *buf = cpudata->buf; >> + >> + WARN_ON(buf->cpudata != cpudata); >> + WARN_ON(cpudata->cpu != smp_processor_id()); >> + WARN_ON(cpudata->drvdata != drvdata); >> + if (cpudata->mode != CS_MODE_PERF) >> + return -EINVAL; >> + >> + trbe_drain_and_disable_local(); >> + buf->cpudata = NULL; >> + cpudata->buf = NULL; >> + cpudata->mode = CS_MODE_DISABLED; >> + return 0; >> +} >> + >> +static void trbe_handle_spurious(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + >> + buf->trbe_limit = compute_trbe_buffer_limit(handle); >> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >> + if (buf->trbe_limit == buf->trbe_base) { >> + trbe_drain_and_disable_local(); >> + return; >> + } >> + trbe_enable_hw(buf); >> +} >> + >> +static void trbe_handle_overflow(struct perf_output_handle *handle) >> +{ >> + struct perf_event *event = handle->event; >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + unsigned long offset, size; >> + struct etm_event_data *event_data; >> + >> + offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); >> + size = offset - PERF_IDX2OFF(handle->head, buf); >> + if (buf->snapshot) >> + handle->head = offset; >> + perf_aux_output_end(handle, size); >> + >> + event_data = perf_aux_output_begin(handle, event); >> + if (!event_data) { >> + trbe_drain_and_disable_local(); >> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; >> + return; >> + } >> + buf->trbe_limit = compute_trbe_buffer_limit(handle); >> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >> + if (buf->trbe_limit == buf->trbe_base) { >> + trbe_stop_and_truncate_event(handle); >> + return; >> + } >> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; >> + trbe_enable_hw(buf); >> +} >> + >> +static bool is_perf_trbe(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + struct trbe_cpudata *cpudata = buf->cpudata; >> + struct trbe_drvdata *drvdata = cpudata->drvdata; >> + int cpu = smp_processor_id(); >> + >> + WARN_ON(buf->trbe_base != get_trbe_base_pointer()); >> + WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); >> + >> + if (cpudata->mode != CS_MODE_PERF) >> + return false; >> + >> + if (cpudata->cpu != cpu) >> + return false; >> + >> + if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) >> + return false; >> + >> + return true; >> +} >> + >> +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) >> +{ >> + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); >> + int ec = get_trbe_ec(trbsr); >> + int bsc = get_trbe_bsc(trbsr); >> + >> + WARN_ON(is_trbe_running(trbsr)); >> + if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) >> + return TRBE_FAULT_ACT_FATAL; >> + >> + if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) >> + return TRBE_FAULT_ACT_FATAL; >> + >> + if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { >> + if (get_trbe_write_pointer() == get_trbe_base_pointer()) >> + return TRBE_FAULT_ACT_WRAP; >> + } >> + return TRBE_FAULT_ACT_SPURIOUS; >> +} >> + >> +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) >> +{ >> + struct perf_output_handle **handle_ptr = dev; >> + struct perf_output_handle *handle = *handle_ptr; >> + enum trbe_fault_action act; >> + >> + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); >> + clr_trbe_irq(); >> + >> + /* >> + * Ensure the trace is visible to the CPUs and >> + * any external aborts have been resolved. >> + */ >> + trbe_drain_buffer(); >> + isb(); >> + >> + if (!perf_get_aux(handle)) >> + return IRQ_NONE; >> + >> + if (!is_perf_trbe(handle)) >> + return IRQ_NONE; >> + >> + irq_work_run(); >> + >> + act = trbe_get_fault_act(handle); >> + switch (act) { >> + case TRBE_FAULT_ACT_WRAP: >> + trbe_handle_overflow(handle); >> + break; >> + case TRBE_FAULT_ACT_SPURIOUS: >> + trbe_handle_spurious(handle); >> + break; >> + case TRBE_FAULT_ACT_FATAL: >> + trbe_stop_and_truncate_event(handle); >> + break; >> + } >> + return IRQ_HANDLED; >> +} >> + >> +static const struct coresight_ops_sink arm_trbe_sink_ops = { >> + .enable = arm_trbe_enable, >> + .disable = arm_trbe_disable, >> + .alloc_buffer = arm_trbe_alloc_buffer, >> + .free_buffer = arm_trbe_free_buffer, >> + .update_buffer = arm_trbe_update_buffer, >> +}; >> + >> +static const struct coresight_ops arm_trbe_cs_ops = { >> + .sink_ops = &arm_trbe_sink_ops, >> +}; >> + >> +static ssize_t align_show(struct device *dev, struct device_attribute *attr, char *buf) >> +{ >> + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); >> + >> + return sprintf(buf, "%llx\n", cpudata->trbe_align); >> +} >> +static DEVICE_ATTR_RO(align); >> + >> +static ssize_t dbm_show(struct device *dev, struct device_attribute *attr, char *buf) >> +{ >> + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); >> + >> + return sprintf(buf, "%d\n", cpudata->trbe_dbm); >> +} >> +static DEVICE_ATTR_RO(dbm); > > What does "dbm" stand for? Looking at the documentation for TRBIDR_EL1.F, I > don't see what "dbm" relates to. I made it up to refer TRBIDR_EL1.F as "Dirty (and Access Flag) Bit Management". Could change it as "afdbm" to be more specific or if it is preferred. > >> + >> +static struct attribute *arm_trbe_attrs[] = { >> + &dev_attr_align.attr, >> + &dev_attr_dbm.attr, >> + NULL, >> +}; >> + >> +static const struct attribute_group arm_trbe_group = { >> + .attrs = arm_trbe_attrs, >> +}; >> + >> +static const struct attribute_group *arm_trbe_groups[] = { >> + &arm_trbe_group, >> + NULL, >> +}; >> + >> +static void arm_trbe_probe_coresight_cpu(void *info) >> +{ >> + struct trbe_drvdata *drvdata = info; >> + struct coresight_desc desc = { 0 }; >> + int cpu = smp_processor_id(); >> + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); >> + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); >> + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); >> + struct device *dev; >> + >> + if (WARN_ON(!cpudata)) >> + goto cpu_clear; >> + >> + if (trbe_csdev) >> + return; >> + >> + cpudata->cpu = smp_processor_id(); >> + cpudata->drvdata = drvdata; >> + dev = &cpudata->drvdata->pdev->dev; >> + >> + if (!is_trbe_available()) { >> + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); >> + goto cpu_clear; >> + } >> + >> + if (!is_trbe_programmable(trbidr)) { >> + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); >> + goto cpu_clear; >> + } >> + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); >> + if (IS_ERR(desc.name)) >> + goto cpu_clear; >> + >> + desc.type = CORESIGHT_DEV_TYPE_SINK; >> + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; >> + desc.ops = &arm_trbe_cs_ops; >> + desc.pdata = dev_get_platdata(dev); >> + desc.groups = arm_trbe_groups; >> + desc.dev = dev; >> + trbe_csdev = coresight_register(&desc); >> + if (IS_ERR(trbe_csdev)) >> + goto cpu_clear; >> + >> + dev_set_drvdata(&trbe_csdev->dev, cpudata); >> + cpudata->trbe_dbm = get_trbe_flag_update(trbidr); >> + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); >> + if (cpudata->trbe_align > SZ_2K) { >> + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); >> + goto cpu_clear; >> + } >> + per_cpu(csdev_sink, cpu) = trbe_csdev; >> + trbe_reset_local(); >> + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); >> + return; >> +cpu_clear: >> + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); >> +} >> + >> +static void arm_trbe_remove_coresight_cpu(void *info) >> +{ >> + int cpu = smp_processor_id(); >> + struct trbe_drvdata *drvdata = info; >> + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); >> + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); >> + >> + if (trbe_csdev) { >> + coresight_unregister(trbe_csdev); >> + cpudata->drvdata = NULL; >> + per_cpu(csdev_sink, cpu) = NULL; >> + } >> + disable_percpu_irq(drvdata->irq); >> + trbe_reset_local(); >> +} >> + >> +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) >> +{ >> + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); >> + if (IS_ERR(drvdata->cpudata)) >> + return PTR_ERR(drvdata->cpudata); >> + >> + arm_trbe_probe_coresight_cpu(drvdata); >> + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); >> + return 0; >> +} >> + >> +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) >> +{ >> + arm_trbe_remove_coresight_cpu(drvdata); >> + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); >> + free_percpu(drvdata->cpudata); >> + return 0; >> +} >> + >> +static int arm_trbe_cpu_startup(unsigned int cpu, struct hlist_node *node) >> +{ >> + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); >> + >> + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { >> + if (!per_cpu(csdev_sink, cpu)) { >> + arm_trbe_probe_coresight_cpu(drvdata); >> + } else { >> + trbe_reset_local(); >> + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); >> + } >> + } >> + return 0; >> +} >> + >> +static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node) >> +{ >> + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); >> + >> + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { >> + disable_percpu_irq(drvdata->irq); >> + trbe_reset_local(); >> + } >> + return 0; >> +} >> + >> +static int arm_trbe_probe_cpuhp(struct trbe_drvdata *drvdata) >> +{ >> + enum cpuhp_state trbe_online; >> + >> + trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, >> + arm_trbe_cpu_startup, arm_trbe_cpu_teardown); > > Indentation Sure, will do. > > trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, > arm_trbe_cpu_startup, > arm_trbe_cpu_teardown); > >> + if (trbe_online < 0) >> + return -EINVAL > > Wrong error code Okay, will return trbe_online here. > >> + >> + if (cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node)) >> + return -EINVAL; > > Wrong error code. Even if the callbacks aren't called it might be a good idea to > properly cleanup the with cpuhp_remove_multi_state(). Okay, will do the following change. --- a/drivers/hwtracing/coresight/coresight-trbe.c +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -916,15 +916,18 @@ static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node) static int arm_trbe_probe_cpuhp(struct trbe_drvdata *drvdata) { enum cpuhp_state trbe_online; + int ret; trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, arm_trbe_cpu_startup, arm_trbe_cpu_teardown); if (trbe_online < 0) return trbe_online; - if (cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node)) - return -EINVAL; - + ret = cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node); + if (ret) { + cpuhp_remove_multi_state(trbe_online); + return ret; + } drvdata->trbe_online = trbe_online; return 0; } > > More to come tomorrow. > > Thanks, > Mathieu > >> + >> + drvdata->trbe_online = trbe_online; >> + return 0; >> +} >> + >> +static void arm_trbe_remove_cpuhp(struct trbe_drvdata *drvdata) >> +{ >> + cpuhp_remove_multi_state(drvdata->trbe_online); >> +} >> + >> +static int arm_trbe_probe_irq(struct platform_device *pdev, >> + struct trbe_drvdata *drvdata) >> +{ >> + drvdata->irq = platform_get_irq(pdev, 0); >> + if (!drvdata->irq) { >> + pr_err("IRQ not found for the platform device\n"); >> + return -ENXIO; >> + } >> + >> + if (!irq_is_percpu(drvdata->irq)) { >> + pr_err("IRQ is not a PPI\n"); >> + return -EINVAL; >> + } >> + >> + if (irq_get_percpu_devid_partition(drvdata->irq, &drvdata->supported_cpus)) >> + return -EINVAL; >> + >> + drvdata->handle = alloc_percpu(typeof(*drvdata->handle)); >> + if (!drvdata->handle) >> + return -ENOMEM; >> + >> + if (request_percpu_irq(drvdata->irq, arm_trbe_irq_handler, DRVNAME, drvdata->handle)) { >> + free_percpu(drvdata->handle); >> + return -EINVAL; >> + } >> + return 0; >> +} >> + >> +static void arm_trbe_remove_irq(struct trbe_drvdata *drvdata) >> +{ >> + free_percpu_irq(drvdata->irq, drvdata->handle); >> + free_percpu(drvdata->handle); >> +} >> + >> +static int arm_trbe_device_probe(struct platform_device *pdev) >> +{ >> + struct coresight_platform_data *pdata; >> + struct trbe_drvdata *drvdata; >> + struct device *dev = &pdev->dev; >> + int ret; >> + >> + drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL); >> + if (IS_ERR(drvdata)) >> + return -ENOMEM; >> + >> + pdata = coresight_get_platform_data(dev); >> + if (IS_ERR(pdata)) { >> + kfree(drvdata); >> + return -ENOMEM; >> + } >> + >> + dev_set_drvdata(dev, drvdata); >> + dev->platform_data = pdata; >> + drvdata->pdev = pdev; >> + ret = arm_trbe_probe_irq(pdev, drvdata); >> + if (ret) >> + goto irq_failed; >> + >> + ret = arm_trbe_probe_coresight(drvdata); >> + if (ret) >> + goto probe_failed; >> + >> + ret = arm_trbe_probe_cpuhp(drvdata); >> + if (ret) >> + goto cpuhp_failed; >> + >> + return 0; >> +cpuhp_failed: >> + arm_trbe_remove_coresight(drvdata); >> +probe_failed: >> + arm_trbe_remove_irq(drvdata); >> +irq_failed: >> + kfree(pdata); >> + kfree(drvdata); >> + return ret; >> +} >> + >> +static int arm_trbe_device_remove(struct platform_device *pdev) >> +{ >> + struct coresight_platform_data *pdata = dev_get_platdata(&pdev->dev); >> + struct trbe_drvdata *drvdata = platform_get_drvdata(pdev); >> + >> + arm_trbe_remove_coresight(drvdata); >> + arm_trbe_remove_cpuhp(drvdata); > > Shouldn't this be called before arm_trbe_remove_coresight()? Yes, will change the order. - Anshuman
On 2/11/21 12:30 AM, Mathieu Poirier wrote: > On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: >> Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is >> accessible via the system registers. The TRBE supports different addressing >> modes including CPU virtual address and buffer modes including the circular >> buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), >> an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the >> access to the trace buffer could be prohibited by a higher exception level >> (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU >> private interrupt (PPI) on address translation errors and when the buffer >> is full. Overall implementation here is inspired from the Arm SPE driver. >> >> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> >> Cc: Mike Leach <mike.leach@linaro.org> >> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> >> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> >> --- >> Changes in V3: >> >> - Added new DT bindings document TRBE.yaml >> - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 >> - Dropped isb() from trbe_reset_local() >> - Dropped gap between (void *) and buf->trbe_base >> - Changed 'int' to 'unsigned int' in is_trbe_available() >> - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), >> set_trbe_enabled() and set_trbe_limit_pointer() >> - Changed get_trbe_flag_update(), is_trbe_programmable() and >> get_trbe_address_align() to accept TRBIDR value >> - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), >> is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value >> - Dropped snapshot mode condition in arm_trbe_alloc_buffer() >> - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled >> - Compute trbe_limit before trbe_write to get the updated handle >> - Added trbe_stop_and_truncate_event() >> - Dropped trbe_handle_fatal() >> >> Documentation/trace/coresight/coresight-trbe.rst | 39 + >> arch/arm64/include/asm/sysreg.h | 1 + >> drivers/hwtracing/coresight/Kconfig | 11 + >> drivers/hwtracing/coresight/Makefile | 1 + >> drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ >> drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ >> 6 files changed, 1235 insertions(+) >> create mode 100644 Documentation/trace/coresight/coresight-trbe.rst >> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c >> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h >> + > > [...] > >> +static void arm_trbe_probe_coresight_cpu(void *info) >> +{ >> + struct trbe_drvdata *drvdata = info; >> + struct coresight_desc desc = { 0 }; >> + int cpu = smp_processor_id(); >> + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); >> + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); >> + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); >> + struct device *dev; >> + >> + if (WARN_ON(!cpudata)) >> + goto cpu_clear; > > There is already a check for this in arm_trbe_probe_coresight(), we couldn't be > here if there was a problem with the allocation. Right but just to be extra cautious. Do you really want this to be dropped ? > >> + >> + if (trbe_csdev) >> + return; > > Now that's a reason to have a WARN_ON(). If we are probing and a sink is > already present in this cpu's slot, something went seriously wrong and we should > be clear about it. Right, will add an WARN_ON(). > >> + >> + cpudata->cpu = smp_processor_id(); >> + cpudata->drvdata = drvdata; >> + dev = &cpudata->drvdata->pdev->dev; >> + >> + if (!is_trbe_available()) { >> + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); >> + goto cpu_clear; >> + } >> + >> + if (!is_trbe_programmable(trbidr)) { >> + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); >> + goto cpu_clear; >> + } >> + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); > > We will end up with "arm_trbe0", "arm_trbe1" and so on in sysfs... Is the > "arm_" part absolutely needed? I think this should be like what we do for etmv3 > and etmv4 where only "etmX" shows up in sysfs. Okay, will drop arm_ here. IIRC this was originally trbeX where X is the cpu number but then ended up using DRVNAME as prefix. > >> + if (IS_ERR(desc.name)) >> + goto cpu_clear; >> + >> + desc.type = CORESIGHT_DEV_TYPE_SINK; >> + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; >> + desc.ops = &arm_trbe_cs_ops; >> + desc.pdata = dev_get_platdata(dev); >> + desc.groups = arm_trbe_groups; >> + desc.dev = dev; >> + trbe_csdev = coresight_register(&desc); >> + if (IS_ERR(trbe_csdev)) >> + goto cpu_clear; >> + >> + dev_set_drvdata(&trbe_csdev->dev, cpudata); >> + cpudata->trbe_dbm = get_trbe_flag_update(trbidr); >> + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); >> + if (cpudata->trbe_align > SZ_2K) { >> + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); >> + goto cpu_clear; > > Here coresight_unregister() should be called. The other option is to call > coresight_register() when everything else is known to be fine, which is the > favoured approach. Okay, will change accordingly. > >> + } >> + per_cpu(csdev_sink, cpu) = trbe_csdev; >> + trbe_reset_local(); >> + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); >> + return; >> +cpu_clear: >> + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); >> +} >> + >> +static void arm_trbe_remove_coresight_cpu(void *info) >> +{ >> + int cpu = smp_processor_id(); >> + struct trbe_drvdata *drvdata = info; >> + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); >> + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); >> + >> + if (trbe_csdev) { > > In what scenario do you see not having a trbe_csdev and still needing to disable > IRQs for the HW? If there is a such a case then a few lines of comment is > needed. > >> + coresight_unregister(trbe_csdev); >> + cpudata->drvdata = NULL; >> + per_cpu(csdev_sink, cpu) = NULL; >> + } >> + disable_percpu_irq(drvdata->irq); >> + trbe_reset_local(); > > Theoretically this code shouldn't run when the TRBE is enabled, because the CS > core will prevent that from happening. As sush disabling interrupts after > coresight_unregister() has been called and setting cpudata->drvdata to NULL > should be fine. But from an outsider's point of view it will look very bizarre. > Either write a comment to explain all that or call the above two before doing > the cleanup. Okay, will move them before the cleanup. > >> +} >> + >> +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) >> +{ >> + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); >> + if (IS_ERR(drvdata->cpudata)) >> + return PTR_ERR(drvdata->cpudata); > > As far as I can tell alloc_percpu() returns NULL on failure and nothing else. Sure, will change the return code as -ENOMEM when alloc_percpu() returns NULL. > >> + >> + arm_trbe_probe_coresight_cpu(drvdata); >> + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); > > The above two calls look racy to me. The executing process could be moved to > another CPU between the call to arm_trbe_probe_coresight_cpu() and > smp_call_function_many(), which would prevent the initialisation of the TRBE on > the new CPU to be done. I suggest using a for_each_cpu() loop where > smp_call_function_single() would be used. That way we are guaranteed all the > TRBEs will be initialised. Okay, will change. > >> + return 0; >> +} >> + >> +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) >> +{ >> + arm_trbe_remove_coresight_cpu(drvdata); >> + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); > > Same as above. Okay, will do. > > I'm out of time for today, more to come tomorrow. Okay. > > Mathieu - Anshuman
[...] > > > > > >> + if (nr_pages < 2) > >> + return NULL; > >> + > >> + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); > >> + if (IS_ERR(buf)) > >> + return ERR_PTR(-ENOMEM); > >> + > >> + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); > >> + if (IS_ERR(pglist)) { > >> + kfree(buf); > >> + return ERR_PTR(-ENOMEM); > >> + } > >> + > >> + for (i = 0; i < nr_pages; i++) > >> + pglist[i] = virt_to_page(pages[i]); > >> + > >> + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); > >> + if (IS_ERR((void *)buf->trbe_base)) { > > > > Why not simply make buf->trbe_base a void * instead of having to do all this > > There are many arithmetic and comparison operations involving trbe_base > element. Hence it might be better to keep it as unsigned long, also to > keeps it consistent with other pointers i.e trbe_write, trbe_limit. That is a fair point. Please add a comment to explain your design choice and make sure the sparse checker is happy with all of it. > > Snippet from $cat drivers/hwtracing/coresight/coresight-trbe.c | grep "trbe_base" > There are just two places type casting trbe_base back to (void *). > > memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); > return buf->trbe_base + offset; > WARN_ON(buf->trbe_write < buf->trbe_base); > set_trbe_base_pointer(buf->trbe_base); > buf->trbe_base = (unsigned long)vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); > if (IS_ERR((void *)buf->trbe_base)) { > return ERR_PTR(buf->trbe_base); > buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; > buf->trbe_write = buf->trbe_base; > vunmap((void *)buf->trbe_base); > base = get_trbe_base_pointer(); > buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > if (buf->trbe_limit == buf->trbe_base) { > buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > if (buf->trbe_limit == buf->trbe_base) { > offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); > buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > if (buf->trbe_limit == buf->trbe_base) { > WARN_ON(buf->trbe_base != get_trbe_base_pointer()); > if (get_trbe_write_pointer() == get_trbe_base_pointer()) > > > casting? And IS_ERR() doesn't work with vmap(). > > Sure, will drop IS_ERR() here. > [...] > > > >> + > >> +static ssize_t dbm_show(struct device *dev, struct device_attribute *attr, char *buf) > >> +{ > >> + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > >> + > >> + return sprintf(buf, "%d\n", cpudata->trbe_dbm); > >> +} > >> +static DEVICE_ATTR_RO(dbm); > > > > What does "dbm" stand for? Looking at the documentation for TRBIDR_EL1.F, I > > don't see what "dbm" relates to. > > I made it up to refer TRBIDR_EL1.F as "Dirty (and Access Flag) Bit Management". > Could change it as "afdbm" to be more specific or if it is preferred. > I don't see "afdbm" being a better solution - why not simply "flag"?
On Fri, Feb 12, 2021 at 11:13:01AM +0530, Anshuman Khandual wrote: > > > On 2/11/21 12:30 AM, Mathieu Poirier wrote: > > On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: > >> Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is > >> accessible via the system registers. The TRBE supports different addressing > >> modes including CPU virtual address and buffer modes including the circular > >> buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), > >> an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the > >> access to the trace buffer could be prohibited by a higher exception level > >> (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU > >> private interrupt (PPI) on address translation errors and when the buffer > >> is full. Overall implementation here is inspired from the Arm SPE driver. > >> > >> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> > >> Cc: Mike Leach <mike.leach@linaro.org> > >> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> > >> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> > >> --- > >> Changes in V3: > >> > >> - Added new DT bindings document TRBE.yaml > >> - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 > >> - Dropped isb() from trbe_reset_local() > >> - Dropped gap between (void *) and buf->trbe_base > >> - Changed 'int' to 'unsigned int' in is_trbe_available() > >> - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), > >> set_trbe_enabled() and set_trbe_limit_pointer() > >> - Changed get_trbe_flag_update(), is_trbe_programmable() and > >> get_trbe_address_align() to accept TRBIDR value > >> - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), > >> is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value > >> - Dropped snapshot mode condition in arm_trbe_alloc_buffer() > >> - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled > >> - Compute trbe_limit before trbe_write to get the updated handle > >> - Added trbe_stop_and_truncate_event() > >> - Dropped trbe_handle_fatal() > >> > >> Documentation/trace/coresight/coresight-trbe.rst | 39 + > >> arch/arm64/include/asm/sysreg.h | 1 + > >> drivers/hwtracing/coresight/Kconfig | 11 + > >> drivers/hwtracing/coresight/Makefile | 1 + > >> drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ > >> drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ > >> 6 files changed, 1235 insertions(+) > >> create mode 100644 Documentation/trace/coresight/coresight-trbe.rst > >> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c > >> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h > >> + > > > > [...] > > > >> +static void arm_trbe_probe_coresight_cpu(void *info) > >> +{ > >> + struct trbe_drvdata *drvdata = info; > >> + struct coresight_desc desc = { 0 }; > >> + int cpu = smp_processor_id(); > >> + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > >> + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > >> + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); > >> + struct device *dev; > >> + > >> + if (WARN_ON(!cpudata)) > >> + goto cpu_clear; > > > > There is already a check for this in arm_trbe_probe_coresight(), we couldn't be > > here if there was a problem with the allocation. > > Right but just to be extra cautious. Do you really want this to be dropped ? I don't think it is necessary but there is no harm in keeping it if you are keen on it. > > > > >> + > >> + if (trbe_csdev) > >> + return; > > > > Now that's a reason to have a WARN_ON(). If we are probing and a sink is > > already present in this cpu's slot, something went seriously wrong and we should > > be clear about it. > > Right, will add an WARN_ON(). > > > > >> + > >> + cpudata->cpu = smp_processor_id(); > >> + cpudata->drvdata = drvdata; > >> + dev = &cpudata->drvdata->pdev->dev; > >> + > >> + if (!is_trbe_available()) { > >> + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); > >> + goto cpu_clear; > >> + } > >> + > >> + if (!is_trbe_programmable(trbidr)) { > >> + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); > >> + goto cpu_clear; > >> + } > >> + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); > > > > We will end up with "arm_trbe0", "arm_trbe1" and so on in sysfs... Is the > > "arm_" part absolutely needed? I think this should be like what we do for etmv3 > > and etmv4 where only "etmX" shows up in sysfs. > > Okay, will drop arm_ here. IIRC this was originally trbeX where X is the cpu number > but then ended up using DRVNAME as prefix. > > > > >> + if (IS_ERR(desc.name)) > >> + goto cpu_clear; > >> + > >> + desc.type = CORESIGHT_DEV_TYPE_SINK; > >> + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; > >> + desc.ops = &arm_trbe_cs_ops; > >> + desc.pdata = dev_get_platdata(dev); > >> + desc.groups = arm_trbe_groups; > >> + desc.dev = dev; > >> + trbe_csdev = coresight_register(&desc); > >> + if (IS_ERR(trbe_csdev)) > >> + goto cpu_clear; > >> + > >> + dev_set_drvdata(&trbe_csdev->dev, cpudata); > >> + cpudata->trbe_dbm = get_trbe_flag_update(trbidr); > >> + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); > >> + if (cpudata->trbe_align > SZ_2K) { > >> + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); > >> + goto cpu_clear; > > > > Here coresight_unregister() should be called. The other option is to call > > coresight_register() when everything else is known to be fine, which is the > > favoured approach. > > Okay, will change accordingly. > > > > >> + } > >> + per_cpu(csdev_sink, cpu) = trbe_csdev; > >> + trbe_reset_local(); > >> + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > >> + return; > >> +cpu_clear: > >> + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); > >> +} > >> + > >> +static void arm_trbe_remove_coresight_cpu(void *info) > >> +{ > >> + int cpu = smp_processor_id(); > >> + struct trbe_drvdata *drvdata = info; > >> + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > >> + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > >> + > >> + if (trbe_csdev) { > > > > In what scenario do you see not having a trbe_csdev and still needing to disable > > IRQs for the HW? If there is a such a case then a few lines of comment is > > needed. > > > >> + coresight_unregister(trbe_csdev); > >> + cpudata->drvdata = NULL; > >> + per_cpu(csdev_sink, cpu) = NULL; > >> + } > >> + disable_percpu_irq(drvdata->irq); > >> + trbe_reset_local(); > > > > Theoretically this code shouldn't run when the TRBE is enabled, because the CS > > core will prevent that from happening. As sush disabling interrupts after > > coresight_unregister() has been called and setting cpudata->drvdata to NULL > > should be fine. But from an outsider's point of view it will look very bizarre. > > Either write a comment to explain all that or call the above two before doing > > the cleanup. > > Okay, will move them before the cleanup. > > > > >> +} > >> + > >> +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) > >> +{ > >> + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); > >> + if (IS_ERR(drvdata->cpudata)) > >> + return PTR_ERR(drvdata->cpudata); > > > > As far as I can tell alloc_percpu() returns NULL on failure and nothing else. > > Sure, will change the return code as -ENOMEM when alloc_percpu() returns NULL. > > > > >> + > >> + arm_trbe_probe_coresight_cpu(drvdata); > >> + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); > > > > The above two calls look racy to me. The executing process could be moved to > > another CPU between the call to arm_trbe_probe_coresight_cpu() and > > smp_call_function_many(), which would prevent the initialisation of the TRBE on > > the new CPU to be done. I suggest using a for_each_cpu() loop where > > smp_call_function_single() would be used. That way we are guaranteed all the > > TRBEs will be initialised. > > Okay, will change. > > > > >> + return 0; > >> +} > >> + > >> +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) > >> +{ > >> + arm_trbe_remove_coresight_cpu(drvdata); > >> + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); > > > > Same as above. > > Okay, will do. > > > > > I'm out of time for today, more to come tomorrow. > > Okay. > > > > > Mathieu > > - Anshuman
On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: > Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is > accessible via the system registers. The TRBE supports different addressing > modes including CPU virtual address and buffer modes including the circular > buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), > an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the > access to the trace buffer could be prohibited by a higher exception level > (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU > private interrupt (PPI) on address translation errors and when the buffer > is full. Overall implementation here is inspired from the Arm SPE driver. > > Cc: Mathieu Poirier <mathieu.poirier@linaro.org> > Cc: Mike Leach <mike.leach@linaro.org> > Cc: Suzuki K Poulose <suzuki.poulose@arm.com> > Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> > --- > Changes in V3: > > - Added new DT bindings document TRBE.yaml > - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 > - Dropped isb() from trbe_reset_local() > - Dropped gap between (void *) and buf->trbe_base > - Changed 'int' to 'unsigned int' in is_trbe_available() > - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), > set_trbe_enabled() and set_trbe_limit_pointer() > - Changed get_trbe_flag_update(), is_trbe_programmable() and > get_trbe_address_align() to accept TRBIDR value > - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), > is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value > - Dropped snapshot mode condition in arm_trbe_alloc_buffer() > - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled > - Compute trbe_limit before trbe_write to get the updated handle > - Added trbe_stop_and_truncate_event() > - Dropped trbe_handle_fatal() > > Documentation/trace/coresight/coresight-trbe.rst | 39 + > arch/arm64/include/asm/sysreg.h | 1 + > drivers/hwtracing/coresight/Kconfig | 11 + > drivers/hwtracing/coresight/Makefile | 1 + > drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ > drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ > 6 files changed, 1235 insertions(+) > create mode 100644 Documentation/trace/coresight/coresight-trbe.rst > create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c > create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h > > diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst > new file mode 100644 > index 0000000..1cbb819 > --- /dev/null > +++ b/Documentation/trace/coresight/coresight-trbe.rst > @@ -0,0 +1,39 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +============================== > +Trace Buffer Extension (TRBE). > +============================== > + > + :Author: Anshuman Khandual <anshuman.khandual@arm.com> > + :Date: November 2020 > + > +Hardware Description > +-------------------- > + > +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system > +memory, CPU traces generated from a corresponding percpu tracing unit. This > +gets plugged in as a coresight sink device because the corresponding trace > +genarators (ETE), are plugged in as source device. > + > +The TRBE is not compliant to CoreSight architecture specifications, but is > +driven via the CoreSight driver framework to support the ETE (which is > +CoreSight compliant) integration. > + > +Sysfs files and directories > +--------------------------- > + > +The TRBE devices appear on the existing coresight bus alongside the other > +coresight devices:: > + > + >$ ls /sys/bus/coresight/devices > + trbe0 trbe1 trbe2 trbe3 > + > +The ``trbe<N>`` named TRBEs are associated with a CPU.:: > + > + >$ ls /sys/bus/coresight/devices/trbe0/ > + align dbm > + > +*Key file items are:-* > + * ``align``: TRBE write pointer alignment > + * ``dbm``: TRBE updates memory with access and dirty flags > + > diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h > index 85ae4db..9e2e9b7 100644 > --- a/arch/arm64/include/asm/sysreg.h > +++ b/arch/arm64/include/asm/sysreg.h > @@ -97,6 +97,7 @@ > #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) > #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) > #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) > +#define TSB_CSYNC __emit_inst(0xd503225f) > > #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) > #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) > diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig > index f154ae7..aa657ab 100644 > --- a/drivers/hwtracing/coresight/Kconfig > +++ b/drivers/hwtracing/coresight/Kconfig > @@ -164,6 +164,17 @@ config CORESIGHT_CTI > To compile this driver as a module, choose M here: the > module will be called coresight-cti. > > +config CORESIGHT_TRBE > + bool "Trace Buffer Extension (TRBE) driver" > + depends on ARM64 > + help > + This driver provides support for percpu Trace Buffer Extension (TRBE). > + TRBE always needs to be used along with it's corresponding percpu ETE > + component. ETE generates trace data which is then captured with TRBE. > + Unlike traditional sink devices, TRBE is a CPU feature accessible via > + system registers. But it's explicit dependency with trace unit (ETE) > + requires it to be plugged in as a coresight sink device. > + > config CORESIGHT_CTI_INTEGRATION_REGS > bool "Access CTI CoreSight Integration Registers" > depends on CORESIGHT_CTI > diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile > index f20e357..d608165 100644 > --- a/drivers/hwtracing/coresight/Makefile > +++ b/drivers/hwtracing/coresight/Makefile > @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o > obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o > obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o > obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o > +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o > coresight-cti-y := coresight-cti-core.o coresight-cti-platform.o \ > coresight-cti-sysfs.o > diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c > new file mode 100644 > index 0000000..1464d8b > --- /dev/null > +++ b/drivers/hwtracing/coresight/coresight-trbe.c > @@ -0,0 +1,1023 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight > + * sink device could then pair with an appropriate per-cpu coresight source > + * device (ETE) thus generating required trace data. Trace can be enabled > + * via the perf framework. > + * > + * Copyright (C) 2020 ARM Ltd. > + * > + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > + */ > +#define DRVNAME "arm_trbe" > + > +#define pr_fmt(fmt) DRVNAME ": " fmt > + > +#include "coresight-trbe.h" > + > +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) > + > +/* > + * A padding packet that will help the user space tools > + * in skipping relevant sections in the captured trace > + * data which could not be decoded. TRBE doesn't support > + * formatting the trace data, unlike the legacy CoreSight > + * sinks and thus we use ETE trace packets to pad the > + * sections of the buffer. > + */ > +#define ETE_IGNORE_PACKET 0x70 > + > +/* > + * Minimum amount of meaningful trace will contain: > + * A-Sync, Trace Info, Trace On, Address, Atom. > + * This is about 44bytes of ETE trace. To be on > + * the safer side, we assume 64bytes is the minimum > + * space required for a meaningful session, before > + * we hit a "WRAP" event. > + */ > +#define TRBE_TRACE_MIN_BUF_SIZE 64 > + > +enum trbe_fault_action { > + TRBE_FAULT_ACT_WRAP, > + TRBE_FAULT_ACT_SPURIOUS, > + TRBE_FAULT_ACT_FATAL, > +}; > + > +struct trbe_buf { > + unsigned long trbe_base; > + unsigned long trbe_limit; > + unsigned long trbe_write; > + int nr_pages; > + void **pages; > + bool snapshot; > + struct trbe_cpudata *cpudata; > +}; > + > +struct trbe_cpudata { > + bool trbe_dbm; > + u64 trbe_align; > + int cpu; > + enum cs_mode mode; > + struct trbe_buf *buf; > + struct trbe_drvdata *drvdata; > +}; > + > +struct trbe_drvdata { > + struct trbe_cpudata __percpu *cpudata; > + struct perf_output_handle __percpu **handle; > + struct hlist_node hotplug_node; > + int irq; > + cpumask_t supported_cpus; > + enum cpuhp_state trbe_online; > + struct platform_device *pdev; > +}; > + > +static int trbe_alloc_node(struct perf_event *event) > +{ > + if (event->cpu == -1) > + return NUMA_NO_NODE; > + return cpu_to_node(event->cpu); > +} > + > +static void trbe_drain_buffer(void) > +{ > + asm(TSB_CSYNC); > + dsb(nsh); > +} > + > +static void trbe_drain_and_disable_local(void) > +{ > + trbe_drain_buffer(); > + write_sysreg_s(0, SYS_TRBLIMITR_EL1); > + isb(); > +} > + > +static void trbe_reset_local(void) > +{ > + trbe_drain_and_disable_local(); > + write_sysreg_s(0, SYS_TRBPTR_EL1); > + write_sysreg_s(0, SYS_TRBBASER_EL1); > + write_sysreg_s(0, SYS_TRBSR_EL1); > +} > + > +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + /* > + * We cannot proceed with the buffer collection and we > + * do not have any data for the current session. The > + * etm_perf driver expects to close out the aux_buffer > + * at event_stop(). So disable the TRBE here and leave > + * the update_buffer() to return a 0 size. > + */ > + trbe_drain_and_disable_local(); > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > +} > + > +/* > + * TRBE Buffer Management > + * > + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, > + * it starts writing trace data from the write pointer onward till the limit pointer. > + * When the write pointer reaches the address just before the limit pointer, it gets > + * wrapped around again to the base pointer. This is called a TRBE wrap event, which > + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver > + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ > + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and > + * LIMIT pointers. > + * > + * Wrap around with an IRQ > + * ------ < ------ < ------- < ----- < ----- > + * | | > + * ------ > ------ > ------- > ----- > ----- > + * > + * +---------------+-----------------------+ > + * | | | > + * +---------------+-----------------------+ > + * Base Pointer Write Pointer Limit Pointer > + * > + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write > + * pointer can be aligned to the implementation defined TRBE trace buffer alignment > + * as captured in trbe_cpudata->trbe_align. > + * > + * > + * head tail wakeup > + * +---------------------------------------+----- ~ ~ ------ > + * |$$$$$$$|################|$$$$$$$$$$$$$$| | > + * +---------------------------------------+----- ~ ~ ------ > + * Base Pointer Write Pointer Limit Pointer > + * > + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing > + * values which tracks all the driver writes and user reads from the perf auxiliary > + * buffer. Generally [head..tail] is the area where the driver can write into unless > + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and > + * configured depending on the perf_output_handle indices, so that the driver does > + * not override into areas in the perf auxiliary buffer which is being or yet to be > + * consumed from the user space. The enabled TRBE buffer area is a moving subset of > + * the allocated perf auxiliary buffer. > + */ > +static void trbe_pad_buf(struct perf_output_handle *handle, int len) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + u64 head = PERF_IDX2OFF(handle->head, buf); > + > + memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); > + if (!buf->snapshot) > + perf_aux_output_skip(handle, len); > +} > + > +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + /* > + * The ETE trace has alignment synchronization packets allowing > + * the decoder to reset in case of an overflow or corruption. > + * So we can use the entire buffer for the snapshot mode. > + */ > + return buf->nr_pages * PAGE_SIZE; > +} > + > +/* > + * TRBE Limit Calculation > + * > + * The following markers are used to illustrate various TRBE buffer situations. > + * > + * $$$$ - Data area, unconsumed captured trace data, not to be overridden > + * #### - Free area, enabled, trace will be written > + * %%%% - Free area, disabled, trace will not be written > + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped > + */ > +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + struct trbe_cpudata *cpudata = buf->cpudata; > + const u64 bufsize = buf->nr_pages * PAGE_SIZE; > + u64 limit = bufsize; > + u64 head, tail, wakeup; > + > + head = PERF_IDX2OFF(handle->head, buf); > + > + /* > + * head > + * ------->| > + * | > + * head TRBE align tail > + * +----|-------|---------------|-------+ > + * |$$$$|=======|###############|$$$$$$$| > + * +----|-------|---------------|-------+ > + * trbe_base trbe_base + nr_pages > + * > + * Perf aux buffer output head position can be misaligned depending on > + * various factors including user space reads. In case misaligned, head > + * needs to be aligned before TRBE can be configured. Pad the alignment > + * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools > + * and skip this section thus advancing the head. > + */ > + if (!IS_ALIGNED(head, cpudata->trbe_align)) { > + unsigned long delta = roundup(head, cpudata->trbe_align) - head; > + > + delta = min(delta, handle->size); > + trbe_pad_buf(handle, delta); > + head = PERF_IDX2OFF(handle->head, buf); > + } > + > + /* > + * head = tail (size = 0) > + * +----|-------------------------------+ > + * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ | > + * +----|-------------------------------+ > + * trbe_base trbe_base + nr_pages > + * > + * Perf aux buffer does not have any space for the driver to write into. > + * Just communicate trace truncation event to the user space by marking > + * it with PERF_AUX_FLAG_TRUNCATED. > + */ > + if (!handle->size) { > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > + return 0; > + } > + > + /* Compute the tail and wakeup indices now that we've aligned head */ > + tail = PERF_IDX2OFF(handle->head + handle->size, buf); > + wakeup = PERF_IDX2OFF(handle->wakeup, buf); > + > + /* > + * Lets calculate the buffer area which TRBE could write into. There > + * are three possible scenarios here. Limit needs to be aligned with > + * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the > + * unconsumed data. > + * > + * 1) head < tail > + * > + * head tail > + * +----|-----------------------|-------+ > + * |$$$$|#######################|$$$$$$$| > + * +----|-----------------------|-------+ > + * trbe_base limit trbe_base + nr_pages > + * > + * TRBE could write into [head..tail] area. Unless the tail is right at > + * the end of the buffer, neither an wrap around nor an IRQ is expected > + * while being enabled. > + * > + * 2) head == tail > + * > + * head = tail (size > 0) > + * +----|-------------------------------+ > + * |%%%%|###############################| > + * +----|-------------------------------+ > + * trbe_base limit = trbe_base + nr_pages > + * > + * TRBE should just write into [head..base + nr_pages] area even though > + * the entire buffer is empty. Reason being, when the trace reaches the > + * end of the buffer, it will just wrap around with an IRQ giving an > + * opportunity to reconfigure the buffer. > + * > + * 3) tail < head > + * > + * tail head > + * +----|-----------------------|-------+ > + * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| > + * +----|-----------------------|-------+ > + * trbe_base limit = trbe_base + nr_pages > + * > + * TRBE should just write into [head..base + nr_pages] area even though > + * the [trbe_base..tail] is also empty. Reason being, when the trace > + * reaches the end of the buffer, it will just wrap around with an IRQ > + * giving an opportunity to reconfigure the buffer. > + */ > + if (head < tail) > + limit = round_down(tail, PAGE_SIZE); > + > + /* > + * Wakeup may be arbitrarily far into the future. If it's not in the > + * current generation, either we'll wrap before hitting it, or it's > + * in the past and has been handled already. > + * > + * If there's a wakeup before we wrap, arrange to be woken up by the > + * page boundary following it. Keep the tail boundary if that's lower. > + * > + * head wakeup tail > + * +----|---------------|-------|-------+ > + * |$$$$|###############|%%%%%%%|$$$$$$$| > + * +----|---------------|-------|-------+ > + * trbe_base limit trbe_base + nr_pages > + */ > + if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) > + limit = min(limit, round_up(wakeup, PAGE_SIZE)); > + > + /* > + * There are two situation when this can happen i.e limit is before > + * the head and hence TRBE cannot be configured. > + * > + * 1) head < tail (aligned down with PAGE_SIZE) and also they are both > + * within the same PAGE size range. > + * > + * PAGE_SIZE > + * |----------------------| > + * > + * limit head tail > + * +------------|------|--------|-------+ > + * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| > + * +------------|------|--------|-------+ > + * trbe_base trbe_base + nr_pages > + * > + * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both > + * head and wakeup are within same PAGE size range. > + * > + * PAGE_SIZE > + * |----------------------| > + * > + * limit head wakeup tail > + * +----|------|-------|--------|-------+ > + * |$$$$$$$$$$$|=======|========|$$$$$$$| > + * +----|------|-------|--------|-------+ > + * trbe_base trbe_base + nr_pages > + */ > + if (limit > head) > + return limit; > + > + trbe_pad_buf(handle, handle->size); > + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > + return 0; > +} > + > +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = perf_get_aux(handle); > + u64 limit = __trbe_normal_offset(handle); > + u64 head = PERF_IDX2OFF(handle->head, buf); > + > + /* > + * If the head is too close to the limit and we don't > + * have space for a meaningful run, we rather pad it > + * and start fresh. > + */ > + if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { > + trbe_pad_buf(handle, limit - head); > + limit = __trbe_normal_offset(handle); > + } > + return limit; > +} > + > +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + unsigned long offset; > + > + if (buf->snapshot) > + offset = trbe_snapshot_offset(handle); > + else > + offset = trbe_normal_offset(handle); > + return buf->trbe_base + offset; > +} I won't review the trace buffer management functions in this revision, I will leave that for the next version. > + > +static void clr_trbe_status(void) > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + > + WARN_ON(is_trbe_enabled()); > + trbsr &= ~TRBSR_IRQ; > + trbsr &= ~TRBSR_TRG; > + trbsr &= ~TRBSR_WRAP; > + trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); > + trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); > + trbsr &= ~TRBSR_STOP; > + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > +} I haven't read the TRBE progammer's manual but looking a the documentation the above looks good. > + > +static void set_trbe_limit_pointer_enabled(unsigned long addr) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + > + trblimitr &= ~TRBLIMITR_NVM; > + trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); > + trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); > + trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); > + > + /* > + * Fill trace buffer mode is used here while configuring the > + * TRBE for trace capture. In this particular mode, the trace > + * collection is stopped and a maintenance interrupt is raised > + * when the current write pointer wraps. This pause in trace > + * collection gives the software an opportunity to capture the > + * trace data in the interrupt handler, before reconfiguring > + * the TRBE. > + */ > + trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; > + > + /* > + * Trigger mode is not used here while configuring the TRBE for > + * the trace capture. Hence just keep this in the ignore mode. > + */ > + trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; > + trblimitr |= (addr & PAGE_MASK); > + > + trblimitr |= TRBLIMITR_ENABLE; > + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > +} Same here > + > +static void trbe_enable_hw(struct trbe_buf *buf) > +{ > + WARN_ON(buf->trbe_write < buf->trbe_base); > + WARN_ON(buf->trbe_write >= buf->trbe_limit); > + set_trbe_disabled(); > + isb(); > + clr_trbe_status(); > + set_trbe_base_pointer(buf->trbe_base); > + set_trbe_write_pointer(buf->trbe_write); > + > + /* > + * Synchronize all the register updates > + * till now before enabling the TRBE. > + */ > + isb(); > + set_trbe_limit_pointer_enabled(buf->trbe_limit); > + > + /* Synchronize the TRBE enable event */ > + isb(); > +} Ok > + > +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, > + struct perf_event *event, void **pages, > + int nr_pages, bool snapshot) > +{ > + struct trbe_buf *buf; > + struct page **pglist; > + int i; > + > + /* > + * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with > + * just a single page, there is not much room left while writing into > + * a partially filled TRBE buffer. Hence restrict the minimum buffer > + * size as two pages. > + */ > + if (nr_pages < 2) > + return NULL; > + > + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); > + if (IS_ERR(buf)) > + return ERR_PTR(-ENOMEM); You know what do to. > + > + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); > + if (IS_ERR(pglist)) { > + kfree(buf); > + return ERR_PTR(-ENOMEM); > + } Here too. > + > + for (i = 0; i < nr_pages; i++) > + pglist[i] = virt_to_page(pages[i]); > + > + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); > + if (IS_ERR((void *)buf->trbe_base)) { > + kfree(pglist); > + kfree(buf); > + return ERR_PTR(buf->trbe_base); > + } Here too. > + buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; > + buf->trbe_write = buf->trbe_base; > + buf->snapshot = snapshot; > + buf->nr_pages = nr_pages; > + buf->pages = pages; > + kfree(pglist); > + return buf; > +} > + > +void arm_trbe_free_buffer(void *config) > +{ > + struct trbe_buf *buf = config; > + > + vunmap((void *)buf->trbe_base); > + kfree(buf); > +} Ok > + > +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, > + struct perf_output_handle *handle, > + void *config) > +{ > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > + struct trbe_buf *buf = config; > + unsigned long size, offset; > + > + WARN_ON(buf->cpudata != cpudata); > + WARN_ON(cpudata->cpu != smp_processor_id()); > + WARN_ON(cpudata->drvdata != drvdata); > + if (cpudata->mode != CS_MODE_PERF) > + return -EINVAL; > + > + /* > + * If the TRBE was disabled due to lack of space in the AUX buffer or a > + * spurious fault, the driver leaves it disabled, truncating the buffer. > + * Since the etm_perf driver expects to close out the AUX buffer, the > + * driver skips it. Thus, just pass in 0 size here to indicate that the > + * buffer was truncated. > + */ > + if (!is_trbe_enabled()) > + return 0; > + /* > + * perf handle structure needs to be shared with the TRBE IRQ handler for > + * capturing trace data and restarting the handle. There is a probability > + * of an undefined reference based crash when etm event is being stopped > + * while a TRBE IRQ also getting processed. This happens due the release > + * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping > + * the TRBE here will ensure that no IRQ could be generated when the perf > + * handle gets freed in etm_event_stop(). > + */ > + trbe_drain_and_disable_local(); > + offset = get_trbe_write_pointer() - get_trbe_base_pointer(); > + size = offset - PERF_IDX2OFF(handle->head, buf); > + if (buf->snapshot) > + handle->head += size; > + return size; > +} Ok - I really appreciate the comments. > + > +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) > +{ > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > + struct perf_output_handle *handle = data; > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + WARN_ON(cpudata->cpu != smp_processor_id()); > + WARN_ON(cpudata->drvdata != drvdata); > + if (mode != CS_MODE_PERF) > + return -EINVAL; > + > + *this_cpu_ptr(drvdata->handle) = handle; > + cpudata->buf = buf; > + cpudata->mode = mode; > + buf->cpudata = cpudata; > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > + if (buf->trbe_limit == buf->trbe_base) { > + trbe_stop_and_truncate_event(handle); > + return 0; > + } > + trbe_enable_hw(buf); > + return 0; > +} Ok > + > +static int arm_trbe_disable(struct coresight_device *csdev) > +{ > + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > + struct trbe_buf *buf = cpudata->buf; > + > + WARN_ON(buf->cpudata != cpudata); > + WARN_ON(cpudata->cpu != smp_processor_id()); > + WARN_ON(cpudata->drvdata != drvdata); > + if (cpudata->mode != CS_MODE_PERF) > + return -EINVAL; > + > + trbe_drain_and_disable_local(); > + buf->cpudata = NULL; > + cpudata->buf = NULL; > + cpudata->mode = CS_MODE_DISABLED; > + return 0; > +} Ok > + > +static void trbe_handle_spurious(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > + if (buf->trbe_limit == buf->trbe_base) { > + trbe_drain_and_disable_local(); > + return; > + } > + trbe_enable_hw(buf); > +} > + > +static void trbe_handle_overflow(struct perf_output_handle *handle) > +{ > + struct perf_event *event = handle->event; > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + unsigned long offset, size; > + struct etm_event_data *event_data; > + > + offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); > + size = offset - PERF_IDX2OFF(handle->head, buf); > + if (buf->snapshot) > + handle->head = offset; > + perf_aux_output_end(handle, size); > + > + event_data = perf_aux_output_begin(handle, event); > + if (!event_data) { > + trbe_drain_and_disable_local(); > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > + return; > + } > + buf->trbe_limit = compute_trbe_buffer_limit(handle); > + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > + if (buf->trbe_limit == buf->trbe_base) { > + trbe_stop_and_truncate_event(handle); > + return; > + } > + *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; > + trbe_enable_hw(buf); > +} > + > +static bool is_perf_trbe(struct perf_output_handle *handle) > +{ > + struct trbe_buf *buf = etm_perf_sink_config(handle); > + struct trbe_cpudata *cpudata = buf->cpudata; > + struct trbe_drvdata *drvdata = cpudata->drvdata; > + int cpu = smp_processor_id(); > + > + WARN_ON(buf->trbe_base != get_trbe_base_pointer()); > + WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); > + > + if (cpudata->mode != CS_MODE_PERF) > + return false; > + > + if (cpudata->cpu != cpu) > + return false; > + > + if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) > + return false; > + > + return true; > +} > + > +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) @handle isn't used for anything. > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + int ec = get_trbe_ec(trbsr); > + int bsc = get_trbe_bsc(trbsr); > + > + WARN_ON(is_trbe_running(trbsr)); > + if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) > + return TRBE_FAULT_ACT_FATAL; > + > + if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) > + return TRBE_FAULT_ACT_FATAL; > + > + if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { > + if (get_trbe_write_pointer() == get_trbe_base_pointer()) > + return TRBE_FAULT_ACT_WRAP; > + } > + return TRBE_FAULT_ACT_SPURIOUS; > +} > + > +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) > +{ > + struct perf_output_handle **handle_ptr = dev; > + struct perf_output_handle *handle = *handle_ptr; > + enum trbe_fault_action act; > + > + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); > + clr_trbe_irq(); > + > + /* > + * Ensure the trace is visible to the CPUs and > + * any external aborts have been resolved. > + */ > + trbe_drain_buffer(); > + isb(); > + > + if (!perf_get_aux(handle)) > + return IRQ_NONE; > + > + if (!is_perf_trbe(handle)) > + return IRQ_NONE; > + > + irq_work_run(); I trust Will that this is the right thing to do. I will stop here for this revision. I will dive more in the mechanic of the TRBE on the next revision. Thanks, Mathieu > + > + act = trbe_get_fault_act(handle); > + switch (act) { > + case TRBE_FAULT_ACT_WRAP: > + trbe_handle_overflow(handle); > + break; > + case TRBE_FAULT_ACT_SPURIOUS: > + trbe_handle_spurious(handle); > + break; > + case TRBE_FAULT_ACT_FATAL: > + trbe_stop_and_truncate_event(handle); > + break; > + } > + return IRQ_HANDLED; > +} > + > +static const struct coresight_ops_sink arm_trbe_sink_ops = { > + .enable = arm_trbe_enable, > + .disable = arm_trbe_disable, > + .alloc_buffer = arm_trbe_alloc_buffer, > + .free_buffer = arm_trbe_free_buffer, > + .update_buffer = arm_trbe_update_buffer, > +}; > + > +static const struct coresight_ops arm_trbe_cs_ops = { > + .sink_ops = &arm_trbe_sink_ops, > +}; > + > +static ssize_t align_show(struct device *dev, struct device_attribute *attr, char *buf) > +{ > + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > + > + return sprintf(buf, "%llx\n", cpudata->trbe_align); > +} > +static DEVICE_ATTR_RO(align); > + > +static ssize_t dbm_show(struct device *dev, struct device_attribute *attr, char *buf) > +{ > + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); > + > + return sprintf(buf, "%d\n", cpudata->trbe_dbm); > +} > +static DEVICE_ATTR_RO(dbm); > + > +static struct attribute *arm_trbe_attrs[] = { > + &dev_attr_align.attr, > + &dev_attr_dbm.attr, > + NULL, > +}; > + > +static const struct attribute_group arm_trbe_group = { > + .attrs = arm_trbe_attrs, > +}; > + > +static const struct attribute_group *arm_trbe_groups[] = { > + &arm_trbe_group, > + NULL, > +}; > + > +static void arm_trbe_probe_coresight_cpu(void *info) > +{ > + struct trbe_drvdata *drvdata = info; > + struct coresight_desc desc = { 0 }; > + int cpu = smp_processor_id(); > + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); > + struct device *dev; > + > + if (WARN_ON(!cpudata)) > + goto cpu_clear; > + > + if (trbe_csdev) > + return; > + > + cpudata->cpu = smp_processor_id(); > + cpudata->drvdata = drvdata; > + dev = &cpudata->drvdata->pdev->dev; > + > + if (!is_trbe_available()) { > + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + > + if (!is_trbe_programmable(trbidr)) { > + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); > + if (IS_ERR(desc.name)) > + goto cpu_clear; > + > + desc.type = CORESIGHT_DEV_TYPE_SINK; > + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; > + desc.ops = &arm_trbe_cs_ops; > + desc.pdata = dev_get_platdata(dev); > + desc.groups = arm_trbe_groups; > + desc.dev = dev; > + trbe_csdev = coresight_register(&desc); > + if (IS_ERR(trbe_csdev)) > + goto cpu_clear; > + > + dev_set_drvdata(&trbe_csdev->dev, cpudata); > + cpudata->trbe_dbm = get_trbe_flag_update(trbidr); > + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); > + if (cpudata->trbe_align > SZ_2K) { > + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); > + goto cpu_clear; > + } > + per_cpu(csdev_sink, cpu) = trbe_csdev; > + trbe_reset_local(); > + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > + return; > +cpu_clear: > + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); > +} > + > +static void arm_trbe_remove_coresight_cpu(void *info) > +{ > + int cpu = smp_processor_id(); > + struct trbe_drvdata *drvdata = info; > + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); > + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); > + > + if (trbe_csdev) { > + coresight_unregister(trbe_csdev); > + cpudata->drvdata = NULL; > + per_cpu(csdev_sink, cpu) = NULL; > + } > + disable_percpu_irq(drvdata->irq); > + trbe_reset_local(); > +} > + > +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) > +{ > + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); > + if (IS_ERR(drvdata->cpudata)) > + return PTR_ERR(drvdata->cpudata); > + > + arm_trbe_probe_coresight_cpu(drvdata); > + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); > + return 0; > +} > + > +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) > +{ > + arm_trbe_remove_coresight_cpu(drvdata); > + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); > + free_percpu(drvdata->cpudata); > + return 0; > +} > + > +static int arm_trbe_cpu_startup(unsigned int cpu, struct hlist_node *node) > +{ > + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > + > + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > + if (!per_cpu(csdev_sink, cpu)) { > + arm_trbe_probe_coresight_cpu(drvdata); > + } else { > + trbe_reset_local(); > + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); > + } > + } > + return 0; > +} > + > +static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node) > +{ > + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); > + > + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { > + disable_percpu_irq(drvdata->irq); > + trbe_reset_local(); > + } > + return 0; > +} > + > +static int arm_trbe_probe_cpuhp(struct trbe_drvdata *drvdata) > +{ > + enum cpuhp_state trbe_online; > + > + trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, > + arm_trbe_cpu_startup, arm_trbe_cpu_teardown); > + if (trbe_online < 0) > + return -EINVAL; > + > + if (cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node)) > + return -EINVAL; > + > + drvdata->trbe_online = trbe_online; > + return 0; > +} > + > +static void arm_trbe_remove_cpuhp(struct trbe_drvdata *drvdata) > +{ > + cpuhp_remove_multi_state(drvdata->trbe_online); > +} > + > +static int arm_trbe_probe_irq(struct platform_device *pdev, > + struct trbe_drvdata *drvdata) > +{ > + drvdata->irq = platform_get_irq(pdev, 0); > + if (!drvdata->irq) { > + pr_err("IRQ not found for the platform device\n"); > + return -ENXIO; > + } > + > + if (!irq_is_percpu(drvdata->irq)) { > + pr_err("IRQ is not a PPI\n"); > + return -EINVAL; > + } > + > + if (irq_get_percpu_devid_partition(drvdata->irq, &drvdata->supported_cpus)) > + return -EINVAL; > + > + drvdata->handle = alloc_percpu(typeof(*drvdata->handle)); > + if (!drvdata->handle) > + return -ENOMEM; > + > + if (request_percpu_irq(drvdata->irq, arm_trbe_irq_handler, DRVNAME, drvdata->handle)) { > + free_percpu(drvdata->handle); > + return -EINVAL; > + } > + return 0; > +} > + > +static void arm_trbe_remove_irq(struct trbe_drvdata *drvdata) > +{ > + free_percpu_irq(drvdata->irq, drvdata->handle); > + free_percpu(drvdata->handle); > +} > + > +static int arm_trbe_device_probe(struct platform_device *pdev) > +{ > + struct coresight_platform_data *pdata; > + struct trbe_drvdata *drvdata; > + struct device *dev = &pdev->dev; > + int ret; > + > + drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL); > + if (IS_ERR(drvdata)) > + return -ENOMEM; > + > + pdata = coresight_get_platform_data(dev); > + if (IS_ERR(pdata)) { > + kfree(drvdata); > + return -ENOMEM; > + } > + > + dev_set_drvdata(dev, drvdata); > + dev->platform_data = pdata; > + drvdata->pdev = pdev; > + ret = arm_trbe_probe_irq(pdev, drvdata); > + if (ret) > + goto irq_failed; > + > + ret = arm_trbe_probe_coresight(drvdata); > + if (ret) > + goto probe_failed; > + > + ret = arm_trbe_probe_cpuhp(drvdata); > + if (ret) > + goto cpuhp_failed; > + > + return 0; > +cpuhp_failed: > + arm_trbe_remove_coresight(drvdata); > +probe_failed: > + arm_trbe_remove_irq(drvdata); > +irq_failed: > + kfree(pdata); > + kfree(drvdata); > + return ret; > +} > + > +static int arm_trbe_device_remove(struct platform_device *pdev) > +{ > + struct coresight_platform_data *pdata = dev_get_platdata(&pdev->dev); > + struct trbe_drvdata *drvdata = platform_get_drvdata(pdev); > + > + arm_trbe_remove_coresight(drvdata); > + arm_trbe_remove_cpuhp(drvdata); > + arm_trbe_remove_irq(drvdata); > + kfree(pdata); > + kfree(drvdata); > + return 0; > +} > + > +static const struct of_device_id arm_trbe_of_match[] = { > + { .compatible = "arm,trace-buffer-extension"}, > + {}, > +}; > +MODULE_DEVICE_TABLE(of, arm_trbe_of_match); > + > +static struct platform_driver arm_trbe_driver = { > + .driver = { > + .name = DRVNAME, > + .of_match_table = of_match_ptr(arm_trbe_of_match), > + .suppress_bind_attrs = true, > + }, > + .probe = arm_trbe_device_probe, > + .remove = arm_trbe_device_remove, > +}; > + > +static int __init arm_trbe_init(void) > +{ > + int ret; > + > + if (arm64_kernel_unmapped_at_el0()) { > + pr_err("TRBE wouldn't work if kernel gets unmapped at EL0\n"); > + return -EOPNOTSUPP; > + } > + > + ret = platform_driver_register(&arm_trbe_driver); > + if (!ret) > + return 0; > + > + pr_err("Error registering %s platform driver\n", DRVNAME); > + return ret; > +} > + > +static void __exit arm_trbe_exit(void) > +{ > + platform_driver_unregister(&arm_trbe_driver); > +} > +module_init(arm_trbe_init); > +module_exit(arm_trbe_exit); > + > +MODULE_AUTHOR("Anshuman Khandual <anshuman.khandual@arm.com>"); > +MODULE_DESCRIPTION("Arm Trace Buffer Extension (TRBE) driver"); > +MODULE_LICENSE("GPL v2"); > diff --git a/drivers/hwtracing/coresight/coresight-trbe.h b/drivers/hwtracing/coresight/coresight-trbe.h > new file mode 100644 > index 0000000..43308bc > --- /dev/null > +++ b/drivers/hwtracing/coresight/coresight-trbe.h > @@ -0,0 +1,160 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * This contains all required hardware related helper functions for > + * Trace Buffer Extension (TRBE) driver in the coresight framework. > + * > + * Copyright (C) 2020 ARM Ltd. > + * > + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > + */ > +#include <linux/coresight.h> > +#include <linux/device.h> > +#include <linux/irq.h> > +#include <linux/kernel.h> > +#include <linux/of.h> > +#include <linux/platform_device.h> > +#include <linux/smp.h> > + > +#include "coresight-etm-perf.h" > + > +DECLARE_PER_CPU(struct coresight_device *, csdev_sink); > + > +static inline bool is_trbe_available(void) > +{ > + u64 aa64dfr0 = read_sysreg_s(SYS_ID_AA64DFR0_EL1); > + unsigned int trbe = cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_TRBE_SHIFT); > + > + return trbe >= 0b0001; > +} > + > +static inline bool is_trbe_enabled(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + return trblimitr & TRBLIMITR_ENABLE; > +} > + > +#define TRBE_EC_OTHERS 0 > +#define TRBE_EC_STAGE1_ABORT 36 > +#define TRBE_EC_STAGE2_ABORT 37 > + > +static inline int get_trbe_ec(u64 trbsr) > +{ > + return (trbsr >> TRBSR_EC_SHIFT) & TRBSR_EC_MASK; > +} > + > +#define TRBE_BSC_NOT_STOPPED 0 > +#define TRBE_BSC_FILLED 1 > +#define TRBE_BSC_TRIGGERED 2 > + > +static inline int get_trbe_bsc(u64 trbsr) > +{ > + return (trbsr >> TRBSR_BSC_SHIFT) & TRBSR_BSC_MASK; > +} > + > +static inline void clr_trbe_irq(void) > +{ > + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > + > + trbsr &= ~TRBSR_IRQ; > + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > +} > + > +static inline bool is_trbe_irq(u64 trbsr) > +{ > + return trbsr & TRBSR_IRQ; > +} > + > +static inline bool is_trbe_trg(u64 trbsr) > +{ > + return trbsr & TRBSR_TRG; > +} > + > +static inline bool is_trbe_wrap(u64 trbsr) > +{ > + return trbsr & TRBSR_WRAP; > +} > + > +static inline bool is_trbe_abort(u64 trbsr) > +{ > + return trbsr & TRBSR_ABORT; > +} > + > +static inline bool is_trbe_running(u64 trbsr) > +{ > + return !(trbsr & TRBSR_STOP); > +} > + > +#define TRBE_TRIG_MODE_STOP 0 > +#define TRBE_TRIG_MODE_IRQ 1 > +#define TRBE_TRIG_MODE_IGNORE 3 > + > +#define TRBE_FILL_MODE_FILL 0 > +#define TRBE_FILL_MODE_WRAP 1 > +#define TRBE_FILL_MODE_CIRCULAR_BUFFER 3 > + > +static inline void set_trbe_disabled(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + > + trblimitr &= ~TRBLIMITR_ENABLE; > + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > +} > + > +static inline bool get_trbe_flag_update(u64 trbidr) > +{ > + return trbidr & TRBIDR_FLAG; > +} > + > +static inline bool is_trbe_programmable(u64 trbidr) > +{ > + return !(trbidr & TRBIDR_PROG); > +} > + > +static inline int get_trbe_address_align(u64 trbidr) > +{ > + return (trbidr >> TRBIDR_ALIGN_SHIFT) & TRBIDR_ALIGN_MASK; > +} > + > +static inline unsigned long get_trbe_write_pointer(void) > +{ > + u64 trbptr = read_sysreg_s(SYS_TRBPTR_EL1); > + unsigned long addr = (trbptr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; > + > + return addr; > +} > + > +static inline void set_trbe_write_pointer(unsigned long addr) > +{ > + WARN_ON(is_trbe_enabled()); > + addr = (addr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; > + write_sysreg_s(addr, SYS_TRBPTR_EL1); > +} > + > +static inline unsigned long get_trbe_limit_pointer(void) > +{ > + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > + unsigned long limit = (trblimitr >> TRBLIMITR_LIMIT_SHIFT) & TRBLIMITR_LIMIT_MASK; > + unsigned long addr = limit << TRBLIMITR_LIMIT_SHIFT; > + > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + return addr; > +} > + > +static inline unsigned long get_trbe_base_pointer(void) > +{ > + u64 trbbaser = read_sysreg_s(SYS_TRBBASER_EL1); > + unsigned long addr = (trbbaser >> TRBBASER_BASE_SHIFT) & TRBBASER_BASE_MASK; > + > + addr = addr << TRBBASER_BASE_SHIFT; > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + return addr; > +} > + > +static inline void set_trbe_base_pointer(unsigned long addr) > +{ > + WARN_ON(is_trbe_enabled()); > + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > + write_sysreg_s(addr, SYS_TRBBASER_EL1); > +} > -- > 2.7.4 >
On 2/12/21 10:27 PM, Mathieu Poirier wrote: > [...] > >>> >>> >>>> + if (nr_pages < 2) >>>> + return NULL; >>>> + >>>> + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); >>>> + if (IS_ERR(buf)) >>>> + return ERR_PTR(-ENOMEM); >>>> + >>>> + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); >>>> + if (IS_ERR(pglist)) { >>>> + kfree(buf); >>>> + return ERR_PTR(-ENOMEM); >>>> + } >>>> + >>>> + for (i = 0; i < nr_pages; i++) >>>> + pglist[i] = virt_to_page(pages[i]); >>>> + >>>> + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); >>>> + if (IS_ERR((void *)buf->trbe_base)) { >>> >>> Why not simply make buf->trbe_base a void * instead of having to do all this >> >> There are many arithmetic and comparison operations involving trbe_base >> element. Hence it might be better to keep it as unsigned long, also to >> keeps it consistent with other pointers i.e trbe_write, trbe_limit. > > That is a fair point. Please add a comment to explain your design choice and > make sure the sparse checker is happy with all of it. Added a comment. > >> >> Snippet from $cat drivers/hwtracing/coresight/coresight-trbe.c | grep "trbe_base" >> There are just two places type casting trbe_base back to (void *). >> >> memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); >> return buf->trbe_base + offset; >> WARN_ON(buf->trbe_write < buf->trbe_base); >> set_trbe_base_pointer(buf->trbe_base); >> buf->trbe_base = (unsigned long)vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); >> if (IS_ERR((void *)buf->trbe_base)) { >> return ERR_PTR(buf->trbe_base); >> buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; >> buf->trbe_write = buf->trbe_base; >> vunmap((void *)buf->trbe_base); >> base = get_trbe_base_pointer(); >> buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >> if (buf->trbe_limit == buf->trbe_base) { >> buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >> if (buf->trbe_limit == buf->trbe_base) { >> offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); >> buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >> if (buf->trbe_limit == buf->trbe_base) { >> WARN_ON(buf->trbe_base != get_trbe_base_pointer()); >> if (get_trbe_write_pointer() == get_trbe_base_pointer()) >> >>> casting? And IS_ERR() doesn't work with vmap(). >> >> Sure, will drop IS_ERR() here. >> > > [...] > > >>> >>>> + >>>> +static ssize_t dbm_show(struct device *dev, struct device_attribute *attr, char *buf) >>>> +{ >>>> + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); >>>> + >>>> + return sprintf(buf, "%d\n", cpudata->trbe_dbm); >>>> +} >>>> +static DEVICE_ATTR_RO(dbm); >>> >>> What does "dbm" stand for? Looking at the documentation for TRBIDR_EL1.F, I >>> don't see what "dbm" relates to. >> >> I made it up to refer TRBIDR_EL1.F as "Dirty (and Access Flag) Bit Management". >> Could change it as "afdbm" to be more specific or if it is preferred. >> > > I don't see "afdbm" being a better solution - why not simply "flag"? Replaced all reference for "dbm" with "flag".
On 2/13/21 1:56 AM, Mathieu Poirier wrote: > On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: >> Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is >> accessible via the system registers. The TRBE supports different addressing >> modes including CPU virtual address and buffer modes including the circular >> buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), >> an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the >> access to the trace buffer could be prohibited by a higher exception level >> (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU >> private interrupt (PPI) on address translation errors and when the buffer >> is full. Overall implementation here is inspired from the Arm SPE driver. >> >> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> >> Cc: Mike Leach <mike.leach@linaro.org> >> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> >> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> >> --- >> Changes in V3: >> >> - Added new DT bindings document TRBE.yaml >> - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 >> - Dropped isb() from trbe_reset_local() >> - Dropped gap between (void *) and buf->trbe_base >> - Changed 'int' to 'unsigned int' in is_trbe_available() >> - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), >> set_trbe_enabled() and set_trbe_limit_pointer() >> - Changed get_trbe_flag_update(), is_trbe_programmable() and >> get_trbe_address_align() to accept TRBIDR value >> - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), >> is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value >> - Dropped snapshot mode condition in arm_trbe_alloc_buffer() >> - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled >> - Compute trbe_limit before trbe_write to get the updated handle >> - Added trbe_stop_and_truncate_event() >> - Dropped trbe_handle_fatal() >> >> Documentation/trace/coresight/coresight-trbe.rst | 39 + >> arch/arm64/include/asm/sysreg.h | 1 + >> drivers/hwtracing/coresight/Kconfig | 11 + >> drivers/hwtracing/coresight/Makefile | 1 + >> drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ >> drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ >> 6 files changed, 1235 insertions(+) >> create mode 100644 Documentation/trace/coresight/coresight-trbe.rst >> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c >> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h >> >> diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst >> new file mode 100644 >> index 0000000..1cbb819 >> --- /dev/null >> +++ b/Documentation/trace/coresight/coresight-trbe.rst >> @@ -0,0 +1,39 @@ >> +.. SPDX-License-Identifier: GPL-2.0 >> + >> +============================== >> +Trace Buffer Extension (TRBE). >> +============================== >> + >> + :Author: Anshuman Khandual <anshuman.khandual@arm.com> >> + :Date: November 2020 >> + >> +Hardware Description >> +-------------------- >> + >> +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system >> +memory, CPU traces generated from a corresponding percpu tracing unit. This >> +gets plugged in as a coresight sink device because the corresponding trace >> +genarators (ETE), are plugged in as source device. >> + >> +The TRBE is not compliant to CoreSight architecture specifications, but is >> +driven via the CoreSight driver framework to support the ETE (which is >> +CoreSight compliant) integration. >> + >> +Sysfs files and directories >> +--------------------------- >> + >> +The TRBE devices appear on the existing coresight bus alongside the other >> +coresight devices:: >> + >> + >$ ls /sys/bus/coresight/devices >> + trbe0 trbe1 trbe2 trbe3 >> + >> +The ``trbe<N>`` named TRBEs are associated with a CPU.:: >> + >> + >$ ls /sys/bus/coresight/devices/trbe0/ >> + align dbm >> + >> +*Key file items are:-* >> + * ``align``: TRBE write pointer alignment >> + * ``dbm``: TRBE updates memory with access and dirty flags >> + >> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h >> index 85ae4db..9e2e9b7 100644 >> --- a/arch/arm64/include/asm/sysreg.h >> +++ b/arch/arm64/include/asm/sysreg.h >> @@ -97,6 +97,7 @@ >> #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) >> #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) >> #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) >> +#define TSB_CSYNC __emit_inst(0xd503225f) >> >> #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) >> #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) >> diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig >> index f154ae7..aa657ab 100644 >> --- a/drivers/hwtracing/coresight/Kconfig >> +++ b/drivers/hwtracing/coresight/Kconfig >> @@ -164,6 +164,17 @@ config CORESIGHT_CTI >> To compile this driver as a module, choose M here: the >> module will be called coresight-cti. >> >> +config CORESIGHT_TRBE >> + bool "Trace Buffer Extension (TRBE) driver" >> + depends on ARM64 >> + help >> + This driver provides support for percpu Trace Buffer Extension (TRBE). >> + TRBE always needs to be used along with it's corresponding percpu ETE >> + component. ETE generates trace data which is then captured with TRBE. >> + Unlike traditional sink devices, TRBE is a CPU feature accessible via >> + system registers. But it's explicit dependency with trace unit (ETE) >> + requires it to be plugged in as a coresight sink device. >> + >> config CORESIGHT_CTI_INTEGRATION_REGS >> bool "Access CTI CoreSight Integration Registers" >> depends on CORESIGHT_CTI >> diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile >> index f20e357..d608165 100644 >> --- a/drivers/hwtracing/coresight/Makefile >> +++ b/drivers/hwtracing/coresight/Makefile >> @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o >> obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o >> obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o >> obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o >> +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o >> coresight-cti-y := coresight-cti-core.o coresight-cti-platform.o \ >> coresight-cti-sysfs.o >> diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c >> new file mode 100644 >> index 0000000..1464d8b >> --- /dev/null >> +++ b/drivers/hwtracing/coresight/coresight-trbe.c >> @@ -0,0 +1,1023 @@ >> +// SPDX-License-Identifier: GPL-2.0 >> +/* >> + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight >> + * sink device could then pair with an appropriate per-cpu coresight source >> + * device (ETE) thus generating required trace data. Trace can be enabled >> + * via the perf framework. >> + * >> + * Copyright (C) 2020 ARM Ltd. >> + * >> + * Author: Anshuman Khandual <anshuman.khandual@arm.com> >> + */ >> +#define DRVNAME "arm_trbe" >> + >> +#define pr_fmt(fmt) DRVNAME ": " fmt >> + >> +#include "coresight-trbe.h" >> + >> +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) >> + >> +/* >> + * A padding packet that will help the user space tools >> + * in skipping relevant sections in the captured trace >> + * data which could not be decoded. TRBE doesn't support >> + * formatting the trace data, unlike the legacy CoreSight >> + * sinks and thus we use ETE trace packets to pad the >> + * sections of the buffer. >> + */ >> +#define ETE_IGNORE_PACKET 0x70 >> + >> +/* >> + * Minimum amount of meaningful trace will contain: >> + * A-Sync, Trace Info, Trace On, Address, Atom. >> + * This is about 44bytes of ETE trace. To be on >> + * the safer side, we assume 64bytes is the minimum >> + * space required for a meaningful session, before >> + * we hit a "WRAP" event. >> + */ >> +#define TRBE_TRACE_MIN_BUF_SIZE 64 >> + >> +enum trbe_fault_action { >> + TRBE_FAULT_ACT_WRAP, >> + TRBE_FAULT_ACT_SPURIOUS, >> + TRBE_FAULT_ACT_FATAL, >> +}; >> + >> +struct trbe_buf { >> + unsigned long trbe_base; >> + unsigned long trbe_limit; >> + unsigned long trbe_write; >> + int nr_pages; >> + void **pages; >> + bool snapshot; >> + struct trbe_cpudata *cpudata; >> +}; >> + >> +struct trbe_cpudata { >> + bool trbe_dbm; >> + u64 trbe_align; >> + int cpu; >> + enum cs_mode mode; >> + struct trbe_buf *buf; >> + struct trbe_drvdata *drvdata; >> +}; >> + >> +struct trbe_drvdata { >> + struct trbe_cpudata __percpu *cpudata; >> + struct perf_output_handle __percpu **handle; >> + struct hlist_node hotplug_node; >> + int irq; >> + cpumask_t supported_cpus; >> + enum cpuhp_state trbe_online; >> + struct platform_device *pdev; >> +}; >> + >> +static int trbe_alloc_node(struct perf_event *event) >> +{ >> + if (event->cpu == -1) >> + return NUMA_NO_NODE; >> + return cpu_to_node(event->cpu); >> +} >> + >> +static void trbe_drain_buffer(void) >> +{ >> + asm(TSB_CSYNC); >> + dsb(nsh); >> +} >> + >> +static void trbe_drain_and_disable_local(void) >> +{ >> + trbe_drain_buffer(); >> + write_sysreg_s(0, SYS_TRBLIMITR_EL1); >> + isb(); >> +} >> + >> +static void trbe_reset_local(void) >> +{ >> + trbe_drain_and_disable_local(); >> + write_sysreg_s(0, SYS_TRBPTR_EL1); >> + write_sysreg_s(0, SYS_TRBBASER_EL1); >> + write_sysreg_s(0, SYS_TRBSR_EL1); >> +} >> + >> +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + >> + /* >> + * We cannot proceed with the buffer collection and we >> + * do not have any data for the current session. The >> + * etm_perf driver expects to close out the aux_buffer >> + * at event_stop(). So disable the TRBE here and leave >> + * the update_buffer() to return a 0 size. >> + */ >> + trbe_drain_and_disable_local(); >> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; >> +} >> + >> +/* >> + * TRBE Buffer Management >> + * >> + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, >> + * it starts writing trace data from the write pointer onward till the limit pointer. >> + * When the write pointer reaches the address just before the limit pointer, it gets >> + * wrapped around again to the base pointer. This is called a TRBE wrap event, which >> + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver >> + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ >> + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and >> + * LIMIT pointers. >> + * >> + * Wrap around with an IRQ >> + * ------ < ------ < ------- < ----- < ----- >> + * | | >> + * ------ > ------ > ------- > ----- > ----- >> + * >> + * +---------------+-----------------------+ >> + * | | | >> + * +---------------+-----------------------+ >> + * Base Pointer Write Pointer Limit Pointer >> + * >> + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write >> + * pointer can be aligned to the implementation defined TRBE trace buffer alignment >> + * as captured in trbe_cpudata->trbe_align. >> + * >> + * >> + * head tail wakeup >> + * +---------------------------------------+----- ~ ~ ------ >> + * |$$$$$$$|################|$$$$$$$$$$$$$$| | >> + * +---------------------------------------+----- ~ ~ ------ >> + * Base Pointer Write Pointer Limit Pointer >> + * >> + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing >> + * values which tracks all the driver writes and user reads from the perf auxiliary >> + * buffer. Generally [head..tail] is the area where the driver can write into unless >> + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and >> + * configured depending on the perf_output_handle indices, so that the driver does >> + * not override into areas in the perf auxiliary buffer which is being or yet to be >> + * consumed from the user space. The enabled TRBE buffer area is a moving subset of >> + * the allocated perf auxiliary buffer. >> + */ >> +static void trbe_pad_buf(struct perf_output_handle *handle, int len) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + u64 head = PERF_IDX2OFF(handle->head, buf); >> + >> + memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); >> + if (!buf->snapshot) >> + perf_aux_output_skip(handle, len); >> +} >> + >> +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + >> + /* >> + * The ETE trace has alignment synchronization packets allowing >> + * the decoder to reset in case of an overflow or corruption. >> + * So we can use the entire buffer for the snapshot mode. >> + */ >> + return buf->nr_pages * PAGE_SIZE; >> +} >> + >> +/* >> + * TRBE Limit Calculation >> + * >> + * The following markers are used to illustrate various TRBE buffer situations. >> + * >> + * $$$$ - Data area, unconsumed captured trace data, not to be overridden >> + * #### - Free area, enabled, trace will be written >> + * %%%% - Free area, disabled, trace will not be written >> + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped >> + */ >> +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + struct trbe_cpudata *cpudata = buf->cpudata; >> + const u64 bufsize = buf->nr_pages * PAGE_SIZE; >> + u64 limit = bufsize; >> + u64 head, tail, wakeup; >> + >> + head = PERF_IDX2OFF(handle->head, buf); >> + >> + /* >> + * head >> + * ------->| >> + * | >> + * head TRBE align tail >> + * +----|-------|---------------|-------+ >> + * |$$$$|=======|###############|$$$$$$$| >> + * +----|-------|---------------|-------+ >> + * trbe_base trbe_base + nr_pages >> + * >> + * Perf aux buffer output head position can be misaligned depending on >> + * various factors including user space reads. In case misaligned, head >> + * needs to be aligned before TRBE can be configured. Pad the alignment >> + * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools >> + * and skip this section thus advancing the head. >> + */ >> + if (!IS_ALIGNED(head, cpudata->trbe_align)) { >> + unsigned long delta = roundup(head, cpudata->trbe_align) - head; >> + >> + delta = min(delta, handle->size); >> + trbe_pad_buf(handle, delta); >> + head = PERF_IDX2OFF(handle->head, buf); >> + } >> + >> + /* >> + * head = tail (size = 0) >> + * +----|-------------------------------+ >> + * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ | >> + * +----|-------------------------------+ >> + * trbe_base trbe_base + nr_pages >> + * >> + * Perf aux buffer does not have any space for the driver to write into. >> + * Just communicate trace truncation event to the user space by marking >> + * it with PERF_AUX_FLAG_TRUNCATED. >> + */ >> + if (!handle->size) { >> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >> + return 0; >> + } >> + >> + /* Compute the tail and wakeup indices now that we've aligned head */ >> + tail = PERF_IDX2OFF(handle->head + handle->size, buf); >> + wakeup = PERF_IDX2OFF(handle->wakeup, buf); >> + >> + /* >> + * Lets calculate the buffer area which TRBE could write into. There >> + * are three possible scenarios here. Limit needs to be aligned with >> + * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the >> + * unconsumed data. >> + * >> + * 1) head < tail >> + * >> + * head tail >> + * +----|-----------------------|-------+ >> + * |$$$$|#######################|$$$$$$$| >> + * +----|-----------------------|-------+ >> + * trbe_base limit trbe_base + nr_pages >> + * >> + * TRBE could write into [head..tail] area. Unless the tail is right at >> + * the end of the buffer, neither an wrap around nor an IRQ is expected >> + * while being enabled. >> + * >> + * 2) head == tail >> + * >> + * head = tail (size > 0) >> + * +----|-------------------------------+ >> + * |%%%%|###############################| >> + * +----|-------------------------------+ >> + * trbe_base limit = trbe_base + nr_pages >> + * >> + * TRBE should just write into [head..base + nr_pages] area even though >> + * the entire buffer is empty. Reason being, when the trace reaches the >> + * end of the buffer, it will just wrap around with an IRQ giving an >> + * opportunity to reconfigure the buffer. >> + * >> + * 3) tail < head >> + * >> + * tail head >> + * +----|-----------------------|-------+ >> + * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| >> + * +----|-----------------------|-------+ >> + * trbe_base limit = trbe_base + nr_pages >> + * >> + * TRBE should just write into [head..base + nr_pages] area even though >> + * the [trbe_base..tail] is also empty. Reason being, when the trace >> + * reaches the end of the buffer, it will just wrap around with an IRQ >> + * giving an opportunity to reconfigure the buffer. >> + */ >> + if (head < tail) >> + limit = round_down(tail, PAGE_SIZE); >> + >> + /* >> + * Wakeup may be arbitrarily far into the future. If it's not in the >> + * current generation, either we'll wrap before hitting it, or it's >> + * in the past and has been handled already. >> + * >> + * If there's a wakeup before we wrap, arrange to be woken up by the >> + * page boundary following it. Keep the tail boundary if that's lower. >> + * >> + * head wakeup tail >> + * +----|---------------|-------|-------+ >> + * |$$$$|###############|%%%%%%%|$$$$$$$| >> + * +----|---------------|-------|-------+ >> + * trbe_base limit trbe_base + nr_pages >> + */ >> + if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) >> + limit = min(limit, round_up(wakeup, PAGE_SIZE)); >> + >> + /* >> + * There are two situation when this can happen i.e limit is before >> + * the head and hence TRBE cannot be configured. >> + * >> + * 1) head < tail (aligned down with PAGE_SIZE) and also they are both >> + * within the same PAGE size range. >> + * >> + * PAGE_SIZE >> + * |----------------------| >> + * >> + * limit head tail >> + * +------------|------|--------|-------+ >> + * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| >> + * +------------|------|--------|-------+ >> + * trbe_base trbe_base + nr_pages >> + * >> + * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both >> + * head and wakeup are within same PAGE size range. >> + * >> + * PAGE_SIZE >> + * |----------------------| >> + * >> + * limit head wakeup tail >> + * +----|------|-------|--------|-------+ >> + * |$$$$$$$$$$$|=======|========|$$$$$$$| >> + * +----|------|-------|--------|-------+ >> + * trbe_base trbe_base + nr_pages >> + */ >> + if (limit > head) >> + return limit; >> + >> + trbe_pad_buf(handle, handle->size); >> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >> + return 0; >> +} >> + >> +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = perf_get_aux(handle); >> + u64 limit = __trbe_normal_offset(handle); >> + u64 head = PERF_IDX2OFF(handle->head, buf); >> + >> + /* >> + * If the head is too close to the limit and we don't >> + * have space for a meaningful run, we rather pad it >> + * and start fresh. >> + */ >> + if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { >> + trbe_pad_buf(handle, limit - head); >> + limit = __trbe_normal_offset(handle); >> + } >> + return limit; >> +} >> + >> +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + unsigned long offset; >> + >> + if (buf->snapshot) >> + offset = trbe_snapshot_offset(handle); >> + else >> + offset = trbe_normal_offset(handle); >> + return buf->trbe_base + offset; >> +} > > I won't review the trace buffer management functions in this revision, I will > leave that for the next version. Okay. > >> + >> +static void clr_trbe_status(void) >> +{ >> + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); >> + >> + WARN_ON(is_trbe_enabled()); >> + trbsr &= ~TRBSR_IRQ; >> + trbsr &= ~TRBSR_TRG; >> + trbsr &= ~TRBSR_WRAP; >> + trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); >> + trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); >> + trbsr &= ~TRBSR_STOP; >> + write_sysreg_s(trbsr, SYS_TRBSR_EL1); >> +} > > I haven't read the TRBE progammer's manual but looking a the documentation the above > looks good. > >> + >> +static void set_trbe_limit_pointer_enabled(unsigned long addr) >> +{ >> + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); >> + >> + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); >> + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); >> + >> + trblimitr &= ~TRBLIMITR_NVM; >> + trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); >> + trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); >> + trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); >> + >> + /* >> + * Fill trace buffer mode is used here while configuring the >> + * TRBE for trace capture. In this particular mode, the trace >> + * collection is stopped and a maintenance interrupt is raised >> + * when the current write pointer wraps. This pause in trace >> + * collection gives the software an opportunity to capture the >> + * trace data in the interrupt handler, before reconfiguring >> + * the TRBE. >> + */ >> + trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; >> + >> + /* >> + * Trigger mode is not used here while configuring the TRBE for >> + * the trace capture. Hence just keep this in the ignore mode. >> + */ >> + trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; >> + trblimitr |= (addr & PAGE_MASK); >> + >> + trblimitr |= TRBLIMITR_ENABLE; >> + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); >> +} > > Same here > >> + >> +static void trbe_enable_hw(struct trbe_buf *buf) >> +{ >> + WARN_ON(buf->trbe_write < buf->trbe_base); >> + WARN_ON(buf->trbe_write >= buf->trbe_limit); >> + set_trbe_disabled(); >> + isb(); >> + clr_trbe_status(); >> + set_trbe_base_pointer(buf->trbe_base); >> + set_trbe_write_pointer(buf->trbe_write); >> + >> + /* >> + * Synchronize all the register updates >> + * till now before enabling the TRBE. >> + */ >> + isb(); >> + set_trbe_limit_pointer_enabled(buf->trbe_limit); >> + >> + /* Synchronize the TRBE enable event */ >> + isb(); >> +} > > Ok > >> + >> +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, >> + struct perf_event *event, void **pages, >> + int nr_pages, bool snapshot) >> +{ >> + struct trbe_buf *buf; >> + struct page **pglist; >> + int i; >> + >> + /* >> + * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with >> + * just a single page, there is not much room left while writing into >> + * a partially filled TRBE buffer. Hence restrict the minimum buffer >> + * size as two pages. >> + */ >> + if (nr_pages < 2) >> + return NULL; >> + >> + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); >> + if (IS_ERR(buf)) >> + return ERR_PTR(-ENOMEM); > > You know what do to. Right, will check for NULL instead and return ERR_PTR(-ENOMEM) as the function return type is (void *). > >> + >> + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); >> + if (IS_ERR(pglist)) { >> + kfree(buf); >> + return ERR_PTR(-ENOMEM); >> + } > > Here too. Yes, changed. > >> + >> + for (i = 0; i < nr_pages; i++) >> + pglist[i] = virt_to_page(pages[i]); >> + >> + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); >> + if (IS_ERR((void *)buf->trbe_base)) { >> + kfree(pglist); >> + kfree(buf); >> + return ERR_PTR(buf->trbe_base); >> + } > > Here too. Yes, changed. > >> + buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; >> + buf->trbe_write = buf->trbe_base; >> + buf->snapshot = snapshot; >> + buf->nr_pages = nr_pages; >> + buf->pages = pages; >> + kfree(pglist); >> + return buf; >> +} >> + >> +void arm_trbe_free_buffer(void *config) Added the missing 'static' here. >> +{ >> + struct trbe_buf *buf = config; >> + >> + vunmap((void *)buf->trbe_base); >> + kfree(buf); >> +} > > Ok > >> + >> +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, >> + struct perf_output_handle *handle, >> + void *config) >> +{ >> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >> + struct trbe_buf *buf = config; >> + unsigned long size, offset; >> + >> + WARN_ON(buf->cpudata != cpudata); >> + WARN_ON(cpudata->cpu != smp_processor_id()); >> + WARN_ON(cpudata->drvdata != drvdata); >> + if (cpudata->mode != CS_MODE_PERF) >> + return -EINVAL; >> + >> + /* >> + * If the TRBE was disabled due to lack of space in the AUX buffer or a >> + * spurious fault, the driver leaves it disabled, truncating the buffer. >> + * Since the etm_perf driver expects to close out the AUX buffer, the >> + * driver skips it. Thus, just pass in 0 size here to indicate that the >> + * buffer was truncated. >> + */ >> + if (!is_trbe_enabled()) >> + return 0; >> + /* >> + * perf handle structure needs to be shared with the TRBE IRQ handler for >> + * capturing trace data and restarting the handle. There is a probability >> + * of an undefined reference based crash when etm event is being stopped >> + * while a TRBE IRQ also getting processed. This happens due the release >> + * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping >> + * the TRBE here will ensure that no IRQ could be generated when the perf >> + * handle gets freed in etm_event_stop(). >> + */ >> + trbe_drain_and_disable_local(); >> + offset = get_trbe_write_pointer() - get_trbe_base_pointer(); >> + size = offset - PERF_IDX2OFF(handle->head, buf); >> + if (buf->snapshot) >> + handle->head += size; >> + return size; >> +} > > Ok - I really appreciate the comments. > >> + >> +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) >> +{ >> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >> + struct perf_output_handle *handle = data; >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + >> + WARN_ON(cpudata->cpu != smp_processor_id()); >> + WARN_ON(cpudata->drvdata != drvdata); >> + if (mode != CS_MODE_PERF) >> + return -EINVAL; >> + >> + *this_cpu_ptr(drvdata->handle) = handle; >> + cpudata->buf = buf; >> + cpudata->mode = mode; >> + buf->cpudata = cpudata; >> + buf->trbe_limit = compute_trbe_buffer_limit(handle); >> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >> + if (buf->trbe_limit == buf->trbe_base) { >> + trbe_stop_and_truncate_event(handle); >> + return 0; >> + } >> + trbe_enable_hw(buf); >> + return 0; >> +} > > Ok > >> + >> +static int arm_trbe_disable(struct coresight_device *csdev) >> +{ >> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >> + struct trbe_buf *buf = cpudata->buf; >> + >> + WARN_ON(buf->cpudata != cpudata); >> + WARN_ON(cpudata->cpu != smp_processor_id()); >> + WARN_ON(cpudata->drvdata != drvdata); >> + if (cpudata->mode != CS_MODE_PERF) >> + return -EINVAL; >> + >> + trbe_drain_and_disable_local(); >> + buf->cpudata = NULL; >> + cpudata->buf = NULL; >> + cpudata->mode = CS_MODE_DISABLED; >> + return 0; >> +} > > Ok > >> + >> +static void trbe_handle_spurious(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + >> + buf->trbe_limit = compute_trbe_buffer_limit(handle); >> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >> + if (buf->trbe_limit == buf->trbe_base) { >> + trbe_drain_and_disable_local(); >> + return; >> + } >> + trbe_enable_hw(buf); >> +} >> + >> +static void trbe_handle_overflow(struct perf_output_handle *handle) >> +{ >> + struct perf_event *event = handle->event; >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + unsigned long offset, size; >> + struct etm_event_data *event_data; >> + >> + offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); >> + size = offset - PERF_IDX2OFF(handle->head, buf); >> + if (buf->snapshot) >> + handle->head = offset; >> + perf_aux_output_end(handle, size); >> + >> + event_data = perf_aux_output_begin(handle, event); >> + if (!event_data) { >> + trbe_drain_and_disable_local(); >> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; >> + return; >> + } >> + buf->trbe_limit = compute_trbe_buffer_limit(handle); >> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >> + if (buf->trbe_limit == buf->trbe_base) { >> + trbe_stop_and_truncate_event(handle); >> + return; >> + } >> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; >> + trbe_enable_hw(buf); >> +} >> + >> +static bool is_perf_trbe(struct perf_output_handle *handle) >> +{ >> + struct trbe_buf *buf = etm_perf_sink_config(handle); >> + struct trbe_cpudata *cpudata = buf->cpudata; >> + struct trbe_drvdata *drvdata = cpudata->drvdata; >> + int cpu = smp_processor_id(); >> + >> + WARN_ON(buf->trbe_base != get_trbe_base_pointer()); >> + WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); >> + >> + if (cpudata->mode != CS_MODE_PERF) >> + return false; >> + >> + if (cpudata->cpu != cpu) >> + return false; >> + >> + if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) >> + return false; >> + >> + return true; >> +} >> + >> +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) > > @handle isn't used for anything. Okay, will drop. > >> +{ >> + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); >> + int ec = get_trbe_ec(trbsr); >> + int bsc = get_trbe_bsc(trbsr); >> + >> + WARN_ON(is_trbe_running(trbsr)); >> + if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) >> + return TRBE_FAULT_ACT_FATAL; >> + >> + if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) >> + return TRBE_FAULT_ACT_FATAL; >> + >> + if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { >> + if (get_trbe_write_pointer() == get_trbe_base_pointer()) >> + return TRBE_FAULT_ACT_WRAP; >> + } >> + return TRBE_FAULT_ACT_SPURIOUS; >> +} >> + >> +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) >> +{ >> + struct perf_output_handle **handle_ptr = dev; >> + struct perf_output_handle *handle = *handle_ptr; >> + enum trbe_fault_action act; >> + >> + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); >> + clr_trbe_irq(); >> + >> + /* >> + * Ensure the trace is visible to the CPUs and >> + * any external aborts have been resolved. >> + */ >> + trbe_drain_buffer(); >> + isb(); >> + >> + if (!perf_get_aux(handle)) >> + return IRQ_NONE; >> + >> + if (!is_perf_trbe(handle)) >> + return IRQ_NONE; >> + >> + irq_work_run(); > > I trust Will that this is the right thing to do. > > I will stop here for this revision. I will dive more in the mechanic of the > TRBE on the next revision. Okay, will collate all the changes till now and respin sooner. - Anshuman
Hi Anshuman, There have been plenty of detailed comments so I will restrict mine to a few general issues:- 1) Currently there appears to be no sysfs support (I cannot see the MODE_SYSFS constants running alongside the MODE_PERF ones present in the other sink drivers). This is present on all other coresight devices, and must be provided for this device. It is useful for testing, and there are users out there who will have scripts to use it. It is not essential it makes it into this set, but should be a follow up set. 2) Using FILL mode for TRBE means that the trace will by definition be lossy. Fill mode will halt collection without cleanly stopping and flushing the source. This will result in the sink missing the last of the data from the source as it stops. Even if taking the exception moves into a prohibited region there is still the possibility the last trace operations will not be seen. Further it is possible that the last few bytes of trace will be an incomplete packet, and indeed the start of the next buffer could contain incomplete packets too. This operation differs from the other sinks which will only halt after the sources have stopped and the path has been flushed. This ensures that the latest trace is complete. The weakness with the older sinks is the lack of interrupt meaning buffers were frequently wrapped so that only the latest trace is available. By using TRBE WRAP mode, with a watermark as described in the TRBE spec, using the interrupts it is possible to approach lossless trace in a way that is not possible with earlier ETR/ETB. This is something that has been requested by partners since trace became available in linux systems. (There is still a possibility of loss due to filling the buffer completely and overflowing the watermark, but that can be flagged). While FILL mode trace is a good start, and suitable for some scenarios - WRAP mode needs implementing as well. 3) Padding: To be clear, it is not safe for the decoder to run off the end of one buffer, into the padding area and continue decoding, or continue through the padding into the next buffer. However I believe the buffer start / stop points are demarked by the aux_output_start / aux_output_end calls? With upcoming perf decode updates this should enable the decoder to correctly be started and stopped on the buffer boundaries. The padding is there primarily to ensure that the decoder does not synchronize with the data stream until a genuine sync point is found. 4) TRBE needs to be a loadable module like the rest of coresight. Regards Mike On Mon, 15 Feb 2021 at 09:46, Anshuman Khandual <anshuman.khandual@arm.com> wrote: > > > On 2/13/21 1:56 AM, Mathieu Poirier wrote: > > On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: > >> Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is > >> accessible via the system registers. The TRBE supports different addressing > >> modes including CPU virtual address and buffer modes including the circular > >> buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), > >> an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the > >> access to the trace buffer could be prohibited by a higher exception level > >> (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU > >> private interrupt (PPI) on address translation errors and when the buffer > >> is full. Overall implementation here is inspired from the Arm SPE driver. > >> > >> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> > >> Cc: Mike Leach <mike.leach@linaro.org> > >> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> > >> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> > >> --- > >> Changes in V3: > >> > >> - Added new DT bindings document TRBE.yaml > >> - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 > >> - Dropped isb() from trbe_reset_local() > >> - Dropped gap between (void *) and buf->trbe_base > >> - Changed 'int' to 'unsigned int' in is_trbe_available() > >> - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), > >> set_trbe_enabled() and set_trbe_limit_pointer() > >> - Changed get_trbe_flag_update(), is_trbe_programmable() and > >> get_trbe_address_align() to accept TRBIDR value > >> - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), > >> is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value > >> - Dropped snapshot mode condition in arm_trbe_alloc_buffer() > >> - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled > >> - Compute trbe_limit before trbe_write to get the updated handle > >> - Added trbe_stop_and_truncate_event() > >> - Dropped trbe_handle_fatal() > >> > >> Documentation/trace/coresight/coresight-trbe.rst | 39 + > >> arch/arm64/include/asm/sysreg.h | 1 + > >> drivers/hwtracing/coresight/Kconfig | 11 + > >> drivers/hwtracing/coresight/Makefile | 1 + > >> drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ > >> drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ > >> 6 files changed, 1235 insertions(+) > >> create mode 100644 Documentation/trace/coresight/coresight-trbe.rst > >> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c > >> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h > >> > >> diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst > >> new file mode 100644 > >> index 0000000..1cbb819 > >> --- /dev/null > >> +++ b/Documentation/trace/coresight/coresight-trbe.rst > >> @@ -0,0 +1,39 @@ > >> +.. SPDX-License-Identifier: GPL-2.0 > >> + > >> +============================== > >> +Trace Buffer Extension (TRBE). > >> +============================== > >> + > >> + :Author: Anshuman Khandual <anshuman.khandual@arm.com> > >> + :Date: November 2020 > >> + > >> +Hardware Description > >> +-------------------- > >> + > >> +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system > >> +memory, CPU traces generated from a corresponding percpu tracing unit. This > >> +gets plugged in as a coresight sink device because the corresponding trace > >> +genarators (ETE), are plugged in as source device. > >> + > >> +The TRBE is not compliant to CoreSight architecture specifications, but is > >> +driven via the CoreSight driver framework to support the ETE (which is > >> +CoreSight compliant) integration. > >> + > >> +Sysfs files and directories > >> +--------------------------- > >> + > >> +The TRBE devices appear on the existing coresight bus alongside the other > >> +coresight devices:: > >> + > >> + >$ ls /sys/bus/coresight/devices > >> + trbe0 trbe1 trbe2 trbe3 > >> + > >> +The ``trbe<N>`` named TRBEs are associated with a CPU.:: > >> + > >> + >$ ls /sys/bus/coresight/devices/trbe0/ > >> + align dbm > >> + > >> +*Key file items are:-* > >> + * ``align``: TRBE write pointer alignment > >> + * ``dbm``: TRBE updates memory with access and dirty flags > >> + > >> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h > >> index 85ae4db..9e2e9b7 100644 > >> --- a/arch/arm64/include/asm/sysreg.h > >> +++ b/arch/arm64/include/asm/sysreg.h > >> @@ -97,6 +97,7 @@ > >> #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) > >> #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) > >> #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) > >> +#define TSB_CSYNC __emit_inst(0xd503225f) > >> > >> #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) > >> #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) > >> diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig > >> index f154ae7..aa657ab 100644 > >> --- a/drivers/hwtracing/coresight/Kconfig > >> +++ b/drivers/hwtracing/coresight/Kconfig > >> @@ -164,6 +164,17 @@ config CORESIGHT_CTI > >> To compile this driver as a module, choose M here: the > >> module will be called coresight-cti. > >> > >> +config CORESIGHT_TRBE > >> + bool "Trace Buffer Extension (TRBE) driver" > >> + depends on ARM64 > >> + help > >> + This driver provides support for percpu Trace Buffer Extension (TRBE). > >> + TRBE always needs to be used along with it's corresponding percpu ETE > >> + component. ETE generates trace data which is then captured with TRBE. > >> + Unlike traditional sink devices, TRBE is a CPU feature accessible via > >> + system registers. But it's explicit dependency with trace unit (ETE) > >> + requires it to be plugged in as a coresight sink device. > >> + > >> config CORESIGHT_CTI_INTEGRATION_REGS > >> bool "Access CTI CoreSight Integration Registers" > >> depends on CORESIGHT_CTI > >> diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile > >> index f20e357..d608165 100644 > >> --- a/drivers/hwtracing/coresight/Makefile > >> +++ b/drivers/hwtracing/coresight/Makefile > >> @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o > >> obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o > >> obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o > >> obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o > >> +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o > >> coresight-cti-y := coresight-cti-core.o coresight-cti-platform.o \ > >> coresight-cti-sysfs.o > >> diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c > >> new file mode 100644 > >> index 0000000..1464d8b > >> --- /dev/null > >> +++ b/drivers/hwtracing/coresight/coresight-trbe.c > >> @@ -0,0 +1,1023 @@ > >> +// SPDX-License-Identifier: GPL-2.0 > >> +/* > >> + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight > >> + * sink device could then pair with an appropriate per-cpu coresight source > >> + * device (ETE) thus generating required trace data. Trace can be enabled > >> + * via the perf framework. > >> + * > >> + * Copyright (C) 2020 ARM Ltd. > >> + * > >> + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > >> + */ > >> +#define DRVNAME "arm_trbe" > >> + > >> +#define pr_fmt(fmt) DRVNAME ": " fmt > >> + > >> +#include "coresight-trbe.h" > >> + > >> +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) > >> + > >> +/* > >> + * A padding packet that will help the user space tools > >> + * in skipping relevant sections in the captured trace > >> + * data which could not be decoded. TRBE doesn't support > >> + * formatting the trace data, unlike the legacy CoreSight > >> + * sinks and thus we use ETE trace packets to pad the > >> + * sections of the buffer. > >> + */ > >> +#define ETE_IGNORE_PACKET 0x70 > >> + > >> +/* > >> + * Minimum amount of meaningful trace will contain: > >> + * A-Sync, Trace Info, Trace On, Address, Atom. > >> + * This is about 44bytes of ETE trace. To be on > >> + * the safer side, we assume 64bytes is the minimum > >> + * space required for a meaningful session, before > >> + * we hit a "WRAP" event. > >> + */ > >> +#define TRBE_TRACE_MIN_BUF_SIZE 64 > >> + > >> +enum trbe_fault_action { > >> + TRBE_FAULT_ACT_WRAP, > >> + TRBE_FAULT_ACT_SPURIOUS, > >> + TRBE_FAULT_ACT_FATAL, > >> +}; > >> + > >> +struct trbe_buf { > >> + unsigned long trbe_base; > >> + unsigned long trbe_limit; > >> + unsigned long trbe_write; > >> + int nr_pages; > >> + void **pages; > >> + bool snapshot; > >> + struct trbe_cpudata *cpudata; > >> +}; > >> + > >> +struct trbe_cpudata { > >> + bool trbe_dbm; > >> + u64 trbe_align; > >> + int cpu; > >> + enum cs_mode mode; > >> + struct trbe_buf *buf; > >> + struct trbe_drvdata *drvdata; > >> +}; > >> + > >> +struct trbe_drvdata { > >> + struct trbe_cpudata __percpu *cpudata; > >> + struct perf_output_handle __percpu **handle; > >> + struct hlist_node hotplug_node; > >> + int irq; > >> + cpumask_t supported_cpus; > >> + enum cpuhp_state trbe_online; > >> + struct platform_device *pdev; > >> +}; > >> + > >> +static int trbe_alloc_node(struct perf_event *event) > >> +{ > >> + if (event->cpu == -1) > >> + return NUMA_NO_NODE; > >> + return cpu_to_node(event->cpu); > >> +} > >> + > >> +static void trbe_drain_buffer(void) > >> +{ > >> + asm(TSB_CSYNC); > >> + dsb(nsh); > >> +} > >> + > >> +static void trbe_drain_and_disable_local(void) > >> +{ > >> + trbe_drain_buffer(); > >> + write_sysreg_s(0, SYS_TRBLIMITR_EL1); > >> + isb(); > >> +} > >> + > >> +static void trbe_reset_local(void) > >> +{ > >> + trbe_drain_and_disable_local(); > >> + write_sysreg_s(0, SYS_TRBPTR_EL1); > >> + write_sysreg_s(0, SYS_TRBBASER_EL1); > >> + write_sysreg_s(0, SYS_TRBSR_EL1); > >> +} > >> + > >> +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) > >> +{ > >> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >> + > >> + /* > >> + * We cannot proceed with the buffer collection and we > >> + * do not have any data for the current session. The > >> + * etm_perf driver expects to close out the aux_buffer > >> + * at event_stop(). So disable the TRBE here and leave > >> + * the update_buffer() to return a 0 size. > >> + */ > >> + trbe_drain_and_disable_local(); > >> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > >> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > >> +} > >> + > >> +/* > >> + * TRBE Buffer Management > >> + * > >> + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, > >> + * it starts writing trace data from the write pointer onward till the limit pointer. > >> + * When the write pointer reaches the address just before the limit pointer, it gets > >> + * wrapped around again to the base pointer. This is called a TRBE wrap event, which > >> + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver > >> + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ > >> + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and > >> + * LIMIT pointers. > >> + * > >> + * Wrap around with an IRQ > >> + * ------ < ------ < ------- < ----- < ----- > >> + * | | > >> + * ------ > ------ > ------- > ----- > ----- > >> + * > >> + * +---------------+-----------------------+ > >> + * | | | > >> + * +---------------+-----------------------+ > >> + * Base Pointer Write Pointer Limit Pointer > >> + * > >> + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write > >> + * pointer can be aligned to the implementation defined TRBE trace buffer alignment > >> + * as captured in trbe_cpudata->trbe_align. > >> + * > >> + * > >> + * head tail wakeup > >> + * +---------------------------------------+----- ~ ~ ------ > >> + * |$$$$$$$|################|$$$$$$$$$$$$$$| | > >> + * +---------------------------------------+----- ~ ~ ------ > >> + * Base Pointer Write Pointer Limit Pointer > >> + * > >> + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing > >> + * values which tracks all the driver writes and user reads from the perf auxiliary > >> + * buffer. Generally [head..tail] is the area where the driver can write into unless > >> + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and > >> + * configured depending on the perf_output_handle indices, so that the driver does > >> + * not override into areas in the perf auxiliary buffer which is being or yet to be > >> + * consumed from the user space. The enabled TRBE buffer area is a moving subset of > >> + * the allocated perf auxiliary buffer. > >> + */ > >> +static void trbe_pad_buf(struct perf_output_handle *handle, int len) > >> +{ > >> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >> + u64 head = PERF_IDX2OFF(handle->head, buf); > >> + > >> + memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); > >> + if (!buf->snapshot) > >> + perf_aux_output_skip(handle, len); > >> +} > >> + > >> +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) > >> +{ > >> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >> + > >> + /* > >> + * The ETE trace has alignment synchronization packets allowing > >> + * the decoder to reset in case of an overflow or corruption. > >> + * So we can use the entire buffer for the snapshot mode. > >> + */ > >> + return buf->nr_pages * PAGE_SIZE; > >> +} > >> + > >> +/* > >> + * TRBE Limit Calculation > >> + * > >> + * The following markers are used to illustrate various TRBE buffer situations. > >> + * > >> + * $$$$ - Data area, unconsumed captured trace data, not to be overridden > >> + * #### - Free area, enabled, trace will be written > >> + * %%%% - Free area, disabled, trace will not be written > >> + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped > >> + */ > >> +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) > >> +{ > >> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >> + struct trbe_cpudata *cpudata = buf->cpudata; > >> + const u64 bufsize = buf->nr_pages * PAGE_SIZE; > >> + u64 limit = bufsize; > >> + u64 head, tail, wakeup; > >> + > >> + head = PERF_IDX2OFF(handle->head, buf); > >> + > >> + /* > >> + * head > >> + * ------->| > >> + * | > >> + * head TRBE align tail > >> + * +----|-------|---------------|-------+ > >> + * |$$$$|=======|###############|$$$$$$$| > >> + * +----|-------|---------------|-------+ > >> + * trbe_base trbe_base + nr_pages > >> + * > >> + * Perf aux buffer output head position can be misaligned depending on > >> + * various factors including user space reads. In case misaligned, head > >> + * needs to be aligned before TRBE can be configured. Pad the alignment > >> + * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools > >> + * and skip this section thus advancing the head. > >> + */ > >> + if (!IS_ALIGNED(head, cpudata->trbe_align)) { > >> + unsigned long delta = roundup(head, cpudata->trbe_align) - head; > >> + > >> + delta = min(delta, handle->size); > >> + trbe_pad_buf(handle, delta); > >> + head = PERF_IDX2OFF(handle->head, buf); > >> + } > >> + > >> + /* > >> + * head = tail (size = 0) > >> + * +----|-------------------------------+ > >> + * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ | > >> + * +----|-------------------------------+ > >> + * trbe_base trbe_base + nr_pages > >> + * > >> + * Perf aux buffer does not have any space for the driver to write into. > >> + * Just communicate trace truncation event to the user space by marking > >> + * it with PERF_AUX_FLAG_TRUNCATED. > >> + */ > >> + if (!handle->size) { > >> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > >> + return 0; > >> + } > >> + > >> + /* Compute the tail and wakeup indices now that we've aligned head */ > >> + tail = PERF_IDX2OFF(handle->head + handle->size, buf); > >> + wakeup = PERF_IDX2OFF(handle->wakeup, buf); > >> + > >> + /* > >> + * Lets calculate the buffer area which TRBE could write into. There > >> + * are three possible scenarios here. Limit needs to be aligned with > >> + * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the > >> + * unconsumed data. > >> + * > >> + * 1) head < tail > >> + * > >> + * head tail > >> + * +----|-----------------------|-------+ > >> + * |$$$$|#######################|$$$$$$$| > >> + * +----|-----------------------|-------+ > >> + * trbe_base limit trbe_base + nr_pages > >> + * > >> + * TRBE could write into [head..tail] area. Unless the tail is right at > >> + * the end of the buffer, neither an wrap around nor an IRQ is expected > >> + * while being enabled. > >> + * > >> + * 2) head == tail > >> + * > >> + * head = tail (size > 0) > >> + * +----|-------------------------------+ > >> + * |%%%%|###############################| > >> + * +----|-------------------------------+ > >> + * trbe_base limit = trbe_base + nr_pages > >> + * > >> + * TRBE should just write into [head..base + nr_pages] area even though > >> + * the entire buffer is empty. Reason being, when the trace reaches the > >> + * end of the buffer, it will just wrap around with an IRQ giving an > >> + * opportunity to reconfigure the buffer. > >> + * > >> + * 3) tail < head > >> + * > >> + * tail head > >> + * +----|-----------------------|-------+ > >> + * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| > >> + * +----|-----------------------|-------+ > >> + * trbe_base limit = trbe_base + nr_pages > >> + * > >> + * TRBE should just write into [head..base + nr_pages] area even though > >> + * the [trbe_base..tail] is also empty. Reason being, when the trace > >> + * reaches the end of the buffer, it will just wrap around with an IRQ > >> + * giving an opportunity to reconfigure the buffer. > >> + */ > >> + if (head < tail) > >> + limit = round_down(tail, PAGE_SIZE); > >> + > >> + /* > >> + * Wakeup may be arbitrarily far into the future. If it's not in the > >> + * current generation, either we'll wrap before hitting it, or it's > >> + * in the past and has been handled already. > >> + * > >> + * If there's a wakeup before we wrap, arrange to be woken up by the > >> + * page boundary following it. Keep the tail boundary if that's lower. > >> + * > >> + * head wakeup tail > >> + * +----|---------------|-------|-------+ > >> + * |$$$$|###############|%%%%%%%|$$$$$$$| > >> + * +----|---------------|-------|-------+ > >> + * trbe_base limit trbe_base + nr_pages > >> + */ > >> + if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) > >> + limit = min(limit, round_up(wakeup, PAGE_SIZE)); > >> + > >> + /* > >> + * There are two situation when this can happen i.e limit is before > >> + * the head and hence TRBE cannot be configured. > >> + * > >> + * 1) head < tail (aligned down with PAGE_SIZE) and also they are both > >> + * within the same PAGE size range. > >> + * > >> + * PAGE_SIZE > >> + * |----------------------| > >> + * > >> + * limit head tail > >> + * +------------|------|--------|-------+ > >> + * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| > >> + * +------------|------|--------|-------+ > >> + * trbe_base trbe_base + nr_pages > >> + * > >> + * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both > >> + * head and wakeup are within same PAGE size range. > >> + * > >> + * PAGE_SIZE > >> + * |----------------------| > >> + * > >> + * limit head wakeup tail > >> + * +----|------|-------|--------|-------+ > >> + * |$$$$$$$$$$$|=======|========|$$$$$$$| > >> + * +----|------|-------|--------|-------+ > >> + * trbe_base trbe_base + nr_pages > >> + */ > >> + if (limit > head) > >> + return limit; > >> + > >> + trbe_pad_buf(handle, handle->size); > >> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > >> + return 0; > >> +} > >> + > >> +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) > >> +{ > >> + struct trbe_buf *buf = perf_get_aux(handle); > >> + u64 limit = __trbe_normal_offset(handle); > >> + u64 head = PERF_IDX2OFF(handle->head, buf); > >> + > >> + /* > >> + * If the head is too close to the limit and we don't > >> + * have space for a meaningful run, we rather pad it > >> + * and start fresh. > >> + */ > >> + if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { > >> + trbe_pad_buf(handle, limit - head); > >> + limit = __trbe_normal_offset(handle); > >> + } > >> + return limit; > >> +} > >> + > >> +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) > >> +{ > >> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >> + unsigned long offset; > >> + > >> + if (buf->snapshot) > >> + offset = trbe_snapshot_offset(handle); > >> + else > >> + offset = trbe_normal_offset(handle); > >> + return buf->trbe_base + offset; > >> +} > > > > I won't review the trace buffer management functions in this revision, I will > > leave that for the next version. > > Okay. > > > > >> + > >> +static void clr_trbe_status(void) > >> +{ > >> + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > >> + > >> + WARN_ON(is_trbe_enabled()); > >> + trbsr &= ~TRBSR_IRQ; > >> + trbsr &= ~TRBSR_TRG; > >> + trbsr &= ~TRBSR_WRAP; > >> + trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); > >> + trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); > >> + trbsr &= ~TRBSR_STOP; > >> + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > >> +} > > > > I haven't read the TRBE progammer's manual but looking a the documentation the above > > looks good. > > > >> + > >> +static void set_trbe_limit_pointer_enabled(unsigned long addr) > >> +{ > >> + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > >> + > >> + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > >> + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > >> + > >> + trblimitr &= ~TRBLIMITR_NVM; > >> + trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); > >> + trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); > >> + trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); > >> + > >> + /* > >> + * Fill trace buffer mode is used here while configuring the > >> + * TRBE for trace capture. In this particular mode, the trace > >> + * collection is stopped and a maintenance interrupt is raised > >> + * when the current write pointer wraps. This pause in trace > >> + * collection gives the software an opportunity to capture the > >> + * trace data in the interrupt handler, before reconfiguring > >> + * the TRBE. > >> + */ > >> + trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; > >> + > >> + /* > >> + * Trigger mode is not used here while configuring the TRBE for > >> + * the trace capture. Hence just keep this in the ignore mode. > >> + */ > >> + trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; > >> + trblimitr |= (addr & PAGE_MASK); > >> + > >> + trblimitr |= TRBLIMITR_ENABLE; > >> + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > >> +} > > > > Same here > > > >> + > >> +static void trbe_enable_hw(struct trbe_buf *buf) > >> +{ > >> + WARN_ON(buf->trbe_write < buf->trbe_base); > >> + WARN_ON(buf->trbe_write >= buf->trbe_limit); > >> + set_trbe_disabled(); > >> + isb(); > >> + clr_trbe_status(); > >> + set_trbe_base_pointer(buf->trbe_base); > >> + set_trbe_write_pointer(buf->trbe_write); > >> + > >> + /* > >> + * Synchronize all the register updates > >> + * till now before enabling the TRBE. > >> + */ > >> + isb(); > >> + set_trbe_limit_pointer_enabled(buf->trbe_limit); > >> + > >> + /* Synchronize the TRBE enable event */ > >> + isb(); > >> +} > > > > Ok > > > >> + > >> +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, > >> + struct perf_event *event, void **pages, > >> + int nr_pages, bool snapshot) > >> +{ > >> + struct trbe_buf *buf; > >> + struct page **pglist; > >> + int i; > >> + > >> + /* > >> + * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with > >> + * just a single page, there is not much room left while writing into > >> + * a partially filled TRBE buffer. Hence restrict the minimum buffer > >> + * size as two pages. > >> + */ > >> + if (nr_pages < 2) > >> + return NULL; > >> + > >> + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); > >> + if (IS_ERR(buf)) > >> + return ERR_PTR(-ENOMEM); > > > > You know what do to. > > Right, will check for NULL instead and return ERR_PTR(-ENOMEM) as > the function return type is (void *). > > > > >> + > >> + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); > >> + if (IS_ERR(pglist)) { > >> + kfree(buf); > >> + return ERR_PTR(-ENOMEM); > >> + } > > > > Here too. > > Yes, changed. > > > > >> + > >> + for (i = 0; i < nr_pages; i++) > >> + pglist[i] = virt_to_page(pages[i]); > >> + > >> + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); > >> + if (IS_ERR((void *)buf->trbe_base)) { > >> + kfree(pglist); > >> + kfree(buf); > >> + return ERR_PTR(buf->trbe_base); > >> + } > > > > Here too. > > Yes, changed. > > > > >> + buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; > >> + buf->trbe_write = buf->trbe_base; > >> + buf->snapshot = snapshot; > >> + buf->nr_pages = nr_pages; > >> + buf->pages = pages; > >> + kfree(pglist); > >> + return buf; > >> +} > >> + > >> +void arm_trbe_free_buffer(void *config) > > Added the missing 'static' here. > > >> +{ > >> + struct trbe_buf *buf = config; > >> + > >> + vunmap((void *)buf->trbe_base); > >> + kfree(buf); > >> +} > > > > Ok > > > >> + > >> +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, > >> + struct perf_output_handle *handle, > >> + void *config) > >> +{ > >> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > >> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > >> + struct trbe_buf *buf = config; > >> + unsigned long size, offset; > >> + > >> + WARN_ON(buf->cpudata != cpudata); > >> + WARN_ON(cpudata->cpu != smp_processor_id()); > >> + WARN_ON(cpudata->drvdata != drvdata); > >> + if (cpudata->mode != CS_MODE_PERF) > >> + return -EINVAL; > >> + > >> + /* > >> + * If the TRBE was disabled due to lack of space in the AUX buffer or a > >> + * spurious fault, the driver leaves it disabled, truncating the buffer. > >> + * Since the etm_perf driver expects to close out the AUX buffer, the > >> + * driver skips it. Thus, just pass in 0 size here to indicate that the > >> + * buffer was truncated. > >> + */ > >> + if (!is_trbe_enabled()) > >> + return 0; > >> + /* > >> + * perf handle structure needs to be shared with the TRBE IRQ handler for > >> + * capturing trace data and restarting the handle. There is a probability > >> + * of an undefined reference based crash when etm event is being stopped > >> + * while a TRBE IRQ also getting processed. This happens due the release > >> + * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping > >> + * the TRBE here will ensure that no IRQ could be generated when the perf > >> + * handle gets freed in etm_event_stop(). > >> + */ > >> + trbe_drain_and_disable_local(); > >> + offset = get_trbe_write_pointer() - get_trbe_base_pointer(); > >> + size = offset - PERF_IDX2OFF(handle->head, buf); > >> + if (buf->snapshot) > >> + handle->head += size; > >> + return size; > >> +} > > > > Ok - I really appreciate the comments. > > > >> + > >> +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) > >> +{ > >> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > >> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > >> + struct perf_output_handle *handle = data; > >> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >> + > >> + WARN_ON(cpudata->cpu != smp_processor_id()); > >> + WARN_ON(cpudata->drvdata != drvdata); > >> + if (mode != CS_MODE_PERF) > >> + return -EINVAL; > >> + > >> + *this_cpu_ptr(drvdata->handle) = handle; > >> + cpudata->buf = buf; > >> + cpudata->mode = mode; > >> + buf->cpudata = cpudata; > >> + buf->trbe_limit = compute_trbe_buffer_limit(handle); > >> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > >> + if (buf->trbe_limit == buf->trbe_base) { > >> + trbe_stop_and_truncate_event(handle); > >> + return 0; > >> + } > >> + trbe_enable_hw(buf); > >> + return 0; > >> +} > > > > Ok > > > >> + > >> +static int arm_trbe_disable(struct coresight_device *csdev) > >> +{ > >> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > >> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > >> + struct trbe_buf *buf = cpudata->buf; > >> + > >> + WARN_ON(buf->cpudata != cpudata); > >> + WARN_ON(cpudata->cpu != smp_processor_id()); > >> + WARN_ON(cpudata->drvdata != drvdata); > >> + if (cpudata->mode != CS_MODE_PERF) > >> + return -EINVAL; > >> + > >> + trbe_drain_and_disable_local(); > >> + buf->cpudata = NULL; > >> + cpudata->buf = NULL; > >> + cpudata->mode = CS_MODE_DISABLED; > >> + return 0; > >> +} > > > > Ok > > > >> + > >> +static void trbe_handle_spurious(struct perf_output_handle *handle) > >> +{ > >> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >> + > >> + buf->trbe_limit = compute_trbe_buffer_limit(handle); > >> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > >> + if (buf->trbe_limit == buf->trbe_base) { > >> + trbe_drain_and_disable_local(); > >> + return; > >> + } > >> + trbe_enable_hw(buf); > >> +} > >> + > >> +static void trbe_handle_overflow(struct perf_output_handle *handle) > >> +{ > >> + struct perf_event *event = handle->event; > >> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >> + unsigned long offset, size; > >> + struct etm_event_data *event_data; > >> + > >> + offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); > >> + size = offset - PERF_IDX2OFF(handle->head, buf); > >> + if (buf->snapshot) > >> + handle->head = offset; > >> + perf_aux_output_end(handle, size); > >> + > >> + event_data = perf_aux_output_begin(handle, event); > >> + if (!event_data) { > >> + trbe_drain_and_disable_local(); > >> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > >> + return; > >> + } > >> + buf->trbe_limit = compute_trbe_buffer_limit(handle); > >> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > >> + if (buf->trbe_limit == buf->trbe_base) { > >> + trbe_stop_and_truncate_event(handle); > >> + return; > >> + } > >> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; > >> + trbe_enable_hw(buf); > >> +} > >> + > >> +static bool is_perf_trbe(struct perf_output_handle *handle) > >> +{ > >> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >> + struct trbe_cpudata *cpudata = buf->cpudata; > >> + struct trbe_drvdata *drvdata = cpudata->drvdata; > >> + int cpu = smp_processor_id(); > >> + > >> + WARN_ON(buf->trbe_base != get_trbe_base_pointer()); > >> + WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); > >> + > >> + if (cpudata->mode != CS_MODE_PERF) > >> + return false; > >> + > >> + if (cpudata->cpu != cpu) > >> + return false; > >> + > >> + if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) > >> + return false; > >> + > >> + return true; > >> +} > >> + > >> +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) > > > > @handle isn't used for anything. > > Okay, will drop. > > > > >> +{ > >> + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > >> + int ec = get_trbe_ec(trbsr); > >> + int bsc = get_trbe_bsc(trbsr); > >> + > >> + WARN_ON(is_trbe_running(trbsr)); > >> + if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) > >> + return TRBE_FAULT_ACT_FATAL; > >> + > >> + if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) > >> + return TRBE_FAULT_ACT_FATAL; > >> + > >> + if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { > >> + if (get_trbe_write_pointer() == get_trbe_base_pointer()) > >> + return TRBE_FAULT_ACT_WRAP; > >> + } > >> + return TRBE_FAULT_ACT_SPURIOUS; > >> +} > >> + > >> +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) > >> +{ > >> + struct perf_output_handle **handle_ptr = dev; > >> + struct perf_output_handle *handle = *handle_ptr; > >> + enum trbe_fault_action act; > >> + > >> + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); > >> + clr_trbe_irq(); > >> + > >> + /* > >> + * Ensure the trace is visible to the CPUs and > >> + * any external aborts have been resolved. > >> + */ > >> + trbe_drain_buffer(); > >> + isb(); > >> + > >> + if (!perf_get_aux(handle)) > >> + return IRQ_NONE; > >> + > >> + if (!is_perf_trbe(handle)) > >> + return IRQ_NONE; > >> + > >> + irq_work_run(); > > > > I trust Will that this is the right thing to do. > > > > I will stop here for this revision. I will dive more in the mechanic of the > > TRBE on the next revision. > > Okay, will collate all the changes till now and respin sooner. > > - Anshuman
Hello Mike, On 2/16/21 2:30 PM, Mike Leach wrote: > Hi Anshuman, > > There have been plenty of detailed comments so I will restrict mine to > a few general issues:- > > 1) Currently there appears to be no sysfs support (I cannot see the > MODE_SYSFS constants running alongside the MODE_PERF ones present in > the other sink drivers). This is present on all other coresight > devices, and must be provided for this device. It is useful for > testing, and there are users out there who will have scripts to use > it. It is not essential it makes it into this set, but should be a > follow up set. Sure, will try and add it in a follow up series. > > 2) Using FILL mode for TRBE means that the trace will by definition be > lossy. Fill mode will halt collection without cleanly stopping and > flushing the source. This will result in the sink missing the last of > the data from the source as it stops. Even if taking the exception > moves into a prohibited region there is still the possibility the last > trace operations will not be seen. Further it is possible that the > last few bytes of trace will be an incomplete packet, and indeed the > start of the next buffer could contain incomplete packets too. Just wondering why TRBE and ETE would not sync with each other in order for the ETE to possibly resend all the lost trace data, when the TRBE runs out of buffer and wrappers around ? Is this ETE/TRBE behavior same for all implementations in the FILL mode ? Just wondering. > > This operation differs from the other sinks which will only halt after > the sources have stopped and the path has been flushed. This ensures > that the latest trace is complete. The weakness with the older sinks > is the lack of interrupt meaning buffers were frequently wrapped so > that only the latest trace is available. Right. > > By using TRBE WRAP mode, with a watermark as described in the TRBE > spec, using the interrupts it is possible to approach lossless trace > in a way that is not possible with earlier ETR/ETB. This is somethin Using TRBTRG_EL1 as the above mentioned watermark ? > that has been requested by partners since trace became available in > linux systems. (There is still a possibility of loss due to filling > the buffer completely and overflowing the watermark, but that can be > flagged). > > While FILL mode trace is a good start, and suitable for some scenarios > - WRAP mode needs implementing as well. I would like to understand this mechanism more. Besides how the perf interface suppose to choose between FILL and WRAP mode ? via a new event attribute ? > > 3) Padding: To be clear, it is not safe for the decoder to run off the > end of one buffer, into the padding area and continue decoding, or > continue through the padding into the next buffer. However I believe > the buffer start / stop points are demarked by the aux_output_start / > aux_output_end calls? Yes. > > With upcoming perf decode updates this should enable the decoder to > correctly be started and stopped on the buffer boundaries. The padding > is there primarily to ensure that the decoder does not synchronize > with the data stream until a genuine sync point is found. Right. > > 4) TRBE needs to be a loadable module like the rest of coresight. Even though the driver has all the module constructs, the Kconfig was missing a tristate value, which is being fixed for the next version. - Anshuman
Hi Anshuman, On Tue, 16 Feb 2021 at 09:44, Anshuman Khandual <anshuman.khandual@arm.com> wrote: > > Hello Mike, > > On 2/16/21 2:30 PM, Mike Leach wrote: > > Hi Anshuman, > > > > There have been plenty of detailed comments so I will restrict mine to > > a few general issues:- > > > > 1) Currently there appears to be no sysfs support (I cannot see the > > MODE_SYSFS constants running alongside the MODE_PERF ones present in > > the other sink drivers). This is present on all other coresight > > devices, and must be provided for this device. It is useful for > > testing, and there are users out there who will have scripts to use > > it. It is not essential it makes it into this set, but should be a > > follow up set. > > Sure, will try and add it in a follow up series. > > > > > 2) Using FILL mode for TRBE means that the trace will by definition be > > lossy. Fill mode will halt collection without cleanly stopping and > > flushing the source. This will result in the sink missing the last of > > the data from the source as it stops. Even if taking the exception > > moves into a prohibited region there is still the possibility the last > > trace operations will not be seen. Further it is possible that the > > last few bytes of trace will be an incomplete packet, and indeed the > > start of the next buffer could contain incomplete packets too. > > Just wondering why TRBE and ETE would not sync with each other in order > for the ETE to possibly resend all the lost trace data, when the TRBE > runs out of buffer and wrappers around ? The ETE and TRBE are separate devices - there is no feedback between them. The ETE can also send to external sinks. Given the rate of trace generation, buffering enough trace in the ETE to resend is not realistic, and would be very complicated in terms of hardware. Therefore the solution is to stop the source (disable ETE or prohibit using TFR), flush (TSB CSYNC), then stop collection. A TSB CSYNC without stopping the ETE, or after TRBE has stopped collection will have no effect in terms of getting cleanly stopped trace into the buffer. > Is this ETE/TRBE behavior same > for all implementations in the FILL mode ? Just wondering. > Yes - there is nothing in either spec that would suggest otherwise. > > > > This operation differs from the other sinks which will only halt after > > the sources have stopped and the path has been flushed. This ensures > > that the latest trace is complete. The weakness with the older sinks > > is the lack of interrupt meaning buffers were frequently wrapped so > > that only the latest trace is available. > > Right. > > > > > By using TRBE WRAP mode, with a watermark as described in the TRBE > > spec, using the interrupts it is possible to approach lossless trace > > in a way that is not possible with earlier ETR/ETB. This is somethin > Using TRBTRG_EL1 as the above mentioned watermark ? > Using TRBTRG_EL1 precludes using the ETE Event triggers for activating and marking trace. It is preferable to use the write pointer offset from the initial base to allow a portion of the buffer to be filled after wrap. This a little more complex but more flexible in terms of ETE usage. > > that has been requested by partners since trace became available in > > linux systems. (There is still a possibility of loss due to filling > > the buffer completely and overflowing the watermark, but that can be > > flagged). > > > > While FILL mode trace is a good start, and suitable for some scenarios > > - WRAP mode needs implementing as well. > > I would like to understand this mechanism more. Besides how the perf > interface suppose to choose between FILL and WRAP mode ? via a new > event attribute ? > That is an open question. Event option is one possibility, configfs or compile time options are others. Probably have to look at the performance of wrap mode and decide if it could be used all the time or if FILL still has value. We are in the early days of ETE / TRBE development here. I do not think there is anything wrong with using FILL as a first step. as long as the limitations are well understood. Regards Mike > > > > 3) Padding: To be clear, it is not safe for the decoder to run off the > > end of one buffer, into the padding area and continue decoding, or > > continue through the padding into the next buffer. However I believe > > the buffer start / stop points are demarked by the aux_output_start / > > aux_output_end calls? > > Yes. > > > > > With upcoming perf decode updates this should enable the decoder to > > correctly be started and stopped on the buffer boundaries. The padding > > is there primarily to ensure that the decoder does not synchronize > > with the data stream until a genuine sync point is found. > > Right. > > > > > 4) TRBE needs to be a loadable module like the rest of coresight. > > Even though the driver has all the module constructs, the Kconfig was > missing a tristate value, which is being fixed for the next version. > > - Anshuman
Hi Mike On 2/16/21 9:00 AM, Mike Leach wrote: > Hi Anshuman, > > There have been plenty of detailed comments so I will restrict mine to > a few general issues:- > > 1) Currently there appears to be no sysfs support (I cannot see the > MODE_SYSFS constants running alongside the MODE_PERF ones present in > the other sink drivers). This is present on all other coresight > devices, and must be provided for this device. It is useful for > testing, and there are users out there who will have scripts to use > it. It is not essential it makes it into this set, but should be a > follow up set. This is mentioned in the cover-letter and as you rightly said we could add this in a later series. > > 2) Using FILL mode for TRBE means that the trace will by definition be > lossy. Fill mode will halt collection without cleanly stopping and > flushing the source. This will result in the sink missing the last of > the data from the source as it stops. Even if taking the exception > moves into a prohibited region there is still the possibility the last > trace operations will not be seen. Further it is possible that the Correct. > last few bytes of trace will be an incomplete packet, and indeed the > start of the next buffer could contain incomplete packets too. Yes, this is possible. > > This operation differs from the other sinks which will only halt after > the sources have stopped and the path has been flushed. This ensures > that the latest trace is complete. The weakness with the older sinks > is the lack of interrupt meaning buffers were frequently wrapped so > that only the latest trace is available. This is true, when there was no overflow. i.e, we follow the normal source-stop-flush, sink-stop. > > By using TRBE WRAP mode, with a watermark as described in the TRBE > spec, using the interrupts it is possible to approach lossless trace > in a way that is not possible with earlier ETR/ETB. This is something It may be possible to do lossless trace, but not without double buffering in perf mode. In perf mode, with a single buffer, we have to honor the boundaries set by the aux_buffer head and tail, otherwise we could be corrupting the trace being consumed by the userland. Please remember that the "water mark" is considered as the END of the buffer by TRBE (unlike the SoC-600 ETR). So the LIMIT pointer could be one of : * Tail pointer ( of the handle space, <= End_of_the_Buffer) * Wake up pointer ( when the userspace would like to be woken up ,<= End_of_the_Buffer) So, if we use WRAP mode for perf, the TRBE would overwrite the from the Base, after we hit the LIMIT, where we should have started writing *after* the LIMIT (when LIMIT < End_of_the_Buffer). Moreover restarting from the Base is going to be even more trouble some as it is most likely the data, perf is still collecting. > that has been requested by partners since trace became available in > linux systems. (There is still a possibility of loss due to filling > the buffer completely and overflowing the watermark, but that can be > flagged). > > While FILL mode trace is a good start, and suitable for some scenarios > - WRAP mode needs implementing as well. Using WRAP mode makes sense only in the case of double buffering. Even with that, we are not guaranteed that we wouldn't loose trace data, with significantly larger buffer than the AUX buffer. So this may not be the right choice looking at the performance and the software expectations. When it comes to sysfs mode, I believe we could use the CIRCULAR_BUFFER mode, as the collection is asynchronous. I understand WRAP is suitable for lossless collection, but unfortunately the Linux sof > 3) Padding: To be clear, it is not safe for the decoder to run off the > end of one buffer, into the padding area and continue decoding, or > continue through the padding into the next buffer. However I believe > the buffer start / stop points are demarked by the aux_output_start / > aux_output_end calls? Yes. Each session is marked by RECORD_AUX. So, as long as we fix the decoding to use the limit, we should be fine. Thanks for raising this point. Suzuki > > With upcoming perf decode updates this should enable the decoder to > correctly be started and stopped on the buffer boundaries. The padding > is there primarily to ensure that the decoder does not synchronize > with the data stream until a genuine sync point is found. > > 4) TRBE needs to be a loadable module like the rest of coresight. > > Regards > > Mike > > On Mon, 15 Feb 2021 at 09:46, Anshuman Khandual > <anshuman.khandual@arm.com> wrote: >> >> >> On 2/13/21 1:56 AM, Mathieu Poirier wrote: >>> On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: >>>> Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is >>>> accessible via the system registers. The TRBE supports different addressing >>>> modes including CPU virtual address and buffer modes including the circular >>>> buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), >>>> an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the >>>> access to the trace buffer could be prohibited by a higher exception level >>>> (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU >>>> private interrupt (PPI) on address translation errors and when the buffer >>>> is full. Overall implementation here is inspired from the Arm SPE driver. >>>> >>>> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> >>>> Cc: Mike Leach <mike.leach@linaro.org> >>>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> >>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> >>>> --- >>>> Changes in V3: >>>> >>>> - Added new DT bindings document TRBE.yaml >>>> - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 >>>> - Dropped isb() from trbe_reset_local() >>>> - Dropped gap between (void *) and buf->trbe_base >>>> - Changed 'int' to 'unsigned int' in is_trbe_available() >>>> - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), >>>> set_trbe_enabled() and set_trbe_limit_pointer() >>>> - Changed get_trbe_flag_update(), is_trbe_programmable() and >>>> get_trbe_address_align() to accept TRBIDR value >>>> - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), >>>> is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value >>>> - Dropped snapshot mode condition in arm_trbe_alloc_buffer() >>>> - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled >>>> - Compute trbe_limit before trbe_write to get the updated handle >>>> - Added trbe_stop_and_truncate_event() >>>> - Dropped trbe_handle_fatal() >>>> >>>> Documentation/trace/coresight/coresight-trbe.rst | 39 + >>>> arch/arm64/include/asm/sysreg.h | 1 + >>>> drivers/hwtracing/coresight/Kconfig | 11 + >>>> drivers/hwtracing/coresight/Makefile | 1 + >>>> drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ >>>> drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ >>>> 6 files changed, 1235 insertions(+) >>>> create mode 100644 Documentation/trace/coresight/coresight-trbe.rst >>>> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c >>>> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h >>>> >>>> diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst >>>> new file mode 100644 >>>> index 0000000..1cbb819 >>>> --- /dev/null >>>> +++ b/Documentation/trace/coresight/coresight-trbe.rst >>>> @@ -0,0 +1,39 @@ >>>> +.. SPDX-License-Identifier: GPL-2.0 >>>> + >>>> +============================== >>>> +Trace Buffer Extension (TRBE). >>>> +============================== >>>> + >>>> + :Author: Anshuman Khandual <anshuman.khandual@arm.com> >>>> + :Date: November 2020 >>>> + >>>> +Hardware Description >>>> +-------------------- >>>> + >>>> +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system >>>> +memory, CPU traces generated from a corresponding percpu tracing unit. This >>>> +gets plugged in as a coresight sink device because the corresponding trace >>>> +genarators (ETE), are plugged in as source device. >>>> + >>>> +The TRBE is not compliant to CoreSight architecture specifications, but is >>>> +driven via the CoreSight driver framework to support the ETE (which is >>>> +CoreSight compliant) integration. >>>> + >>>> +Sysfs files and directories >>>> +--------------------------- >>>> + >>>> +The TRBE devices appear on the existing coresight bus alongside the other >>>> +coresight devices:: >>>> + >>>> + >$ ls /sys/bus/coresight/devices >>>> + trbe0 trbe1 trbe2 trbe3 >>>> + >>>> +The ``trbe<N>`` named TRBEs are associated with a CPU.:: >>>> + >>>> + >$ ls /sys/bus/coresight/devices/trbe0/ >>>> + align dbm >>>> + >>>> +*Key file items are:-* >>>> + * ``align``: TRBE write pointer alignment >>>> + * ``dbm``: TRBE updates memory with access and dirty flags >>>> + >>>> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h >>>> index 85ae4db..9e2e9b7 100644 >>>> --- a/arch/arm64/include/asm/sysreg.h >>>> +++ b/arch/arm64/include/asm/sysreg.h >>>> @@ -97,6 +97,7 @@ >>>> #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) >>>> #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) >>>> #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) >>>> +#define TSB_CSYNC __emit_inst(0xd503225f) >>>> >>>> #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) >>>> #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) >>>> diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig >>>> index f154ae7..aa657ab 100644 >>>> --- a/drivers/hwtracing/coresight/Kconfig >>>> +++ b/drivers/hwtracing/coresight/Kconfig >>>> @@ -164,6 +164,17 @@ config CORESIGHT_CTI >>>> To compile this driver as a module, choose M here: the >>>> module will be called coresight-cti. >>>> >>>> +config CORESIGHT_TRBE >>>> + bool "Trace Buffer Extension (TRBE) driver" >>>> + depends on ARM64 >>>> + help >>>> + This driver provides support for percpu Trace Buffer Extension (TRBE). >>>> + TRBE always needs to be used along with it's corresponding percpu ETE >>>> + component. ETE generates trace data which is then captured with TRBE. >>>> + Unlike traditional sink devices, TRBE is a CPU feature accessible via >>>> + system registers. But it's explicit dependency with trace unit (ETE) >>>> + requires it to be plugged in as a coresight sink device. >>>> + >>>> config CORESIGHT_CTI_INTEGRATION_REGS >>>> bool "Access CTI CoreSight Integration Registers" >>>> depends on CORESIGHT_CTI >>>> diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile >>>> index f20e357..d608165 100644 >>>> --- a/drivers/hwtracing/coresight/Makefile >>>> +++ b/drivers/hwtracing/coresight/Makefile >>>> @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o >>>> obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o >>>> obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o >>>> obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o >>>> +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o >>>> coresight-cti-y := coresight-cti-core.o coresight-cti-platform.o \ >>>> coresight-cti-sysfs.o >>>> diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c >>>> new file mode 100644 >>>> index 0000000..1464d8b >>>> --- /dev/null >>>> +++ b/drivers/hwtracing/coresight/coresight-trbe.c >>>> @@ -0,0 +1,1023 @@ >>>> +// SPDX-License-Identifier: GPL-2.0 >>>> +/* >>>> + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight >>>> + * sink device could then pair with an appropriate per-cpu coresight source >>>> + * device (ETE) thus generating required trace data. Trace can be enabled >>>> + * via the perf framework. >>>> + * >>>> + * Copyright (C) 2020 ARM Ltd. >>>> + * >>>> + * Author: Anshuman Khandual <anshuman.khandual@arm.com> >>>> + */ >>>> +#define DRVNAME "arm_trbe" >>>> + >>>> +#define pr_fmt(fmt) DRVNAME ": " fmt >>>> + >>>> +#include "coresight-trbe.h" >>>> + >>>> +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) >>>> + >>>> +/* >>>> + * A padding packet that will help the user space tools >>>> + * in skipping relevant sections in the captured trace >>>> + * data which could not be decoded. TRBE doesn't support >>>> + * formatting the trace data, unlike the legacy CoreSight >>>> + * sinks and thus we use ETE trace packets to pad the >>>> + * sections of the buffer. >>>> + */ >>>> +#define ETE_IGNORE_PACKET 0x70 >>>> + >>>> +/* >>>> + * Minimum amount of meaningful trace will contain: >>>> + * A-Sync, Trace Info, Trace On, Address, Atom. >>>> + * This is about 44bytes of ETE trace. To be on >>>> + * the safer side, we assume 64bytes is the minimum >>>> + * space required for a meaningful session, before >>>> + * we hit a "WRAP" event. >>>> + */ >>>> +#define TRBE_TRACE_MIN_BUF_SIZE 64 >>>> + >>>> +enum trbe_fault_action { >>>> + TRBE_FAULT_ACT_WRAP, >>>> + TRBE_FAULT_ACT_SPURIOUS, >>>> + TRBE_FAULT_ACT_FATAL, >>>> +}; >>>> + >>>> +struct trbe_buf { >>>> + unsigned long trbe_base; >>>> + unsigned long trbe_limit; >>>> + unsigned long trbe_write; >>>> + int nr_pages; >>>> + void **pages; >>>> + bool snapshot; >>>> + struct trbe_cpudata *cpudata; >>>> +}; >>>> + >>>> +struct trbe_cpudata { >>>> + bool trbe_dbm; >>>> + u64 trbe_align; >>>> + int cpu; >>>> + enum cs_mode mode; >>>> + struct trbe_buf *buf; >>>> + struct trbe_drvdata *drvdata; >>>> +}; >>>> + >>>> +struct trbe_drvdata { >>>> + struct trbe_cpudata __percpu *cpudata; >>>> + struct perf_output_handle __percpu **handle; >>>> + struct hlist_node hotplug_node; >>>> + int irq; >>>> + cpumask_t supported_cpus; >>>> + enum cpuhp_state trbe_online; >>>> + struct platform_device *pdev; >>>> +}; >>>> + >>>> +static int trbe_alloc_node(struct perf_event *event) >>>> +{ >>>> + if (event->cpu == -1) >>>> + return NUMA_NO_NODE; >>>> + return cpu_to_node(event->cpu); >>>> +} >>>> + >>>> +static void trbe_drain_buffer(void) >>>> +{ >>>> + asm(TSB_CSYNC); >>>> + dsb(nsh); >>>> +} >>>> + >>>> +static void trbe_drain_and_disable_local(void) >>>> +{ >>>> + trbe_drain_buffer(); >>>> + write_sysreg_s(0, SYS_TRBLIMITR_EL1); >>>> + isb(); >>>> +} >>>> + >>>> +static void trbe_reset_local(void) >>>> +{ >>>> + trbe_drain_and_disable_local(); >>>> + write_sysreg_s(0, SYS_TRBPTR_EL1); >>>> + write_sysreg_s(0, SYS_TRBBASER_EL1); >>>> + write_sysreg_s(0, SYS_TRBSR_EL1); >>>> +} >>>> + >>>> +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) >>>> +{ >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>>> + >>>> + /* >>>> + * We cannot proceed with the buffer collection and we >>>> + * do not have any data for the current session. The >>>> + * etm_perf driver expects to close out the aux_buffer >>>> + * at event_stop(). So disable the TRBE here and leave >>>> + * the update_buffer() to return a 0 size. >>>> + */ >>>> + trbe_drain_and_disable_local(); >>>> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >>>> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; >>>> +} >>>> + >>>> +/* >>>> + * TRBE Buffer Management >>>> + * >>>> + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, >>>> + * it starts writing trace data from the write pointer onward till the limit pointer. >>>> + * When the write pointer reaches the address just before the limit pointer, it gets >>>> + * wrapped around again to the base pointer. This is called a TRBE wrap event, which >>>> + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver >>>> + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ >>>> + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and >>>> + * LIMIT pointers. >>>> + * >>>> + * Wrap around with an IRQ >>>> + * ------ < ------ < ------- < ----- < ----- >>>> + * | | >>>> + * ------ > ------ > ------- > ----- > ----- >>>> + * >>>> + * +---------------+-----------------------+ >>>> + * | | | >>>> + * +---------------+-----------------------+ >>>> + * Base Pointer Write Pointer Limit Pointer >>>> + * >>>> + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write >>>> + * pointer can be aligned to the implementation defined TRBE trace buffer alignment >>>> + * as captured in trbe_cpudata->trbe_align. >>>> + * >>>> + * >>>> + * head tail wakeup >>>> + * +---------------------------------------+----- ~ ~ ------ >>>> + * |$$$$$$$|################|$$$$$$$$$$$$$$| | >>>> + * +---------------------------------------+----- ~ ~ ------ >>>> + * Base Pointer Write Pointer Limit Pointer >>>> + * >>>> + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing >>>> + * values which tracks all the driver writes and user reads from the perf auxiliary >>>> + * buffer. Generally [head..tail] is the area where the driver can write into unless >>>> + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and >>>> + * configured depending on the perf_output_handle indices, so that the driver does >>>> + * not override into areas in the perf auxiliary buffer which is being or yet to be >>>> + * consumed from the user space. The enabled TRBE buffer area is a moving subset of >>>> + * the allocated perf auxiliary buffer. >>>> + */ >>>> +static void trbe_pad_buf(struct perf_output_handle *handle, int len) >>>> +{ >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>>> + u64 head = PERF_IDX2OFF(handle->head, buf); >>>> + >>>> + memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); >>>> + if (!buf->snapshot) >>>> + perf_aux_output_skip(handle, len); >>>> +} >>>> + >>>> +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) >>>> +{ >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>>> + >>>> + /* >>>> + * The ETE trace has alignment synchronization packets allowing >>>> + * the decoder to reset in case of an overflow or corruption. >>>> + * So we can use the entire buffer for the snapshot mode. >>>> + */ >>>> + return buf->nr_pages * PAGE_SIZE; >>>> +} >>>> + >>>> +/* >>>> + * TRBE Limit Calculation >>>> + * >>>> + * The following markers are used to illustrate various TRBE buffer situations. >>>> + * >>>> + * $$$$ - Data area, unconsumed captured trace data, not to be overridden >>>> + * #### - Free area, enabled, trace will be written >>>> + * %%%% - Free area, disabled, trace will not be written >>>> + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped >>>> + */ >>>> +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) >>>> +{ >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>>> + struct trbe_cpudata *cpudata = buf->cpudata; >>>> + const u64 bufsize = buf->nr_pages * PAGE_SIZE; >>>> + u64 limit = bufsize; >>>> + u64 head, tail, wakeup; >>>> + >>>> + head = PERF_IDX2OFF(handle->head, buf); >>>> + >>>> + /* >>>> + * head >>>> + * ------->| >>>> + * | >>>> + * head TRBE align tail >>>> + * +----|-------|---------------|-------+ >>>> + * |$$$$|=======|###############|$$$$$$$| >>>> + * +----|-------|---------------|-------+ >>>> + * trbe_base trbe_base + nr_pages >>>> + * >>>> + * Perf aux buffer output head position can be misaligned depending on >>>> + * various factors including user space reads. In case misaligned, head >>>> + * needs to be aligned before TRBE can be configured. Pad the alignment >>>> + * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools >>>> + * and skip this section thus advancing the head. >>>> + */ >>>> + if (!IS_ALIGNED(head, cpudata->trbe_align)) { >>>> + unsigned long delta = roundup(head, cpudata->trbe_align) - head; >>>> + >>>> + delta = min(delta, handle->size); >>>> + trbe_pad_buf(handle, delta); >>>> + head = PERF_IDX2OFF(handle->head, buf); >>>> + } >>>> + >>>> + /* >>>> + * head = tail (size = 0) >>>> + * +----|-------------------------------+ >>>> + * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ | >>>> + * +----|-------------------------------+ >>>> + * trbe_base trbe_base + nr_pages >>>> + * >>>> + * Perf aux buffer does not have any space for the driver to write into. >>>> + * Just communicate trace truncation event to the user space by marking >>>> + * it with PERF_AUX_FLAG_TRUNCATED. >>>> + */ >>>> + if (!handle->size) { >>>> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >>>> + return 0; >>>> + } >>>> + >>>> + /* Compute the tail and wakeup indices now that we've aligned head */ >>>> + tail = PERF_IDX2OFF(handle->head + handle->size, buf); >>>> + wakeup = PERF_IDX2OFF(handle->wakeup, buf); >>>> + >>>> + /* >>>> + * Lets calculate the buffer area which TRBE could write into. There >>>> + * are three possible scenarios here. Limit needs to be aligned with >>>> + * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the >>>> + * unconsumed data. >>>> + * >>>> + * 1) head < tail >>>> + * >>>> + * head tail >>>> + * +----|-----------------------|-------+ >>>> + * |$$$$|#######################|$$$$$$$| >>>> + * +----|-----------------------|-------+ >>>> + * trbe_base limit trbe_base + nr_pages >>>> + * >>>> + * TRBE could write into [head..tail] area. Unless the tail is right at >>>> + * the end of the buffer, neither an wrap around nor an IRQ is expected >>>> + * while being enabled. >>>> + * >>>> + * 2) head == tail >>>> + * >>>> + * head = tail (size > 0) >>>> + * +----|-------------------------------+ >>>> + * |%%%%|###############################| >>>> + * +----|-------------------------------+ >>>> + * trbe_base limit = trbe_base + nr_pages >>>> + * >>>> + * TRBE should just write into [head..base + nr_pages] area even though >>>> + * the entire buffer is empty. Reason being, when the trace reaches the >>>> + * end of the buffer, it will just wrap around with an IRQ giving an >>>> + * opportunity to reconfigure the buffer. >>>> + * >>>> + * 3) tail < head >>>> + * >>>> + * tail head >>>> + * +----|-----------------------|-------+ >>>> + * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| >>>> + * +----|-----------------------|-------+ >>>> + * trbe_base limit = trbe_base + nr_pages >>>> + * >>>> + * TRBE should just write into [head..base + nr_pages] area even though >>>> + * the [trbe_base..tail] is also empty. Reason being, when the trace >>>> + * reaches the end of the buffer, it will just wrap around with an IRQ >>>> + * giving an opportunity to reconfigure the buffer. >>>> + */ >>>> + if (head < tail) >>>> + limit = round_down(tail, PAGE_SIZE); >>>> + >>>> + /* >>>> + * Wakeup may be arbitrarily far into the future. If it's not in the >>>> + * current generation, either we'll wrap before hitting it, or it's >>>> + * in the past and has been handled already. >>>> + * >>>> + * If there's a wakeup before we wrap, arrange to be woken up by the >>>> + * page boundary following it. Keep the tail boundary if that's lower. >>>> + * >>>> + * head wakeup tail >>>> + * +----|---------------|-------|-------+ >>>> + * |$$$$|###############|%%%%%%%|$$$$$$$| >>>> + * +----|---------------|-------|-------+ >>>> + * trbe_base limit trbe_base + nr_pages >>>> + */ >>>> + if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) >>>> + limit = min(limit, round_up(wakeup, PAGE_SIZE)); >>>> + >>>> + /* >>>> + * There are two situation when this can happen i.e limit is before >>>> + * the head and hence TRBE cannot be configured. >>>> + * >>>> + * 1) head < tail (aligned down with PAGE_SIZE) and also they are both >>>> + * within the same PAGE size range. >>>> + * >>>> + * PAGE_SIZE >>>> + * |----------------------| >>>> + * >>>> + * limit head tail >>>> + * +------------|------|--------|-------+ >>>> + * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| >>>> + * +------------|------|--------|-------+ >>>> + * trbe_base trbe_base + nr_pages >>>> + * >>>> + * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both >>>> + * head and wakeup are within same PAGE size range. >>>> + * >>>> + * PAGE_SIZE >>>> + * |----------------------| >>>> + * >>>> + * limit head wakeup tail >>>> + * +----|------|-------|--------|-------+ >>>> + * |$$$$$$$$$$$|=======|========|$$$$$$$| >>>> + * +----|------|-------|--------|-------+ >>>> + * trbe_base trbe_base + nr_pages >>>> + */ >>>> + if (limit > head) >>>> + return limit; >>>> + >>>> + trbe_pad_buf(handle, handle->size); >>>> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); >>>> + return 0; >>>> +} >>>> + >>>> +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) >>>> +{ >>>> + struct trbe_buf *buf = perf_get_aux(handle); >>>> + u64 limit = __trbe_normal_offset(handle); >>>> + u64 head = PERF_IDX2OFF(handle->head, buf); >>>> + >>>> + /* >>>> + * If the head is too close to the limit and we don't >>>> + * have space for a meaningful run, we rather pad it >>>> + * and start fresh. >>>> + */ >>>> + if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { >>>> + trbe_pad_buf(handle, limit - head); >>>> + limit = __trbe_normal_offset(handle); >>>> + } >>>> + return limit; >>>> +} >>>> + >>>> +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) >>>> +{ >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>>> + unsigned long offset; >>>> + >>>> + if (buf->snapshot) >>>> + offset = trbe_snapshot_offset(handle); >>>> + else >>>> + offset = trbe_normal_offset(handle); >>>> + return buf->trbe_base + offset; >>>> +} >>> >>> I won't review the trace buffer management functions in this revision, I will >>> leave that for the next version. >> >> Okay. >> >>> >>>> + >>>> +static void clr_trbe_status(void) >>>> +{ >>>> + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); >>>> + >>>> + WARN_ON(is_trbe_enabled()); >>>> + trbsr &= ~TRBSR_IRQ; >>>> + trbsr &= ~TRBSR_TRG; >>>> + trbsr &= ~TRBSR_WRAP; >>>> + trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); >>>> + trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); >>>> + trbsr &= ~TRBSR_STOP; >>>> + write_sysreg_s(trbsr, SYS_TRBSR_EL1); >>>> +} >>> >>> I haven't read the TRBE progammer's manual but looking a the documentation the above >>> looks good. >>> >>>> + >>>> +static void set_trbe_limit_pointer_enabled(unsigned long addr) >>>> +{ >>>> + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); >>>> + >>>> + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); >>>> + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); >>>> + >>>> + trblimitr &= ~TRBLIMITR_NVM; >>>> + trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); >>>> + trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); >>>> + trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); >>>> + >>>> + /* >>>> + * Fill trace buffer mode is used here while configuring the >>>> + * TRBE for trace capture. In this particular mode, the trace >>>> + * collection is stopped and a maintenance interrupt is raised >>>> + * when the current write pointer wraps. This pause in trace >>>> + * collection gives the software an opportunity to capture the >>>> + * trace data in the interrupt handler, before reconfiguring >>>> + * the TRBE. >>>> + */ >>>> + trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; >>>> + >>>> + /* >>>> + * Trigger mode is not used here while configuring the TRBE for >>>> + * the trace capture. Hence just keep this in the ignore mode. >>>> + */ >>>> + trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; >>>> + trblimitr |= (addr & PAGE_MASK); >>>> + >>>> + trblimitr |= TRBLIMITR_ENABLE; >>>> + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); >>>> +} >>> >>> Same here >>> >>>> + >>>> +static void trbe_enable_hw(struct trbe_buf *buf) >>>> +{ >>>> + WARN_ON(buf->trbe_write < buf->trbe_base); >>>> + WARN_ON(buf->trbe_write >= buf->trbe_limit); >>>> + set_trbe_disabled(); >>>> + isb(); >>>> + clr_trbe_status(); >>>> + set_trbe_base_pointer(buf->trbe_base); >>>> + set_trbe_write_pointer(buf->trbe_write); >>>> + >>>> + /* >>>> + * Synchronize all the register updates >>>> + * till now before enabling the TRBE. >>>> + */ >>>> + isb(); >>>> + set_trbe_limit_pointer_enabled(buf->trbe_limit); >>>> + >>>> + /* Synchronize the TRBE enable event */ >>>> + isb(); >>>> +} >>> >>> Ok >>> >>>> + >>>> +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, >>>> + struct perf_event *event, void **pages, >>>> + int nr_pages, bool snapshot) >>>> +{ >>>> + struct trbe_buf *buf; >>>> + struct page **pglist; >>>> + int i; >>>> + >>>> + /* >>>> + * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with >>>> + * just a single page, there is not much room left while writing into >>>> + * a partially filled TRBE buffer. Hence restrict the minimum buffer >>>> + * size as two pages. >>>> + */ >>>> + if (nr_pages < 2) >>>> + return NULL; >>>> + >>>> + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); >>>> + if (IS_ERR(buf)) >>>> + return ERR_PTR(-ENOMEM); >>> >>> You know what do to. >> >> Right, will check for NULL instead and return ERR_PTR(-ENOMEM) as >> the function return type is (void *). >> >>> >>>> + >>>> + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); >>>> + if (IS_ERR(pglist)) { >>>> + kfree(buf); >>>> + return ERR_PTR(-ENOMEM); >>>> + } >>> >>> Here too. >> >> Yes, changed. >> >>> >>>> + >>>> + for (i = 0; i < nr_pages; i++) >>>> + pglist[i] = virt_to_page(pages[i]); >>>> + >>>> + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); >>>> + if (IS_ERR((void *)buf->trbe_base)) { >>>> + kfree(pglist); >>>> + kfree(buf); >>>> + return ERR_PTR(buf->trbe_base); >>>> + } >>> >>> Here too. >> >> Yes, changed. >> >>> >>>> + buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; >>>> + buf->trbe_write = buf->trbe_base; >>>> + buf->snapshot = snapshot; >>>> + buf->nr_pages = nr_pages; >>>> + buf->pages = pages; >>>> + kfree(pglist); >>>> + return buf; >>>> +} >>>> + >>>> +void arm_trbe_free_buffer(void *config) >> >> Added the missing 'static' here. >> >>>> +{ >>>> + struct trbe_buf *buf = config; >>>> + >>>> + vunmap((void *)buf->trbe_base); >>>> + kfree(buf); >>>> +} >>> >>> Ok >>> >>>> + >>>> +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, >>>> + struct perf_output_handle *handle, >>>> + void *config) >>>> +{ >>>> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >>>> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >>>> + struct trbe_buf *buf = config; >>>> + unsigned long size, offset; >>>> + >>>> + WARN_ON(buf->cpudata != cpudata); >>>> + WARN_ON(cpudata->cpu != smp_processor_id()); >>>> + WARN_ON(cpudata->drvdata != drvdata); >>>> + if (cpudata->mode != CS_MODE_PERF) >>>> + return -EINVAL; >>>> + >>>> + /* >>>> + * If the TRBE was disabled due to lack of space in the AUX buffer or a >>>> + * spurious fault, the driver leaves it disabled, truncating the buffer. >>>> + * Since the etm_perf driver expects to close out the AUX buffer, the >>>> + * driver skips it. Thus, just pass in 0 size here to indicate that the >>>> + * buffer was truncated. >>>> + */ >>>> + if (!is_trbe_enabled()) >>>> + return 0; >>>> + /* >>>> + * perf handle structure needs to be shared with the TRBE IRQ handler for >>>> + * capturing trace data and restarting the handle. There is a probability >>>> + * of an undefined reference based crash when etm event is being stopped >>>> + * while a TRBE IRQ also getting processed. This happens due the release >>>> + * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping >>>> + * the TRBE here will ensure that no IRQ could be generated when the perf >>>> + * handle gets freed in etm_event_stop(). >>>> + */ >>>> + trbe_drain_and_disable_local(); >>>> + offset = get_trbe_write_pointer() - get_trbe_base_pointer(); >>>> + size = offset - PERF_IDX2OFF(handle->head, buf); >>>> + if (buf->snapshot) >>>> + handle->head += size; >>>> + return size; >>>> +} >>> >>> Ok - I really appreciate the comments. >>> >>>> + >>>> +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) >>>> +{ >>>> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >>>> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >>>> + struct perf_output_handle *handle = data; >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>>> + >>>> + WARN_ON(cpudata->cpu != smp_processor_id()); >>>> + WARN_ON(cpudata->drvdata != drvdata); >>>> + if (mode != CS_MODE_PERF) >>>> + return -EINVAL; >>>> + >>>> + *this_cpu_ptr(drvdata->handle) = handle; >>>> + cpudata->buf = buf; >>>> + cpudata->mode = mode; >>>> + buf->cpudata = cpudata; >>>> + buf->trbe_limit = compute_trbe_buffer_limit(handle); >>>> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >>>> + if (buf->trbe_limit == buf->trbe_base) { >>>> + trbe_stop_and_truncate_event(handle); >>>> + return 0; >>>> + } >>>> + trbe_enable_hw(buf); >>>> + return 0; >>>> +} >>> >>> Ok >>> >>>> + >>>> +static int arm_trbe_disable(struct coresight_device *csdev) >>>> +{ >>>> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); >>>> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); >>>> + struct trbe_buf *buf = cpudata->buf; >>>> + >>>> + WARN_ON(buf->cpudata != cpudata); >>>> + WARN_ON(cpudata->cpu != smp_processor_id()); >>>> + WARN_ON(cpudata->drvdata != drvdata); >>>> + if (cpudata->mode != CS_MODE_PERF) >>>> + return -EINVAL; >>>> + >>>> + trbe_drain_and_disable_local(); >>>> + buf->cpudata = NULL; >>>> + cpudata->buf = NULL; >>>> + cpudata->mode = CS_MODE_DISABLED; >>>> + return 0; >>>> +} >>> >>> Ok >>> >>>> + >>>> +static void trbe_handle_spurious(struct perf_output_handle *handle) >>>> +{ >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>>> + >>>> + buf->trbe_limit = compute_trbe_buffer_limit(handle); >>>> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >>>> + if (buf->trbe_limit == buf->trbe_base) { >>>> + trbe_drain_and_disable_local(); >>>> + return; >>>> + } >>>> + trbe_enable_hw(buf); >>>> +} >>>> + >>>> +static void trbe_handle_overflow(struct perf_output_handle *handle) >>>> +{ >>>> + struct perf_event *event = handle->event; >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>>> + unsigned long offset, size; >>>> + struct etm_event_data *event_data; >>>> + >>>> + offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); >>>> + size = offset - PERF_IDX2OFF(handle->head, buf); >>>> + if (buf->snapshot) >>>> + handle->head = offset; >>>> + perf_aux_output_end(handle, size); >>>> + >>>> + event_data = perf_aux_output_begin(handle, event); >>>> + if (!event_data) { >>>> + trbe_drain_and_disable_local(); >>>> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; >>>> + return; >>>> + } >>>> + buf->trbe_limit = compute_trbe_buffer_limit(handle); >>>> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); >>>> + if (buf->trbe_limit == buf->trbe_base) { >>>> + trbe_stop_and_truncate_event(handle); >>>> + return; >>>> + } >>>> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; >>>> + trbe_enable_hw(buf); >>>> +} >>>> + >>>> +static bool is_perf_trbe(struct perf_output_handle *handle) >>>> +{ >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); >>>> + struct trbe_cpudata *cpudata = buf->cpudata; >>>> + struct trbe_drvdata *drvdata = cpudata->drvdata; >>>> + int cpu = smp_processor_id(); >>>> + >>>> + WARN_ON(buf->trbe_base != get_trbe_base_pointer()); >>>> + WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); >>>> + >>>> + if (cpudata->mode != CS_MODE_PERF) >>>> + return false; >>>> + >>>> + if (cpudata->cpu != cpu) >>>> + return false; >>>> + >>>> + if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) >>>> + return false; >>>> + >>>> + return true; >>>> +} >>>> + >>>> +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) >>> >>> @handle isn't used for anything. >> >> Okay, will drop. >> >>> >>>> +{ >>>> + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); >>>> + int ec = get_trbe_ec(trbsr); >>>> + int bsc = get_trbe_bsc(trbsr); >>>> + >>>> + WARN_ON(is_trbe_running(trbsr)); >>>> + if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) >>>> + return TRBE_FAULT_ACT_FATAL; >>>> + >>>> + if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) >>>> + return TRBE_FAULT_ACT_FATAL; >>>> + >>>> + if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { >>>> + if (get_trbe_write_pointer() == get_trbe_base_pointer()) >>>> + return TRBE_FAULT_ACT_WRAP; >>>> + } >>>> + return TRBE_FAULT_ACT_SPURIOUS; >>>> +} >>>> + >>>> +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) >>>> +{ >>>> + struct perf_output_handle **handle_ptr = dev; >>>> + struct perf_output_handle *handle = *handle_ptr; >>>> + enum trbe_fault_action act; >>>> + >>>> + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); >>>> + clr_trbe_irq(); >>>> + >>>> + /* >>>> + * Ensure the trace is visible to the CPUs and >>>> + * any external aborts have been resolved. >>>> + */ >>>> + trbe_drain_buffer(); >>>> + isb(); >>>> + >>>> + if (!perf_get_aux(handle)) >>>> + return IRQ_NONE; >>>> + >>>> + if (!is_perf_trbe(handle)) >>>> + return IRQ_NONE; >>>> + >>>> + irq_work_run(); >>> >>> I trust Will that this is the right thing to do. >>> >>> I will stop here for this revision. I will dive more in the mechanic of the >>> TRBE on the next revision. >> >> Okay, will collate all the changes till now and respin sooner. >> >> - Anshuman > > >
HI Suzuki, On Thu, 18 Feb 2021 at 07:50, Suzuki K Poulose <suzuki.poulose@arm.com> wrote: > > Hi Mike > > On 2/16/21 9:00 AM, Mike Leach wrote: > > Hi Anshuman, > > > > There have been plenty of detailed comments so I will restrict mine to > > a few general issues:- > > > > 1) Currently there appears to be no sysfs support (I cannot see the > > MODE_SYSFS constants running alongside the MODE_PERF ones present in > > the other sink drivers). This is present on all other coresight > > devices, and must be provided for this device. It is useful for > > testing, and there are users out there who will have scripts to use > > it. It is not essential it makes it into this set, but should be a > > follow up set. > > This is mentioned in the cover-letter and as you rightly said > we could add this in a later series. > Yes - I see that it was mentioned at the end as an open question - so I guess this is my answer! > > > > 2) Using FILL mode for TRBE means that the trace will by definition be > > lossy. Fill mode will halt collection without cleanly stopping and > > flushing the source. This will result in the sink missing the last of > > the data from the source as it stops. Even if taking the exception > > moves into a prohibited region there is still the possibility the last > > trace operations will not be seen. Further it is possible that the > > Correct. > > > last few bytes of trace will be an incomplete packet, and indeed the > > start of the next buffer could contain incomplete packets too. > > Yes, this is possible. > > > > > This operation differs from the other sinks which will only halt after > > the sources have stopped and the path has been flushed. This ensures > > that the latest trace is complete. The weakness with the older sinks > > is the lack of interrupt meaning buffers were frequently wrapped so > > that only the latest trace is available. > > This is true, when there was no overflow. i.e, we follow the normal > source-stop-flush, sink-stop. > > > > > By using TRBE WRAP mode, with a watermark as described in the TRBE > > spec, using the interrupts it is possible to approach lossless trace > > in a way that is not possible with earlier ETR/ETB. This is something > > It may be possible to do lossless trace, but not without double buffering > in perf mode. In perf mode, with a single buffer, we have to honor the > boundaries set by the aux_buffer head and tail, otherwise we could be > corrupting the trace being consumed by the userland. > > Please remember that the "water mark" is considered as the END of the > buffer by TRBE (unlike the SoC-600 ETR). So the LIMIT pointer could be > one of : > > * Tail pointer ( of the handle space, <= End_of_the_Buffer) > * Wake up pointer ( when the userspace would like to be woken up ,<= End_of_the_Buffer) > > So, if we use WRAP mode for perf, the TRBE would overwrite the from > the Base, after we hit the LIMIT, where we should have started > writing *after* the LIMIT (when LIMIT < End_of_the_Buffer). Moreover > restarting from the Base is going to be even more trouble some > as it is most likely the data, perf is still collecting. > I agree that the TRBE must write inbetween head and tail / wakeup. Howver, there is no reason that I can see why the trbe_base register has to remain constant @ the start of the vmapped aux buffer. A valid trbe write buffer could be set by: trbe_base >= head (rounded up to page boundary) trbe_limit <= min(tail, wakeup) (rounded down to page boundary) trbe_write is then trbe_base + "watermark" offset. - as suggested in the TRBE spec. The issue then becomes unravelling the buffer. Given what we know now, and the work on aux buffers, I would suggest that we can easily insert meta data to do this in the front of the buffer, saving any trace overwirtten at the end of the buffer, and setting a new flag in the aux buffer to tell userspace decode to sort it out. Thus the only copy needed is in the region of 8 bytes perhaps. Of course there are potential inefficiencies here in usage of buffer space, and yes we cannot guarantee lossless trace, but FILL mode guarantees lossy trace and a truncated buffer for every time it wraps (by definition, if FILL mode wraps then you cannot be sure that you have all the possible trace so it has to be marked as truncated in the same way we mark wrapped ETR buffers as truncated whenever they are wrapped). > > that has been requested by partners since trace became available in > > linux systems. (There is still a possibility of loss due to filling > > the buffer completely and overflowing the watermark, but that can be > > flagged). > > > > While FILL mode trace is a good start, and suitable for some scenarios > > - WRAP mode needs implementing as well. > > Using WRAP mode makes sense only in the case of double buffering. Even > with that, we are not guaranteed that we wouldn't loose trace data, with > significantly larger buffer than the AUX buffer. So this may not be the > right choice looking at the performance and the software expectations. > > When it comes to sysfs mode, I believe we could use the CIRCULAR_BUFFER > mode, as the collection is asynchronous. Agreed - this makes the most sense here. >I understand WRAP is suitable > for lossless collection, but unfortunately the Linux sof > sentence unexpectedly truncated?! Regards Mike > > 3) Padding: To be clear, it is not safe for the decoder to run off the > > end of one buffer, into the padding area and continue decoding, or > > continue through the padding into the next buffer. However I believe > > the buffer start / stop points are demarked by the aux_output_start / > > aux_output_end calls? > > Yes. Each session is marked by RECORD_AUX. So, as long as we fix > the decoding to use the limit, we should be fine. > > Thanks for raising this point. > > Suzuki > > > > > With upcoming perf decode updates this should enable the decoder to > > correctly be started and stopped on the buffer boundaries. The padding > > is there primarily to ensure that the decoder does not synchronize > > with the data stream until a genuine sync point is found. > > > > 4) TRBE needs to be a loadable module like the rest of coresight. > > > > Regards > > > > Mike > > > > On Mon, 15 Feb 2021 at 09:46, Anshuman Khandual > > <anshuman.khandual@arm.com> wrote: > >> > >> > >> On 2/13/21 1:56 AM, Mathieu Poirier wrote: > >>> On Wed, Jan 27, 2021 at 02:25:35PM +0530, Anshuman Khandual wrote: > >>>> Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is > >>>> accessible via the system registers. The TRBE supports different addressing > >>>> modes including CPU virtual address and buffer modes including the circular > >>>> buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), > >>>> an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the > >>>> access to the trace buffer could be prohibited by a higher exception level > >>>> (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU > >>>> private interrupt (PPI) on address translation errors and when the buffer > >>>> is full. Overall implementation here is inspired from the Arm SPE driver. > >>>> > >>>> Cc: Mathieu Poirier <mathieu.poirier@linaro.org> > >>>> Cc: Mike Leach <mike.leach@linaro.org> > >>>> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> > >>>> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> > >>>> --- > >>>> Changes in V3: > >>>> > >>>> - Added new DT bindings document TRBE.yaml > >>>> - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 > >>>> - Dropped isb() from trbe_reset_local() > >>>> - Dropped gap between (void *) and buf->trbe_base > >>>> - Changed 'int' to 'unsigned int' in is_trbe_available() > >>>> - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), > >>>> set_trbe_enabled() and set_trbe_limit_pointer() > >>>> - Changed get_trbe_flag_update(), is_trbe_programmable() and > >>>> get_trbe_address_align() to accept TRBIDR value > >>>> - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), > >>>> is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value > >>>> - Dropped snapshot mode condition in arm_trbe_alloc_buffer() > >>>> - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled > >>>> - Compute trbe_limit before trbe_write to get the updated handle > >>>> - Added trbe_stop_and_truncate_event() > >>>> - Dropped trbe_handle_fatal() > >>>> > >>>> Documentation/trace/coresight/coresight-trbe.rst | 39 + > >>>> arch/arm64/include/asm/sysreg.h | 1 + > >>>> drivers/hwtracing/coresight/Kconfig | 11 + > >>>> drivers/hwtracing/coresight/Makefile | 1 + > >>>> drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ > >>>> drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ > >>>> 6 files changed, 1235 insertions(+) > >>>> create mode 100644 Documentation/trace/coresight/coresight-trbe.rst > >>>> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c > >>>> create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h > >>>> > >>>> diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst > >>>> new file mode 100644 > >>>> index 0000000..1cbb819 > >>>> --- /dev/null > >>>> +++ b/Documentation/trace/coresight/coresight-trbe.rst > >>>> @@ -0,0 +1,39 @@ > >>>> +.. SPDX-License-Identifier: GPL-2.0 > >>>> + > >>>> +============================== > >>>> +Trace Buffer Extension (TRBE). > >>>> +============================== > >>>> + > >>>> + :Author: Anshuman Khandual <anshuman.khandual@arm.com> > >>>> + :Date: November 2020 > >>>> + > >>>> +Hardware Description > >>>> +-------------------- > >>>> + > >>>> +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system > >>>> +memory, CPU traces generated from a corresponding percpu tracing unit. This > >>>> +gets plugged in as a coresight sink device because the corresponding trace > >>>> +genarators (ETE), are plugged in as source device. > >>>> + > >>>> +The TRBE is not compliant to CoreSight architecture specifications, but is > >>>> +driven via the CoreSight driver framework to support the ETE (which is > >>>> +CoreSight compliant) integration. > >>>> + > >>>> +Sysfs files and directories > >>>> +--------------------------- > >>>> + > >>>> +The TRBE devices appear on the existing coresight bus alongside the other > >>>> +coresight devices:: > >>>> + > >>>> + >$ ls /sys/bus/coresight/devices > >>>> + trbe0 trbe1 trbe2 trbe3 > >>>> + > >>>> +The ``trbe<N>`` named TRBEs are associated with a CPU.:: > >>>> + > >>>> + >$ ls /sys/bus/coresight/devices/trbe0/ > >>>> + align dbm > >>>> + > >>>> +*Key file items are:-* > >>>> + * ``align``: TRBE write pointer alignment > >>>> + * ``dbm``: TRBE updates memory with access and dirty flags > >>>> + > >>>> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h > >>>> index 85ae4db..9e2e9b7 100644 > >>>> --- a/arch/arm64/include/asm/sysreg.h > >>>> +++ b/arch/arm64/include/asm/sysreg.h > >>>> @@ -97,6 +97,7 @@ > >>>> #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) > >>>> #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) > >>>> #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) > >>>> +#define TSB_CSYNC __emit_inst(0xd503225f) > >>>> > >>>> #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) > >>>> #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) > >>>> diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig > >>>> index f154ae7..aa657ab 100644 > >>>> --- a/drivers/hwtracing/coresight/Kconfig > >>>> +++ b/drivers/hwtracing/coresight/Kconfig > >>>> @@ -164,6 +164,17 @@ config CORESIGHT_CTI > >>>> To compile this driver as a module, choose M here: the > >>>> module will be called coresight-cti. > >>>> > >>>> +config CORESIGHT_TRBE > >>>> + bool "Trace Buffer Extension (TRBE) driver" > >>>> + depends on ARM64 > >>>> + help > >>>> + This driver provides support for percpu Trace Buffer Extension (TRBE). > >>>> + TRBE always needs to be used along with it's corresponding percpu ETE > >>>> + component. ETE generates trace data which is then captured with TRBE. > >>>> + Unlike traditional sink devices, TRBE is a CPU feature accessible via > >>>> + system registers. But it's explicit dependency with trace unit (ETE) > >>>> + requires it to be plugged in as a coresight sink device. > >>>> + > >>>> config CORESIGHT_CTI_INTEGRATION_REGS > >>>> bool "Access CTI CoreSight Integration Registers" > >>>> depends on CORESIGHT_CTI > >>>> diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile > >>>> index f20e357..d608165 100644 > >>>> --- a/drivers/hwtracing/coresight/Makefile > >>>> +++ b/drivers/hwtracing/coresight/Makefile > >>>> @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o > >>>> obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o > >>>> obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o > >>>> obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o > >>>> +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o > >>>> coresight-cti-y := coresight-cti-core.o coresight-cti-platform.o \ > >>>> coresight-cti-sysfs.o > >>>> diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c > >>>> new file mode 100644 > >>>> index 0000000..1464d8b > >>>> --- /dev/null > >>>> +++ b/drivers/hwtracing/coresight/coresight-trbe.c > >>>> @@ -0,0 +1,1023 @@ > >>>> +// SPDX-License-Identifier: GPL-2.0 > >>>> +/* > >>>> + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight > >>>> + * sink device could then pair with an appropriate per-cpu coresight source > >>>> + * device (ETE) thus generating required trace data. Trace can be enabled > >>>> + * via the perf framework. > >>>> + * > >>>> + * Copyright (C) 2020 ARM Ltd. > >>>> + * > >>>> + * Author: Anshuman Khandual <anshuman.khandual@arm.com> > >>>> + */ > >>>> +#define DRVNAME "arm_trbe" > >>>> + > >>>> +#define pr_fmt(fmt) DRVNAME ": " fmt > >>>> + > >>>> +#include "coresight-trbe.h" > >>>> + > >>>> +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) > >>>> + > >>>> +/* > >>>> + * A padding packet that will help the user space tools > >>>> + * in skipping relevant sections in the captured trace > >>>> + * data which could not be decoded. TRBE doesn't support > >>>> + * formatting the trace data, unlike the legacy CoreSight > >>>> + * sinks and thus we use ETE trace packets to pad the > >>>> + * sections of the buffer. > >>>> + */ > >>>> +#define ETE_IGNORE_PACKET 0x70 > >>>> + > >>>> +/* > >>>> + * Minimum amount of meaningful trace will contain: > >>>> + * A-Sync, Trace Info, Trace On, Address, Atom. > >>>> + * This is about 44bytes of ETE trace. To be on > >>>> + * the safer side, we assume 64bytes is the minimum > >>>> + * space required for a meaningful session, before > >>>> + * we hit a "WRAP" event. > >>>> + */ > >>>> +#define TRBE_TRACE_MIN_BUF_SIZE 64 > >>>> + > >>>> +enum trbe_fault_action { > >>>> + TRBE_FAULT_ACT_WRAP, > >>>> + TRBE_FAULT_ACT_SPURIOUS, > >>>> + TRBE_FAULT_ACT_FATAL, > >>>> +}; > >>>> + > >>>> +struct trbe_buf { > >>>> + unsigned long trbe_base; > >>>> + unsigned long trbe_limit; > >>>> + unsigned long trbe_write; > >>>> + int nr_pages; > >>>> + void **pages; > >>>> + bool snapshot; > >>>> + struct trbe_cpudata *cpudata; > >>>> +}; > >>>> + > >>>> +struct trbe_cpudata { > >>>> + bool trbe_dbm; > >>>> + u64 trbe_align; > >>>> + int cpu; > >>>> + enum cs_mode mode; > >>>> + struct trbe_buf *buf; > >>>> + struct trbe_drvdata *drvdata; > >>>> +}; > >>>> + > >>>> +struct trbe_drvdata { > >>>> + struct trbe_cpudata __percpu *cpudata; > >>>> + struct perf_output_handle __percpu **handle; > >>>> + struct hlist_node hotplug_node; > >>>> + int irq; > >>>> + cpumask_t supported_cpus; > >>>> + enum cpuhp_state trbe_online; > >>>> + struct platform_device *pdev; > >>>> +}; > >>>> + > >>>> +static int trbe_alloc_node(struct perf_event *event) > >>>> +{ > >>>> + if (event->cpu == -1) > >>>> + return NUMA_NO_NODE; > >>>> + return cpu_to_node(event->cpu); > >>>> +} > >>>> + > >>>> +static void trbe_drain_buffer(void) > >>>> +{ > >>>> + asm(TSB_CSYNC); > >>>> + dsb(nsh); > >>>> +} > >>>> + > >>>> +static void trbe_drain_and_disable_local(void) > >>>> +{ > >>>> + trbe_drain_buffer(); > >>>> + write_sysreg_s(0, SYS_TRBLIMITR_EL1); > >>>> + isb(); > >>>> +} > >>>> + > >>>> +static void trbe_reset_local(void) > >>>> +{ > >>>> + trbe_drain_and_disable_local(); > >>>> + write_sysreg_s(0, SYS_TRBPTR_EL1); > >>>> + write_sysreg_s(0, SYS_TRBBASER_EL1); > >>>> + write_sysreg_s(0, SYS_TRBSR_EL1); > >>>> +} > >>>> + > >>>> +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) > >>>> +{ > >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >>>> + > >>>> + /* > >>>> + * We cannot proceed with the buffer collection and we > >>>> + * do not have any data for the current session. The > >>>> + * etm_perf driver expects to close out the aux_buffer > >>>> + * at event_stop(). So disable the TRBE here and leave > >>>> + * the update_buffer() to return a 0 size. > >>>> + */ > >>>> + trbe_drain_and_disable_local(); > >>>> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > >>>> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > >>>> +} > >>>> + > >>>> +/* > >>>> + * TRBE Buffer Management > >>>> + * > >>>> + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, > >>>> + * it starts writing trace data from the write pointer onward till the limit pointer. > >>>> + * When the write pointer reaches the address just before the limit pointer, it gets > >>>> + * wrapped around again to the base pointer. This is called a TRBE wrap event, which > >>>> + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver > >>>> + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ > >>>> + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and > >>>> + * LIMIT pointers. > >>>> + * > >>>> + * Wrap around with an IRQ > >>>> + * ------ < ------ < ------- < ----- < ----- > >>>> + * | | > >>>> + * ------ > ------ > ------- > ----- > ----- > >>>> + * > >>>> + * +---------------+-----------------------+ > >>>> + * | | | > >>>> + * +---------------+-----------------------+ > >>>> + * Base Pointer Write Pointer Limit Pointer > >>>> + * > >>>> + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write > >>>> + * pointer can be aligned to the implementation defined TRBE trace buffer alignment > >>>> + * as captured in trbe_cpudata->trbe_align. > >>>> + * > >>>> + * > >>>> + * head tail wakeup > >>>> + * +---------------------------------------+----- ~ ~ ------ > >>>> + * |$$$$$$$|################|$$$$$$$$$$$$$$| | > >>>> + * +---------------------------------------+----- ~ ~ ------ > >>>> + * Base Pointer Write Pointer Limit Pointer > >>>> + * > >>>> + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing > >>>> + * values which tracks all the driver writes and user reads from the perf auxiliary > >>>> + * buffer. Generally [head..tail] is the area where the driver can write into unless > >>>> + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and > >>>> + * configured depending on the perf_output_handle indices, so that the driver does > >>>> + * not override into areas in the perf auxiliary buffer which is being or yet to be > >>>> + * consumed from the user space. The enabled TRBE buffer area is a moving subset of > >>>> + * the allocated perf auxiliary buffer. > >>>> + */ > >>>> +static void trbe_pad_buf(struct perf_output_handle *handle, int len) > >>>> +{ > >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >>>> + u64 head = PERF_IDX2OFF(handle->head, buf); > >>>> + > >>>> + memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); > >>>> + if (!buf->snapshot) > >>>> + perf_aux_output_skip(handle, len); > >>>> +} > >>>> + > >>>> +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) > >>>> +{ > >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >>>> + > >>>> + /* > >>>> + * The ETE trace has alignment synchronization packets allowing > >>>> + * the decoder to reset in case of an overflow or corruption. > >>>> + * So we can use the entire buffer for the snapshot mode. > >>>> + */ > >>>> + return buf->nr_pages * PAGE_SIZE; > >>>> +} > >>>> + > >>>> +/* > >>>> + * TRBE Limit Calculation > >>>> + * > >>>> + * The following markers are used to illustrate various TRBE buffer situations. > >>>> + * > >>>> + * $$$$ - Data area, unconsumed captured trace data, not to be overridden > >>>> + * #### - Free area, enabled, trace will be written > >>>> + * %%%% - Free area, disabled, trace will not be written > >>>> + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped > >>>> + */ > >>>> +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) > >>>> +{ > >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >>>> + struct trbe_cpudata *cpudata = buf->cpudata; > >>>> + const u64 bufsize = buf->nr_pages * PAGE_SIZE; > >>>> + u64 limit = bufsize; > >>>> + u64 head, tail, wakeup; > >>>> + > >>>> + head = PERF_IDX2OFF(handle->head, buf); > >>>> + > >>>> + /* > >>>> + * head > >>>> + * ------->| > >>>> + * | > >>>> + * head TRBE align tail > >>>> + * +----|-------|---------------|-------+ > >>>> + * |$$$$|=======|###############|$$$$$$$| > >>>> + * +----|-------|---------------|-------+ > >>>> + * trbe_base trbe_base + nr_pages > >>>> + * > >>>> + * Perf aux buffer output head position can be misaligned depending on > >>>> + * various factors including user space reads. In case misaligned, head > >>>> + * needs to be aligned before TRBE can be configured. Pad the alignment > >>>> + * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools > >>>> + * and skip this section thus advancing the head. > >>>> + */ > >>>> + if (!IS_ALIGNED(head, cpudata->trbe_align)) { > >>>> + unsigned long delta = roundup(head, cpudata->trbe_align) - head; > >>>> + > >>>> + delta = min(delta, handle->size); > >>>> + trbe_pad_buf(handle, delta); > >>>> + head = PERF_IDX2OFF(handle->head, buf); > >>>> + } > >>>> + > >>>> + /* > >>>> + * head = tail (size = 0) > >>>> + * +----|-------------------------------+ > >>>> + * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ | > >>>> + * +----|-------------------------------+ > >>>> + * trbe_base trbe_base + nr_pages > >>>> + * > >>>> + * Perf aux buffer does not have any space for the driver to write into. > >>>> + * Just communicate trace truncation event to the user space by marking > >>>> + * it with PERF_AUX_FLAG_TRUNCATED. > >>>> + */ > >>>> + if (!handle->size) { > >>>> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > >>>> + return 0; > >>>> + } > >>>> + > >>>> + /* Compute the tail and wakeup indices now that we've aligned head */ > >>>> + tail = PERF_IDX2OFF(handle->head + handle->size, buf); > >>>> + wakeup = PERF_IDX2OFF(handle->wakeup, buf); > >>>> + > >>>> + /* > >>>> + * Lets calculate the buffer area which TRBE could write into. There > >>>> + * are three possible scenarios here. Limit needs to be aligned with > >>>> + * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the > >>>> + * unconsumed data. > >>>> + * > >>>> + * 1) head < tail > >>>> + * > >>>> + * head tail > >>>> + * +----|-----------------------|-------+ > >>>> + * |$$$$|#######################|$$$$$$$| > >>>> + * +----|-----------------------|-------+ > >>>> + * trbe_base limit trbe_base + nr_pages > >>>> + * > >>>> + * TRBE could write into [head..tail] area. Unless the tail is right at > >>>> + * the end of the buffer, neither an wrap around nor an IRQ is expected > >>>> + * while being enabled. > >>>> + * > >>>> + * 2) head == tail > >>>> + * > >>>> + * head = tail (size > 0) > >>>> + * +----|-------------------------------+ > >>>> + * |%%%%|###############################| > >>>> + * +----|-------------------------------+ > >>>> + * trbe_base limit = trbe_base + nr_pages > >>>> + * > >>>> + * TRBE should just write into [head..base + nr_pages] area even though > >>>> + * the entire buffer is empty. Reason being, when the trace reaches the > >>>> + * end of the buffer, it will just wrap around with an IRQ giving an > >>>> + * opportunity to reconfigure the buffer. > >>>> + * > >>>> + * 3) tail < head > >>>> + * > >>>> + * tail head > >>>> + * +----|-----------------------|-------+ > >>>> + * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| > >>>> + * +----|-----------------------|-------+ > >>>> + * trbe_base limit = trbe_base + nr_pages > >>>> + * > >>>> + * TRBE should just write into [head..base + nr_pages] area even though > >>>> + * the [trbe_base..tail] is also empty. Reason being, when the trace > >>>> + * reaches the end of the buffer, it will just wrap around with an IRQ > >>>> + * giving an opportunity to reconfigure the buffer. > >>>> + */ > >>>> + if (head < tail) > >>>> + limit = round_down(tail, PAGE_SIZE); > >>>> + > >>>> + /* > >>>> + * Wakeup may be arbitrarily far into the future. If it's not in the > >>>> + * current generation, either we'll wrap before hitting it, or it's > >>>> + * in the past and has been handled already. > >>>> + * > >>>> + * If there's a wakeup before we wrap, arrange to be woken up by the > >>>> + * page boundary following it. Keep the tail boundary if that's lower. > >>>> + * > >>>> + * head wakeup tail > >>>> + * +----|---------------|-------|-------+ > >>>> + * |$$$$|###############|%%%%%%%|$$$$$$$| > >>>> + * +----|---------------|-------|-------+ > >>>> + * trbe_base limit trbe_base + nr_pages > >>>> + */ > >>>> + if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) > >>>> + limit = min(limit, round_up(wakeup, PAGE_SIZE)); > >>>> + > >>>> + /* > >>>> + * There are two situation when this can happen i.e limit is before > >>>> + * the head and hence TRBE cannot be configured. > >>>> + * > >>>> + * 1) head < tail (aligned down with PAGE_SIZE) and also they are both > >>>> + * within the same PAGE size range. > >>>> + * > >>>> + * PAGE_SIZE > >>>> + * |----------------------| > >>>> + * > >>>> + * limit head tail > >>>> + * +------------|------|--------|-------+ > >>>> + * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| > >>>> + * +------------|------|--------|-------+ > >>>> + * trbe_base trbe_base + nr_pages > >>>> + * > >>>> + * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both > >>>> + * head and wakeup are within same PAGE size range. > >>>> + * > >>>> + * PAGE_SIZE > >>>> + * |----------------------| > >>>> + * > >>>> + * limit head wakeup tail > >>>> + * +----|------|-------|--------|-------+ > >>>> + * |$$$$$$$$$$$|=======|========|$$$$$$$| > >>>> + * +----|------|-------|--------|-------+ > >>>> + * trbe_base trbe_base + nr_pages > >>>> + */ > >>>> + if (limit > head) > >>>> + return limit; > >>>> + > >>>> + trbe_pad_buf(handle, handle->size); > >>>> + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); > >>>> + return 0; > >>>> +} > >>>> + > >>>> +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) > >>>> +{ > >>>> + struct trbe_buf *buf = perf_get_aux(handle); > >>>> + u64 limit = __trbe_normal_offset(handle); > >>>> + u64 head = PERF_IDX2OFF(handle->head, buf); > >>>> + > >>>> + /* > >>>> + * If the head is too close to the limit and we don't > >>>> + * have space for a meaningful run, we rather pad it > >>>> + * and start fresh. > >>>> + */ > >>>> + if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { > >>>> + trbe_pad_buf(handle, limit - head); > >>>> + limit = __trbe_normal_offset(handle); > >>>> + } > >>>> + return limit; > >>>> +} > >>>> + > >>>> +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) > >>>> +{ > >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >>>> + unsigned long offset; > >>>> + > >>>> + if (buf->snapshot) > >>>> + offset = trbe_snapshot_offset(handle); > >>>> + else > >>>> + offset = trbe_normal_offset(handle); > >>>> + return buf->trbe_base + offset; > >>>> +} > >>> > >>> I won't review the trace buffer management functions in this revision, I will > >>> leave that for the next version. > >> > >> Okay. > >> > >>> > >>>> + > >>>> +static void clr_trbe_status(void) > >>>> +{ > >>>> + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > >>>> + > >>>> + WARN_ON(is_trbe_enabled()); > >>>> + trbsr &= ~TRBSR_IRQ; > >>>> + trbsr &= ~TRBSR_TRG; > >>>> + trbsr &= ~TRBSR_WRAP; > >>>> + trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); > >>>> + trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); > >>>> + trbsr &= ~TRBSR_STOP; > >>>> + write_sysreg_s(trbsr, SYS_TRBSR_EL1); > >>>> +} > >>> > >>> I haven't read the TRBE progammer's manual but looking a the documentation the above > >>> looks good. > >>> > >>>> + > >>>> +static void set_trbe_limit_pointer_enabled(unsigned long addr) > >>>> +{ > >>>> + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); > >>>> + > >>>> + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); > >>>> + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); > >>>> + > >>>> + trblimitr &= ~TRBLIMITR_NVM; > >>>> + trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); > >>>> + trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); > >>>> + trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); > >>>> + > >>>> + /* > >>>> + * Fill trace buffer mode is used here while configuring the > >>>> + * TRBE for trace capture. In this particular mode, the trace > >>>> + * collection is stopped and a maintenance interrupt is raised > >>>> + * when the current write pointer wraps. This pause in trace > >>>> + * collection gives the software an opportunity to capture the > >>>> + * trace data in the interrupt handler, before reconfiguring > >>>> + * the TRBE. > >>>> + */ > >>>> + trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; > >>>> + > >>>> + /* > >>>> + * Trigger mode is not used here while configuring the TRBE for > >>>> + * the trace capture. Hence just keep this in the ignore mode. > >>>> + */ > >>>> + trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; > >>>> + trblimitr |= (addr & PAGE_MASK); > >>>> + > >>>> + trblimitr |= TRBLIMITR_ENABLE; > >>>> + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); > >>>> +} > >>> > >>> Same here > >>> > >>>> + > >>>> +static void trbe_enable_hw(struct trbe_buf *buf) > >>>> +{ > >>>> + WARN_ON(buf->trbe_write < buf->trbe_base); > >>>> + WARN_ON(buf->trbe_write >= buf->trbe_limit); > >>>> + set_trbe_disabled(); > >>>> + isb(); > >>>> + clr_trbe_status(); > >>>> + set_trbe_base_pointer(buf->trbe_base); > >>>> + set_trbe_write_pointer(buf->trbe_write); > >>>> + > >>>> + /* > >>>> + * Synchronize all the register updates > >>>> + * till now before enabling the TRBE. > >>>> + */ > >>>> + isb(); > >>>> + set_trbe_limit_pointer_enabled(buf->trbe_limit); > >>>> + > >>>> + /* Synchronize the TRBE enable event */ > >>>> + isb(); > >>>> +} > >>> > >>> Ok > >>> > >>>> + > >>>> +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, > >>>> + struct perf_event *event, void **pages, > >>>> + int nr_pages, bool snapshot) > >>>> +{ > >>>> + struct trbe_buf *buf; > >>>> + struct page **pglist; > >>>> + int i; > >>>> + > >>>> + /* > >>>> + * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with > >>>> + * just a single page, there is not much room left while writing into > >>>> + * a partially filled TRBE buffer. Hence restrict the minimum buffer > >>>> + * size as two pages. > >>>> + */ > >>>> + if (nr_pages < 2) > >>>> + return NULL; > >>>> + > >>>> + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); > >>>> + if (IS_ERR(buf)) > >>>> + return ERR_PTR(-ENOMEM); > >>> > >>> You know what do to. > >> > >> Right, will check for NULL instead and return ERR_PTR(-ENOMEM) as > >> the function return type is (void *). > >> > >>> > >>>> + > >>>> + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); > >>>> + if (IS_ERR(pglist)) { > >>>> + kfree(buf); > >>>> + return ERR_PTR(-ENOMEM); > >>>> + } > >>> > >>> Here too. > >> > >> Yes, changed. > >> > >>> > >>>> + > >>>> + for (i = 0; i < nr_pages; i++) > >>>> + pglist[i] = virt_to_page(pages[i]); > >>>> + > >>>> + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); > >>>> + if (IS_ERR((void *)buf->trbe_base)) { > >>>> + kfree(pglist); > >>>> + kfree(buf); > >>>> + return ERR_PTR(buf->trbe_base); > >>>> + } > >>> > >>> Here too. > >> > >> Yes, changed. > >> > >>> > >>>> + buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; > >>>> + buf->trbe_write = buf->trbe_base; > >>>> + buf->snapshot = snapshot; > >>>> + buf->nr_pages = nr_pages; > >>>> + buf->pages = pages; > >>>> + kfree(pglist); > >>>> + return buf; > >>>> +} > >>>> + > >>>> +void arm_trbe_free_buffer(void *config) > >> > >> Added the missing 'static' here. > >> > >>>> +{ > >>>> + struct trbe_buf *buf = config; > >>>> + > >>>> + vunmap((void *)buf->trbe_base); > >>>> + kfree(buf); > >>>> +} > >>> > >>> Ok > >>> > >>>> + > >>>> +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, > >>>> + struct perf_output_handle *handle, > >>>> + void *config) > >>>> +{ > >>>> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > >>>> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > >>>> + struct trbe_buf *buf = config; > >>>> + unsigned long size, offset; > >>>> + > >>>> + WARN_ON(buf->cpudata != cpudata); > >>>> + WARN_ON(cpudata->cpu != smp_processor_id()); > >>>> + WARN_ON(cpudata->drvdata != drvdata); > >>>> + if (cpudata->mode != CS_MODE_PERF) > >>>> + return -EINVAL; > >>>> + > >>>> + /* > >>>> + * If the TRBE was disabled due to lack of space in the AUX buffer or a > >>>> + * spurious fault, the driver leaves it disabled, truncating the buffer. > >>>> + * Since the etm_perf driver expects to close out the AUX buffer, the > >>>> + * driver skips it. Thus, just pass in 0 size here to indicate that the > >>>> + * buffer was truncated. > >>>> + */ > >>>> + if (!is_trbe_enabled()) > >>>> + return 0; > >>>> + /* > >>>> + * perf handle structure needs to be shared with the TRBE IRQ handler for > >>>> + * capturing trace data and restarting the handle. There is a probability > >>>> + * of an undefined reference based crash when etm event is being stopped > >>>> + * while a TRBE IRQ also getting processed. This happens due the release > >>>> + * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping > >>>> + * the TRBE here will ensure that no IRQ could be generated when the perf > >>>> + * handle gets freed in etm_event_stop(). > >>>> + */ > >>>> + trbe_drain_and_disable_local(); > >>>> + offset = get_trbe_write_pointer() - get_trbe_base_pointer(); > >>>> + size = offset - PERF_IDX2OFF(handle->head, buf); > >>>> + if (buf->snapshot) > >>>> + handle->head += size; > >>>> + return size; > >>>> +} > >>> > >>> Ok - I really appreciate the comments. > >>> > >>>> + > >>>> +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) > >>>> +{ > >>>> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > >>>> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > >>>> + struct perf_output_handle *handle = data; > >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >>>> + > >>>> + WARN_ON(cpudata->cpu != smp_processor_id()); > >>>> + WARN_ON(cpudata->drvdata != drvdata); > >>>> + if (mode != CS_MODE_PERF) > >>>> + return -EINVAL; > >>>> + > >>>> + *this_cpu_ptr(drvdata->handle) = handle; > >>>> + cpudata->buf = buf; > >>>> + cpudata->mode = mode; > >>>> + buf->cpudata = cpudata; > >>>> + buf->trbe_limit = compute_trbe_buffer_limit(handle); > >>>> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > >>>> + if (buf->trbe_limit == buf->trbe_base) { > >>>> + trbe_stop_and_truncate_event(handle); > >>>> + return 0; > >>>> + } > >>>> + trbe_enable_hw(buf); > >>>> + return 0; > >>>> +} > >>> > >>> Ok > >>> > >>>> + > >>>> +static int arm_trbe_disable(struct coresight_device *csdev) > >>>> +{ > >>>> + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); > >>>> + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); > >>>> + struct trbe_buf *buf = cpudata->buf; > >>>> + > >>>> + WARN_ON(buf->cpudata != cpudata); > >>>> + WARN_ON(cpudata->cpu != smp_processor_id()); > >>>> + WARN_ON(cpudata->drvdata != drvdata); > >>>> + if (cpudata->mode != CS_MODE_PERF) > >>>> + return -EINVAL; > >>>> + > >>>> + trbe_drain_and_disable_local(); > >>>> + buf->cpudata = NULL; > >>>> + cpudata->buf = NULL; > >>>> + cpudata->mode = CS_MODE_DISABLED; > >>>> + return 0; > >>>> +} > >>> > >>> Ok > >>> > >>>> + > >>>> +static void trbe_handle_spurious(struct perf_output_handle *handle) > >>>> +{ > >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >>>> + > >>>> + buf->trbe_limit = compute_trbe_buffer_limit(handle); > >>>> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > >>>> + if (buf->trbe_limit == buf->trbe_base) { > >>>> + trbe_drain_and_disable_local(); > >>>> + return; > >>>> + } > >>>> + trbe_enable_hw(buf); > >>>> +} > >>>> + > >>>> +static void trbe_handle_overflow(struct perf_output_handle *handle) > >>>> +{ > >>>> + struct perf_event *event = handle->event; > >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >>>> + unsigned long offset, size; > >>>> + struct etm_event_data *event_data; > >>>> + > >>>> + offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); > >>>> + size = offset - PERF_IDX2OFF(handle->head, buf); > >>>> + if (buf->snapshot) > >>>> + handle->head = offset; > >>>> + perf_aux_output_end(handle, size); > >>>> + > >>>> + event_data = perf_aux_output_begin(handle, event); > >>>> + if (!event_data) { > >>>> + trbe_drain_and_disable_local(); > >>>> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; > >>>> + return; > >>>> + } > >>>> + buf->trbe_limit = compute_trbe_buffer_limit(handle); > >>>> + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); > >>>> + if (buf->trbe_limit == buf->trbe_base) { > >>>> + trbe_stop_and_truncate_event(handle); > >>>> + return; > >>>> + } > >>>> + *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; > >>>> + trbe_enable_hw(buf); > >>>> +} > >>>> + > >>>> +static bool is_perf_trbe(struct perf_output_handle *handle) > >>>> +{ > >>>> + struct trbe_buf *buf = etm_perf_sink_config(handle); > >>>> + struct trbe_cpudata *cpudata = buf->cpudata; > >>>> + struct trbe_drvdata *drvdata = cpudata->drvdata; > >>>> + int cpu = smp_processor_id(); > >>>> + > >>>> + WARN_ON(buf->trbe_base != get_trbe_base_pointer()); > >>>> + WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); > >>>> + > >>>> + if (cpudata->mode != CS_MODE_PERF) > >>>> + return false; > >>>> + > >>>> + if (cpudata->cpu != cpu) > >>>> + return false; > >>>> + > >>>> + if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) > >>>> + return false; > >>>> + > >>>> + return true; > >>>> +} > >>>> + > >>>> +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) > >>> > >>> @handle isn't used for anything. > >> > >> Okay, will drop. > >> > >>> > >>>> +{ > >>>> + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); > >>>> + int ec = get_trbe_ec(trbsr); > >>>> + int bsc = get_trbe_bsc(trbsr); > >>>> + > >>>> + WARN_ON(is_trbe_running(trbsr)); > >>>> + if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) > >>>> + return TRBE_FAULT_ACT_FATAL; > >>>> + > >>>> + if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) > >>>> + return TRBE_FAULT_ACT_FATAL; > >>>> + > >>>> + if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { > >>>> + if (get_trbe_write_pointer() == get_trbe_base_pointer()) > >>>> + return TRBE_FAULT_ACT_WRAP; > >>>> + } > >>>> + return TRBE_FAULT_ACT_SPURIOUS; > >>>> +} > >>>> + > >>>> +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) > >>>> +{ > >>>> + struct perf_output_handle **handle_ptr = dev; > >>>> + struct perf_output_handle *handle = *handle_ptr; > >>>> + enum trbe_fault_action act; > >>>> + > >>>> + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); > >>>> + clr_trbe_irq(); > >>>> + > >>>> + /* > >>>> + * Ensure the trace is visible to the CPUs and > >>>> + * any external aborts have been resolved. > >>>> + */ > >>>> + trbe_drain_buffer(); > >>>> + isb(); > >>>> + > >>>> + if (!perf_get_aux(handle)) > >>>> + return IRQ_NONE; > >>>> + > >>>> + if (!is_perf_trbe(handle)) > >>>> + return IRQ_NONE; > >>>> + > >>>> + irq_work_run(); > >>> > >>> I trust Will that this is the right thing to do. > >>> > >>> I will stop here for this revision. I will dive more in the mechanic of the > >>> TRBE on the next revision. > >> > >> Okay, will collate all the changes till now and respin sooner. > >> > >> - Anshuman > > > > > > >
On 2/18/21 2:30 PM, Mike Leach wrote: > HI Suzuki, > > On Thu, 18 Feb 2021 at 07:50, Suzuki K Poulose <suzuki.poulose@arm.com> wrote: >> >> Hi Mike >> >> On 2/16/21 9:00 AM, Mike Leach wrote: >>> Hi Anshuman, >>> >>> There have been plenty of detailed comments so I will restrict mine to >>> a few general issues:- >>> >>> 1) Currently there appears to be no sysfs support (I cannot see the >>> MODE_SYSFS constants running alongside the MODE_PERF ones present in >>> the other sink drivers). This is present on all other coresight >>> devices, and must be provided for this device. It is useful for >>> testing, and there are users out there who will have scripts to use >>> it. It is not essential it makes it into this set, but should be a >>> follow up set. >> >> This is mentioned in the cover-letter and as you rightly said >> we could add this in a later series. >> > > Yes - I see that it was mentioned at the end as an open question - so > I guess this is my answer! > >>> >>> 2) Using FILL mode for TRBE means that the trace will by definition be >>> lossy. Fill mode will halt collection without cleanly stopping and >>> flushing the source. This will result in the sink missing the last of >>> the data from the source as it stops. Even if taking the exception >>> moves into a prohibited region there is still the possibility the last >>> trace operations will not be seen. Further it is possible that the >> >> Correct. >> >>> last few bytes of trace will be an incomplete packet, and indeed the >>> start of the next buffer could contain incomplete packets too. >> >> Yes, this is possible. >> >>> >>> This operation differs from the other sinks which will only halt after >>> the sources have stopped and the path has been flushed. This ensures >>> that the latest trace is complete. The weakness with the older sinks >>> is the lack of interrupt meaning buffers were frequently wrapped so >>> that only the latest trace is available. >> >> This is true, when there was no overflow. i.e, we follow the normal >> source-stop-flush, sink-stop. >> >>> >>> By using TRBE WRAP mode, with a watermark as described in the TRBE >>> spec, using the interrupts it is possible to approach lossless trace >>> in a way that is not possible with earlier ETR/ETB. This is something >> >> It may be possible to do lossless trace, but not without double buffering >> in perf mode. In perf mode, with a single buffer, we have to honor the >> boundaries set by the aux_buffer head and tail, otherwise we could be >> corrupting the trace being consumed by the userland. >> >> Please remember that the "water mark" is considered as the END of the >> buffer by TRBE (unlike the SoC-600 ETR). So the LIMIT pointer could be >> one of : >> >> * Tail pointer ( of the handle space, <= End_of_the_Buffer) >> * Wake up pointer ( when the userspace would like to be woken up ,<= End_of_the_Buffer) >> >> So, if we use WRAP mode for perf, the TRBE would overwrite the from >> the Base, after we hit the LIMIT, where we should have started >> writing *after* the LIMIT (when LIMIT < End_of_the_Buffer). Moreover >> restarting from the Base is going to be even more trouble some >> as it is most likely the data, perf is still collecting. >> > > I agree that the TRBE must write inbetween head and tail / wakeup. > Howver, there is no reason that I can see why the trbe_base register > has to remain constant @ the start of the vmapped aux buffer. > A valid trbe write buffer could be set by: > trbe_base >= head (rounded up to page boundary) > trbe_limit <= min(tail, wakeup) (rounded down to page boundary) > trbe_write is then trbe_base + "watermark" offset. - as suggested in > the TRBE spec. The problem is we are dealing with separate entities. The producer and the consumer are separate entities playing with a single, infinite running ring buffer. Had this been a double buffering scheme, this would work really nice. In fact, I had some plans to do this for SoC-600 ETR. Coming back, with the proposed approach : head write (watermark) end-of-real buffer / | ^---v-------^----------v---------------| / \ base(aligned) limit (wakeup/tail) In this case, assuming single shared buffer, the TRBE would start writing from "write" watermark and WRAP at limit, going back to base. The issues here are : - If there were no overflow, we cant update the AUX handle unless we pad from head to write (which might be significant). If there was overflow : - You have wrongly ordered data. i.e, the older trace is at "write" and newer trace is at "base". Unless we copy the data written from base to the end of "limit", the userspace can't consume it. Or else it thinks the data at base is older and this gets the trace decoding gone for a toss. As you can see, two players dealing with a single buffer doesn't allow for the kind of flow possible with WRAP. On the other hand this works perfectly with double buffering. We could copy the data from the "write" to limit, followed by from "base" to the current write ptr. This could well be used for sysfs, but since we have asynchronous collection, we don't need the interrupt and thus fall back to CIRCULAR_BUFFER mode. > > The issue then becomes unravelling the buffer. Given what we know now, > and the work on aux buffers, I would suggest that we can easily insert > meta data to do this in the front of the buffer, saving any trace > overwirtten at the end of the buffer, and setting a new flag in the > aux buffer to tell userspace decode to sort it out. Thus the only copy > needed is in the region of 8 bytes perhaps. > > Of course there are potential inefficiencies here in usage of buffer > space, and yes we cannot guarantee lossless trace, but FILL mode > guarantees lossy trace and a truncated buffer for every time it wraps > (by definition, if FILL mode wraps then you cannot be sure that you > have all the possible trace so it has to be marked as truncated in the > same way we mark wrapped ETR buffers as truncated whenever they are > wrapped). > >>> that has been requested by partners since trace became available in >>> linux systems. (There is still a possibility of loss due to filling >>> the buffer completely and overflowing the watermark, but that can be >>> flagged). >>> >>> While FILL mode trace is a good start, and suitable for some scenarios >>> - WRAP mode needs implementing as well. >> >> Using WRAP mode makes sense only in the case of double buffering. Even >> with that, we are not guaranteed that we wouldn't loose trace data, with >> significantly larger buffer than the AUX buffer. So this may not be the >> right choice looking at the performance and the software expectations. >> >> When it comes to sysfs mode, I believe we could use the CIRCULAR_BUFFER >> mode, as the collection is asynchronous. > > Agreed - this makes the most sense here. > >> I understand WRAP is suitable >> for lossless collection, but unfortunately the Linux sof >> > > sentence unexpectedly truncated?! Sorry, needed some more coffee after the break :-). It should have been : "... but unfortunately the Linux software semantics (especially perf) doesn't make this any better usable than the FILL mode". Cheers Suzuki
Hi Suzuki, On Thu, 18 Feb 2021 at 15:14, Suzuki K Poulose <suzuki.poulose@arm.com> wrote: > > On 2/18/21 2:30 PM, Mike Leach wrote: > > HI Suzuki, > > > > On Thu, 18 Feb 2021 at 07:50, Suzuki K Poulose <suzuki.poulose@arm.com> wrote: > >> > >> Hi Mike > >> > >> On 2/16/21 9:00 AM, Mike Leach wrote: > >>> Hi Anshuman, > >>> > >>> There have been plenty of detailed comments so I will restrict mine to > >>> a few general issues:- > >>> > >>> 1) Currently there appears to be no sysfs support (I cannot see the > >>> MODE_SYSFS constants running alongside the MODE_PERF ones present in > >>> the other sink drivers). This is present on all other coresight > >>> devices, and must be provided for this device. It is useful for > >>> testing, and there are users out there who will have scripts to use > >>> it. It is not essential it makes it into this set, but should be a > >>> follow up set. > >> > >> This is mentioned in the cover-letter and as you rightly said > >> we could add this in a later series. > >> > > > > Yes - I see that it was mentioned at the end as an open question - so > > I guess this is my answer! > > > >>> > >>> 2) Using FILL mode for TRBE means that the trace will by definition be > >>> lossy. Fill mode will halt collection without cleanly stopping and > >>> flushing the source. This will result in the sink missing the last of > >>> the data from the source as it stops. Even if taking the exception > >>> moves into a prohibited region there is still the possibility the last > >>> trace operations will not be seen. Further it is possible that the > >> > >> Correct. > >> > >>> last few bytes of trace will be an incomplete packet, and indeed the > >>> start of the next buffer could contain incomplete packets too. > >> > >> Yes, this is possible. > >> > >>> > >>> This operation differs from the other sinks which will only halt after > >>> the sources have stopped and the path has been flushed. This ensures > >>> that the latest trace is complete. The weakness with the older sinks > >>> is the lack of interrupt meaning buffers were frequently wrapped so > >>> that only the latest trace is available. > >> > >> This is true, when there was no overflow. i.e, we follow the normal > >> source-stop-flush, sink-stop. > >> > >>> > >>> By using TRBE WRAP mode, with a watermark as described in the TRBE > >>> spec, using the interrupts it is possible to approach lossless trace > >>> in a way that is not possible with earlier ETR/ETB. This is something > >> > >> It may be possible to do lossless trace, but not without double buffering > >> in perf mode. In perf mode, with a single buffer, we have to honor the > >> boundaries set by the aux_buffer head and tail, otherwise we could be > >> corrupting the trace being consumed by the userland. > >> > >> Please remember that the "water mark" is considered as the END of the > >> buffer by TRBE (unlike the SoC-600 ETR). So the LIMIT pointer could be > >> one of : > >> > >> * Tail pointer ( of the handle space, <= End_of_the_Buffer) > >> * Wake up pointer ( when the userspace would like to be woken up ,<= End_of_the_Buffer) > >> > >> So, if we use WRAP mode for perf, the TRBE would overwrite the from > >> the Base, after we hit the LIMIT, where we should have started > >> writing *after* the LIMIT (when LIMIT < End_of_the_Buffer). Moreover > >> restarting from the Base is going to be even more trouble some > >> as it is most likely the data, perf is still collecting. > >> > > > > I agree that the TRBE must write inbetween head and tail / wakeup. > > Howver, there is no reason that I can see why the trbe_base register > > has to remain constant @ the start of the vmapped aux buffer. > > A valid trbe write buffer could be set by: > > trbe_base >= head (rounded up to page boundary) > > trbe_limit <= min(tail, wakeup) (rounded down to page boundary) > > trbe_write is then trbe_base + "watermark" offset. - as suggested in > > the TRBE spec. > > The problem is we are dealing with separate entities. The producer > and the consumer are separate entities playing with a single, > infinite running ring buffer. Had this been a double buffering scheme, > this would work really nice. In fact, I had some plans to do this for > SoC-600 ETR. > > Coming back, with the proposed approach : > > head write (watermark) end-of-real buffer > / | > ^---v-------^----------v---------------| > / \ > base(aligned) limit (wakeup/tail) > > In this case, assuming single shared buffer, the TRBE would start > writing from "write" watermark and WRAP at limit, going back to base. > > The issues here are : > - If there were no overflow, we cant update the AUX handle > unless we pad from head to write (which might be significant). > > If there was overflow : > - You have wrongly ordered data. i.e, the older trace is at "write" > and newer trace is at "base". Unless we copy the data written from > base to the end of "limit", the userspace can't consume it. Or else > it thinks the data at base is older and this gets the trace decoding > gone for a toss. > Both of these issues are dealt with if we allow for the insertion of metadata into the buffer as I described below. The consumer - as the copier to user space - does not care about the internal structure of the buffer. The consumer cannot touch the area that the producer is filling until it is given permission by the call that updates the head pointer. Thus the problem becomes one for the decoder to sort out. If it knows the internal format of the buffer - then it can unwrap, and decode the correct trace. Either way - without a wrap flow, we cannot match the performance of intel-pt as our trace will always be truncated on the FILL IRQ. The intel-pt IRQ will both service the aux buffer, but also stop and start the trace generation - something that will happen in the WRAP flow, but cannot happen cleanly in the FILL flow. So WRAP flow is required - either by double buffering or a scheme that avoids this. Regards Mike > As you can see, two players dealing with a single buffer doesn't allow > for the kind of flow possible with WRAP. > > On the other hand this works perfectly with double buffering. We could > copy the data from the "write" to limit, followed by from "base" to the > current write ptr. This could well be used for sysfs, but since we > have asynchronous collection, we don't need the interrupt and thus > fall back to CIRCULAR_BUFFER mode. > > > > > The issue then becomes unravelling the buffer. Given what we know now, > > and the work on aux buffers, I would suggest that we can easily insert > > meta data to do this in the front of the buffer, saving any trace > > overwirtten at the end of the buffer, and setting a new flag in the > > aux buffer to tell userspace decode to sort it out. Thus the only copy > > needed is in the region of 8 bytes perhaps. > > > > Of course there are potential inefficiencies here in usage of buffer > > space, and yes we cannot guarantee lossless trace, but FILL mode > > guarantees lossy trace and a truncated buffer for every time it wraps > > (by definition, if FILL mode wraps then you cannot be sure that you > > have all the possible trace so it has to be marked as truncated in the > > same way we mark wrapped ETR buffers as truncated whenever they are > > wrapped). > > > >>> that has been requested by partners since trace became available in > >>> linux systems. (There is still a possibility of loss due to filling > >>> the buffer completely and overflowing the watermark, but that can be > >>> flagged). > >>> > >>> While FILL mode trace is a good start, and suitable for some scenarios > >>> - WRAP mode needs implementing as well. > >> > >> Using WRAP mode makes sense only in the case of double buffering. Even > >> with that, we are not guaranteed that we wouldn't loose trace data, with > >> significantly larger buffer than the AUX buffer. So this may not be the > >> right choice looking at the performance and the software expectations. > >> > >> When it comes to sysfs mode, I believe we could use the CIRCULAR_BUFFER > >> mode, as the collection is asynchronous. > > > > Agreed - this makes the most sense here. > > > >> I understand WRAP is suitable > >> for lossless collection, but unfortunately the Linux sof > >> > > > > sentence unexpectedly truncated?! > > Sorry, needed some more coffee after the break :-). It should have been : > > "... but unfortunately the Linux software semantics (especially perf) > doesn't make this any better usable than the FILL mode". > > Cheers > Suzuki
diff --git a/Documentation/trace/coresight/coresight-trbe.rst b/Documentation/trace/coresight/coresight-trbe.rst new file mode 100644 index 0000000..1cbb819 --- /dev/null +++ b/Documentation/trace/coresight/coresight-trbe.rst @@ -0,0 +1,39 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================== +Trace Buffer Extension (TRBE). +============================== + + :Author: Anshuman Khandual <anshuman.khandual@arm.com> + :Date: November 2020 + +Hardware Description +-------------------- + +Trace Buffer Extension (TRBE) is a percpu hardware which captures in system +memory, CPU traces generated from a corresponding percpu tracing unit. This +gets plugged in as a coresight sink device because the corresponding trace +genarators (ETE), are plugged in as source device. + +The TRBE is not compliant to CoreSight architecture specifications, but is +driven via the CoreSight driver framework to support the ETE (which is +CoreSight compliant) integration. + +Sysfs files and directories +--------------------------- + +The TRBE devices appear on the existing coresight bus alongside the other +coresight devices:: + + >$ ls /sys/bus/coresight/devices + trbe0 trbe1 trbe2 trbe3 + +The ``trbe<N>`` named TRBEs are associated with a CPU.:: + + >$ ls /sys/bus/coresight/devices/trbe0/ + align dbm + +*Key file items are:-* + * ``align``: TRBE write pointer alignment + * ``dbm``: TRBE updates memory with access and dirty flags + diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 85ae4db..9e2e9b7 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -97,6 +97,7 @@ #define SET_PSTATE_UAO(x) __emit_inst(0xd500401f | PSTATE_UAO | ((!!x) << PSTATE_Imm_shift)) #define SET_PSTATE_SSBS(x) __emit_inst(0xd500401f | PSTATE_SSBS | ((!!x) << PSTATE_Imm_shift)) #define SET_PSTATE_TCO(x) __emit_inst(0xd500401f | PSTATE_TCO | ((!!x) << PSTATE_Imm_shift)) +#define TSB_CSYNC __emit_inst(0xd503225f) #define set_pstate_pan(x) asm volatile(SET_PSTATE_PAN(x)) #define set_pstate_uao(x) asm volatile(SET_PSTATE_UAO(x)) diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig index f154ae7..aa657ab 100644 --- a/drivers/hwtracing/coresight/Kconfig +++ b/drivers/hwtracing/coresight/Kconfig @@ -164,6 +164,17 @@ config CORESIGHT_CTI To compile this driver as a module, choose M here: the module will be called coresight-cti. +config CORESIGHT_TRBE + bool "Trace Buffer Extension (TRBE) driver" + depends on ARM64 + help + This driver provides support for percpu Trace Buffer Extension (TRBE). + TRBE always needs to be used along with it's corresponding percpu ETE + component. ETE generates trace data which is then captured with TRBE. + Unlike traditional sink devices, TRBE is a CPU feature accessible via + system registers. But it's explicit dependency with trace unit (ETE) + requires it to be plugged in as a coresight sink device. + config CORESIGHT_CTI_INTEGRATION_REGS bool "Access CTI CoreSight Integration Registers" depends on CORESIGHT_CTI diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile index f20e357..d608165 100644 --- a/drivers/hwtracing/coresight/Makefile +++ b/drivers/hwtracing/coresight/Makefile @@ -21,5 +21,6 @@ obj-$(CONFIG_CORESIGHT_STM) += coresight-stm.o obj-$(CONFIG_CORESIGHT_CPU_DEBUG) += coresight-cpu-debug.o obj-$(CONFIG_CORESIGHT_CATU) += coresight-catu.o obj-$(CONFIG_CORESIGHT_CTI) += coresight-cti.o +obj-$(CONFIG_CORESIGHT_TRBE) += coresight-trbe.o coresight-cti-y := coresight-cti-core.o coresight-cti-platform.o \ coresight-cti-sysfs.o diff --git a/drivers/hwtracing/coresight/coresight-trbe.c b/drivers/hwtracing/coresight/coresight-trbe.c new file mode 100644 index 0000000..1464d8b --- /dev/null +++ b/drivers/hwtracing/coresight/coresight-trbe.c @@ -0,0 +1,1023 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * This driver enables Trace Buffer Extension (TRBE) as a per-cpu coresight + * sink device could then pair with an appropriate per-cpu coresight source + * device (ETE) thus generating required trace data. Trace can be enabled + * via the perf framework. + * + * Copyright (C) 2020 ARM Ltd. + * + * Author: Anshuman Khandual <anshuman.khandual@arm.com> + */ +#define DRVNAME "arm_trbe" + +#define pr_fmt(fmt) DRVNAME ": " fmt + +#include "coresight-trbe.h" + +#define PERF_IDX2OFF(idx, buf) ((idx) % ((buf)->nr_pages << PAGE_SHIFT)) + +/* + * A padding packet that will help the user space tools + * in skipping relevant sections in the captured trace + * data which could not be decoded. TRBE doesn't support + * formatting the trace data, unlike the legacy CoreSight + * sinks and thus we use ETE trace packets to pad the + * sections of the buffer. + */ +#define ETE_IGNORE_PACKET 0x70 + +/* + * Minimum amount of meaningful trace will contain: + * A-Sync, Trace Info, Trace On, Address, Atom. + * This is about 44bytes of ETE trace. To be on + * the safer side, we assume 64bytes is the minimum + * space required for a meaningful session, before + * we hit a "WRAP" event. + */ +#define TRBE_TRACE_MIN_BUF_SIZE 64 + +enum trbe_fault_action { + TRBE_FAULT_ACT_WRAP, + TRBE_FAULT_ACT_SPURIOUS, + TRBE_FAULT_ACT_FATAL, +}; + +struct trbe_buf { + unsigned long trbe_base; + unsigned long trbe_limit; + unsigned long trbe_write; + int nr_pages; + void **pages; + bool snapshot; + struct trbe_cpudata *cpudata; +}; + +struct trbe_cpudata { + bool trbe_dbm; + u64 trbe_align; + int cpu; + enum cs_mode mode; + struct trbe_buf *buf; + struct trbe_drvdata *drvdata; +}; + +struct trbe_drvdata { + struct trbe_cpudata __percpu *cpudata; + struct perf_output_handle __percpu **handle; + struct hlist_node hotplug_node; + int irq; + cpumask_t supported_cpus; + enum cpuhp_state trbe_online; + struct platform_device *pdev; +}; + +static int trbe_alloc_node(struct perf_event *event) +{ + if (event->cpu == -1) + return NUMA_NO_NODE; + return cpu_to_node(event->cpu); +} + +static void trbe_drain_buffer(void) +{ + asm(TSB_CSYNC); + dsb(nsh); +} + +static void trbe_drain_and_disable_local(void) +{ + trbe_drain_buffer(); + write_sysreg_s(0, SYS_TRBLIMITR_EL1); + isb(); +} + +static void trbe_reset_local(void) +{ + trbe_drain_and_disable_local(); + write_sysreg_s(0, SYS_TRBPTR_EL1); + write_sysreg_s(0, SYS_TRBBASER_EL1); + write_sysreg_s(0, SYS_TRBSR_EL1); +} + +static void trbe_stop_and_truncate_event(struct perf_output_handle *handle) +{ + struct trbe_buf *buf = etm_perf_sink_config(handle); + + /* + * We cannot proceed with the buffer collection and we + * do not have any data for the current session. The + * etm_perf driver expects to close out the aux_buffer + * at event_stop(). So disable the TRBE here and leave + * the update_buffer() to return a 0 size. + */ + trbe_drain_and_disable_local(); + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; +} + +/* + * TRBE Buffer Management + * + * The TRBE buffer spans from the base pointer till the limit pointer. When enabled, + * it starts writing trace data from the write pointer onward till the limit pointer. + * When the write pointer reaches the address just before the limit pointer, it gets + * wrapped around again to the base pointer. This is called a TRBE wrap event, which + * generates a maintenance interrupt when operated in WRAP or FILL mode. This driver + * uses FILL mode, where the TRBE stops the trace collection at wrap event. The IRQ + * handler updates the AUX buffer and re-enables the TRBE with updated WRITE and + * LIMIT pointers. + * + * Wrap around with an IRQ + * ------ < ------ < ------- < ----- < ----- + * | | + * ------ > ------ > ------- > ----- > ----- + * + * +---------------+-----------------------+ + * | | | + * +---------------+-----------------------+ + * Base Pointer Write Pointer Limit Pointer + * + * The base and limit pointers always needs to be PAGE_SIZE aligned. But the write + * pointer can be aligned to the implementation defined TRBE trace buffer alignment + * as captured in trbe_cpudata->trbe_align. + * + * + * head tail wakeup + * +---------------------------------------+----- ~ ~ ------ + * |$$$$$$$|################|$$$$$$$$$$$$$$| | + * +---------------------------------------+----- ~ ~ ------ + * Base Pointer Write Pointer Limit Pointer + * + * The perf_output_handle indices (head, tail, wakeup) are monotonically increasing + * values which tracks all the driver writes and user reads from the perf auxiliary + * buffer. Generally [head..tail] is the area where the driver can write into unless + * the wakeup is behind the tail. Enabled TRBE buffer span needs to be adjusted and + * configured depending on the perf_output_handle indices, so that the driver does + * not override into areas in the perf auxiliary buffer which is being or yet to be + * consumed from the user space. The enabled TRBE buffer area is a moving subset of + * the allocated perf auxiliary buffer. + */ +static void trbe_pad_buf(struct perf_output_handle *handle, int len) +{ + struct trbe_buf *buf = etm_perf_sink_config(handle); + u64 head = PERF_IDX2OFF(handle->head, buf); + + memset((void *)buf->trbe_base + head, ETE_IGNORE_PACKET, len); + if (!buf->snapshot) + perf_aux_output_skip(handle, len); +} + +static unsigned long trbe_snapshot_offset(struct perf_output_handle *handle) +{ + struct trbe_buf *buf = etm_perf_sink_config(handle); + + /* + * The ETE trace has alignment synchronization packets allowing + * the decoder to reset in case of an overflow or corruption. + * So we can use the entire buffer for the snapshot mode. + */ + return buf->nr_pages * PAGE_SIZE; +} + +/* + * TRBE Limit Calculation + * + * The following markers are used to illustrate various TRBE buffer situations. + * + * $$$$ - Data area, unconsumed captured trace data, not to be overridden + * #### - Free area, enabled, trace will be written + * %%%% - Free area, disabled, trace will not be written + * ==== - Free area, padded with ETE_IGNORE_PACKET, trace will be skipped + */ +static unsigned long __trbe_normal_offset(struct perf_output_handle *handle) +{ + struct trbe_buf *buf = etm_perf_sink_config(handle); + struct trbe_cpudata *cpudata = buf->cpudata; + const u64 bufsize = buf->nr_pages * PAGE_SIZE; + u64 limit = bufsize; + u64 head, tail, wakeup; + + head = PERF_IDX2OFF(handle->head, buf); + + /* + * head + * ------->| + * | + * head TRBE align tail + * +----|-------|---------------|-------+ + * |$$$$|=======|###############|$$$$$$$| + * +----|-------|---------------|-------+ + * trbe_base trbe_base + nr_pages + * + * Perf aux buffer output head position can be misaligned depending on + * various factors including user space reads. In case misaligned, head + * needs to be aligned before TRBE can be configured. Pad the alignment + * gap with ETE_IGNORE_PACKET bytes that will be ignored by user tools + * and skip this section thus advancing the head. + */ + if (!IS_ALIGNED(head, cpudata->trbe_align)) { + unsigned long delta = roundup(head, cpudata->trbe_align) - head; + + delta = min(delta, handle->size); + trbe_pad_buf(handle, delta); + head = PERF_IDX2OFF(handle->head, buf); + } + + /* + * head = tail (size = 0) + * +----|-------------------------------+ + * |$$$$|$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ | + * +----|-------------------------------+ + * trbe_base trbe_base + nr_pages + * + * Perf aux buffer does not have any space for the driver to write into. + * Just communicate trace truncation event to the user space by marking + * it with PERF_AUX_FLAG_TRUNCATED. + */ + if (!handle->size) { + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); + return 0; + } + + /* Compute the tail and wakeup indices now that we've aligned head */ + tail = PERF_IDX2OFF(handle->head + handle->size, buf); + wakeup = PERF_IDX2OFF(handle->wakeup, buf); + + /* + * Lets calculate the buffer area which TRBE could write into. There + * are three possible scenarios here. Limit needs to be aligned with + * PAGE_SIZE per the TRBE requirement. Always avoid clobbering the + * unconsumed data. + * + * 1) head < tail + * + * head tail + * +----|-----------------------|-------+ + * |$$$$|#######################|$$$$$$$| + * +----|-----------------------|-------+ + * trbe_base limit trbe_base + nr_pages + * + * TRBE could write into [head..tail] area. Unless the tail is right at + * the end of the buffer, neither an wrap around nor an IRQ is expected + * while being enabled. + * + * 2) head == tail + * + * head = tail (size > 0) + * +----|-------------------------------+ + * |%%%%|###############################| + * +----|-------------------------------+ + * trbe_base limit = trbe_base + nr_pages + * + * TRBE should just write into [head..base + nr_pages] area even though + * the entire buffer is empty. Reason being, when the trace reaches the + * end of the buffer, it will just wrap around with an IRQ giving an + * opportunity to reconfigure the buffer. + * + * 3) tail < head + * + * tail head + * +----|-----------------------|-------+ + * |%%%%|$$$$$$$$$$$$$$$$$$$$$$$|#######| + * +----|-----------------------|-------+ + * trbe_base limit = trbe_base + nr_pages + * + * TRBE should just write into [head..base + nr_pages] area even though + * the [trbe_base..tail] is also empty. Reason being, when the trace + * reaches the end of the buffer, it will just wrap around with an IRQ + * giving an opportunity to reconfigure the buffer. + */ + if (head < tail) + limit = round_down(tail, PAGE_SIZE); + + /* + * Wakeup may be arbitrarily far into the future. If it's not in the + * current generation, either we'll wrap before hitting it, or it's + * in the past and has been handled already. + * + * If there's a wakeup before we wrap, arrange to be woken up by the + * page boundary following it. Keep the tail boundary if that's lower. + * + * head wakeup tail + * +----|---------------|-------|-------+ + * |$$$$|###############|%%%%%%%|$$$$$$$| + * +----|---------------|-------|-------+ + * trbe_base limit trbe_base + nr_pages + */ + if (handle->wakeup < (handle->head + handle->size) && head <= wakeup) + limit = min(limit, round_up(wakeup, PAGE_SIZE)); + + /* + * There are two situation when this can happen i.e limit is before + * the head and hence TRBE cannot be configured. + * + * 1) head < tail (aligned down with PAGE_SIZE) and also they are both + * within the same PAGE size range. + * + * PAGE_SIZE + * |----------------------| + * + * limit head tail + * +------------|------|--------|-------+ + * |$$$$$$$$$$$$$$$$$$$|========|$$$$$$$| + * +------------|------|--------|-------+ + * trbe_base trbe_base + nr_pages + * + * 2) head < wakeup (aligned up with PAGE_SIZE) < tail and also both + * head and wakeup are within same PAGE size range. + * + * PAGE_SIZE + * |----------------------| + * + * limit head wakeup tail + * +----|------|-------|--------|-------+ + * |$$$$$$$$$$$|=======|========|$$$$$$$| + * +----|------|-------|--------|-------+ + * trbe_base trbe_base + nr_pages + */ + if (limit > head) + return limit; + + trbe_pad_buf(handle, handle->size); + perf_aux_output_flag(handle, PERF_AUX_FLAG_TRUNCATED); + return 0; +} + +static unsigned long trbe_normal_offset(struct perf_output_handle *handle) +{ + struct trbe_buf *buf = perf_get_aux(handle); + u64 limit = __trbe_normal_offset(handle); + u64 head = PERF_IDX2OFF(handle->head, buf); + + /* + * If the head is too close to the limit and we don't + * have space for a meaningful run, we rather pad it + * and start fresh. + */ + if (limit && (limit - head < TRBE_TRACE_MIN_BUF_SIZE)) { + trbe_pad_buf(handle, limit - head); + limit = __trbe_normal_offset(handle); + } + return limit; +} + +static unsigned long compute_trbe_buffer_limit(struct perf_output_handle *handle) +{ + struct trbe_buf *buf = etm_perf_sink_config(handle); + unsigned long offset; + + if (buf->snapshot) + offset = trbe_snapshot_offset(handle); + else + offset = trbe_normal_offset(handle); + return buf->trbe_base + offset; +} + +static void clr_trbe_status(void) +{ + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); + + WARN_ON(is_trbe_enabled()); + trbsr &= ~TRBSR_IRQ; + trbsr &= ~TRBSR_TRG; + trbsr &= ~TRBSR_WRAP; + trbsr &= ~(TRBSR_EC_MASK << TRBSR_EC_SHIFT); + trbsr &= ~(TRBSR_BSC_MASK << TRBSR_BSC_SHIFT); + trbsr &= ~TRBSR_STOP; + write_sysreg_s(trbsr, SYS_TRBSR_EL1); +} + +static void set_trbe_limit_pointer_enabled(unsigned long addr) +{ + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); + + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); + + trblimitr &= ~TRBLIMITR_NVM; + trblimitr &= ~(TRBLIMITR_FILL_MODE_MASK << TRBLIMITR_FILL_MODE_SHIFT); + trblimitr &= ~(TRBLIMITR_TRIG_MODE_MASK << TRBLIMITR_TRIG_MODE_SHIFT); + trblimitr &= ~(TRBLIMITR_LIMIT_MASK << TRBLIMITR_LIMIT_SHIFT); + + /* + * Fill trace buffer mode is used here while configuring the + * TRBE for trace capture. In this particular mode, the trace + * collection is stopped and a maintenance interrupt is raised + * when the current write pointer wraps. This pause in trace + * collection gives the software an opportunity to capture the + * trace data in the interrupt handler, before reconfiguring + * the TRBE. + */ + trblimitr |= (TRBE_FILL_MODE_FILL & TRBLIMITR_FILL_MODE_MASK) << TRBLIMITR_FILL_MODE_SHIFT; + + /* + * Trigger mode is not used here while configuring the TRBE for + * the trace capture. Hence just keep this in the ignore mode. + */ + trblimitr |= (TRBE_TRIG_MODE_IGNORE & TRBLIMITR_TRIG_MODE_MASK) << TRBLIMITR_TRIG_MODE_SHIFT; + trblimitr |= (addr & PAGE_MASK); + + trblimitr |= TRBLIMITR_ENABLE; + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); +} + +static void trbe_enable_hw(struct trbe_buf *buf) +{ + WARN_ON(buf->trbe_write < buf->trbe_base); + WARN_ON(buf->trbe_write >= buf->trbe_limit); + set_trbe_disabled(); + isb(); + clr_trbe_status(); + set_trbe_base_pointer(buf->trbe_base); + set_trbe_write_pointer(buf->trbe_write); + + /* + * Synchronize all the register updates + * till now before enabling the TRBE. + */ + isb(); + set_trbe_limit_pointer_enabled(buf->trbe_limit); + + /* Synchronize the TRBE enable event */ + isb(); +} + +static void *arm_trbe_alloc_buffer(struct coresight_device *csdev, + struct perf_event *event, void **pages, + int nr_pages, bool snapshot) +{ + struct trbe_buf *buf; + struct page **pglist; + int i; + + /* + * TRBE LIMIT and TRBE WRITE pointers must be page aligned. But with + * just a single page, there is not much room left while writing into + * a partially filled TRBE buffer. Hence restrict the minimum buffer + * size as two pages. + */ + if (nr_pages < 2) + return NULL; + + buf = kzalloc_node(sizeof(*buf), GFP_KERNEL, trbe_alloc_node(event)); + if (IS_ERR(buf)) + return ERR_PTR(-ENOMEM); + + pglist = kcalloc(nr_pages, sizeof(*pglist), GFP_KERNEL); + if (IS_ERR(pglist)) { + kfree(buf); + return ERR_PTR(-ENOMEM); + } + + for (i = 0; i < nr_pages; i++) + pglist[i] = virt_to_page(pages[i]); + + buf->trbe_base = (unsigned long) vmap(pglist, nr_pages, VM_MAP, PAGE_KERNEL); + if (IS_ERR((void *)buf->trbe_base)) { + kfree(pglist); + kfree(buf); + return ERR_PTR(buf->trbe_base); + } + buf->trbe_limit = buf->trbe_base + nr_pages * PAGE_SIZE; + buf->trbe_write = buf->trbe_base; + buf->snapshot = snapshot; + buf->nr_pages = nr_pages; + buf->pages = pages; + kfree(pglist); + return buf; +} + +void arm_trbe_free_buffer(void *config) +{ + struct trbe_buf *buf = config; + + vunmap((void *)buf->trbe_base); + kfree(buf); +} + +static unsigned long arm_trbe_update_buffer(struct coresight_device *csdev, + struct perf_output_handle *handle, + void *config) +{ + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); + struct trbe_buf *buf = config; + unsigned long size, offset; + + WARN_ON(buf->cpudata != cpudata); + WARN_ON(cpudata->cpu != smp_processor_id()); + WARN_ON(cpudata->drvdata != drvdata); + if (cpudata->mode != CS_MODE_PERF) + return -EINVAL; + + /* + * If the TRBE was disabled due to lack of space in the AUX buffer or a + * spurious fault, the driver leaves it disabled, truncating the buffer. + * Since the etm_perf driver expects to close out the AUX buffer, the + * driver skips it. Thus, just pass in 0 size here to indicate that the + * buffer was truncated. + */ + if (!is_trbe_enabled()) + return 0; + /* + * perf handle structure needs to be shared with the TRBE IRQ handler for + * capturing trace data and restarting the handle. There is a probability + * of an undefined reference based crash when etm event is being stopped + * while a TRBE IRQ also getting processed. This happens due the release + * of perf handle via perf_aux_output_end() in etm_event_stop(). Stopping + * the TRBE here will ensure that no IRQ could be generated when the perf + * handle gets freed in etm_event_stop(). + */ + trbe_drain_and_disable_local(); + offset = get_trbe_write_pointer() - get_trbe_base_pointer(); + size = offset - PERF_IDX2OFF(handle->head, buf); + if (buf->snapshot) + handle->head += size; + return size; +} + +static int arm_trbe_enable(struct coresight_device *csdev, u32 mode, void *data) +{ + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); + struct perf_output_handle *handle = data; + struct trbe_buf *buf = etm_perf_sink_config(handle); + + WARN_ON(cpudata->cpu != smp_processor_id()); + WARN_ON(cpudata->drvdata != drvdata); + if (mode != CS_MODE_PERF) + return -EINVAL; + + *this_cpu_ptr(drvdata->handle) = handle; + cpudata->buf = buf; + cpudata->mode = mode; + buf->cpudata = cpudata; + buf->trbe_limit = compute_trbe_buffer_limit(handle); + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); + if (buf->trbe_limit == buf->trbe_base) { + trbe_stop_and_truncate_event(handle); + return 0; + } + trbe_enable_hw(buf); + return 0; +} + +static int arm_trbe_disable(struct coresight_device *csdev) +{ + struct trbe_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); + struct trbe_cpudata *cpudata = dev_get_drvdata(&csdev->dev); + struct trbe_buf *buf = cpudata->buf; + + WARN_ON(buf->cpudata != cpudata); + WARN_ON(cpudata->cpu != smp_processor_id()); + WARN_ON(cpudata->drvdata != drvdata); + if (cpudata->mode != CS_MODE_PERF) + return -EINVAL; + + trbe_drain_and_disable_local(); + buf->cpudata = NULL; + cpudata->buf = NULL; + cpudata->mode = CS_MODE_DISABLED; + return 0; +} + +static void trbe_handle_spurious(struct perf_output_handle *handle) +{ + struct trbe_buf *buf = etm_perf_sink_config(handle); + + buf->trbe_limit = compute_trbe_buffer_limit(handle); + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); + if (buf->trbe_limit == buf->trbe_base) { + trbe_drain_and_disable_local(); + return; + } + trbe_enable_hw(buf); +} + +static void trbe_handle_overflow(struct perf_output_handle *handle) +{ + struct perf_event *event = handle->event; + struct trbe_buf *buf = etm_perf_sink_config(handle); + unsigned long offset, size; + struct etm_event_data *event_data; + + offset = get_trbe_limit_pointer() - get_trbe_base_pointer(); + size = offset - PERF_IDX2OFF(handle->head, buf); + if (buf->snapshot) + handle->head = offset; + perf_aux_output_end(handle, size); + + event_data = perf_aux_output_begin(handle, event); + if (!event_data) { + trbe_drain_and_disable_local(); + *this_cpu_ptr(buf->cpudata->drvdata->handle) = NULL; + return; + } + buf->trbe_limit = compute_trbe_buffer_limit(handle); + buf->trbe_write = buf->trbe_base + PERF_IDX2OFF(handle->head, buf); + if (buf->trbe_limit == buf->trbe_base) { + trbe_stop_and_truncate_event(handle); + return; + } + *this_cpu_ptr(buf->cpudata->drvdata->handle) = handle; + trbe_enable_hw(buf); +} + +static bool is_perf_trbe(struct perf_output_handle *handle) +{ + struct trbe_buf *buf = etm_perf_sink_config(handle); + struct trbe_cpudata *cpudata = buf->cpudata; + struct trbe_drvdata *drvdata = cpudata->drvdata; + int cpu = smp_processor_id(); + + WARN_ON(buf->trbe_base != get_trbe_base_pointer()); + WARN_ON(buf->trbe_limit != get_trbe_limit_pointer()); + + if (cpudata->mode != CS_MODE_PERF) + return false; + + if (cpudata->cpu != cpu) + return false; + + if (!cpumask_test_cpu(cpu, &drvdata->supported_cpus)) + return false; + + return true; +} + +static enum trbe_fault_action trbe_get_fault_act(struct perf_output_handle *handle) +{ + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); + int ec = get_trbe_ec(trbsr); + int bsc = get_trbe_bsc(trbsr); + + WARN_ON(is_trbe_running(trbsr)); + if (is_trbe_trg(trbsr) || is_trbe_abort(trbsr)) + return TRBE_FAULT_ACT_FATAL; + + if ((ec == TRBE_EC_STAGE1_ABORT) || (ec == TRBE_EC_STAGE2_ABORT)) + return TRBE_FAULT_ACT_FATAL; + + if (is_trbe_wrap(trbsr) && (ec == TRBE_EC_OTHERS) && (bsc == TRBE_BSC_FILLED)) { + if (get_trbe_write_pointer() == get_trbe_base_pointer()) + return TRBE_FAULT_ACT_WRAP; + } + return TRBE_FAULT_ACT_SPURIOUS; +} + +static irqreturn_t arm_trbe_irq_handler(int irq, void *dev) +{ + struct perf_output_handle **handle_ptr = dev; + struct perf_output_handle *handle = *handle_ptr; + enum trbe_fault_action act; + + WARN_ON(!is_trbe_irq(read_sysreg_s(SYS_TRBSR_EL1))); + clr_trbe_irq(); + + /* + * Ensure the trace is visible to the CPUs and + * any external aborts have been resolved. + */ + trbe_drain_buffer(); + isb(); + + if (!perf_get_aux(handle)) + return IRQ_NONE; + + if (!is_perf_trbe(handle)) + return IRQ_NONE; + + irq_work_run(); + + act = trbe_get_fault_act(handle); + switch (act) { + case TRBE_FAULT_ACT_WRAP: + trbe_handle_overflow(handle); + break; + case TRBE_FAULT_ACT_SPURIOUS: + trbe_handle_spurious(handle); + break; + case TRBE_FAULT_ACT_FATAL: + trbe_stop_and_truncate_event(handle); + break; + } + return IRQ_HANDLED; +} + +static const struct coresight_ops_sink arm_trbe_sink_ops = { + .enable = arm_trbe_enable, + .disable = arm_trbe_disable, + .alloc_buffer = arm_trbe_alloc_buffer, + .free_buffer = arm_trbe_free_buffer, + .update_buffer = arm_trbe_update_buffer, +}; + +static const struct coresight_ops arm_trbe_cs_ops = { + .sink_ops = &arm_trbe_sink_ops, +}; + +static ssize_t align_show(struct device *dev, struct device_attribute *attr, char *buf) +{ + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); + + return sprintf(buf, "%llx\n", cpudata->trbe_align); +} +static DEVICE_ATTR_RO(align); + +static ssize_t dbm_show(struct device *dev, struct device_attribute *attr, char *buf) +{ + struct trbe_cpudata *cpudata = dev_get_drvdata(dev); + + return sprintf(buf, "%d\n", cpudata->trbe_dbm); +} +static DEVICE_ATTR_RO(dbm); + +static struct attribute *arm_trbe_attrs[] = { + &dev_attr_align.attr, + &dev_attr_dbm.attr, + NULL, +}; + +static const struct attribute_group arm_trbe_group = { + .attrs = arm_trbe_attrs, +}; + +static const struct attribute_group *arm_trbe_groups[] = { + &arm_trbe_group, + NULL, +}; + +static void arm_trbe_probe_coresight_cpu(void *info) +{ + struct trbe_drvdata *drvdata = info; + struct coresight_desc desc = { 0 }; + int cpu = smp_processor_id(); + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); + u64 trbidr = read_sysreg_s(SYS_TRBIDR_EL1); + struct device *dev; + + if (WARN_ON(!cpudata)) + goto cpu_clear; + + if (trbe_csdev) + return; + + cpudata->cpu = smp_processor_id(); + cpudata->drvdata = drvdata; + dev = &cpudata->drvdata->pdev->dev; + + if (!is_trbe_available()) { + pr_err("TRBE is not implemented on cpu %d\n", cpudata->cpu); + goto cpu_clear; + } + + if (!is_trbe_programmable(trbidr)) { + pr_err("TRBE is owned in higher exception level on cpu %d\n", cpudata->cpu); + goto cpu_clear; + } + desc.name = devm_kasprintf(dev, GFP_KERNEL, "%s%d", DRVNAME, smp_processor_id()); + if (IS_ERR(desc.name)) + goto cpu_clear; + + desc.type = CORESIGHT_DEV_TYPE_SINK; + desc.subtype.sink_subtype = CORESIGHT_DEV_SUBTYPE_SINK_PERCPU_SYSMEM; + desc.ops = &arm_trbe_cs_ops; + desc.pdata = dev_get_platdata(dev); + desc.groups = arm_trbe_groups; + desc.dev = dev; + trbe_csdev = coresight_register(&desc); + if (IS_ERR(trbe_csdev)) + goto cpu_clear; + + dev_set_drvdata(&trbe_csdev->dev, cpudata); + cpudata->trbe_dbm = get_trbe_flag_update(trbidr); + cpudata->trbe_align = 1ULL << get_trbe_address_align(trbidr); + if (cpudata->trbe_align > SZ_2K) { + pr_err("Unsupported alignment on cpu %d\n", cpudata->cpu); + goto cpu_clear; + } + per_cpu(csdev_sink, cpu) = trbe_csdev; + trbe_reset_local(); + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); + return; +cpu_clear: + cpumask_clear_cpu(cpudata->cpu, &cpudata->drvdata->supported_cpus); +} + +static void arm_trbe_remove_coresight_cpu(void *info) +{ + int cpu = smp_processor_id(); + struct trbe_drvdata *drvdata = info; + struct trbe_cpudata *cpudata = per_cpu_ptr(drvdata->cpudata, cpu); + struct coresight_device *trbe_csdev = per_cpu(csdev_sink, cpu); + + if (trbe_csdev) { + coresight_unregister(trbe_csdev); + cpudata->drvdata = NULL; + per_cpu(csdev_sink, cpu) = NULL; + } + disable_percpu_irq(drvdata->irq); + trbe_reset_local(); +} + +static int arm_trbe_probe_coresight(struct trbe_drvdata *drvdata) +{ + drvdata->cpudata = alloc_percpu(typeof(*drvdata->cpudata)); + if (IS_ERR(drvdata->cpudata)) + return PTR_ERR(drvdata->cpudata); + + arm_trbe_probe_coresight_cpu(drvdata); + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_probe_coresight_cpu, drvdata, 1); + return 0; +} + +static int arm_trbe_remove_coresight(struct trbe_drvdata *drvdata) +{ + arm_trbe_remove_coresight_cpu(drvdata); + smp_call_function_many(&drvdata->supported_cpus, arm_trbe_remove_coresight_cpu, drvdata, 1); + free_percpu(drvdata->cpudata); + return 0; +} + +static int arm_trbe_cpu_startup(unsigned int cpu, struct hlist_node *node) +{ + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); + + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { + if (!per_cpu(csdev_sink, cpu)) { + arm_trbe_probe_coresight_cpu(drvdata); + } else { + trbe_reset_local(); + enable_percpu_irq(drvdata->irq, IRQ_TYPE_NONE); + } + } + return 0; +} + +static int arm_trbe_cpu_teardown(unsigned int cpu, struct hlist_node *node) +{ + struct trbe_drvdata *drvdata = hlist_entry_safe(node, struct trbe_drvdata, hotplug_node); + + if (cpumask_test_cpu(cpu, &drvdata->supported_cpus)) { + disable_percpu_irq(drvdata->irq); + trbe_reset_local(); + } + return 0; +} + +static int arm_trbe_probe_cpuhp(struct trbe_drvdata *drvdata) +{ + enum cpuhp_state trbe_online; + + trbe_online = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, DRVNAME, + arm_trbe_cpu_startup, arm_trbe_cpu_teardown); + if (trbe_online < 0) + return -EINVAL; + + if (cpuhp_state_add_instance(trbe_online, &drvdata->hotplug_node)) + return -EINVAL; + + drvdata->trbe_online = trbe_online; + return 0; +} + +static void arm_trbe_remove_cpuhp(struct trbe_drvdata *drvdata) +{ + cpuhp_remove_multi_state(drvdata->trbe_online); +} + +static int arm_trbe_probe_irq(struct platform_device *pdev, + struct trbe_drvdata *drvdata) +{ + drvdata->irq = platform_get_irq(pdev, 0); + if (!drvdata->irq) { + pr_err("IRQ not found for the platform device\n"); + return -ENXIO; + } + + if (!irq_is_percpu(drvdata->irq)) { + pr_err("IRQ is not a PPI\n"); + return -EINVAL; + } + + if (irq_get_percpu_devid_partition(drvdata->irq, &drvdata->supported_cpus)) + return -EINVAL; + + drvdata->handle = alloc_percpu(typeof(*drvdata->handle)); + if (!drvdata->handle) + return -ENOMEM; + + if (request_percpu_irq(drvdata->irq, arm_trbe_irq_handler, DRVNAME, drvdata->handle)) { + free_percpu(drvdata->handle); + return -EINVAL; + } + return 0; +} + +static void arm_trbe_remove_irq(struct trbe_drvdata *drvdata) +{ + free_percpu_irq(drvdata->irq, drvdata->handle); + free_percpu(drvdata->handle); +} + +static int arm_trbe_device_probe(struct platform_device *pdev) +{ + struct coresight_platform_data *pdata; + struct trbe_drvdata *drvdata; + struct device *dev = &pdev->dev; + int ret; + + drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL); + if (IS_ERR(drvdata)) + return -ENOMEM; + + pdata = coresight_get_platform_data(dev); + if (IS_ERR(pdata)) { + kfree(drvdata); + return -ENOMEM; + } + + dev_set_drvdata(dev, drvdata); + dev->platform_data = pdata; + drvdata->pdev = pdev; + ret = arm_trbe_probe_irq(pdev, drvdata); + if (ret) + goto irq_failed; + + ret = arm_trbe_probe_coresight(drvdata); + if (ret) + goto probe_failed; + + ret = arm_trbe_probe_cpuhp(drvdata); + if (ret) + goto cpuhp_failed; + + return 0; +cpuhp_failed: + arm_trbe_remove_coresight(drvdata); +probe_failed: + arm_trbe_remove_irq(drvdata); +irq_failed: + kfree(pdata); + kfree(drvdata); + return ret; +} + +static int arm_trbe_device_remove(struct platform_device *pdev) +{ + struct coresight_platform_data *pdata = dev_get_platdata(&pdev->dev); + struct trbe_drvdata *drvdata = platform_get_drvdata(pdev); + + arm_trbe_remove_coresight(drvdata); + arm_trbe_remove_cpuhp(drvdata); + arm_trbe_remove_irq(drvdata); + kfree(pdata); + kfree(drvdata); + return 0; +} + +static const struct of_device_id arm_trbe_of_match[] = { + { .compatible = "arm,trace-buffer-extension"}, + {}, +}; +MODULE_DEVICE_TABLE(of, arm_trbe_of_match); + +static struct platform_driver arm_trbe_driver = { + .driver = { + .name = DRVNAME, + .of_match_table = of_match_ptr(arm_trbe_of_match), + .suppress_bind_attrs = true, + }, + .probe = arm_trbe_device_probe, + .remove = arm_trbe_device_remove, +}; + +static int __init arm_trbe_init(void) +{ + int ret; + + if (arm64_kernel_unmapped_at_el0()) { + pr_err("TRBE wouldn't work if kernel gets unmapped at EL0\n"); + return -EOPNOTSUPP; + } + + ret = platform_driver_register(&arm_trbe_driver); + if (!ret) + return 0; + + pr_err("Error registering %s platform driver\n", DRVNAME); + return ret; +} + +static void __exit arm_trbe_exit(void) +{ + platform_driver_unregister(&arm_trbe_driver); +} +module_init(arm_trbe_init); +module_exit(arm_trbe_exit); + +MODULE_AUTHOR("Anshuman Khandual <anshuman.khandual@arm.com>"); +MODULE_DESCRIPTION("Arm Trace Buffer Extension (TRBE) driver"); +MODULE_LICENSE("GPL v2"); diff --git a/drivers/hwtracing/coresight/coresight-trbe.h b/drivers/hwtracing/coresight/coresight-trbe.h new file mode 100644 index 0000000..43308bc --- /dev/null +++ b/drivers/hwtracing/coresight/coresight-trbe.h @@ -0,0 +1,160 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * This contains all required hardware related helper functions for + * Trace Buffer Extension (TRBE) driver in the coresight framework. + * + * Copyright (C) 2020 ARM Ltd. + * + * Author: Anshuman Khandual <anshuman.khandual@arm.com> + */ +#include <linux/coresight.h> +#include <linux/device.h> +#include <linux/irq.h> +#include <linux/kernel.h> +#include <linux/of.h> +#include <linux/platform_device.h> +#include <linux/smp.h> + +#include "coresight-etm-perf.h" + +DECLARE_PER_CPU(struct coresight_device *, csdev_sink); + +static inline bool is_trbe_available(void) +{ + u64 aa64dfr0 = read_sysreg_s(SYS_ID_AA64DFR0_EL1); + unsigned int trbe = cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_TRBE_SHIFT); + + return trbe >= 0b0001; +} + +static inline bool is_trbe_enabled(void) +{ + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); + + return trblimitr & TRBLIMITR_ENABLE; +} + +#define TRBE_EC_OTHERS 0 +#define TRBE_EC_STAGE1_ABORT 36 +#define TRBE_EC_STAGE2_ABORT 37 + +static inline int get_trbe_ec(u64 trbsr) +{ + return (trbsr >> TRBSR_EC_SHIFT) & TRBSR_EC_MASK; +} + +#define TRBE_BSC_NOT_STOPPED 0 +#define TRBE_BSC_FILLED 1 +#define TRBE_BSC_TRIGGERED 2 + +static inline int get_trbe_bsc(u64 trbsr) +{ + return (trbsr >> TRBSR_BSC_SHIFT) & TRBSR_BSC_MASK; +} + +static inline void clr_trbe_irq(void) +{ + u64 trbsr = read_sysreg_s(SYS_TRBSR_EL1); + + trbsr &= ~TRBSR_IRQ; + write_sysreg_s(trbsr, SYS_TRBSR_EL1); +} + +static inline bool is_trbe_irq(u64 trbsr) +{ + return trbsr & TRBSR_IRQ; +} + +static inline bool is_trbe_trg(u64 trbsr) +{ + return trbsr & TRBSR_TRG; +} + +static inline bool is_trbe_wrap(u64 trbsr) +{ + return trbsr & TRBSR_WRAP; +} + +static inline bool is_trbe_abort(u64 trbsr) +{ + return trbsr & TRBSR_ABORT; +} + +static inline bool is_trbe_running(u64 trbsr) +{ + return !(trbsr & TRBSR_STOP); +} + +#define TRBE_TRIG_MODE_STOP 0 +#define TRBE_TRIG_MODE_IRQ 1 +#define TRBE_TRIG_MODE_IGNORE 3 + +#define TRBE_FILL_MODE_FILL 0 +#define TRBE_FILL_MODE_WRAP 1 +#define TRBE_FILL_MODE_CIRCULAR_BUFFER 3 + +static inline void set_trbe_disabled(void) +{ + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); + + trblimitr &= ~TRBLIMITR_ENABLE; + write_sysreg_s(trblimitr, SYS_TRBLIMITR_EL1); +} + +static inline bool get_trbe_flag_update(u64 trbidr) +{ + return trbidr & TRBIDR_FLAG; +} + +static inline bool is_trbe_programmable(u64 trbidr) +{ + return !(trbidr & TRBIDR_PROG); +} + +static inline int get_trbe_address_align(u64 trbidr) +{ + return (trbidr >> TRBIDR_ALIGN_SHIFT) & TRBIDR_ALIGN_MASK; +} + +static inline unsigned long get_trbe_write_pointer(void) +{ + u64 trbptr = read_sysreg_s(SYS_TRBPTR_EL1); + unsigned long addr = (trbptr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; + + return addr; +} + +static inline void set_trbe_write_pointer(unsigned long addr) +{ + WARN_ON(is_trbe_enabled()); + addr = (addr >> TRBPTR_PTR_SHIFT) & TRBPTR_PTR_MASK; + write_sysreg_s(addr, SYS_TRBPTR_EL1); +} + +static inline unsigned long get_trbe_limit_pointer(void) +{ + u64 trblimitr = read_sysreg_s(SYS_TRBLIMITR_EL1); + unsigned long limit = (trblimitr >> TRBLIMITR_LIMIT_SHIFT) & TRBLIMITR_LIMIT_MASK; + unsigned long addr = limit << TRBLIMITR_LIMIT_SHIFT; + + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); + return addr; +} + +static inline unsigned long get_trbe_base_pointer(void) +{ + u64 trbbaser = read_sysreg_s(SYS_TRBBASER_EL1); + unsigned long addr = (trbbaser >> TRBBASER_BASE_SHIFT) & TRBBASER_BASE_MASK; + + addr = addr << TRBBASER_BASE_SHIFT; + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); + return addr; +} + +static inline void set_trbe_base_pointer(unsigned long addr) +{ + WARN_ON(is_trbe_enabled()); + WARN_ON(!IS_ALIGNED(addr, (1UL << TRBLIMITR_LIMIT_SHIFT))); + WARN_ON(!IS_ALIGNED(addr, PAGE_SIZE)); + write_sysreg_s(addr, SYS_TRBBASER_EL1); +}
Trace Buffer Extension (TRBE) implements a trace buffer per CPU which is accessible via the system registers. The TRBE supports different addressing modes including CPU virtual address and buffer modes including the circular buffer mode. The TRBE buffer is addressed by a base pointer (TRBBASER_EL1), an write pointer (TRBPTR_EL1) and a limit pointer (TRBLIMITR_EL1). But the access to the trace buffer could be prohibited by a higher exception level (EL3 or EL2), indicated by TRBIDR_EL1.P. The TRBE can also generate a CPU private interrupt (PPI) on address translation errors and when the buffer is full. Overall implementation here is inspired from the Arm SPE driver. Cc: Mathieu Poirier <mathieu.poirier@linaro.org> Cc: Mike Leach <mike.leach@linaro.org> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com> --- Changes in V3: - Added new DT bindings document TRBE.yaml - Changed TRBLIMITR_TRIG_MODE_SHIFT from 2 to 3 - Dropped isb() from trbe_reset_local() - Dropped gap between (void *) and buf->trbe_base - Changed 'int' to 'unsigned int' in is_trbe_available() - Dropped unused function set_trbe_running(), set_trbe_virtual_mode(), set_trbe_enabled() and set_trbe_limit_pointer() - Changed get_trbe_flag_update(), is_trbe_programmable() and get_trbe_address_align() to accept TRBIDR value - Changed is_trbe_running(), is_trbe_abort(), is_trbe_wrap(), is_trbe_trg(), is_trbe_irq(), get_trbe_bsc() and get_trbe_ec() to accept TRBSR value - Dropped snapshot mode condition in arm_trbe_alloc_buffer() - Exit arm_trbe_init() when arm64_kernel_unmapped_at_el0() is enabled - Compute trbe_limit before trbe_write to get the updated handle - Added trbe_stop_and_truncate_event() - Dropped trbe_handle_fatal() Documentation/trace/coresight/coresight-trbe.rst | 39 + arch/arm64/include/asm/sysreg.h | 1 + drivers/hwtracing/coresight/Kconfig | 11 + drivers/hwtracing/coresight/Makefile | 1 + drivers/hwtracing/coresight/coresight-trbe.c | 1023 ++++++++++++++++++++++ drivers/hwtracing/coresight/coresight-trbe.h | 160 ++++ 6 files changed, 1235 insertions(+) create mode 100644 Documentation/trace/coresight/coresight-trbe.rst create mode 100644 drivers/hwtracing/coresight/coresight-trbe.c create mode 100644 drivers/hwtracing/coresight/coresight-trbe.h