| Message ID | 1618654631-42454-1-git-send-email-yangyicong@hisilicon.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add support for HiSilicon PCIe Tune and Trace device | expand |
Yicong Yang <yangyicong@hisilicon.com> writes: > The reason for not using perf is because there is no current support > for uncore tracing in the perf facilities. Not unless you count $ perf list|grep -ic uncore 77 > We have our own format > of data and don't need perf doing the parsing. Perf has AUX buffers, which are used for all kinds of own formats. > A similar approach for implementing this function is ETM, which use > sysfs for configuring and a character device for dumping data. And also perf. One reason ETM has a sysfs interface is because the driver predates perf's AUX buffers. Can't say if it's the only reason. I'm assuming you're talking about Coresight ETM. > Greg has some comments on our implementation and doesn't advocate > to build driver on debugfs [1]. So I resend this series to > collect more feedbacks on the implementation of this driver. > > Hi perf and ETM related experts, is it suggested to adapt this driver > to perf? Or is the debugfs approach acceptable? Otherwise use Aside from the above, I don't think the use of debugfs for kernel ABIs is ever encouraged. Regards, -- Ale
On 17/04/2021 11:17, Yicong Yang wrote: > [RESEND with perf and coresight folks Cc'ed] > > HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex > integrated Endpoint (RCiEP) device, providing the capability > to dynamically monitor and tune the PCIe traffic (tune), > and trace the TLP headers (trace). > > PTT tune is designed for monitoring and adjusting PCIe link parameters. > We provide several parameters of the PCIe link. Through the driver, > user can adjust the value of certain parameter to affect the PCIe link > for the purpose of enhancing the performance in certian situation. ... > > The reason for not using perf is because there is no current support > for uncore tracing in the perf facilities. We have our own format > of data and don't need perf doing the parsing. The setting through > perf tools doesn't seem to be friendly as well. For example, > we cannot count on perf to decode the usual format BDF number like > <domain>:<bus>:<dev>.<fn>, which user can use to filter the TLP > headers through the PTT device. > > A similar approach for implementing this function is ETM, which use > sysfs for configuring and a character device for dumping data. > > Greg has some comments on our implementation and doesn't advocate > to build driver on debugfs [1]. So I resend this series to > collect more feedbacks on the implementation of this driver. > > Hi perf and ETM related experts, is it suggested to adapt this driver > to perf? Or is the debugfs approach acceptable? Otherwise use > sysfs + character device like ETM and use perf tools for decoding it? > Any comments is welcomed. Please use perf. Debugfs / sysfs is not the right place for these things. Also, please move your driver to drivers/perf/ As Alex mentioned, the ETM drivers were initially developed when the AUX buffer was not available. The sysfs interface is there only for the backward compatibility and for bring up ( due to the nature of the connections between the CoreSight components and sometimes the missing engineering spec). Suzuki
On 2021/4/17 21:56, Alexander Shishkin wrote: > Yicong Yang <yangyicong@hisilicon.com> writes: > >> The reason for not using perf is because there is no current support >> for uncore tracing in the perf facilities. > > Not unless you count > > $ perf list|grep -ic uncore > 77 > these are uncore events probably do not support sampling. I tried on x86: # ./perf record -e uncore_imc_0/cas_count_read/ Error: The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (uncore_imc_0/cas_count_read/). /bin/dmesg | grep -i perf may provide additional information. For HiSilicon uncore PMUs, we don't support uncore sampling: 'The current driver does not support sampling. So "perf record" is unsupported. ' [1] and also in another PMU: 'PMU doesn't support process specific events and cannot be used in sampling mode.' [2] [1] Documentation/admin-guide/perf/hisi-pmu.rst [2] Documentation/admin-guide/perf/arm_dsu_pmu.rst >> We have our own format >> of data and don't need perf doing the parsing. > > Perf has AUX buffers, which are used for all kinds of own formats. > ok. we thought perf will break the data format but AUX buffers seems won't. do we need to add full support for tracing as well as parsing or it's ok for not parsing it through perf? >> A similar approach for implementing this function is ETM, which use >> sysfs for configuring and a character device for dumping data. > > And also perf. One reason ETM has a sysfs interface is because the > driver predates perf's AUX buffers. Can't say if it's the only > reason. I'm assuming you're talking about Coresight ETM. > got it. thanks. >> Greg has some comments on our implementation and doesn't advocate >> to build driver on debugfs [1]. So I resend this series to >> collect more feedbacks on the implementation of this driver. >> >> Hi perf and ETM related experts, is it suggested to adapt this driver >> to perf? Or is the debugfs approach acceptable? Otherwise use > > Aside from the above, I don't think the use of debugfs for kernel ABIs > is ever encouraged. > ok. thanks for the suggestions. Regards, Yicong > Regards, > -- > Ale > > . >
On 2021/4/19 19:17, Suzuki K Poulose wrote: > On 17/04/2021 11:17, Yicong Yang wrote: >> [RESEND with perf and coresight folks Cc'ed] >> >> HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex >> integrated Endpoint (RCiEP) device, providing the capability >> to dynamically monitor and tune the PCIe traffic (tune), >> and trace the TLP headers (trace). >> >> PTT tune is designed for monitoring and adjusting PCIe link parameters. >> We provide several parameters of the PCIe link. Through the driver, >> user can adjust the value of certain parameter to affect the PCIe link >> for the purpose of enhancing the performance in certian situation. > > ... > >> >> The reason for not using perf is because there is no current support >> for uncore tracing in the perf facilities. We have our own format >> of data and don't need perf doing the parsing. The setting through >> perf tools doesn't seem to be friendly as well. For example, >> we cannot count on perf to decode the usual format BDF number like >> <domain>:<bus>:<dev>.<fn>, which user can use to filter the TLP >> headers through the PTT device. >> >> A similar approach for implementing this function is ETM, which use >> sysfs for configuring and a character device for dumping data. >> >> Greg has some comments on our implementation and doesn't advocate >> to build driver on debugfs [1]. So I resend this series to >> collect more feedbacks on the implementation of this driver. >> >> Hi perf and ETM related experts, is it suggested to adapt this driver >> to perf? Or is the debugfs approach acceptable? Otherwise use >> sysfs + character device like ETM and use perf tools for decoding it? >> Any comments is welcomed. > > Please use perf. Debugfs / sysfs is not the right place for these things. > ok. > Also, please move your driver to drivers/perf/ > Does it make sense as it's a tuning and tracing device, and doesn't have counters nor do the sampling like usual PMU device under drivers/perf/. > As Alex mentioned, the ETM drivers were initially developed when the AUX > buffer was not available. The sysfs interface is there only for the backward compatibility and for bring up ( due to the nature of the > connections between the CoreSight components and sometimes the missing engineering spec). > got it. thanks for the explanation. Regards, Yicong > Suzuki > > .
On 19/04/2021 14:21, Yicong Yang wrote: > On 2021/4/19 19:17, Suzuki K Poulose wrote: >> On 17/04/2021 11:17, Yicong Yang wrote: >>> [RESEND with perf and coresight folks Cc'ed] >>> >>> HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex >>> integrated Endpoint (RCiEP) device, providing the capability >>> to dynamically monitor and tune the PCIe traffic (tune), >>> and trace the TLP headers (trace). >>> >>> PTT tune is designed for monitoring and adjusting PCIe link parameters. >>> We provide several parameters of the PCIe link. Through the driver, >>> user can adjust the value of certain parameter to affect the PCIe link >>> for the purpose of enhancing the performance in certian situation. >> >> ... >> >>> >>> The reason for not using perf is because there is no current support >>> for uncore tracing in the perf facilities. We have our own format >>> of data and don't need perf doing the parsing. The setting through >>> perf tools doesn't seem to be friendly as well. For example, >>> we cannot count on perf to decode the usual format BDF number like >>> <domain>:<bus>:<dev>.<fn>, which user can use to filter the TLP >>> headers through the PTT device. >>> >>> A similar approach for implementing this function is ETM, which use >>> sysfs for configuring and a character device for dumping data. >>> >>> Greg has some comments on our implementation and doesn't advocate >>> to build driver on debugfs [1]. So I resend this series to >>> collect more feedbacks on the implementation of this driver. >>> >>> Hi perf and ETM related experts, is it suggested to adapt this driver >>> to perf? Or is the debugfs approach acceptable? Otherwise use >>> sysfs + character device like ETM and use perf tools for decoding it? >>> Any comments is welcomed. >> >> Please use perf. Debugfs / sysfs is not the right place for these things. >> > > ok. > >> Also, please move your driver to drivers/perf/ >> > > Does it make sense as it's a tuning and tracing device, and doesn't have counters > nor do the sampling like usual PMU device under drivers/perf/. It doesn't matter. As long as you can drive it via the perf interface, it can live there. The CoreSight was added way before there was kind of a suitable place like the above. You could find other uncore PMUs under drivers/perf. Suzuki
On Mon, Apr 19, 2021 at 09:03:18PM +0800, Yicong Yang wrote: > On 2021/4/17 21:56, Alexander Shishkin wrote: > > Yicong Yang <yangyicong@hisilicon.com> writes: > > > >> The reason for not using perf is because there is no current support > >> for uncore tracing in the perf facilities. > > > > Not unless you count > > > > $ perf list|grep -ic uncore > > 77 > > > > these are uncore events probably do not support sampling. > > I tried on x86: > > # ./perf record -e uncore_imc_0/cas_count_read/ > Error: > The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (uncore_imc_0/cas_count_read/). > /bin/dmesg | grep -i perf may provide additional information. > > For HiSilicon uncore PMUs, we don't support uncore sampling: > > 'The current driver does not support sampling. So "perf record" is unsupported. ' [1] > > and also in another PMU: > > 'PMU doesn't support process specific events and cannot be used in sampling mode.' [2] > > [1] Documentation/admin-guide/perf/hisi-pmu.rst > [2] Documentation/admin-guide/perf/arm_dsu_pmu.rst I did some debugging for this, and yes, it's related with the event doesn't support sampling for these x86 uncore events. So I can use below commands for the uncore event 'uncore_imc/data_reads/' in my experiment: # perf record -e 'uncore_imc/data_reads/' --no-samples -- ls # perf stat -e 'uncore_imc/data_reads/' -- ls For your case, I think you need to write the callback pmu::event_init(), it should not forbid any tracing even if set sampling, just like other perf event drive for support AUX tracing. > >> We have our own format > >> of data and don't need perf doing the parsing. > > > > Perf has AUX buffers, which are used for all kinds of own formats. > > > > ok. we thought perf will break the data format but AUX buffers seems won't. > do we need to add full support for tracing as well as parsing or it's ok for > not parsing it through perf? IMHO, this could divide into two parts. The first part is to enable perf drive with support AUX tracing, and perf tool can capture the trace data. The second part is to add the decoder in the perf tool so that the developers can *consume* the trace data; for the decoder, you could refer the codes: tools/perf/util/intel-pt-decoder/ tools/perf/util/cs-etm-decoder/ Or Arm SPE case: tools/perf/util/arm-spe-decoder/ > >> A similar approach for implementing this function is ETM, which use > >> sysfs for configuring and a character device for dumping data. > > > > And also perf. One reason ETM has a sysfs interface is because the > > driver predates perf's AUX buffers. Can't say if it's the only > > reason. I'm assuming you're talking about Coresight ETM. I am not the best person to give background for this. Mathieu or Mike could give more info for this. From my undersanding, Sysfs nodes can be used as knobs for configuration, but it's difficult for profiling. Let's think about for the profiling, if one developer uses the Sysfs for the setting and read out the trace data, these informations are discrete. If another developer wants to review the profiling result, then all these info need to be shared together. So we can benefit much from the perf tool for the usage, since all the profiling context will be gathered (DSOs, hardware configuration which can be saved into metadata), so the final profiling file can be easily shared and more friendly for reviewing. Thanks, Leo
On 2021/4/22 11:49, Leo Yan wrote: > On Mon, Apr 19, 2021 at 09:03:18PM +0800, Yicong Yang wrote: >> On 2021/4/17 21:56, Alexander Shishkin wrote: >>> Yicong Yang <yangyicong@hisilicon.com> writes: >>> >>>> The reason for not using perf is because there is no current support >>>> for uncore tracing in the perf facilities. >>> >>> Not unless you count >>> >>> $ perf list|grep -ic uncore >>> 77 >>> >> >> these are uncore events probably do not support sampling. >> >> I tried on x86: >> >> # ./perf record -e uncore_imc_0/cas_count_read/ >> Error: >> The sys_perf_event_open() syscall returned with 22 (Invalid argument) for event (uncore_imc_0/cas_count_read/). >> /bin/dmesg | grep -i perf may provide additional information. >> >> For HiSilicon uncore PMUs, we don't support uncore sampling: >> >> 'The current driver does not support sampling. So "perf record" is unsupported. ' [1] >> >> and also in another PMU: >> >> 'PMU doesn't support process specific events and cannot be used in sampling mode.' [2] >> >> [1] Documentation/admin-guide/perf/hisi-pmu.rst >> [2] Documentation/admin-guide/perf/arm_dsu_pmu.rst > > I did some debugging for this, and yes, it's related with the event > doesn't support sampling for these x86 uncore events. > > So I can use below commands for the uncore event > 'uncore_imc/data_reads/' in my experiment: > > # perf record -e 'uncore_imc/data_reads/' --no-samples -- ls > # perf stat -e 'uncore_imc/data_reads/' -- ls > > For your case, I think you need to write the callback > pmu::event_init(), it should not forbid any tracing even if set > sampling, just like other perf event drive for support AUX tracing. > thanks for the hint! I didn't know much about perf so I only do the basic test. will further investigate on this. >>>> We have our own format >>>> of data and don't need perf doing the parsing. >>> >>> Perf has AUX buffers, which are used for all kinds of own formats. >>> >> >> ok. we thought perf will break the data format but AUX buffers seems won't. >> do we need to add full support for tracing as well as parsing or it's ok for >> not parsing it through perf? > > IMHO, this could divide into two parts. The first part is to enable > perf drive with support AUX tracing, and perf tool can capture the trace > data. The second part is to add the decoder in the perf tool so that > the developers can *consume* the trace data; for the decoder, you > could refer the codes: > > tools/perf/util/intel-pt-decoder/ > tools/perf/util/cs-etm-decoder/ > > Or Arm SPE case: > > tools/perf/util/arm-spe-decoder/ > will refer to these implementation to see how to add the decoder for our traced data. very detailed guidance! >>>> A similar approach for implementing this function is ETM, which use >>>> sysfs for configuring and a character device for dumping data. >>> >>> And also perf. One reason ETM has a sysfs interface is because the >>> driver predates perf's AUX buffers. Can't say if it's the only >>> reason. I'm assuming you're talking about Coresight ETM. > > I am not the best person to give background for this. Mathieu or Mike > could give more info for this. From my undersanding, Sysfs nodes can > be used as knobs for configuration, but it's difficult for profiling. > as explained by the maintainers that there are some historical reasons for having sysfs interfaces for ETM as there is no perf AUX buffers at beginning. I thought sysfs interface as an option but perf AUX buffer is better as suggested. > Let's think about for the profiling, if one developer uses the Sysfs > for the setting and read out the trace data, these informations are > discrete. If another developer wants to review the profiling result, > then all these info need to be shared together. > ok. make sense to me. > So we can benefit much from the perf tool for the usage, since all the > profiling context will be gathered (DSOs, hardware configuration which > can be saved into metadata), so the final profiling file can be easily > shared and more friendly for reviewing. > ok. it will be beneficial if we use perf for both tracing and decoding, as we'll also get addition information attached to the trace data. Considering we have two functions: tracing and tuning. For tracing we can make use of perf AUX buffer but for tuning, I still cannot see how to make use of perf. So probably we can make tuning go through sysfs? And Daniel suggested so. Appreciate for the suggestion and guidance! Regards, Yicong > Thanks, > Leo > > . >
[RESEND with perf and coresight folks Cc'ed] HiSilicon PCIe tune and trace device (PTT) is a PCIe Root Complex integrated Endpoint (RCiEP) device, providing the capability to dynamically monitor and tune the PCIe traffic (tune), and trace the TLP headers (trace). PTT tune is designed for monitoring and adjusting PCIe link parameters. We provide several parameters of the PCIe link. Through the driver, user can adjust the value of certain parameter to affect the PCIe link for the purpose of enhancing the performance in certian situation. PTT trace is designed for dumping the TLP headers to the memory, which can be used to analyze the transactions and usage condition of the PCIe Link. Users can choose filters to trace headers, by either requester ID, or those downstream of a set of Root Ports on the same core of the PTT device. It's also supported to trace the headers of certain type and of certain direction. We use debugfs to expose the interface. For tune, one parameter is a debugfs file and user can set/get the value by reading/writing the file. For trace, we have several control files for the user to configure the trace parameters like filters, TLP type and format, the desired trace data size and so on. There is one data file for dumping the traced data to the user. The traced data maybe hundreds of megabytes so sysfs cannot support. The reason for debugfs rather than character device is that we don't want to have additional userspace tools. The operation through debugfs is easier and a bit like ftrace. The reason for not using perf is because there is no current support for uncore tracing in the perf facilities. We have our own format of data and don't need perf doing the parsing. The setting through perf tools doesn't seem to be friendly as well. For example, we cannot count on perf to decode the usual format BDF number like <domain>:<bus>:<dev>.<fn>, which user can use to filter the TLP headers through the PTT device. A similar approach for implementing this function is ETM, which use sysfs for configuring and a character device for dumping data. Greg has some comments on our implementation and doesn't advocate to build driver on debugfs [1]. So I resend this series to collect more feedbacks on the implementation of this driver. Hi perf and ETM related experts, is it suggested to adapt this driver to perf? Or is the debugfs approach acceptable? Otherwise use sysfs + character device like ETM and use perf tools for decoding it? Any comments is welcomed. [1] https://lore.kernel.org/linux-pci/1617713154-35533-1-git-send-email-yangyicong@hisilicon.com/ Yicong Yang (4): hwtracing: Add trace function support for HiSilicon PCIe Tune and Trace device hwtracing: Add tune function support for HiSilicon PCIe Tune and Trace device docs: Add HiSilicon PTT device driver documentation MAINTAINERS: Add maintainer for HiSilicon PTT driver Documentation/trace/hisi-ptt.rst | 326 +++++++ MAINTAINERS | 7 + drivers/Makefile | 1 + drivers/hwtracing/Kconfig | 2 + drivers/hwtracing/hisilicon/Kconfig | 11 + drivers/hwtracing/hisilicon/Makefile | 2 + drivers/hwtracing/hisilicon/hisi_ptt.c | 1636 ++++++++++++++++++++++++++++++++ 7 files changed, 1985 insertions(+) create mode 100644 Documentation/trace/hisi-ptt.rst create mode 100644 drivers/hwtracing/hisilicon/Kconfig create mode 100644 drivers/hwtracing/hisilicon/Makefile create mode 100644 drivers/hwtracing/hisilicon/hisi_ptt.c