| Message ID | 20201015124522.1876-1-saiprakash.ranjan@codeaurora.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | coresight: etm4x: Add config to exclude kernel mode tracing | expand |
Hi Sai, On 10/15/2020 01:45 PM, Sai Prakash Ranjan wrote: > On production systems with ETMs enabled, it is preferred to > exclude kernel mode(NS EL1) tracing for security concerns and > support only userspace(NS EL0) tracing. So provide an option > via kconfig to exclude kernel mode tracing if it is required. > This config is disabled by default and would not affect the > current configuration which has both kernel and userspace > tracing enabled by default. While this solution works for ETM4x, I would prefer if we did this in a more generic way. There are other hardware tracing PMUs that provide instruction tracing (e.g, Intel PT, even ETM3x) and it would be good to have a single option that works everywhere. Something like EXCLUDE_KERNEL_HW_ITRACE, which can be honored by all tracing drivers ? > > Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> > --- > drivers/hwtracing/coresight/Kconfig | 9 +++++++++ > drivers/hwtracing/coresight/coresight-etm4x-core.c | 6 +++++- > 2 files changed, 14 insertions(+), 1 deletion(-) > > diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig > index c1198245461d..52435de8824c 100644 > --- a/drivers/hwtracing/coresight/Kconfig > +++ b/drivers/hwtracing/coresight/Kconfig > @@ -110,6 +110,15 @@ config CORESIGHT_SOURCE_ETM4X > To compile this driver as a module, choose M here: the > module will be called coresight-etm4x. > > +config CORESIGHT_ETM4X_EXCL_KERN > + bool "Coresight ETM 4.x exclude kernel mode tracing" > + depends on CORESIGHT_SOURCE_ETM4X > + help > + This will exclude kernel mode(NS EL1) tracing if enabled. This option > + will be useful to provide more flexible options on production systems > + where only userspace(NS EL0) tracing might be preferred for security > + reasons. > + > config CORESIGHT_STM > tristate "CoreSight System Trace Macrocell driver" > depends on (ARM && !(CPU_32v3 || CPU_32v4 || CPU_32v4T)) || ARM64 > diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c > index abd706b216ac..7e5669e5cd1f 100644 > --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c > +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c > @@ -832,6 +832,9 @@ static u64 etm4_get_ns_access_type(struct etmv4_config *config) > { > u64 access_type = 0; > > + if (IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN)) > + config->mode |= ETM_MODE_EXCL_KERN; > + Rather than hacking the mode behind the back, could we always make sure that mode is not set in the first place and return this back to the user with proper errors (see below) ? > /* > * EXLEVEL_NS, bits[15:12] > * The Exception levels are: > @@ -849,7 +852,8 @@ static u64 etm4_get_ns_access_type(struct etmv4_config *config) > access_type = ETM_EXLEVEL_NS_HYP; > } > > - if (config->mode & ETM_MODE_EXCL_USER) > + if (config->mode & ETM_MODE_EXCL_USER && > + !IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN)) > access_type |= ETM_EXLEVEL_NS_APP; Why is this needed ? Also we should return an error if the sysfs mode ever tries to clear the mode bit for kernel in config->mode. Suzuki
On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote: > On production systems with ETMs enabled, it is preferred to > exclude kernel mode(NS EL1) tracing for security concerns and > support only userspace(NS EL0) tracing. So provide an option > via kconfig to exclude kernel mode tracing if it is required. > This config is disabled by default and would not affect the > current configuration which has both kernel and userspace > tracing enabled by default. > One requires root access (or be part of a special trace group) to be able to use the cs_etm PMU. With this kind of elevated access restricting tracing at EL1 provides little in terms of security. Thanks, Mathieu > Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> > --- > drivers/hwtracing/coresight/Kconfig | 9 +++++++++ > drivers/hwtracing/coresight/coresight-etm4x-core.c | 6 +++++- > 2 files changed, 14 insertions(+), 1 deletion(-) > > diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig > index c1198245461d..52435de8824c 100644 > --- a/drivers/hwtracing/coresight/Kconfig > +++ b/drivers/hwtracing/coresight/Kconfig > @@ -110,6 +110,15 @@ config CORESIGHT_SOURCE_ETM4X > To compile this driver as a module, choose M here: the > module will be called coresight-etm4x. > > +config CORESIGHT_ETM4X_EXCL_KERN > + bool "Coresight ETM 4.x exclude kernel mode tracing" > + depends on CORESIGHT_SOURCE_ETM4X > + help > + This will exclude kernel mode(NS EL1) tracing if enabled. This option > + will be useful to provide more flexible options on production systems > + where only userspace(NS EL0) tracing might be preferred for security > + reasons. > + > config CORESIGHT_STM > tristate "CoreSight System Trace Macrocell driver" > depends on (ARM && !(CPU_32v3 || CPU_32v4 || CPU_32v4T)) || ARM64 > diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c > index abd706b216ac..7e5669e5cd1f 100644 > --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c > +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c > @@ -832,6 +832,9 @@ static u64 etm4_get_ns_access_type(struct etmv4_config *config) > { > u64 access_type = 0; > > + if (IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN)) > + config->mode |= ETM_MODE_EXCL_KERN; > + > /* > * EXLEVEL_NS, bits[15:12] > * The Exception levels are: > @@ -849,7 +852,8 @@ static u64 etm4_get_ns_access_type(struct etmv4_config *config) > access_type = ETM_EXLEVEL_NS_HYP; > } > > - if (config->mode & ETM_MODE_EXCL_USER) > + if (config->mode & ETM_MODE_EXCL_USER && > + !IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN)) > access_type |= ETM_EXLEVEL_NS_APP; > > return access_type; > > base-commit: 3477326277451000bc667dfcc4fd0774c039184c > -- > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member > of Code Aurora Forum, hosted by The Linux Foundation >
On Thu, Oct 15, 2020 at 11:40:05PM -0700, Denis Nikitin wrote: > Hi Mathieu, > > I think one of the use cases could be VMs. > Is there isolation between EL1 guest kernels which we can control from perf > in a system wide mode? Sorry for suddenly jumping in. For KVM, I think we need to implement mechanism for saving/restoring CoreSight context for every guest OS, the CPU PMUs has implemented related features [1]. Thanks, Leo [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/pmu.c
Hi Suzuki, On 2020-10-15 19:57, Suzuki K Poulose wrote: > Hi Sai, > > On 10/15/2020 01:45 PM, Sai Prakash Ranjan wrote: >> On production systems with ETMs enabled, it is preferred to >> exclude kernel mode(NS EL1) tracing for security concerns and >> support only userspace(NS EL0) tracing. So provide an option >> via kconfig to exclude kernel mode tracing if it is required. >> This config is disabled by default and would not affect the >> current configuration which has both kernel and userspace >> tracing enabled by default. > > While this solution works for ETM4x, I would prefer if we did > this in a more generic way. There are other hardware tracing > PMUs that provide instruction tracing (e.g, Intel PT, even ETM3x) > and it would be good to have a single option that works everywhere. > > Something like EXCLUDE_KERNEL_HW_ITRACE, which can be honored by > all tracing drivers ? I can add this for ETM3x as well but I have zero idea regarding Intel PTs, is there any code in those hwtracing PMUs actually excluding kernel mode tracing currently? >> >> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> >> --- >> drivers/hwtracing/coresight/Kconfig | 9 +++++++++ >> drivers/hwtracing/coresight/coresight-etm4x-core.c | 6 +++++- >> 2 files changed, 14 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/hwtracing/coresight/Kconfig >> b/drivers/hwtracing/coresight/Kconfig >> index c1198245461d..52435de8824c 100644 >> --- a/drivers/hwtracing/coresight/Kconfig >> +++ b/drivers/hwtracing/coresight/Kconfig >> @@ -110,6 +110,15 @@ config CORESIGHT_SOURCE_ETM4X >> To compile this driver as a module, choose M here: the >> module will be called coresight-etm4x. >> +config CORESIGHT_ETM4X_EXCL_KERN >> + bool "Coresight ETM 4.x exclude kernel mode tracing" >> + depends on CORESIGHT_SOURCE_ETM4X >> + help >> + This will exclude kernel mode(NS EL1) tracing if enabled. This >> option >> + will be useful to provide more flexible options on production >> systems >> + where only userspace(NS EL0) tracing might be preferred for >> security >> + reasons. >> + >> config CORESIGHT_STM >> tristate "CoreSight System Trace Macrocell driver" >> depends on (ARM && !(CPU_32v3 || CPU_32v4 || CPU_32v4T)) || ARM64 >> diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c >> b/drivers/hwtracing/coresight/coresight-etm4x-core.c >> index abd706b216ac..7e5669e5cd1f 100644 >> --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c >> +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c >> @@ -832,6 +832,9 @@ static u64 etm4_get_ns_access_type(struct >> etmv4_config *config) >> { >> u64 access_type = 0; >> + if (IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN)) >> + config->mode |= ETM_MODE_EXCL_KERN; >> + > > Rather than hacking the mode behind the back, could we always make sure > that > mode is not set in the first place and return this back to the user > with > proper errors (see below) ? > Sure, this was the minimal change with which I could keep the check in one place which would work for both sysfs and perf, but I'll change as you suggested and move the check to etm4_parse_event_config() and etm4_config_trace_mode() and return errors properly. >> /* >> * EXLEVEL_NS, bits[15:12] >> * The Exception levels are: >> @@ -849,7 +852,8 @@ static u64 etm4_get_ns_access_type(struct >> etmv4_config *config) >> access_type = ETM_EXLEVEL_NS_HYP; >> } >> - if (config->mode & ETM_MODE_EXCL_USER) >> + if (config->mode & ETM_MODE_EXCL_USER && >> + !IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN)) >> access_type |= ETM_EXLEVEL_NS_APP; > > Why is this needed ? > Yes this will not be required as excluding both doesn't make sense and we print warning in that case already, will drop this. > Also we should return an error if the sysfs mode ever tries to clear > the mode bit > for kernel in config->mode. > Yes will change as explained in above comment. Thanks, Sai
Hi Leo, On 2020-10-16 12:54, Leo Yan wrote: > On Thu, Oct 15, 2020 at 11:40:05PM -0700, Denis Nikitin wrote: >> Hi Mathieu, >> >> I think one of the use cases could be VMs. >> Is there isolation between EL1 guest kernels which we can control from >> perf >> in a system wide mode? > > Sorry for suddenly jumping in. > > For KVM, I think we need to implement mechanism for saving/restoring > CoreSight context for every guest OS, the CPU PMUs has implemented > related features [1]. > > Thanks, > Leo > > [1] > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/pmu.c > What happens to the sysfs mode of tracing? For that we would still need a config right to exclude kernel mode tracing completely. Thanks, Sai
Hi Sai, On Fri, Oct 16, 2020 at 02:10:47PM +0530, Sai Prakash Ranjan wrote: > Hi Leo, > > On 2020-10-16 12:54, Leo Yan wrote: > > On Thu, Oct 15, 2020 at 11:40:05PM -0700, Denis Nikitin wrote: > > > Hi Mathieu, > > > > > > I think one of the use cases could be VMs. > > > Is there isolation between EL1 guest kernels which we can control > > > from perf > > > in a system wide mode? > > > > Sorry for suddenly jumping in. > > > > For KVM, I think we need to implement mechanism for saving/restoring > > CoreSight context for every guest OS, the CPU PMUs has implemented > > related features [1]. > > > > Thanks, > > Leo > > > > [1] > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/pmu.c > > > > What happens to the sysfs mode of tracing? For that we would still > need a config right to exclude kernel mode tracing completely. IIUC, sysfs mode and perf mode both can apply the same approach, the guest OS runs a thread context for the host, so when a guest OS is switched in or out, the hypervisor can save/restore the context for the guest OS; thus every guest OS will have its dedicated context and trace data ideally. Thanks, Leo
Hi Leo, On 2020-10-16 14:54, Leo Yan wrote: > Hi Sai, > > On Fri, Oct 16, 2020 at 02:10:47PM +0530, Sai Prakash Ranjan wrote: >> Hi Leo, >> >> On 2020-10-16 12:54, Leo Yan wrote: >> > On Thu, Oct 15, 2020 at 11:40:05PM -0700, Denis Nikitin wrote: >> > > Hi Mathieu, >> > > >> > > I think one of the use cases could be VMs. >> > > Is there isolation between EL1 guest kernels which we can control >> > > from perf >> > > in a system wide mode? >> > >> > Sorry for suddenly jumping in. >> > >> > For KVM, I think we need to implement mechanism for saving/restoring >> > CoreSight context for every guest OS, the CPU PMUs has implemented >> > related features [1]. >> > >> > Thanks, >> > Leo >> > >> > [1] >> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/pmu.c >> > >> >> What happens to the sysfs mode of tracing? For that we would still >> need a config right to exclude kernel mode tracing completely. > > IIUC, sysfs mode and perf mode both can apply the same approach, the > guest OS runs a thread context for the host, so when a guest OS is > switched in or out, the hypervisor can save/restore the context for > the guest OS; thus every guest OS will have its dedicated context and > trace data ideally. > Thanks for the explanation, so for this usecase then we would have to implement something as you suggested, not sure how hard would that be looking at my KVM knowledge(which at the moment is almost nil) when compared to a kconfig ;) Thanks, Sai
On 10/16/20 7:40 AM, Denis Nikitin wrote: > Hi Mathieu, > > I think one of the use cases could be VMs. > Is there isolation between EL1 guest kernels which we can control from > perf in a system wide mode? The proposed solution doesn't solve this for VMs anyway. It only excludes EL1 *OR* EL2, depending on the host kernel's running EL. We cannot support Virtual ETM access for VMs with memory mapped accesses. Unforutnately, trace filtering is the solution for preventing tracing for EL1 guest/kernel (available from v8.4 Self Hosted extensions). Other option is to add support for "exclude_guest" support for CoreSight for perf. But again this can't be controlled by sysfs. And it can't be enforced for perf, if not specified. Again it all goes back to the root permission hammer lock which Mathieu pointed out. With the v8.4 Self hosted trace extensions, Guest and Host both could control individually if they can be traced (both EL0 and EL1/2). Suzuki > > Thanks, > Denis > > On Thu, Oct 15, 2020 at 9:03 AM Mathieu Poirier > <mathieu.poirier@linaro.org <mailto:mathieu.poirier@linaro.org>> wrote: > > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote: > > On production systems with ETMs enabled, it is preferred to > > exclude kernel mode(NS EL1) tracing for security concerns and > > support only userspace(NS EL0) tracing. So provide an option > > via kconfig to exclude kernel mode tracing if it is required. > > This config is disabled by default and would not affect the > > current configuration which has both kernel and userspace > > tracing enabled by default. > > > > One requires root access (or be part of a special trace group) to be > able to use > the cs_etm PMU. With this kind of elevated access restricting > tracing at EL1 > provides little in terms of security. > > Thanks, > Mathieu > > > Signed-off-by: Sai Prakash Ranjan > <saiprakash.ranjan@codeaurora.org > <mailto:saiprakash.ranjan@codeaurora.org>> > > --- > > drivers/hwtracing/coresight/Kconfig | 9 +++++++++ > > drivers/hwtracing/coresight/coresight-etm4x-core.c | 6 +++++- > > 2 files changed, 14 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/hwtracing/coresight/Kconfig > b/drivers/hwtracing/coresight/Kconfig > > index c1198245461d..52435de8824c 100644 > > --- a/drivers/hwtracing/coresight/Kconfig > > +++ b/drivers/hwtracing/coresight/Kconfig > > @@ -110,6 +110,15 @@ config CORESIGHT_SOURCE_ETM4X > > To compile this driver as a module, choose M here: the > > module will be called coresight-etm4x. > > > > +config CORESIGHT_ETM4X_EXCL_KERN > > + bool "Coresight ETM 4.x exclude kernel mode tracing" > > + depends on CORESIGHT_SOURCE_ETM4X > > + help > > + This will exclude kernel mode(NS EL1) tracing if enabled. > This option > > + will be useful to provide more flexible options on > production systems > > + where only userspace(NS EL0) tracing might be preferred > for security > > + reasons. > > + > > config CORESIGHT_STM > > tristate "CoreSight System Trace Macrocell driver" > > depends on (ARM && !(CPU_32v3 || CPU_32v4 || CPU_32v4T)) || > ARM64 > > diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c > b/drivers/hwtracing/coresight/coresight-etm4x-core.c > > index abd706b216ac..7e5669e5cd1f 100644 > > --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c > > +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c > > @@ -832,6 +832,9 @@ static u64 etm4_get_ns_access_type(struct > etmv4_config *config) > > { > > u64 access_type = 0; > > > > + if (IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN)) > > + config->mode |= ETM_MODE_EXCL_KERN; > > + > > /* > > * EXLEVEL_NS, bits[15:12] > > * The Exception levels are: > > @@ -849,7 +852,8 @@ static u64 etm4_get_ns_access_type(struct > etmv4_config *config) > > access_type = ETM_EXLEVEL_NS_HYP; > > } > > > > - if (config->mode & ETM_MODE_EXCL_USER) > > + if (config->mode & ETM_MODE_EXCL_USER && > > + !IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN)) > > access_type |= ETM_EXLEVEL_NS_APP; > > > > return access_type; > > > > base-commit: 3477326277451000bc667dfcc4fd0774c039184c > > -- > > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is > a member > > of Code Aurora Forum, hosted by The Linux Foundation > > >
On 10/16/20 10:24 AM, Leo Yan wrote: > Hi Sai, > > On Fri, Oct 16, 2020 at 02:10:47PM +0530, Sai Prakash Ranjan wrote: >> Hi Leo, >> >> On 2020-10-16 12:54, Leo Yan wrote: >>> On Thu, Oct 15, 2020 at 11:40:05PM -0700, Denis Nikitin wrote: >>>> Hi Mathieu, >>>> >>>> I think one of the use cases could be VMs. >>>> Is there isolation between EL1 guest kernels which we can control >>>> from perf >>>> in a system wide mode? >>> >>> Sorry for suddenly jumping in. >>> >>> For KVM, I think we need to implement mechanism for saving/restoring >>> CoreSight context for every guest OS, the CPU PMUs has implemented >>> related features [1]. >>> >>> Thanks, >>> Leo >>> >>> [1] >>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/kvm/pmu.c >>> Its not as easy as the CPU PMU for virtualizing the ETMs (with memory mapped access only), i.e supporting ETMs from VMs. We could definitely stop/resume on guest entry/exit, to support attr.exclude_guest. >> >> What happens to the sysfs mode of tracing? For that we would still >> need a config right to exclude kernel mode tracing completely. > > IIUC, sysfs mode and perf mode both can apply the same approach, the > guest OS runs a thread context for the host, so when a guest OS is > switched in or out, the hypervisor can save/restore the context for > the guest OS; thus every guest OS will have its dedicated context and > trace data ideally. I don't think Guest Context is something we can support as mentioned above, at least for systems without sysreg access for ETMs (and virtualizing ETRs is a different story !) Cheers Suzuki
On Fri, Oct 16, 2020 at 12:38:47PM +0100, Suzuki Kuruppassery Poulose wrote: [...] > > > What happens to the sysfs mode of tracing? For that we would still > > > need a config right to exclude kernel mode tracing completely. > > > > IIUC, sysfs mode and perf mode both can apply the same approach, the > > guest OS runs a thread context for the host, so when a guest OS is > > switched in or out, the hypervisor can save/restore the context for > > the guest OS; thus every guest OS will have its dedicated context and > > trace data ideally. > > I don't think Guest Context is something we can support as mentioned > above, at least for systems without sysreg access for ETMs (and virtualizing > ETRs is a different story !) Thanks for sharing thoughts, Suzuki. I missed the device virtulisation. Here should virtualize all devices (includes CoreSight ETM/funnel/ETR/ETF)? Or only need to virtualize ETRs? Obviously, this is a difficult task :) Thanks, Leo
On 10/16/20 2:14 PM, Leo Yan wrote: > On Fri, Oct 16, 2020 at 12:38:47PM +0100, Suzuki Kuruppassery Poulose wrote: > > [...] > >>>> What happens to the sysfs mode of tracing? For that we would still >>>> need a config right to exclude kernel mode tracing completely. >>> >>> IIUC, sysfs mode and perf mode both can apply the same approach, the >>> guest OS runs a thread context for the host, so when a guest OS is >>> switched in or out, the hypervisor can save/restore the context for >>> the guest OS; thus every guest OS will have its dedicated context and >>> trace data ideally. >> >> I don't think Guest Context is something we can support as mentioned >> above, at least for systems without sysreg access for ETMs (and virtualizing >> ETRs is a different story !) > > Thanks for sharing thoughts, Suzuki. > > I missed the device virtulisation. Here should virtualize all devices > (includes CoreSight ETM/funnel/ETR/ETF)? Or only need to virtualize > ETRs? I wouldn't even think of virtualizing the components without sysreg access. So let us not worry about it :-) Cheers Suzuki
On Fri, Jan 15, 2021 at 6:46 AM Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> wrote: > > Hello Mathieu, Suzuki > > On 2020-10-15 21:32, Mathieu Poirier wrote: > > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote: > >> On production systems with ETMs enabled, it is preferred to > >> exclude kernel mode(NS EL1) tracing for security concerns and > >> support only userspace(NS EL0) tracing. So provide an option > >> via kconfig to exclude kernel mode tracing if it is required. > >> This config is disabled by default and would not affect the > >> current configuration which has both kernel and userspace > >> tracing enabled by default. > >> > > > > One requires root access (or be part of a special trace group) to be > > able to use > > the cs_etm PMU. With this kind of elevated access restricting tracing > > at EL1 > > provides little in terms of security. > > > > Apart from the VM usecase discussed, I am told there are other > security concerns here regarding need to exclude kernel mode tracing > even for the privileged users/root. One such case being the ability > to analyze cryptographic code execution since ETMs can record all > branch instructions including timestamps in the kernel and there may > be other cases as well which I may not be aware of and hence have > added Denis and Mattias. Please let us know if you have any questions > further regarding this not being a security concern. Well, the idea that root privileges != full control over the kernel isn't new and at the very least since lockdown became part of mainline [1] no longer an esoteric edge case. Regarding the use case Sai hints at (namely protection of secrets in the kernel), Matthew Garret actually has some more thoughts about confidentiality mode for lockdown for secret protection [2]. And thus, unless someone can make a compelling case that instruction-level tracing will not leak secrets held by the kernel, I think an option for the kernel to prevent itself from being traced (even by root) is valuable. Finally, to sketch a practical use case scenario: Consider a system where disk contents are encrypted and the encryption key is set up by the user when mounting the file system. From that point on the encryption key resides in the kernel. It seems reasonable to expect that the disk encryption key be protected from exfiltration even if the system later suffers a root compromise (or even against insiders that have root access), at least as long as the attacker doesn't manage to compromise the kernel. [1] https://lwn.net/Articles/796866/ [2] https://mjg59.dreamwidth.org/55105.html > > After this discussion, I would like to post a v2 based on Suzuki's > feedback earlier. @Suzuki, I have a common config for ETM3 and ETM4 > but couldn't get much idea on how to implement it for Intel PTs, if > you have any suggestions there, please do share or we can have this > only for Coresight ETMs. > > Thanks, > Sai > > -- > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a > member > of Code Aurora Forum, hosted by The Linux Foundation
On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote: > Hello Mathieu, Suzuki > > On 2020-10-15 21:32, Mathieu Poirier wrote: > > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote: > > > On production systems with ETMs enabled, it is preferred to > > > exclude kernel mode(NS EL1) tracing for security concerns and > > > support only userspace(NS EL0) tracing. So provide an option > > > via kconfig to exclude kernel mode tracing if it is required. > > > This config is disabled by default and would not affect the > > > current configuration which has both kernel and userspace > > > tracing enabled by default. > > > > > > > One requires root access (or be part of a special trace group) to be > > able to use > > the cs_etm PMU. With this kind of elevated access restricting tracing > > at EL1 > > provides little in terms of security. > > > > Apart from the VM usecase discussed, I am told there are other > security concerns here regarding need to exclude kernel mode tracing > even for the privileged users/root. One such case being the ability > to analyze cryptographic code execution since ETMs can record all > branch instructions including timestamps in the kernel and there may > be other cases as well which I may not be aware of and hence have > added Denis and Mattias. Please let us know if you have any questions > further regarding this not being a security concern. Even if we were to apply this patch there are many ways to compromise a system or get the kernel to reveal important information using the perf subsystem. I would perfer to tackle the problem at that level rather than concentrating on coresight. > > After this discussion, I would like to post a v2 based on Suzuki's > feedback earlier. @Suzuki, I have a common config for ETM3 and ETM4 > but couldn't get much idea on how to implement it for Intel PTs, if > you have any suggestions there, please do share or we can have this > only for Coresight ETMs. > > Thanks, > Sai > > -- > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member > of Code Aurora Forum, hosted by The Linux Foundation
Hi Sai, > From: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org > Hi Mathieu, > > On 2021-01-19 01:53, Mathieu Poirier wrote: > > On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote: > >> Hello Mathieu, Suzuki > >> > >> On 2020-10-15 21:32, Mathieu Poirier wrote: > >> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote: > >> > > On production systems with ETMs enabled, it is preferred to > >> > > exclude kernel mode(NS EL1) tracing for security concerns and > >> > > support only userspace(NS EL0) tracing. So provide an option via > >> > > kconfig to exclude kernel mode tracing if it is required. > >> > > This config is disabled by default and would not affect the > >> > > current configuration which has both kernel and userspace tracing > >> > > enabled by default. > >> > > > >> > > >> > One requires root access (or be part of a special trace group) to > >> > be able to use the cs_etm PMU. With this kind of elevated access > >> > restricting tracing at EL1 provides little in terms of security. > >> > > >> > >> Apart from the VM usecase discussed, I am told there are other > >> security concerns here regarding need to exclude kernel mode tracing > >> even for the privileged users/root. One such case being the ability > >> to analyze cryptographic code execution since ETMs can record all > >> branch instructions including timestamps in the kernel and there may > >> be other cases as well which I may not be aware of and hence have > >> added Denis and Mattias. Please let us know if you have any questions > >> further regarding this not being a security concern. > > > > Even if we were to apply this patch there are many ways to compromise > > a system or get the kernel to reveal important information using the > > perf subsystem. I would perfer to tackle the problem at that level > > rather than concentrating on coresight. > > > > Sorry but I did not understand your point. We are talking about the capabilities > of coresight etm tracing which has the instruction level tracing and a lot more. > Perf subsystem is just the framework used for it. > In other words, its not the perf subsystem which does instruction level tracing, > its the coresight etm. Why the perf subsystem should be modified to lockdown > kernel mode? If we were to let perf handle all the trace filtering for different > exception levels, then why do we need the register settings in coresight etm > driver to filter out NS EL* tracing? And more importantly, how do you suppose > we handle sysfs mode of coresight tracing with perf subsystem? You both have good points. Mathieu is right that this is not a CoreSight issue specifically, it is a matter of kernel security policy, and other hardware tracing mechanisms ought to be within its scope. There should be a general "anti kernel exfiltration" config that applies to all mechanisms within its scope, and we'd definitely expect that to include Intel PT as well as ETM. A kernel config that forced exclude_kernel on all perf events would deal with ETM and PT in one place, but miss the sysfs interface to ETM. On the other hand, doing it in the ETM drivers would cover the perf and sysfs interfaces to ETM, but would miss Intel PT. So I think what is needed is a general config option that is both implemented in perf (excluding all kernel tracing events) and by any drivers that provide an alternative interface to hardware tracing events. Al > > Thanks, > Sai > > -- > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member > of Code Aurora Forum, hosted by The Linux Foundation
On 1/19/21 9:51 AM, Sai Prakash Ranjan wrote: > Hi Al, > > On 2021-01-19 14:06, Al Grant wrote: >> Hi Sai, >> >>> From: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org >>> Hi Mathieu, >>> >>> On 2021-01-19 01:53, Mathieu Poirier wrote: >>> > On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote: >>> >> Hello Mathieu, Suzuki >>> >> >>> >> On 2020-10-15 21:32, Mathieu Poirier wrote: >>> >> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote: >>> >> > > On production systems with ETMs enabled, it is preferred to >>> >> > > exclude kernel mode(NS EL1) tracing for security concerns and >>> >> > > support only userspace(NS EL0) tracing. So provide an option via >>> >> > > kconfig to exclude kernel mode tracing if it is required. >>> >> > > This config is disabled by default and would not affect the >>> >> > > current configuration which has both kernel and userspace tracing >>> >> > > enabled by default. >>> >> > > >>> >> > >>> >> > One requires root access (or be part of a special trace group) to >>> >> > be able to use the cs_etm PMU. With this kind of elevated access >>> >> > restricting tracing at EL1 provides little in terms of security. >>> >> > >>> >> >>> >> Apart from the VM usecase discussed, I am told there are other >>> >> security concerns here regarding need to exclude kernel mode tracing >>> >> even for the privileged users/root. One such case being the ability >>> >> to analyze cryptographic code execution since ETMs can record all >>> >> branch instructions including timestamps in the kernel and there may >>> >> be other cases as well which I may not be aware of and hence have >>> >> added Denis and Mattias. Please let us know if you have any questions >>> >> further regarding this not being a security concern. >>> > >>> > Even if we were to apply this patch there are many ways to compromise >>> > a system or get the kernel to reveal important information using the >>> > perf subsystem. I would perfer to tackle the problem at that level >>> > rather than concentrating on coresight. >>> > >>> >>> Sorry but I did not understand your point. We are talking about the capabilities >>> of coresight etm tracing which has the instruction level tracing and a lot more. >>> Perf subsystem is just the framework used for it. >>> In other words, its not the perf subsystem which does instruction level tracing, >>> its the coresight etm. Why the perf subsystem should be modified to lockdown >>> kernel mode? If we were to let perf handle all the trace filtering for different >>> exception levels, then why do we need the register settings in coresight etm >>> driver to filter out NS EL* tracing? And more importantly, how do you suppose >>> we handle sysfs mode of coresight tracing with perf subsystem? >> >> You both have good points. Mathieu is right that this is not a CoreSight >> issue specifically, it is a matter of kernel security policy, and other hardware >> tracing mechanisms ought to be within its scope. There should be a general >> "anti kernel exfiltration" config that applies to all mechanisms within >> its scope, and we'd definitely expect that to include Intel PT as well as ETM. >> > > I agree with this part where there should be a generic config for all > hardware tracing families(atleast for Intel PT and ARM Coresight), > Suzuki suggested that as well. I am under the impression that Mathieu > didn't like adding such a config and wanted perf subsystem to handle > it since initial discussion was around whether root compromise meant > everything is lost already and such a kconfig would not help, but > Mattias already gave some good examples where that is not true. > >> A kernel config that forced exclude_kernel on all perf events would deal with >> ETM and PT in one place, but miss the sysfs interface to ETM. >> >> On the other hand, doing it in the ETM drivers would cover the perf and sysfs >> interfaces to ETM, but would miss Intel PT. >> >> So I think what is needed is a general config option that is both implemented >> in perf (excluding all kernel tracing events) and by any drivers that provide >> an alternative interface to hardware tracing events. >> > > I am good with this approach, once Mathieu confirms, I can add a kernel > wide kconfig as Suzuki suggested earlier and make ETM{3,4}x as the > initial users. Someone more familiar with Intel PTs can then make use > of this kconfig. Instead of adding the support for individual drivers, you could handle this in the generic perf layer. e.g, Fail perf_event create with an attribute which allows kernel tracing ? if (!attr.exclude_kernel) return -EINVAL; Or even exclude the kernel silently always. This could also be limited to PMUs with PERF_PMU_CAP_ITRACE, if you want to limit this to PMUs that instruction level tracing. Suzuki
> From: Suzuki K Poulose <suzuki.poulose@arm.com> > On 1/19/21 9:51 AM, Sai Prakash Ranjan wrote: > > Hi Al, > > > > On 2021-01-19 14:06, Al Grant wrote: > >> Hi Sai, > >> > >>> From: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org > >>> Hi Mathieu, > >>> > >>> On 2021-01-19 01:53, Mathieu Poirier wrote: > >>> > On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote: > >>> >> Hello Mathieu, Suzuki > >>> >> > >>> >> On 2020-10-15 21:32, Mathieu Poirier wrote: > >>> >> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote: > >>> >> > > On production systems with ETMs enabled, it is preferred to > >>> >> > > exclude kernel mode(NS EL1) tracing for security concerns and > >>> >> > > support only userspace(NS EL0) tracing. So provide an option > >>> >> > > via kconfig to exclude kernel mode tracing if it is required. > >>> >> > > This config is disabled by default and would not affect the > >>> >> > > current configuration which has both kernel and userspace > >>> >> > > tracing enabled by default. > >>> >> > > > >>> >> > > >>> >> > One requires root access (or be part of a special trace group) > >>> >> > to be able to use the cs_etm PMU. With this kind of elevated > >>> >> > access restricting tracing at EL1 provides little in terms of security. > >>> >> > > >>> >> > >>> >> Apart from the VM usecase discussed, I am told there are other > >>> >> security concerns here regarding need to exclude kernel mode > >>> >> tracing even for the privileged users/root. One such case being > >>> >> the ability to analyze cryptographic code execution since ETMs > >>> >> can record all branch instructions including timestamps in the > >>> >> kernel and there may be other cases as well which I may not be > >>> >> aware of and hence have added Denis and Mattias. Please let us > >>> >> know if you have any questions further regarding this not being a security > concern. > >>> > > >>> > Even if we were to apply this patch there are many ways to > >>> > compromise a system or get the kernel to reveal important > >>> > information using the perf subsystem. I would perfer to tackle > >>> > the problem at that level rather than concentrating on coresight. > >>> > > >>> > >>> Sorry but I did not understand your point. We are talking about the > >>> capabilities of coresight etm tracing which has the instruction level tracing > and a lot more. > >>> Perf subsystem is just the framework used for it. > >>> In other words, its not the perf subsystem which does instruction > >>> level tracing, its the coresight etm. Why the perf subsystem should > >>> be modified to lockdown kernel mode? If we were to let perf handle > >>> all the trace filtering for different exception levels, then why do > >>> we need the register settings in coresight etm driver to filter out > >>> NS EL* tracing? And more importantly, how do you suppose we handle sysfs > mode of coresight tracing with perf subsystem? > >> > >> You both have good points. Mathieu is right that this is not a > >> CoreSight issue specifically, it is a matter of kernel security > >> policy, and other hardware tracing mechanisms ought to be within its > >> scope. There should be a general "anti kernel exfiltration" config > >> that applies to all mechanisms within its scope, and we'd definitely expect > that to include Intel PT as well as ETM. > >> > > > > I agree with this part where there should be a generic config for all > > hardware tracing families(atleast for Intel PT and ARM Coresight), > > Suzuki suggested that as well. I am under the impression that Mathieu > > didn't like adding such a config and wanted perf subsystem to handle > > it since initial discussion was around whether root compromise meant > > everything is lost already and such a kconfig would not help, but > > Mattias already gave some good examples where that is not true. > > > >> A kernel config that forced exclude_kernel on all perf events would > >> deal with ETM and PT in one place, but miss the sysfs interface to ETM. > >> > >> On the other hand, doing it in the ETM drivers would cover the perf > >> and sysfs interfaces to ETM, but would miss Intel PT. > >> > >> So I think what is needed is a general config option that is both > >> implemented in perf (excluding all kernel tracing events) and by any > >> drivers that provide an alternative interface to hardware tracing events. > >> > > > > I am good with this approach, once Mathieu confirms, I can add a > > kernel wide kconfig as Suzuki suggested earlier and make ETM{3,4}x as > > the initial users. Someone more familiar with Intel PTs can then make > > use of this kconfig. > > Instead of adding the support for individual drivers, you could handle this in the > generic perf layer. e.g, Fail perf_event create with an attribute which allows > kernel tracing ? > > if (!attr.exclude_kernel) > return -EINVAL; > > Or even exclude the kernel silently always. > > This could also be limited to PMUs with PERF_PMU_CAP_ITRACE, if you want to > limit this to PMUs that instruction level tracing. The sysfs interface to ETM also needs to deny access to kernel trace, so it's safest to enforce it in the drivers in addition to any enforcement done in perf. Also, forcing exclude_kernel on all perf events may be too strong. Including the kernel in counted events e.g. cache misses can help understand the effect of system calls on performance, and isn't a big side channel compared to userspace event counts. It doesn't reveal detailed timings in the way trace does. So there's an argument for locking out kernel trace specifically (ETM or PT on the kernel); or even, for locking out timed trace with timestamps and cycle counts, and allowing untimed trace. So, that could be done in perf, with a more specific test on the type of event, before it forced exclude_kernel. Al > > Suzuki
On Tue, Jan 19, 2021 at 08:36:22AM +0000, Al Grant wrote: > Hi Sai, > > > From: saiprakash.ranjan=codeaurora.org@mg.codeaurora.org > > Hi Mathieu, > > > > On 2021-01-19 01:53, Mathieu Poirier wrote: > > > On Fri, Jan 15, 2021 at 11:16:24AM +0530, Sai Prakash Ranjan wrote: > > >> Hello Mathieu, Suzuki > > >> > > >> On 2020-10-15 21:32, Mathieu Poirier wrote: > > >> > On Thu, Oct 15, 2020 at 06:15:22PM +0530, Sai Prakash Ranjan wrote: > > >> > > On production systems with ETMs enabled, it is preferred to > > >> > > exclude kernel mode(NS EL1) tracing for security concerns and > > >> > > support only userspace(NS EL0) tracing. So provide an option via > > >> > > kconfig to exclude kernel mode tracing if it is required. > > >> > > This config is disabled by default and would not affect the > > >> > > current configuration which has both kernel and userspace tracing > > >> > > enabled by default. > > >> > > > > >> > > > >> > One requires root access (or be part of a special trace group) to > > >> > be able to use the cs_etm PMU. With this kind of elevated access > > >> > restricting tracing at EL1 provides little in terms of security. > > >> > > > >> > > >> Apart from the VM usecase discussed, I am told there are other > > >> security concerns here regarding need to exclude kernel mode tracing > > >> even for the privileged users/root. One such case being the ability > > >> to analyze cryptographic code execution since ETMs can record all > > >> branch instructions including timestamps in the kernel and there may > > >> be other cases as well which I may not be aware of and hence have > > >> added Denis and Mattias. Please let us know if you have any questions > > >> further regarding this not being a security concern. > > > > > > Even if we were to apply this patch there are many ways to compromise > > > a system or get the kernel to reveal important information using the > > > perf subsystem. I would perfer to tackle the problem at that level > > > rather than concentrating on coresight. > > > > > > > Sorry but I did not understand your point. We are talking about the capabilities > > of coresight etm tracing which has the instruction level tracing and a lot more. > > Perf subsystem is just the framework used for it. > > In other words, its not the perf subsystem which does instruction level tracing, > > its the coresight etm. Why the perf subsystem should be modified to lockdown > > kernel mode? If we were to let perf handle all the trace filtering for different > > exception levels, then why do we need the register settings in coresight etm > > driver to filter out NS EL* tracing? And more importantly, how do you suppose > > we handle sysfs mode of coresight tracing with perf subsystem? > > You both have good points. Mathieu is right that this is not a CoreSight > issue specifically, it is a matter of kernel security policy, and other hardware > tracing mechanisms ought to be within its scope. There should be a general > "anti kernel exfiltration" config that applies to all mechanisms within > its scope, and we'd definitely expect that to include Intel PT as well as ETM. > > A kernel config that forced exclude_kernel on all perf events would deal with > ETM and PT in one place, but miss the sysfs interface to ETM. > > On the other hand, doing it in the ETM drivers would cover the perf and sysfs > interfaces to ETM, but would miss Intel PT. > > So I think what is needed is a general config option that is both implemented > in perf (excluding all kernel tracing events) and by any drivers that provide > an alternative interface to hardware tracing events. > I also think this is the right solution. Thanks, Mathieu > Al > > > > > > Thanks, > > Sai > > > > -- > > QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member > > of Code Aurora Forum, hosted by The Linux Foundation
diff --git a/drivers/hwtracing/coresight/Kconfig b/drivers/hwtracing/coresight/Kconfig index c1198245461d..52435de8824c 100644 --- a/drivers/hwtracing/coresight/Kconfig +++ b/drivers/hwtracing/coresight/Kconfig @@ -110,6 +110,15 @@ config CORESIGHT_SOURCE_ETM4X To compile this driver as a module, choose M here: the module will be called coresight-etm4x. +config CORESIGHT_ETM4X_EXCL_KERN + bool "Coresight ETM 4.x exclude kernel mode tracing" + depends on CORESIGHT_SOURCE_ETM4X + help + This will exclude kernel mode(NS EL1) tracing if enabled. This option + will be useful to provide more flexible options on production systems + where only userspace(NS EL0) tracing might be preferred for security + reasons. + config CORESIGHT_STM tristate "CoreSight System Trace Macrocell driver" depends on (ARM && !(CPU_32v3 || CPU_32v4 || CPU_32v4T)) || ARM64 diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c index abd706b216ac..7e5669e5cd1f 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c @@ -832,6 +832,9 @@ static u64 etm4_get_ns_access_type(struct etmv4_config *config) { u64 access_type = 0; + if (IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN)) + config->mode |= ETM_MODE_EXCL_KERN; + /* * EXLEVEL_NS, bits[15:12] * The Exception levels are: @@ -849,7 +852,8 @@ static u64 etm4_get_ns_access_type(struct etmv4_config *config) access_type = ETM_EXLEVEL_NS_HYP; } - if (config->mode & ETM_MODE_EXCL_USER) + if (config->mode & ETM_MODE_EXCL_USER && + !IS_ENABLED(CONFIG_CORESIGHT_ETM4X_EXCL_KERN)) access_type |= ETM_EXLEVEL_NS_APP; return access_type;
On production systems with ETMs enabled, it is preferred to exclude kernel mode(NS EL1) tracing for security concerns and support only userspace(NS EL0) tracing. So provide an option via kconfig to exclude kernel mode tracing if it is required. This config is disabled by default and would not affect the current configuration which has both kernel and userspace tracing enabled by default. Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan@codeaurora.org> --- drivers/hwtracing/coresight/Kconfig | 9 +++++++++ drivers/hwtracing/coresight/coresight-etm4x-core.c | 6 +++++- 2 files changed, 14 insertions(+), 1 deletion(-) base-commit: 3477326277451000bc667dfcc4fd0774c039184c