[0/3] arm64: perf: Add support for Perf NMI interrupts

Message

Lecopzer Chen May 16, 2020, 12:48 p.m. UTC

These series implement Perf NMI funxtionality and depends on
Pseudo NMI [1] which has been upstreamed.

In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts.
That can be extended to Perf NMI which is the prerequisite for hard-lockup
detector which had already a standard interface inside Linux.

Thus the first step we need to implement perf NMI interface and make sure
it works fine.

Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2]
did.

[1] https://lkml.org/lkml/2019/1/31/535
[2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiq


Lecopzer Chen (3):
  arm_pmu: Add support for perf NMI interrupts registration
  arm64: perf: Support NMI context for perf event ISR
  arm64: Kconfig: Add support for the Perf NMI

 arch/arm64/Kconfig             | 10 +++++++
 arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------
 drivers/perf/arm_pmu.c         | 51 ++++++++++++++++++++++++++++++----
 include/linux/perf/arm_pmu.h   |  6 ++++
 4 files changed, 88 insertions(+), 15 deletions(-)

Comments

Sumit Garg May 18, 2020, 5:46 a.m. UTC | #1

+ Julien

Hi Lecopzer,

On Sat, 16 May 2020 at 18:20, Lecopzer Chen <[e-mail]> wrote:
>
> These series implement Perf NMI funxtionality and depends on
> Pseudo NMI [1] which has been upstreamed.
>
> In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts.
> That can be extended to Perf NMI which is the prerequisite for hard-lockup
> detector which had already a standard interface inside Linux.
>
> Thus the first step we need to implement perf NMI interface and make sure
> it works fine.
>

This is something that is already implemented via Julien's patch-set
[1]. Its v4 has been floating since July, 2019 and I couldn't find any
major blocking comments but not sure why things haven't progressed
further.

Maybe Julien or Arm maintainers can provide updates on existing
patch-set [1] and how we should proceed further with this interesting
feature.

And regarding hard-lockup detection, I have been able to enable it
based on perf NMI events using Julien's perf patch-set [1]. Have a
look at the patch here [2].

[1] https://patchwork.kernel.org/cover/11047407/
[2] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html

-Sumit

> Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2]
> did.
>
> [1] https://lkml.org/lkml/2019/1/31/535
> [2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiq
>
>
> Lecopzer Chen (3):
>   arm_pmu: Add support for perf NMI interrupts registration
>   arm64: perf: Support NMI context for perf event ISR
>   arm64: Kconfig: Add support for the Perf NMI
>
>  arch/arm64/Kconfig             | 10 +++++++
>  arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------
>  drivers/perf/arm_pmu.c         | 51 ++++++++++++++++++++++++++++++----
>  include/linux/perf/arm_pmu.h   |  6 ++++
>  4 files changed, 88 insertions(+), 15 deletions(-)
>
> --
> 2.25.1
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> [e-mail]
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Lecopzer Chen May 18, 2020, 6:26 a.m. UTC | #2

HI Sumit,

Thanks for your information.

I've already implemented IPI (same as you did [1], little difference
in detail), hardlockup detector and perf in last year(2019) for
debuggability.
And now we tend to upstream to reduce kernel maintaining effort.
I'm glad if someone in ARM can do this work :)

Hi Julien,

Does any Arm maintainers can proceed this action?
This is really useful in debugging.
Thank you!!



[1] https://lkml.org/lkml/2020/4/24/328


Lecopzer

Sumit Garg <[e-mail]> 於 2020年5月18日 週一 下午1:46寫道：
>
> + Julien
>
> Hi Lecopzer,
>
> On Sat, 16 May 2020 at 18:20, Lecopzer Chen <[e-mail]> wrote:
> >
> > These series implement Perf NMI funxtionality and depends on
> > Pseudo NMI [1] which has been upstreamed.
> >
> > In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts.
> > That can be extended to Perf NMI which is the prerequisite for hard-lockup
> > detector which had already a standard interface inside Linux.
> >
> > Thus the first step we need to implement perf NMI interface and make sure
> > it works fine.
> >
>
> This is something that is already implemented via Julien's patch-set
> [1]. Its v4 has been floating since July, 2019 and I couldn't find any
> major blocking comments but not sure why things haven't progressed
> further.
>
> Maybe Julien or Arm maintainers can provide updates on existing
> patch-set [1] and how we should proceed further with this interesting
> feature.
>
> And regarding hard-lockup detection, I have been able to enable it
> based on perf NMI events using Julien's perf patch-set [1]. Have a
> look at the patch here [2].
>
> [1] https://patchwork.kernel.org/cover/11047407/
> [2] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html
>
> -Sumit
>
> > Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2]
> > did.
> >
> > [1] https://lkml.org/lkml/2019/1/31/535
> > [2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiq
> >
> >
> > Lecopzer Chen (3):
> >   arm_pmu: Add support for perf NMI interrupts registration
> >   arm64: perf: Support NMI context for perf event ISR
> >   arm64: Kconfig: Add support for the Perf NMI
> >
> >  arch/arm64/Kconfig             | 10 +++++++
> >  arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------
> >  drivers/perf/arm_pmu.c         | 51 ++++++++++++++++++++++++++++++----
> >  include/linux/perf/arm_pmu.h   |  6 ++++
> >  4 files changed, 88 insertions(+), 15 deletions(-)
> >
> > --
> > 2.25.1
> >
> >
> > _______________________________________________
> > linux-arm-kernel mailing list
> > [e-mail]
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Mark Rutland May 18, 2020, 10:45 a.m. UTC | #3

Hi all,

On Mon, May 18, 2020 at 02:26:00PM +0800, Lecopzer Chen wrote:
> HI Sumit,
> 
> Thanks for your information.
> 
> I've already implemented IPI (same as you did [1], little difference
> in detail), hardlockup detector and perf in last year(2019) for
> debuggability.
> And now we tend to upstream to reduce kernel maintaining effort.
> I'm glad if someone in ARM can do this work :)
> 
> Hi Julien,
> 
> Does any Arm maintainers can proceed this action?

Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which
is my preferred approach.

I understand that's not quite ready for posting since he's investigating
some of the nastier subtleties (e.g. mutual exclusion with the NMI), but
maybe we can put the work-in-progress patches somewhere in the mean
time.

Alexandru, do you have an idea of what needs to be done, and/or when you
expect you could post that?

Thanks,
Mark.

> This is really useful in debugging.
> Thank you!!
> 
> 
> 
> [1] https://lkml.org/lkml/2020/4/24/328
> 
> 
> Lecopzer
> 
> Sumit Garg <[e-mail]> 於 2020年5月18日 週一 下午1:46寫道：
> >
> > + Julien
> >
> > Hi Lecopzer,
> >
> > On Sat, 16 May 2020 at 18:20, Lecopzer Chen <[e-mail]> wrote:
> > >
> > > These series implement Perf NMI funxtionality and depends on
> > > Pseudo NMI [1] which has been upstreamed.
> > >
> > > In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts.
> > > That can be extended to Perf NMI which is the prerequisite for hard-lockup
> > > detector which had already a standard interface inside Linux.
> > >
> > > Thus the first step we need to implement perf NMI interface and make sure
> > > it works fine.
> > >
> >
> > This is something that is already implemented via Julien's patch-set
> > [1]. Its v4 has been floating since July, 2019 and I couldn't find any
> > major blocking comments but not sure why things haven't progressed
> > further.
> >
> > Maybe Julien or Arm maintainers can provide updates on existing
> > patch-set [1] and how we should proceed further with this interesting
> > feature.
> >
> > And regarding hard-lockup detection, I have been able to enable it
> > based on perf NMI events using Julien's perf patch-set [1]. Have a
> > look at the patch here [2].
> >
> > [1] https://patchwork.kernel.org/cover/11047407/
> > [2] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html
> >
> > -Sumit
> >
> > > Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2]
> > > did.
> > >
> > > [1] https://lkml.org/lkml/2019/1/31/535
> > > [2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiq
> > >
> > >
> > > Lecopzer Chen (3):
> > >   arm_pmu: Add support for perf NMI interrupts registration
> > >   arm64: perf: Support NMI context for perf event ISR
> > >   arm64: Kconfig: Add support for the Perf NMI
> > >
> > >  arch/arm64/Kconfig             | 10 +++++++
> > >  arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------
> > >  drivers/perf/arm_pmu.c         | 51 ++++++++++++++++++++++++++++++----
> > >  include/linux/perf/arm_pmu.h   |  6 ++++
> > >  4 files changed, 88 insertions(+), 15 deletions(-)
> > >
> > > --
> > > 2.25.1
> > >
> > >
> > > _______________________________________________
> > > linux-arm-kernel mailing list
> > > [e-mail]
> > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Alexandru Elisei May 18, 2020, 11:17 a.m. UTC | #4

Hi,

On 5/18/20 11:45 AM, Mark Rutland wrote:
> Hi all,
>
> On Mon, May 18, 2020 at 02:26:00PM +0800, Lecopzer Chen wrote:
>> HI Sumit,
>>
>> Thanks for your information.
>>
>> I've already implemented IPI (same as you did [1], little difference
>> in detail), hardlockup detector and perf in last year(2019) for
>> debuggability.
>> And now we tend to upstream to reduce kernel maintaining effort.
>> I'm glad if someone in ARM can do this work :)
>>
>> Hi Julien,
>>
>> Does any Arm maintainers can proceed this action?
> Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which
> is my preferred approach.
>
> I understand that's not quite ready for posting since he's investigating
> some of the nastier subtleties (e.g. mutual exclusion with the NMI), but
> maybe we can put the work-in-progress patches somewhere in the mean
> time.
>
> Alexandru, do you have an idea of what needs to be done, and/or when you
> expect you could post that?

I'm currently working on rebasing the patches on top of 5.7-rc5, when I have
something usable I'll post a link (should be a couple of days). After that I will
address the review comments, and I plan to do a thorough testing because I'm not
100% confident that some of the assumptions around the locks that were removed are
correct. My guess is this will take a few weeks.

Thanks,
Alex
>
> Thanks,
> Mark.
>
>> This is really useful in debugging.
>> Thank you!!
>>
>>
>>
>> [1] https://lkml.org/lkml/2020/4/24/328
>>
>>
>> Lecopzer
>>
>> Sumit Garg <[e-mail]> 於 2020年5月18日 週一 下午1:46寫道：
>>> + Julien
>>>
>>> Hi Lecopzer,
>>>
>>> On Sat, 16 May 2020 at 18:20, Lecopzer Chen <[e-mail]> wrote:
>>>> These series implement Perf NMI funxtionality and depends on
>>>> Pseudo NMI [1] which has been upstreamed.
>>>>
>>>> In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts.
>>>> That can be extended to Perf NMI which is the prerequisite for hard-lockup
>>>> detector which had already a standard interface inside Linux.
>>>>
>>>> Thus the first step we need to implement perf NMI interface and make sure
>>>> it works fine.
>>>>
>>> This is something that is already implemented via Julien's patch-set
>>> [1]. Its v4 has been floating since July, 2019 and I couldn't find any
>>> major blocking comments but not sure why things haven't progressed
>>> further.
>>>
>>> Maybe Julien or Arm maintainers can provide updates on existing
>>> patch-set [1] and how we should proceed further with this interesting
>>> feature.
>>>
>>> And regarding hard-lockup detection, I have been able to enable it
>>> based on perf NMI events using Julien's perf patch-set [1]. Have a
>>> look at the patch here [2].
>>>
>>> [1] https://patchwork.kernel.org/cover/11047407/
>>> [2] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html
>>>
>>> -Sumit
>>>
>>>> Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2]
>>>> did.
>>>>
>>>> [1] https://lkml.org/lkml/2019/1/31/535
>>>> [2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiq
>>>>
>>>>
>>>> Lecopzer Chen (3):
>>>>   arm_pmu: Add support for perf NMI interrupts registration
>>>>   arm64: perf: Support NMI context for perf event ISR
>>>>   arm64: Kconfig: Add support for the Perf NMI
>>>>
>>>>  arch/arm64/Kconfig             | 10 +++++++
>>>>  arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------
>>>>  drivers/perf/arm_pmu.c         | 51 ++++++++++++++++++++++++++++++----
>>>>  include/linux/perf/arm_pmu.h   |  6 ++++
>>>>  4 files changed, 88 insertions(+), 15 deletions(-)
>>>>
>>>> --
>>>> 2.25.1
>>>>
>>>>
>>>> _______________________________________________
>>>> linux-arm-kernel mailing list
>>>> [e-mail]
>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Sumit Garg May 18, 2020, 2:09 p.m. UTC | #5

On Mon, 18 May 2020 at 16:47, Alexandru Elisei <[e-mail]> wrote:
>
> Hi,
>
> On 5/18/20 11:45 AM, Mark Rutland wrote:
> > Hi all,
> >
> > On Mon, May 18, 2020 at 02:26:00PM +0800, Lecopzer Chen wrote:
> >> HI Sumit,
> >>
> >> Thanks for your information.
> >>
> >> I've already implemented IPI (same as you did [1], little difference
> >> in detail), hardlockup detector and perf in last year(2019) for
> >> debuggability.
> >> And now we tend to upstream to reduce kernel maintaining effort.
> >> I'm glad if someone in ARM can do this work :)
> >>
> >> Hi Julien,
> >>
> >> Does any Arm maintainers can proceed this action?
> > Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which
> > is my preferred approach.
> >
> > I understand that's not quite ready for posting since he's investigating
> > some of the nastier subtleties (e.g. mutual exclusion with the NMI), but
> > maybe we can put the work-in-progress patches somewhere in the mean
> > time.
> >
> > Alexandru, do you have an idea of what needs to be done, and/or when you
> > expect you could post that?
>
> I'm currently working on rebasing the patches on top of 5.7-rc5, when I have
> something usable I'll post a link (should be a couple of days). After that I will
> address the review comments, and I plan to do a thorough testing because I'm not
> 100% confident that some of the assumptions around the locks that were removed are
> correct. My guess is this will take a few weeks.
>

Thanks Mark, Alex for the status updates on perf NMI feature.

Alex,

As the hard-lockup detection patch [1] has a dependency on perf NMI
patch-set, I will rebase and test hard-lockup detector when you have
got a working tree. But due to the dependency, I think patch [1]
should be accepted along with perf NMI patch-set. So would you be open
to include this patch as part of your series?

[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html

-Sumit

> Thanks,
> Alex
> >
> > Thanks,
> > Mark.
> >
> >> This is really useful in debugging.
> >> Thank you!!
> >>
> >>
> >>
> >> [1] https://lkml.org/lkml/2020/4/24/328
> >>
> >>
> >> Lecopzer
> >>
> >> Sumit Garg <[e-mail]> 於 2020年5月18日 週一 下午1:46寫道：
> >>> + Julien
> >>>
> >>> Hi Lecopzer,
> >>>
> >>> On Sat, 16 May 2020 at 18:20, Lecopzer Chen <[e-mail]> wrote:
> >>>> These series implement Perf NMI funxtionality and depends on
> >>>> Pseudo NMI [1] which has been upstreamed.
> >>>>
> >>>> In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts.
> >>>> That can be extended to Perf NMI which is the prerequisite for hard-lockup
> >>>> detector which had already a standard interface inside Linux.
> >>>>
> >>>> Thus the first step we need to implement perf NMI interface and make sure
> >>>> it works fine.
> >>>>
> >>> This is something that is already implemented via Julien's patch-set
> >>> [1]. Its v4 has been floating since July, 2019 and I couldn't find any
> >>> major blocking comments but not sure why things haven't progressed
> >>> further.
> >>>
> >>> Maybe Julien or Arm maintainers can provide updates on existing
> >>> patch-set [1] and how we should proceed further with this interesting
> >>> feature.
> >>>
> >>> And regarding hard-lockup detection, I have been able to enable it
> >>> based on perf NMI events using Julien's perf patch-set [1]. Have a
> >>> look at the patch here [2].
> >>>
> >>> [1] https://patchwork.kernel.org/cover/11047407/
> >>> [2] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html
> >>>
> >>> -Sumit
> >>>
> >>>> Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2]
> >>>> did.
> >>>>
> >>>> [1] https://lkml.org/lkml/2019/1/31/535
> >>>> [2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiq
> >>>>
> >>>>
> >>>> Lecopzer Chen (3):
> >>>>   arm_pmu: Add support for perf NMI interrupts registration
> >>>>   arm64: perf: Support NMI context for perf event ISR
> >>>>   arm64: Kconfig: Add support for the Perf NMI
> >>>>
> >>>>  arch/arm64/Kconfig             | 10 +++++++
> >>>>  arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------
> >>>>  drivers/perf/arm_pmu.c         | 51 ++++++++++++++++++++++++++++++----
> >>>>  include/linux/perf/arm_pmu.h   |  6 ++++
> >>>>  4 files changed, 88 insertions(+), 15 deletions(-)
> >>>>
> >>>> --
> >>>> 2.25.1
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> linux-arm-kernel mailing list
> >>>> [e-mail]
> >>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Mark Rutland May 18, 2020, 2:19 p.m. UTC | #6

On Mon, May 18, 2020 at 07:39:23PM +0530, Sumit Garg wrote:
> On Mon, 18 May 2020 at 16:47, Alexandru Elisei <[e-mail]> wrote:
> > On 5/18/20 11:45 AM, Mark Rutland wrote:
> > > On Mon, May 18, 2020 at 02:26:00PM +0800, Lecopzer Chen wrote:
> > >> HI Sumit,
> > >>
> > >> Thanks for your information.
> > >>
> > >> I've already implemented IPI (same as you did [1], little difference
> > >> in detail), hardlockup detector and perf in last year(2019) for
> > >> debuggability.
> > >> And now we tend to upstream to reduce kernel maintaining effort.
> > >> I'm glad if someone in ARM can do this work :)
> > >>
> > >> Hi Julien,
> > >>
> > >> Does any Arm maintainers can proceed this action?
> > > Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which
> > > is my preferred approach.
> > >
> > > I understand that's not quite ready for posting since he's investigating
> > > some of the nastier subtleties (e.g. mutual exclusion with the NMI), but
> > > maybe we can put the work-in-progress patches somewhere in the mean
> > > time.
> > >
> > > Alexandru, do you have an idea of what needs to be done, and/or when you
> > > expect you could post that?
> >
> > I'm currently working on rebasing the patches on top of 5.7-rc5, when I have
> > something usable I'll post a link (should be a couple of days). After that I will
> > address the review comments, and I plan to do a thorough testing because I'm not
> > 100% confident that some of the assumptions around the locks that were removed are
> > correct. My guess is this will take a few weeks.
> >
> 
> Thanks Mark, Alex for the status updates on perf NMI feature.
> 
> Alex,
> 
> As the hard-lockup detection patch [1] has a dependency on perf NMI
> patch-set, I will rebase and test hard-lockup detector when you have
> got a working tree. But due to the dependency, I think patch [1]
> should be accepted along with perf NMI patch-set. So would you be open
> to include this patch as part of your series?
> 
> [1] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html

While it depends on the perf NMI bits, I don't think it makes sense to
tie that into the series given it's trying to achieve something very
different.

I think that should be reposted separately once the perf NMI bits are in
shape.

Thanks,
Mark.

Sumit Garg May 19, 2020, 6:48 a.m. UTC | #7

On Mon, 18 May 2020 at 19:49, Mark Rutland <[e-mail]> wrote:
>
> On Mon, May 18, 2020 at 07:39:23PM +0530, Sumit Garg wrote:
> > On Mon, 18 May 2020 at 16:47, Alexandru Elisei <[e-mail]> wrote:
> > > On 5/18/20 11:45 AM, Mark Rutland wrote:
> > > > On Mon, May 18, 2020 at 02:26:00PM +0800, Lecopzer Chen wrote:
> > > >> HI Sumit,
> > > >>
> > > >> Thanks for your information.
> > > >>
> > > >> I've already implemented IPI (same as you did [1], little difference
> > > >> in detail), hardlockup detector and perf in last year(2019) for
> > > >> debuggability.
> > > >> And now we tend to upstream to reduce kernel maintaining effort.
> > > >> I'm glad if someone in ARM can do this work :)
> > > >>
> > > >> Hi Julien,
> > > >>
> > > >> Does any Arm maintainers can proceed this action?
> > > > Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which
> > > > is my preferred approach.
> > > >
> > > > I understand that's not quite ready for posting since he's investigating
> > > > some of the nastier subtleties (e.g. mutual exclusion with the NMI), but
> > > > maybe we can put the work-in-progress patches somewhere in the mean
> > > > time.
> > > >
> > > > Alexandru, do you have an idea of what needs to be done, and/or when you
> > > > expect you could post that?
> > >
> > > I'm currently working on rebasing the patches on top of 5.7-rc5, when I have
> > > something usable I'll post a link (should be a couple of days). After that I will
> > > address the review comments, and I plan to do a thorough testing because I'm not
> > > 100% confident that some of the assumptions around the locks that were removed are
> > > correct. My guess is this will take a few weeks.
> > >
> >
> > Thanks Mark, Alex for the status updates on perf NMI feature.
> >
> > Alex,
> >
> > As the hard-lockup detection patch [1] has a dependency on perf NMI
> > patch-set, I will rebase and test hard-lockup detector when you have
> > got a working tree. But due to the dependency, I think patch [1]
> > should be accepted along with perf NMI patch-set. So would you be open
> > to include this patch as part of your series?
> >
> > [1] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html
>
> While it depends on the perf NMI bits, I don't think it makes sense to
> tie that into the series given it's trying to achieve something very
> different.
>
> I think that should be reposted separately once the perf NMI bits are in
> shape.

Okay, fair enough. Will keep it as a separate patch then.

-Sumit

>
> Thanks,
> Mark.

Song Bao Hua (Barry Song) May 20, 2020, 6:55 a.m. UTC | #8

> 
> On 5/18/20 11:45 AM, Mark Rutland wrote:
> > Hi all,
> >
> > On Mon, May 18, 2020 at 02:26:00PM +0800, Lecopzer Chen wrote:
> >> HI Sumit,
> >>
> >> Thanks for your information.
> >>
> >> I've already implemented IPI (same as you did [1], little difference
> >> in detail), hardlockup detector and perf in last year(2019) for
> >> debuggability.
> >> And now we tend to upstream to reduce kernel maintaining effort.
> >> I'm glad if someone in ARM can do this work :)
> >>
> >> Hi Julien,
> >>
> >> Does any Arm maintainers can proceed this action?
> > Alexandru (Cc'd) has been rebasing and reworking Julien's patches,
> > which is my preferred approach.
> >
> > I understand that's not quite ready for posting since he's
> > investigating some of the nastier subtleties (e.g. mutual exclusion
> > with the NMI), but maybe we can put the work-in-progress patches
> > somewhere in the mean time.
> >
> > Alexandru, do you have an idea of what needs to be done, and/or when
> > you expect you could post that?
> 
> I'm currently working on rebasing the patches on top of 5.7-rc5, when I have
> something usable I'll post a link (should be a couple of days). After that I will
> address the review comments, and I plan to do a thorough testing because I'm
> not 100% confident that some of the assumptions around the locks that were
> removed are correct. My guess is this will take a few weeks.

+1
I would be awesome if perf NMI patches could be re-activated. Right now, it seems it is hard to
do "perf annotate" on a kernel function with local_irq disabled.

func()
{
local_irq_save();
.....
....
local_irq_restore();
return;
}

Perf will report all cycles are used by the last moment of the func().

Thanks,
Barry

> 
> Thanks,
> Alex
> >
> > Thanks,
> > Mark.
> >
> >> This is really useful in debugging.
> >> Thank you!!
> >>
> >>
> >>
> >> [1] https://lkml.org/lkml/2020/4/24/328
> >>
> >>
> >> Lecopzer
> >>
> >> Sumit Garg <[e-mail]> 於 2020年5月18日 週一 下午
> 1:46寫道：
> >>> + Julien
> >>>
> >>> Hi Lecopzer,
> >>>
> >>> On Sat, 16 May 2020 at 18:20, Lecopzer Chen <[e-mail]>
> wrote:
> >>>> These series implement Perf NMI funxtionality and depends on Pseudo
> >>>> NMI [1] which has been upstreamed.
> >>>>
> >>>> In arm64 with GICv3, Pseudo NMI was implemented for NMI-like
> interruts.
> >>>> That can be extended to Perf NMI which is the prerequisite for
> >>>> hard-lockup detector which had already a standard interface inside Linux.
> >>>>
> >>>> Thus the first step we need to implement perf NMI interface and
> >>>> make sure it works fine.
> >>>>
> >>> This is something that is already implemented via Julien's patch-set
> >>> [1]. Its v4 has been floating since July, 2019 and I couldn't find
> >>> any major blocking comments but not sure why things haven't
> >>> progressed further.
> >>>
> >>> Maybe Julien or Arm maintainers can provide updates on existing
> >>> patch-set [1] and how we should proceed further with this
> >>> interesting feature.
> >>>
> >>> And regarding hard-lockup detection, I have been able to enable it
> >>> based on perf NMI events using Julien's perf patch-set [1]. Have a
> >>> look at the patch here [2].
> >>>
> >>> [1] https://patchwork.kernel.org/cover/11047407/
> >>> [2]
> >>> http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/73222
> >>> 7.html
> >>>
> >>> -Sumit
> >>>
> >>>> Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the
> >>>> link [2] did.
> >>>>
> >>>> [1] https://lkml.org/lkml/2019/1/31/535
> >>>> [2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiq
> >>>>
> >>>>
> >>>> Lecopzer Chen (3):
> >>>>   arm_pmu: Add support for perf NMI interrupts registration
> >>>>   arm64: perf: Support NMI context for perf event ISR
> >>>>   arm64: Kconfig: Add support for the Perf NMI
> >>>>
> >>>>  arch/arm64/Kconfig             | 10 +++++++
> >>>>  arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------
> >>>>  drivers/perf/arm_pmu.c         | 51
> ++++++++++++++++++++++++++++++----
> >>>>  include/linux/perf/arm_pmu.h   |  6 ++++
> >>>>  4 files changed, 88 insertions(+), 15 deletions(-)
> >>>>
> >>>> --
> >>>> 2.25.1

Alexandru Elisei May 20, 2020, 10:30 a.m. UTC | #9

Hi,

On 5/18/20 12:17 PM, Alexandru Elisei wrote:
> Hi,
>
> On 5/18/20 11:45 AM, Mark Rutland wrote:
>> Hi all,
>>
>> On Mon, May 18, 2020 at 02:26:00PM +0800, Lecopzer Chen wrote:
>>> HI Sumit,
>>>
>>> Thanks for your information.
>>>
>>> I've already implemented IPI (same as you did [1], little difference
>>> in detail), hardlockup detector and perf in last year(2019) for
>>> debuggability.
>>> And now we tend to upstream to reduce kernel maintaining effort.
>>> I'm glad if someone in ARM can do this work :)
>>>
>>> Hi Julien,
>>>
>>> Does any Arm maintainers can proceed this action?
>> Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which
>> is my preferred approach.
>>
>> I understand that's not quite ready for posting since he's investigating
>> some of the nastier subtleties (e.g. mutual exclusion with the NMI), but
>> maybe we can put the work-in-progress patches somewhere in the mean
>> time.
>>
>> Alexandru, do you have an idea of what needs to be done, and/or when you
>> expect you could post that?
> I'm currently working on rebasing the patches on top of 5.7-rc5, when I have
> something usable I'll post a link (should be a couple of days). After that I will
> address the review comments, and I plan to do a thorough testing because I'm not
> 100% confident that some of the assumptions around the locks that were removed are
> correct. My guess is this will take a few weeks.

Pushed a WIP branch on linux-arm.org [1]:

git clone -b WIP-pmu-nmi git://linux-arm.org/linux-ae

Practically untested, I only did perf record on a defconfig kernel running on the
model.

[1] http://www.linux-arm.org/git?p=linux-ae.git;a=shortlog;h=refs/heads/WIP-pmu-nmi

Thanks,
Alex
>
> Thanks,
> Alex
>> Thanks,
>> Mark.
>>
>>> This is really useful in debugging.
>>> Thank you!!
>>>
>>>
>>>
>>> [1] https://lkml.org/lkml/2020/4/24/328
>>>
>>>
>>> Lecopzer
>>>
>>> Sumit Garg <[e-mail]> 於 2020年5月18日 週一 下午1:46寫道：
>>>> + Julien
>>>>
>>>> Hi Lecopzer,
>>>>
>>>> On Sat, 16 May 2020 at 18:20, Lecopzer Chen <[e-mail]> wrote:
>>>>> These series implement Perf NMI funxtionality and depends on
>>>>> Pseudo NMI [1] which has been upstreamed.
>>>>>
>>>>> In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts.
>>>>> That can be extended to Perf NMI which is the prerequisite for hard-lockup
>>>>> detector which had already a standard interface inside Linux.
>>>>>
>>>>> Thus the first step we need to implement perf NMI interface and make sure
>>>>> it works fine.
>>>>>
>>>> This is something that is already implemented via Julien's patch-set
>>>> [1]. Its v4 has been floating since July, 2019 and I couldn't find any
>>>> major blocking comments but not sure why things haven't progressed
>>>> further.
>>>>
>>>> Maybe Julien or Arm maintainers can provide updates on existing
>>>> patch-set [1] and how we should proceed further with this interesting
>>>> feature.
>>>>
>>>> And regarding hard-lockup detection, I have been able to enable it
>>>> based on perf NMI events using Julien's perf patch-set [1]. Have a
>>>> look at the patch here [2].
>>>>
>>>> [1] https://patchwork.kernel.org/cover/11047407/
>>>> [2] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html
>>>>
>>>> -Sumit
>>>>
>>>>> Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2]
>>>>> did.
>>>>>
>>>>> [1] https://lkml.org/lkml/2019/1/31/535
>>>>> [2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiq
>>>>>
>>>>>
>>>>> Lecopzer Chen (3):
>>>>>   arm_pmu: Add support for perf NMI interrupts registration
>>>>>   arm64: perf: Support NMI context for perf event ISR
>>>>>   arm64: Kconfig: Add support for the Perf NMI
>>>>>
>>>>>  arch/arm64/Kconfig             | 10 +++++++
>>>>>  arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------
>>>>>  drivers/perf/arm_pmu.c         | 51 ++++++++++++++++++++++++++++++----
>>>>>  include/linux/perf/arm_pmu.h   |  6 ++++
>>>>>  4 files changed, 88 insertions(+), 15 deletions(-)
>>>>>
>>>>> --
>>>>> 2.25.1
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> linux-arm-kernel mailing list
>>>>> [e-mail]
>>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> _______________________________________________
> linux-arm-kernel mailing list
> [e-mail]
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Song Bao Hua (Barry Song) May 21, 2020, 3 a.m. UTC | #10

> -----Original Message-----
> From: linux-arm-kernel [mailto:[e-mail]]
> On Behalf Of Alexandru Elisei
> Sent: Wednesday, May 20, 2020 10:31 PM> 
> Hi,
> 
> On 5/18/20 12:17 PM, Alexandru Elisei wrote:
> > Hi,
> >
> > On 5/18/20 11:45 AM, Mark Rutland wrote:
> >> Hi all,
> >>
> >> On Mon, May 18, 2020 at 02:26:00PM +0800, Lecopzer Chen wrote:
> >>> HI Sumit,
> >>>
> >>> Thanks for your information.
> >>>
> >>> I've already implemented IPI (same as you did [1], little difference
> >>> in detail), hardlockup detector and perf in last year(2019) for
> >>> debuggability.
> >>> And now we tend to upstream to reduce kernel maintaining effort.
> >>> I'm glad if someone in ARM can do this work :)
> >>>
> >>> Hi Julien,
> >>>
> >>> Does any Arm maintainers can proceed this action?
> >> Alexandru (Cc'd) has been rebasing and reworking Julien's patches,
> >> which is my preferred approach.
> >>
> >> I understand that's not quite ready for posting since he's
> >> investigating some of the nastier subtleties (e.g. mutual exclusion
> >> with the NMI), but maybe we can put the work-in-progress patches
> >> somewhere in the mean time.
> >>
> >> Alexandru, do you have an idea of what needs to be done, and/or when
> >> you expect you could post that?
> > I'm currently working on rebasing the patches on top of 5.7-rc5, when
> > I have something usable I'll post a link (should be a couple of days).
> > After that I will address the review comments, and I plan to do a
> > thorough testing because I'm not 100% confident that some of the
> > assumptions around the locks that were removed are correct. My guess is
> this will take a few weeks.
> 
> Pushed a WIP branch on linux-arm.org [1]:
> 
> git clone -b WIP-pmu-nmi git://linux-arm.org/linux-ae
> 
> Practically untested, I only did perf record on a defconfig kernel running on the
> model.
> 
> [1]
> http://www.linux-arm.org/git?p=linux-ae.git;a=shortlog;h=refs/heads/WIP-pm
> u-nmi

Fortunately, it does work. I used this tree to perf annotate arm_smmu_cmdq_issue_cmdlist() which
is completely disabling IRQ. Luckily, it reports correct data. Before that, it reported all time was spent by
the code which enabled IRQ .


Barry

> 
> Thanks,
> Alex
> >
> > Thanks,
> > Alex
> >> Thanks,
> >> Mark.
> >>
> >>> This is really useful in debugging.
> >>> Thank you!!
> >>>
> >>>
> >>>
> >>> [1] https://lkml.org/lkml/2020/4/24/328
> >>>
> >>>
> >>> Lecopzer
> >>>
> >>> Sumit Garg <[e-mail]> 於 2020年5月18日 週一 下午
> 1:46寫道：
> >>>> + Julien
> >>>>
> >>>> Hi Lecopzer,
> >>>>
> >>>> On Sat, 16 May 2020 at 18:20, Lecopzer Chen <[e-mail]>
> wrote:
> >>>>> These series implement Perf NMI funxtionality and depends on
> >>>>> Pseudo NMI [1] which has been upstreamed.
> >>>>>
> >>>>> In arm64 with GICv3, Pseudo NMI was implemented for NMI-like
> interruts.
> >>>>> That can be extended to Perf NMI which is the prerequisite for
> >>>>> hard-lockup detector which had already a standard interface inside
> Linux.
> >>>>>
> >>>>> Thus the first step we need to implement perf NMI interface and
> >>>>> make sure it works fine.
> >>>>>
> >>>> This is something that is already implemented via Julien's
> >>>> patch-set [1]. Its v4 has been floating since July, 2019 and I
> >>>> couldn't find any major blocking comments but not sure why things
> >>>> haven't progressed further.
> >>>>
> >>>> Maybe Julien or Arm maintainers can provide updates on existing
> >>>> patch-set [1] and how we should proceed further with this
> >>>> interesting feature.
> >>>>
> >>>> And regarding hard-lockup detection, I have been able to enable it
> >>>> based on perf NMI events using Julien's perf patch-set [1]. Have a
> >>>> look at the patch here [2].
> >>>>
> >>>> [1] https://patchwork.kernel.org/cover/11047407/
> >>>> [2]
> >>>> http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/7322
> >>>> 27.html
> >>>>
> >>>> -Sumit
> >>>>
> >>>>> Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the
> >>>>> link [2] did.
> >>>>>
> >>>>> [1] https://lkml.org/lkml/2019/1/31/535
> >>>>> [2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiq
> >>>>>
> >>>>>
> >>>>> Lecopzer Chen (3):
> >>>>>   arm_pmu: Add support for perf NMI interrupts registration
> >>>>>   arm64: perf: Support NMI context for perf event ISR
> >>>>>   arm64: Kconfig: Add support for the Perf NMI
> >>>>>
> >>>>>  arch/arm64/Kconfig             | 10 +++++++
> >>>>>  arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------
> >>>>>  drivers/perf/arm_pmu.c         | 51
> ++++++++++++++++++++++++++++++----
> >>>>>  include/linux/perf/arm_pmu.h   |  6 ++++
> >>>>>  4 files changed, 88 insertions(+), 15 deletions(-)
> >>>>>
> >>>>> --
> >>>>> 2.25.1

Alexandru Elisei May 21, 2020, 12:36 p.m. UTC | #11

Hi,

On 5/21/20 4:00 AM, Song Bao Hua (Barry Song) wrote:

>
>> -----Original Message-----
>> From: linux-arm-kernel [mailto:[e-mail]]
>> On Behalf Of Alexandru Elisei
>> Sent: Wednesday, May 20, 2020 10:31 PM> 
>> Hi,
>>
>> On 5/18/20 12:17 PM, Alexandru Elisei wrote:
>>> Hi,
>>>
>>> On 5/18/20 11:45 AM, Mark Rutland wrote:
>>>> Hi all,
>>>>
>>>> On Mon, May 18, 2020 at 02:26:00PM +0800, Lecopzer Chen wrote:
>>>>> HI Sumit,
>>>>>
>>>>> Thanks for your information.
>>>>>
>>>>> I've already implemented IPI (same as you did [1], little difference
>>>>> in detail), hardlockup detector and perf in last year(2019) for
>>>>> debuggability.
>>>>> And now we tend to upstream to reduce kernel maintaining effort.
>>>>> I'm glad if someone in ARM can do this work :)
>>>>>
>>>>> Hi Julien,
>>>>>
>>>>> Does any Arm maintainers can proceed this action?
>>>> Alexandru (Cc'd) has been rebasing and reworking Julien's patches,
>>>> which is my preferred approach.
>>>>
>>>> I understand that's not quite ready for posting since he's
>>>> investigating some of the nastier subtleties (e.g. mutual exclusion
>>>> with the NMI), but maybe we can put the work-in-progress patches
>>>> somewhere in the mean time.
>>>>
>>>> Alexandru, do you have an idea of what needs to be done, and/or when
>>>> you expect you could post that?
>>> I'm currently working on rebasing the patches on top of 5.7-rc5, when
>>> I have something usable I'll post a link (should be a couple of days).
>>> After that I will address the review comments, and I plan to do a
>>> thorough testing because I'm not 100% confident that some of the
>>> assumptions around the locks that were removed are correct. My guess is
>> this will take a few weeks.
>>
>> Pushed a WIP branch on linux-arm.org [1]:
>>
>> git clone -b WIP-pmu-nmi git://linux-arm.org/linux-ae
>>
>> Practically untested, I only did perf record on a defconfig kernel running on the
>> model.
>>
>> [1]
>> http://www.linux-arm.org/git?p=linux-ae.git;a=shortlog;h=refs/heads/WIP-pm
>> u-nmi
> Fortunately, it does work. I used this tree to perf annotate arm_smmu_cmdq_issue_cmdlist() which
> is completely disabling IRQ. Luckily, it reports correct data. Before that, it reported all time was spent by
> the code which enabled IRQ .

That's good news that it works for you, thanks for letting me know.

Alex