| Message ID | 1553692845-20983-1-git-send-email-abel.vesa@nxp.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | cpuidle: Add poking mechanism to support non-IPI wakeup | expand |
Hi Abel, Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > This work is a workaround I'm looking into (more as a background task) > in order to add support for cpuidle on i.MX8MQ based platforms. > > The main idea here is getting around the missing GIC wake_request signal > (due to integration design issue) by waking up a each individual core through > some dedicated SW power-up bits inside the power controller (GPC) right before > every IPI is requested for that each individual core. Just a general comment, without going into the details of this series: this issue is not only affecting IPIs, but also MSIs terminated at the GIC. Currently MSIs are terminated at the PCIe core, but terminating them at the GIC is clearly preferable, as this allows assigning CPU affinity to individual MSIs and lowers IRQ service overhead. I'm not sure what the consequences are for upstream Linux support yet, but we should keep in mind that having a workaround for IPIs is only solving part of the issue. Regards, Lucas
On 27/03/2019 15:44, Lucas Stach wrote: > Hi Abel, > > Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: >> This work is a workaround I'm looking into (more as a background task) >> in order to add support for cpuidle on i.MX8MQ based platforms. >> >> The main idea here is getting around the missing GIC wake_request signal >> (due to integration design issue) by waking up a each individual core through >> some dedicated SW power-up bits inside the power controller (GPC) right before >> every IPI is requested for that each individual core. > > Just a general comment, without going into the details of this series: > this issue is not only affecting IPIs, but also MSIs terminated at the > GIC. Currently MSIs are terminated at the PCIe core, but terminating > them at the GIC is clearly preferable, as this allows assigning CPU > affinity to individual MSIs and lowers IRQ service overhead. > > I'm not sure what the consequences are for upstream Linux support yet, > but we should keep in mind that having a workaround for IPIs is only > solving part of the issue. If this erratum is affecting more than just IPIs, then indeed I don't see how this patch series solves anything. But the erratum documentation seems to imply that only SGIs are affected, and goes as far as suggesting to use an external interrupt would solve it. How comes this is not the case? Or is it that anything directly routed to a redistributor is also affected? This would break LPIs (and thus MSIs) and PPIs (the CPU timer, among others). What is the *exact* status of this thing? I have the ugly feeling that the true workaround is just to disable cpuidle. Thanks, M.
Hi Marc, Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: > On 27/03/2019 15:44, Lucas Stach wrote: > > Hi Abel, > > > > Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > > > This work is a workaround I'm looking into (more as a background task) > > > in order to add support for cpuidle on i.MX8MQ based platforms. > > > > > > The main idea here is getting around the missing GIC wake_request signal > > > (due to integration design issue) by waking up a each individual core through > > > some dedicated SW power-up bits inside the power controller (GPC) right before > > > every IPI is requested for that each individual core. > > > > Just a general comment, without going into the details of this series: > > this issue is not only affecting IPIs, but also MSIs terminated at the > > GIC. Currently MSIs are terminated at the PCIe core, but terminating > > them at the GIC is clearly preferable, as this allows assigning CPU > > affinity to individual MSIs and lowers IRQ service overhead. > > > > I'm not sure what the consequences are for upstream Linux support yet, > > but we should keep in mind that having a workaround for IPIs is only > > solving part of the issue. > > If this erratum is affecting more than just IPIs, then indeed I don't > see how this patch series solves anything. > > But the erratum documentation seems to imply that only SGIs are > affected, and goes as far as suggesting to use an external interrupt > would solve it. How comes this is not the case? Or is it that anything > directly routed to a redistributor is also affected? This would break > LPIs (and thus MSIs) and PPIs (the CPU timer, among others). > > What is the *exact* status of this thing? I have the ugly feeling that > the true workaround is just to disable cpuidle. As far as I understand the erratum, the basic issue is that the GIC wake_request signals are not connected to the GPC (the CPU/peripheral power sequencer). The SPIs are routed through the GPC and thus are visible as wakeup sources, which is why the workaround of using an external SPI as wakeup trigger for the IPI works. Anything that isn't visible to the GPC and requires the GIC wake_request signal to behave as specified is broken by this erratum. You probably know the GIC better than any of us to tell what this means. Regards, Lucas
On Wed, 2019-03-27 at 17:06 +0100, Lucas Stach wrote: > Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: > > On 27/03/2019 15:44, Lucas Stach wrote: > > > Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > > > > This work is a workaround I'm looking into (more as a background task) > > > > in order to add support for cpuidle on i.MX8MQ based platforms. > > > > > > > > The main idea here is getting around the missing GIC wake_request signal > > > > (due to integration design issue) by waking up a each individual core through > > > > some dedicated SW power-up bits inside the power controller (GPC) right before > > > > every IPI is requested for that each individual core. > > > > > > Just a general comment, without going into the details of this series: > > > this issue is not only affecting IPIs, but also MSIs terminated at the > > > GIC. Currently MSIs are terminated at the PCIe core, but terminating > > > them at the GIC is clearly preferable, as this allows assigning CPU > > > affinity to individual MSIs and lowers IRQ service overhead. > > > > > > I'm not sure what the consequences are for upstream Linux support yet, > > > but we should keep in mind that having a workaround for IPIs is only > > > solving part of the issue. > > > > If this erratum is affecting more than just IPIs, then indeed I don't > > see how this patch series solves anything. > > > > But the erratum documentation seems to imply that only SGIs are > > affected, and goes as far as suggesting to use an external interrupt > > would solve it. How comes this is not the case? Or is it that anything > > directly routed to a redistributor is also affected? This would break > > LPIs (and thus MSIs) and PPIs (the CPU timer, among others). > > > > What is the *exact* status of this thing? I have the ugly feeling that > > the true workaround is just to disable cpuidle. > > As far as I understand the erratum, the basic issue is that the GIC > wake_request signals are not connected to the GPC (the CPU/peripheral > power sequencer). The SPIs are routed through the GPC and thus are > visible as wakeup sources, which is why the workaround of using an > external SPI as wakeup trigger for the IPI works. We had a kernel workaround for IPIs in our internal tree for a long time and I don't think we do anything special for PCI. Does PCI MSI really bypass the GPC on 8mq? Adding Richard/Jacky, they might know about this. This seems like something of a corner case to me, don't many imx boards ship without PCI; especially for low-power scenarios? If required it might be reasonable to add an additional workaround to disable all cpuidle if pci msis are used. -- Regards, Leonard
Am Mittwoch, den 27.03.2019, 17:00 +0000 schrieb Leonard Crestez: > On Wed, 2019-03-27 at 17:06 +0100, Lucas Stach wrote: > > Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: > > > On 27/03/2019 15:44, Lucas Stach wrote: > > > > Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > > > > > This work is a workaround I'm looking into (more as a > > > > > background task) > > > > > in order to add support for cpuidle on i.MX8MQ based > > > > > platforms. > > > > > > > > > > The main idea here is getting around the missing GIC > > > > > wake_request signal > > > > > (due to integration design issue) by waking up a each > > > > > individual core through > > > > > some dedicated SW power-up bits inside the power controller > > > > > (GPC) right before > > > > > every IPI is requested for that each individual core. > > > > > > > > Just a general comment, without going into the details of this > > > > series: > > > > this issue is not only affecting IPIs, but also MSIs terminated > > > > at the > > > > GIC. Currently MSIs are terminated at the PCIe core, but > > > > terminating > > > > them at the GIC is clearly preferable, as this allows assigning > > > > CPU > > > > affinity to individual MSIs and lowers IRQ service overhead. > > > > > > > > I'm not sure what the consequences are for upstream Linux > > > > support yet, > > > > but we should keep in mind that having a workaround for IPIs is > > > > only > > > > solving part of the issue. > > > > > > If this erratum is affecting more than just IPIs, then indeed I > > > don't > > > see how this patch series solves anything. > > > > > > But the erratum documentation seems to imply that only SGIs are > > > affected, and goes as far as suggesting to use an external > > > interrupt > > > would solve it. How comes this is not the case? Or is it that > > > anything > > > directly routed to a redistributor is also affected? This would > > > break > > > LPIs (and thus MSIs) and PPIs (the CPU timer, among others). > > > > > > What is the *exact* status of this thing? I have the ugly feeling > > > that > > > the true workaround is just to disable cpuidle. > > > > As far as I understand the erratum, the basic issue is that the GIC > > wake_request signals are not connected to the GPC (the > > CPU/peripheral > > power sequencer). The SPIs are routed through the GPC and thus are > > visible as wakeup sources, which is why the workaround of using an > > external SPI as wakeup trigger for the IPI works. > > We had a kernel workaround for IPIs in our internal tree for a long > time and I don't think we do anything special for PCI. Does PCI MSI > really bypass the GPC on 8mq? > > Adding Richard/Jacky, they might know about this. Currently the MSIs are terminated at the PCIe controller and routed to the CPU via a normal interrupt line that is going through the GPC, so there are no workaround required today. But then this setup severely limits the usefulness of PCI MSIs, as they incur an additional overhead of working with the DWC MSI controller and are unable to target a specific CPU, as they are all routed via a single IRQ line. > This seems like something of a corner case to me, don't many imx > boards > ship without PCI; especially for low-power scenarios? If required it > might be reasonable to add an additional workaround to disable all > cpuidle if pci msis are used. I don't know how common using PCIe with the i.MX8M is, but even the reference board ships with the WLAN connected to PCIe. I'm working with a design that has both a multi-queue and TSN capable ethernet card connected to one PCIe controller and a NVMe SSD with multiple queues connected to the second controller. Being able to terminate the MSIs at the GIC level and have proper CPU affinity makes a lot of sense in that scenario. Regards, Lucas
On 27/03/2019 16:06, Lucas Stach wrote: > Hi Marc, > > Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: >> On 27/03/2019 15:44, Lucas Stach wrote: >>> Hi Abel, >>> >>> Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: >>>> This work is a workaround I'm looking into (more as a background task) >>>> in order to add support for cpuidle on i.MX8MQ based platforms. >>>> >>>> The main idea here is getting around the missing GIC wake_request signal >>>> (due to integration design issue) by waking up a each individual core through >>>> some dedicated SW power-up bits inside the power controller (GPC) right before >>>> every IPI is requested for that each individual core. >>> >>> Just a general comment, without going into the details of this series: >>> this issue is not only affecting IPIs, but also MSIs terminated at the >>> GIC. Currently MSIs are terminated at the PCIe core, but terminating >>> them at the GIC is clearly preferable, as this allows assigning CPU >>> affinity to individual MSIs and lowers IRQ service overhead. >>> >>> I'm not sure what the consequences are for upstream Linux support yet, >>> but we should keep in mind that having a workaround for IPIs is only >>> solving part of the issue. >> >> If this erratum is affecting more than just IPIs, then indeed I don't >> see how this patch series solves anything. >> >> But the erratum documentation seems to imply that only SGIs are >> affected, and goes as far as suggesting to use an external interrupt >> would solve it. How comes this is not the case? Or is it that anything >> directly routed to a redistributor is also affected? This would break >> LPIs (and thus MSIs) and PPIs (the CPU timer, among others). >> >> What is the *exact* status of this thing? I have the ugly feeling that >> the true workaround is just to disable cpuidle. > > As far as I understand the erratum, the basic issue is that the GIC > wake_request signals are not connected to the GPC (the CPU/peripheral > power sequencer). The SPIs are routed through the GPC and thus are > visible as wakeup sources, which is why the workaround of using an > external SPI as wakeup trigger for the IPI works. Are all SPIs connected to the GPC? > Anything that isn't visible to the GPC and requires the GIC > wake_request signal to behave as specified is broken by this erratum. I really wonder how a timer interrupt (a PPI, hence not routed through the GPC) can wake up the CPU in this case. It really feels like something like "program CNTV_CVAL_EL0 to expire at some later point; WFI" could result in the CPU going to a deep sleep state, and not wake-up at all. This would indicate that not only cpuidle is broken with this, but absolutely every interrupt that is not routed through the GPC. > You probably know the GIC better than any of us to tell what this > means. Yeah, and that's a very unfortunate state of things... :-/ Thanks, M.
Am Mittwoch, den 27.03.2019, 17:45 +0000 schrieb Marc Zyngier: > On 27/03/2019 16:06, Lucas Stach wrote: > > Hi Marc, > > > > Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: > > > On 27/03/2019 15:44, Lucas Stach wrote: > > > > Hi Abel, > > > > > > > > Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > > > > > This work is a workaround I'm looking into (more as a background task) > > > > > in order to add support for cpuidle on i.MX8MQ based platforms. > > > > > > > > > > The main idea here is getting around the missing GIC wake_request signal > > > > > (due to integration design issue) by waking up a each individual core through > > > > > some dedicated SW power-up bits inside the power controller (GPC) right before > > > > > every IPI is requested for that each individual core. > > > > > > > > Just a general comment, without going into the details of this series: > > > > this issue is not only affecting IPIs, but also MSIs terminated at the > > > > GIC. Currently MSIs are terminated at the PCIe core, but terminating > > > > them at the GIC is clearly preferable, as this allows assigning CPU > > > > affinity to individual MSIs and lowers IRQ service overhead. > > > > > > > > I'm not sure what the consequences are for upstream Linux support yet, > > > > but we should keep in mind that having a workaround for IPIs is only > > > > solving part of the issue. > > > > > > If this erratum is affecting more than just IPIs, then indeed I don't > > > see how this patch series solves anything. > > > > > > But the erratum documentation seems to imply that only SGIs are > > > affected, and goes as far as suggesting to use an external interrupt > > > would solve it. How comes this is not the case? Or is it that anything > > > directly routed to a redistributor is also affected? This would break > > > LPIs (and thus MSIs) and PPIs (the CPU timer, among others). > > > > > > What is the *exact* status of this thing? I have the ugly feeling that > > > the true workaround is just to disable cpuidle. > > > > As far as I understand the erratum, the basic issue is that the GIC > > wake_request signals are not connected to the GPC (the CPU/peripheral > > power sequencer). The SPIs are routed through the GPC and thus are > > visible as wakeup sources, which is why the workaround of using an > > external SPI as wakeup trigger for the IPI works. > > Are all SPIs connected to the GPC? AFAICS yes. > > Anything that isn't visible to the GPC and requires the GIC > > wake_request signal to behave as specified is broken by this erratum. > > I really wonder how a timer interrupt (a PPI, hence not routed through > the GPC) can wake up the CPU in this case. It really feels like > something like "program CNTV_CVAL_EL0 to expire at some later point; > WFI" could result in the CPU going to a deep sleep state, and not > wake-up at all. I guess it's broken in the same way. The downstream DT claims "local-timer-stop" for the CPU sleep state and "arm,no-tick-in-suspend" for the armv8-timer, which I guess is not the timer actually stopping in suspend, but the CPU being unable to wake up due to the timer IRQ. > This would indicate that not only cpuidle is broken with this, but > absolutely every interrupt that is not routed through the GPC. That's my understanding as well. Note that I have no NXP internal information and can only infer from the published reference manual, errata notice and downstream kernel. Regards, Lucas
On 27/03/2019 17:00, Leonard Crestez wrote: > On Wed, 2019-03-27 at 17:06 +0100, Lucas Stach wrote: >> Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: >>> On 27/03/2019 15:44, Lucas Stach wrote: >>>> Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: >>>>> This work is a workaround I'm looking into (more as a background task) >>>>> in order to add support for cpuidle on i.MX8MQ based platforms. >>>>> >>>>> The main idea here is getting around the missing GIC wake_request signal >>>>> (due to integration design issue) by waking up a each individual core through >>>>> some dedicated SW power-up bits inside the power controller (GPC) right before >>>>> every IPI is requested for that each individual core. >>>> >>>> Just a general comment, without going into the details of this series: >>>> this issue is not only affecting IPIs, but also MSIs terminated at the >>>> GIC. Currently MSIs are terminated at the PCIe core, but terminating >>>> them at the GIC is clearly preferable, as this allows assigning CPU >>>> affinity to individual MSIs and lowers IRQ service overhead. >>>> >>>> I'm not sure what the consequences are for upstream Linux support yet, >>>> but we should keep in mind that having a workaround for IPIs is only >>>> solving part of the issue. >>> >>> If this erratum is affecting more than just IPIs, then indeed I don't >>> see how this patch series solves anything. >>> >>> But the erratum documentation seems to imply that only SGIs are >>> affected, and goes as far as suggesting to use an external interrupt >>> would solve it. How comes this is not the case? Or is it that anything >>> directly routed to a redistributor is also affected? This would break >>> LPIs (and thus MSIs) and PPIs (the CPU timer, among others). >>> >>> What is the *exact* status of this thing? I have the ugly feeling that >>> the true workaround is just to disable cpuidle. >> >> As far as I understand the erratum, the basic issue is that the GIC >> wake_request signals are not connected to the GPC (the CPU/peripheral >> power sequencer). The SPIs are routed through the GPC and thus are >> visible as wakeup sources, which is why the workaround of using an >> external SPI as wakeup trigger for the IPI works. > > We had a kernel workaround for IPIs in our internal tree for a long > time and I don't think we do anything special for PCI. Does PCI MSI > really bypass the GPC on 8mq? If you have an ITS, certainly. If you don't, it depends. MSIs can hit the distributor's MBI registers and generate non-wired SPIs, which I assume will bypass the GPC altogether. > Adding Richard/Jacky, they might know about this. > > This seems like something of a corner case to me, don't many imx boards > ship without PCI; especially for low-power scenarios? If required it > might be reasonable to add an additional workaround to disable all > cpuidle if pci msis are used. Establishing a link between cpuidle and PCI in the kernel would be pretty invasive, and that would come on top of what this series also mandates. At that level of apparent brokenness, it is far safer to get cpuidle out of the picture altogether, and I'd rather see these patches in a vendor tree (for once). Thanks, M.
On Wed, 2019-03-27 at 17:45 +0000, Marc Zyngier wrote: > On 27/03/2019 16:06, Lucas Stach wrote: > > Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: > > > On 27/03/2019 15:44, Lucas Stach wrote: > > > > Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > > > > > This work is a workaround I'm looking into (more as a background task) > > > > > in order to add support for cpuidle on i.MX8MQ based platforms. > > > > > > > > > > The main idea here is getting around the missing GIC wake_request signal > > > > > (due to integration design issue) by waking up a each individual core through > > > > > some dedicated SW power-up bits inside the power controller (GPC) right before > > > > > every IPI is requested for that each individual core. > > > > > > > > Just a general comment, without going into the details of this series: > > > > this issue is not only affecting IPIs, but also MSIs terminated at the > > > > GIC. Currently MSIs are terminated at the PCIe core, but terminating > > > > them at the GIC is clearly preferable, as this allows assigning CPU > > > > affinity to individual MSIs and lowers IRQ service overhead. > > > > > > > > I'm not sure what the consequences are for upstream Linux support yet, > > > > but we should keep in mind that having a workaround for IPIs is only > > > > solving part of the issue. > > > > > > If this erratum is affecting more than just IPIs, then indeed I don't > > > see how this patch series solves anything. > > > > > > But the erratum documentation seems to imply that only SGIs are > > > affected, and goes as far as suggesting to use an external interrupt > > > would solve it. How comes this is not the case? Or is it that anything > > > directly routed to a redistributor is also affected? This would break > > > LPIs (and thus MSIs) and PPIs (the CPU timer, among others). > > > > Anything that isn't visible to the GPC and requires the GIC > > wake_request signal to behave as specified is broken by this erratum. > > I really wonder how a timer interrupt (a PPI, hence not routed through > the GPC) can wake up the CPU in this case. It really feels like > something like "program CNTV_CVAL_EL0 to expire at some later point; > WFI" could result in the CPU going to a deep sleep state, and not > wake-up at all. This is already a common issue for cpuidle implementions handled by the "local-timer-stop" property. imx has other timer blocks in the SOC, they generate SPIs which are connected to GPC. > This would indicate that not only cpuidle is broken with this, but > absolutely every interrupt that is not routed through the GPC. Yes, cpuidle is broken for irqs not routed through GPC. However: * All SPIs are connected to GPC in a 1:1 mapping * This series deals with SGIs * The timer PPIs are not required; covered by local-timer-stop * LPIs are currently unused (I understand imx-pci uses SPI by default from Lucas) Anything missing? My understanding is that this wake request feature via GIC is new in v3 and this is maybe why HW team missed it during integration. Older imx6/7 has GICv2 and has deep idle states which always rely on GPC to wakeup so the approach can work. -- Regards, Leonard
On 27/03/2019 18:40, Leonard Crestez wrote: > On Wed, 2019-03-27 at 17:45 +0000, Marc Zyngier wrote: >> On 27/03/2019 16:06, Lucas Stach wrote: >>> Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: >>>> On 27/03/2019 15:44, Lucas Stach wrote: >>>>> Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: >>>>>> This work is a workaround I'm looking into (more as a background task) >>>>>> in order to add support for cpuidle on i.MX8MQ based platforms. >>>>>> >>>>>> The main idea here is getting around the missing GIC wake_request signal >>>>>> (due to integration design issue) by waking up a each individual core through >>>>>> some dedicated SW power-up bits inside the power controller (GPC) right before >>>>>> every IPI is requested for that each individual core. >>>>> >>>>> Just a general comment, without going into the details of this series: >>>>> this issue is not only affecting IPIs, but also MSIs terminated at the >>>>> GIC. Currently MSIs are terminated at the PCIe core, but terminating >>>>> them at the GIC is clearly preferable, as this allows assigning CPU >>>>> affinity to individual MSIs and lowers IRQ service overhead. >>>>> >>>>> I'm not sure what the consequences are for upstream Linux support yet, >>>>> but we should keep in mind that having a workaround for IPIs is only >>>>> solving part of the issue. >>>> >>>> If this erratum is affecting more than just IPIs, then indeed I don't >>>> see how this patch series solves anything. >>>> >>>> But the erratum documentation seems to imply that only SGIs are >>>> affected, and goes as far as suggesting to use an external interrupt >>>> would solve it. How comes this is not the case? Or is it that anything >>>> directly routed to a redistributor is also affected? This would break >>>> LPIs (and thus MSIs) and PPIs (the CPU timer, among others). >>> >>> Anything that isn't visible to the GPC and requires the GIC >>> wake_request signal to behave as specified is broken by this erratum. >> >> I really wonder how a timer interrupt (a PPI, hence not routed through >> the GPC) can wake up the CPU in this case. It really feels like >> something like "program CNTV_CVAL_EL0 to expire at some later point; >> WFI" could result in the CPU going to a deep sleep state, and not >> wake-up at all. > > This is already a common issue for cpuidle implementions handled by the > "local-timer-stop" property. imx has other timer blocks in the SOC, > they generate SPIs which are connected to GPC. > >> This would indicate that not only cpuidle is broken with this, but >> absolutely every interrupt that is not routed through the GPC. > > Yes, cpuidle is broken for irqs not routed through GPC. However: > > * All SPIs are connected to GPC in a 1:1 mapping > * This series deals with SGIs > * The timer PPIs are not required; covered by local-timer-stop > * LPIs are currently unused (I understand imx-pci uses SPI by default > from Lucas) > > Anything missing? > > My understanding is that this wake request feature via GIC is new in v3 > and this is maybe why HW team missed it during integration. Older > imx6/7 has GICv2 and has deep idle states which always rely on GPC to > wakeup so the approach can work. Certainly the approach can work. The question is whether we want to support this in a mainline kernel, spreading random hooks in the generic code and adding a firmware interface on top of that. By all accounts, this HW is broken. You can indeed impose limitations (dumb down PCI, mandate the use of a broadcast timer), or you can just flag cpuidle as unsupported on this HW. My vote is on the latter. Thanks, M.
On Thursday, March 28, 2019 11:35:23 AM CET Marc Zyngier wrote: > On 27/03/2019 18:40, Leonard Crestez wrote: > > On Wed, 2019-03-27 at 17:45 +0000, Marc Zyngier wrote: > >> On 27/03/2019 16:06, Lucas Stach wrote: > >>> Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: > >>>> On 27/03/2019 15:44, Lucas Stach wrote: > >>>>> Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > >>>>>> This work is a workaround I'm looking into (more as a background task) > >>>>>> in order to add support for cpuidle on i.MX8MQ based platforms. > >>>>>> > >>>>>> The main idea here is getting around the missing GIC wake_request signal > >>>>>> (due to integration design issue) by waking up a each individual core through > >>>>>> some dedicated SW power-up bits inside the power controller (GPC) right before > >>>>>> every IPI is requested for that each individual core. > >>>>> > >>>>> Just a general comment, without going into the details of this series: > >>>>> this issue is not only affecting IPIs, but also MSIs terminated at the > >>>>> GIC. Currently MSIs are terminated at the PCIe core, but terminating > >>>>> them at the GIC is clearly preferable, as this allows assigning CPU > >>>>> affinity to individual MSIs and lowers IRQ service overhead. > >>>>> > >>>>> I'm not sure what the consequences are for upstream Linux support yet, > >>>>> but we should keep in mind that having a workaround for IPIs is only > >>>>> solving part of the issue. > >>>> > >>>> If this erratum is affecting more than just IPIs, then indeed I don't > >>>> see how this patch series solves anything. > >>>> > >>>> But the erratum documentation seems to imply that only SGIs are > >>>> affected, and goes as far as suggesting to use an external interrupt > >>>> would solve it. How comes this is not the case? Or is it that anything > >>>> directly routed to a redistributor is also affected? This would break > >>>> LPIs (and thus MSIs) and PPIs (the CPU timer, among others). > >>> > >>> Anything that isn't visible to the GPC and requires the GIC > >>> wake_request signal to behave as specified is broken by this erratum. > >> > >> I really wonder how a timer interrupt (a PPI, hence not routed through > >> the GPC) can wake up the CPU in this case. It really feels like > >> something like "program CNTV_CVAL_EL0 to expire at some later point; > >> WFI" could result in the CPU going to a deep sleep state, and not > >> wake-up at all. > > > > This is already a common issue for cpuidle implementions handled by the > > "local-timer-stop" property. imx has other timer blocks in the SOC, > > they generate SPIs which are connected to GPC. > > > >> This would indicate that not only cpuidle is broken with this, but > >> absolutely every interrupt that is not routed through the GPC. > > > > Yes, cpuidle is broken for irqs not routed through GPC. However: > > > > * All SPIs are connected to GPC in a 1:1 mapping > > * This series deals with SGIs > > * The timer PPIs are not required; covered by local-timer-stop > > * LPIs are currently unused (I understand imx-pci uses SPI by default > > from Lucas) > > > > Anything missing? > > > > My understanding is that this wake request feature via GIC is new in v3 > > and this is maybe why HW team missed it during integration. Older > > imx6/7 has GICv2 and has deep idle states which always rely on GPC to > > wakeup so the approach can work. > > Certainly the approach can work. The question is whether we want to > support this in a mainline kernel, spreading random hooks in the generic > code and adding a firmware interface on top of that. Not really. > By all accounts, this HW is broken. You can indeed impose limitations > (dumb down PCI, mandate the use of a broadcast timer), or you can just > flag cpuidle as unsupported on this HW. My vote is on the latter. Agreed. Thanks, Rafael
On Wed, Mar 27, 2019 at 06:40:07PM +0000, Leonard Crestez wrote: > On Wed, 2019-03-27 at 17:45 +0000, Marc Zyngier wrote: > > On 27/03/2019 16:06, Lucas Stach wrote: > > > Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: > > > > On 27/03/2019 15:44, Lucas Stach wrote: > > > > > Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > > > > > > This work is a workaround I'm looking into (more as a background task) > > > > > > in order to add support for cpuidle on i.MX8MQ based platforms. > > > > > > > > > > > > The main idea here is getting around the missing GIC wake_request signal > > > > > > (due to integration design issue) by waking up a each individual core through > > > > > > some dedicated SW power-up bits inside the power controller (GPC) right before > > > > > > every IPI is requested for that each individual core. > > > > > > > > > > Just a general comment, without going into the details of this series: > > > > > this issue is not only affecting IPIs, but also MSIs terminated at the > > > > > GIC. Currently MSIs are terminated at the PCIe core, but terminating > > > > > them at the GIC is clearly preferable, as this allows assigning CPU > > > > > affinity to individual MSIs and lowers IRQ service overhead. > > > > > > > > > > I'm not sure what the consequences are for upstream Linux support yet, > > > > > but we should keep in mind that having a workaround for IPIs is only > > > > > solving part of the issue. > > > > > > > > If this erratum is affecting more than just IPIs, then indeed I don't > > > > see how this patch series solves anything. > > > > > > > > But the erratum documentation seems to imply that only SGIs are > > > > affected, and goes as far as suggesting to use an external interrupt > > > > would solve it. How comes this is not the case? Or is it that anything > > > > directly routed to a redistributor is also affected? This would break > > > > LPIs (and thus MSIs) and PPIs (the CPU timer, among others). > > > > > > Anything that isn't visible to the GPC and requires the GIC > > > wake_request signal to behave as specified is broken by this erratum. > > > > I really wonder how a timer interrupt (a PPI, hence not routed through > > the GPC) can wake up the CPU in this case. It really feels like > > something like "program CNTV_CVAL_EL0 to expire at some later point; > > WFI" could result in the CPU going to a deep sleep state, and not > > wake-up at all. > > This is already a common issue for cpuidle implementions handled by the > "local-timer-stop" property. imx has other timer blocks in the SOC, > they generate SPIs which are connected to GPC. It is not a common issue. The tick-broadcast mechanism relies on IPIs that are sent to specific CPUs upon timer expiry. If IPIs don't work for CPUs in shutdown state (which is what this patch is fixing AFAIU), the only reason I can see how a CPU can resume from idle on a timer expiry is the GPC waking up all cores upon the global timer SPI; if that's the case there is precious little point in implementing CPUidle at all - too bad people worked hard to implement NOHZ in a power efficient manner. > > This would indicate that not only cpuidle is broken with this, but > > absolutely every interrupt that is not routed through the GPC. > > Yes, cpuidle is broken for irqs not routed through GPC. However: > > * All SPIs are connected to GPC in a 1:1 mapping > * This series deals with SGIs > * The timer PPIs are not required; covered by local-timer-stop > * LPIs are currently unused (I understand imx-pci uses SPI by default > from Lucas) > > Anything missing? Yes, LPIs must be able to wake up CPUs and only the CPU for which an IRQ is actually pending. From an architectural perspective, an ARM core executing the WFI instruction must resume execution upon an IRQ occurrence targeted at it and that's true regardless of the idle state entered. Anything deviating from this behaviour is not architecture compliant. > My understanding is that this wake request feature via GIC is new in v3 > and this is maybe why HW team missed it during integration. Older > imx6/7 has GICv2 and has deep idle states which always rely on GPC to > wakeup so the approach can work. If HW designers really wanted to have sensible power management policy in this SoC they would have paid attention, I am against patching the kernel heavily to fix a platform bug. Thanks, Lorenzo
> From: Marc Zyngier [mailto:marc.zyngier@arm.com] > Sent: Thursday, March 28, 2019 2:13 AM > On 27/03/2019 17:00, Leonard Crestez wrote: > > On Wed, 2019-03-27 at 17:06 +0100, Lucas Stach wrote: > >> Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: > >>> On 27/03/2019 15:44, Lucas Stach wrote: > >>>> Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > >>>>> This work is a workaround I'm looking into (more as a background > >>>>> task) in order to add support for cpuidle on i.MX8MQ based platforms. > >>>>> > >>>>> The main idea here is getting around the missing GIC wake_request > >>>>> signal (due to integration design issue) by waking up a each > >>>>> individual core through some dedicated SW power-up bits inside the > >>>>> power controller (GPC) right before every IPI is requested for that each > individual core. > >>>> > >>>> Just a general comment, without going into the details of this series: > >>>> this issue is not only affecting IPIs, but also MSIs terminated at > >>>> the GIC. Currently MSIs are terminated at the PCIe core, but > >>>> terminating them at the GIC is clearly preferable, as this allows > >>>> assigning CPU affinity to individual MSIs and lowers IRQ service overhead. > >>>> > >>>> I'm not sure what the consequences are for upstream Linux support > >>>> yet, but we should keep in mind that having a workaround for IPIs > >>>> is only solving part of the issue. > >>> > >>> If this erratum is affecting more than just IPIs, then indeed I > >>> don't see how this patch series solves anything. > >>> > >>> But the erratum documentation seems to imply that only SGIs are > >>> affected, and goes as far as suggesting to use an external interrupt > >>> would solve it. How comes this is not the case? Or is it that > >>> anything directly routed to a redistributor is also affected? This > >>> would break LPIs (and thus MSIs) and PPIs (the CPU timer, among others). > >>> > >>> What is the *exact* status of this thing? I have the ugly feeling > >>> that the true workaround is just to disable cpuidle. > >> > >> As far as I understand the erratum, the basic issue is that the GIC > >> wake_request signals are not connected to the GPC (the CPU/peripheral > >> power sequencer). The SPIs are routed through the GPC and thus are > >> visible as wakeup sources, which is why the workaround of using an > >> external SPI as wakeup trigger for the IPI works. > > > > We had a kernel workaround for IPIs in our internal tree for a long > > time and I don't think we do anything special for PCI. Does PCI MSI > > really bypass the GPC on 8mq? > > If you have an ITS, certainly. If you don't, it depends. MSIs can hit the > distributor's MBI registers and generate non-wired SPIs, which I assume will > bypass the GPC altogether. > Richard & Jacky, Can you double check if this issue affect PCI MSI function? Regards Dong Aisheng > > Adding Richard/Jacky, they might know about this. > > > > This seems like something of a corner case to me, don't many imx > > boards ship without PCI; especially for low-power scenarios? If > > required it might be reasonable to add an additional workaround to > > disable all cpuidle if pci msis are used. > > Establishing a link between cpuidle and PCI in the kernel would be pretty > invasive, and that would come on top of what this series also mandates. > > At that level of apparent brokenness, it is far safer to get cpuidle out of the > picture altogether, and I'd rather see these patches in a vendor tree (for once). > > Thanks, > > M. > -- > Jazz is not dead. It just smells funny...
[...] > > > Anything that isn't visible to the GPC and requires the GIC > > > wake_request signal to behave as specified is broken by this erratum. > > > > I really wonder how a timer interrupt (a PPI, hence not routed through > > the GPC) can wake up the CPU in this case. It really feels like > > something like "program CNTV_CVAL_EL0 to expire at some later point; > > WFI" could result in the CPU going to a deep sleep state, and not > > wake-up at all. > > I guess it's broken in the same way. The downstream DT claims > "local-timer-stop" for the CPU sleep state and "arm,no-tick-in-suspend" > for the armv8-timer, which I guess is not the timer actually stopping in suspend, > but the CPU being unable to wake up due to the timer IRQ. > > > This would indicate that not only cpuidle is broken with this, but > > absolutely every interrupt that is not routed through the GPC. > > That's my understanding as well. Note that I have no NXP internal information > and can only infer from the published reference manual, errata notice and > downstream kernel. > We will double check it. Thanks for the information. Regards Dong Aisheng > Regards, > Lucas
[...] > > * All SPIs are connected to GPC in a 1:1 mapping > > * This series deals with SGIs > > * The timer PPIs are not required; covered by local-timer-stop > > * LPIs are currently unused (I understand imx-pci uses SPI by default > > from Lucas) > > > > Anything missing? > > > > My understanding is that this wake request feature via GIC is new in > > v3 and this is maybe why HW team missed it during integration. Older > > imx6/7 has GICv2 and has deep idle states which always rely on GPC to > > wakeup so the approach can work. > > Certainly the approach can work. The question is whether we want to support > this in a mainline kernel, spreading random hooks in the generic code and > adding a firmware interface on top of that. > > By all accounts, this HW is broken. You can indeed impose limitations (dumb > down PCI, mandate the use of a broadcast timer), or you can just flag cpuidle > as unsupported on this HW. My vote is on the latter. > Hi Marc, Rafael, Lorenzo Thanks for the suggestion. I fully understand the concern. Do you think we can patch the platform code to address the issue to avoid the big churn on kernel core code? If yes, we could try to investigate if there's a suitable place to do that. The main thing we need to do seems like to manually wakeup cpu core during the sending IPI path when exit idle. We could see if there's chance to do it on that path. Regards Dong Aisheng > Thanks, > > M. > -- > Jazz is not dead. It just smells funny...
> -----Original Message----- > From: Aisheng Dong > Sent: 2019年3月28日 19:21 > To: Marc Zyngier <marc.zyngier@arm.com>; Leonard Crestez > <leonard.crestez@nxp.com>; l.stach@pengutronix.de; Richard Zhu > <hongxing.zhu@nxp.com>; Jacky Bai <ping.bai@nxp.com> > Cc: Fabio Estevam <fabio.estevam@nxp.com>; Cosmin Samoila > <cosmin.samoila@nxp.com>; Robin Gong <yibin.gong@nxp.com>; Mircea Pop > <mircea.pop@nxp.com>; Daniel Baluta <daniel.baluta@nxp.com>; > catalin.marinas@arm.com; shawnguo@kernel.org; Robert Chiras > <robert.chiras@nxp.com>; Anson Huang <anson.huang@nxp.com>; Jun Li > <jun.li@nxp.com>; Abel Vesa <abel.vesa@nxp.com>; robh@kernel.org; > Zening Wang <zening.wang@nxp.com>; dl-linux-imx <linux-imx@nxp.com>; > BOUGH CHEN <haibo.chen@nxp.com>; Horia Geanta > <horia.geanta@nxp.com>; Peter Chen <peter.chen@nxp.com>; Joakim Zhang > <qiangqing.zhang@nxp.com>; rjw@rjwysocki.net; Leo Zhang > <leo.zhang@nxp.com>; Shenwei Wang <shenwei.wang@nxp.com>; > linux-pm@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Ranjani > Vaidyanathan <ranjani.vaidyanathan@nxp.com>; Han Xu <han.xu@nxp.com>; > will.deacon@arm.com; Iuliana Prodan <iuliana.prodan@nxp.com>; > sudeep.holla@arm.com; lorenzo.pieralisi@arm.com; > linux-kernel@vger.kernel.org; mark.rutland@arm.com; Peng Fan > <peng.fan@nxp.com>; kernel@pengutronix.de; Viorel Suman > <viorel.suman@nxp.com> > Subject: RE: [RFC 0/7] cpuidle: Add poking mechanism to support non-IPI > wakeup > > > From: Marc Zyngier [mailto:marc.zyngier@arm.com] > > Sent: Thursday, March 28, 2019 2:13 AM On 27/03/2019 17:00, Leonard > > Crestez wrote: > > > On Wed, 2019-03-27 at 17:06 +0100, Lucas Stach wrote: > > >> Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: > > >>> On 27/03/2019 15:44, Lucas Stach wrote: > > >>>> Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > > >>>>> This work is a workaround I'm looking into (more as a background > > >>>>> task) in order to add support for cpuidle on i.MX8MQ based > platforms. > > >>>>> > > >>>>> The main idea here is getting around the missing GIC > > >>>>> wake_request signal (due to integration design issue) by waking > > >>>>> up a each individual core through some dedicated SW power-up > > >>>>> bits inside the power controller (GPC) right before every IPI is > > >>>>> requested for that each > > individual core. > > >>>> > > >>>> Just a general comment, without going into the details of this series: > > >>>> this issue is not only affecting IPIs, but also MSIs terminated > > >>>> at the GIC. Currently MSIs are terminated at the PCIe core, but > > >>>> terminating them at the GIC is clearly preferable, as this allows > > >>>> assigning CPU affinity to individual MSIs and lowers IRQ service > overhead. > > >>>> > > >>>> I'm not sure what the consequences are for upstream Linux support > > >>>> yet, but we should keep in mind that having a workaround for IPIs > > >>>> is only solving part of the issue. > > >>> > > >>> If this erratum is affecting more than just IPIs, then indeed I > > >>> don't see how this patch series solves anything. > > >>> > > >>> But the erratum documentation seems to imply that only SGIs are > > >>> affected, and goes as far as suggesting to use an external > > >>> interrupt would solve it. How comes this is not the case? Or is it > > >>> that anything directly routed to a redistributor is also affected? > > >>> This would break LPIs (and thus MSIs) and PPIs (the CPU timer, among > others). > > >>> > > >>> What is the *exact* status of this thing? I have the ugly feeling > > >>> that the true workaround is just to disable cpuidle. > > >> > > >> As far as I understand the erratum, the basic issue is that the GIC > > >> wake_request signals are not connected to the GPC (the > > >> CPU/peripheral power sequencer). The SPIs are routed through the > > >> GPC and thus are visible as wakeup sources, which is why the > > >> workaround of using an external SPI as wakeup trigger for the IPI works. > > > > > > We had a kernel workaround for IPIs in our internal tree for a long > > > time and I don't think we do anything special for PCI. Does PCI MSI > > > really bypass the GPC on 8mq? > > > > If you have an ITS, certainly. If you don't, it depends. MSIs can hit > > the distributor's MBI registers and generate non-wired SPIs, which I > > assume will bypass the GPC altogether. > > > > Richard & Jacky, > > Can you double check if this issue affect PCI MSI function? > [Richard Zhu] GIC V3 has the ITS/LPIs features. That can be used by PCIe MSI functions. BTW, the PCIe MSI ITS mode is not enabled in vendor tree. Best Regards Richard Zhu > Regards > Dong Aisheng > > > > Adding Richard/Jacky, they might know about this. > > > > > > This seems like something of a corner case to me, don't many imx > > > boards ship without PCI; especially for low-power scenarios? If > > > required it might be reasonable to add an additional workaround to > > > disable all cpuidle if pci msis are used. > > > > Establishing a link between cpuidle and PCI in the kernel would be > > pretty invasive, and that would come on top of what this series also > mandates. > > > > At that level of apparent brokenness, it is far safer to get cpuidle > > out of the picture altogether, and I'd rather see these patches in a vendor > tree (for once). > > > > Thanks, > > > > M. > > -- > > Jazz is not dead. It just smells funny...
On 3/28/19 3:45 AM, Lorenzo Pieralisi wrote: > On Wed, Mar 27, 2019 at 06:40:07PM +0000, Leonard Crestez wrote: >> On Wed, 2019-03-27 at 17:45 +0000, Marc Zyngier wrote: >>> On 27/03/2019 16:06, Lucas Stach wrote: >>>> Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: >>>>> On 27/03/2019 15:44, Lucas Stach wrote: >>>>>> Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: >>>>>>> This work is a workaround I'm looking into (more as a background task) >>>>>>> in order to add support for cpuidle on i.MX8MQ based platforms. >>>>>>> >>>>>>> The main idea here is getting around the missing GIC wake_request signal >>>>>>> (due to integration design issue) by waking up a each individual core through >>>>>>> some dedicated SW power-up bits inside the power controller (GPC) right before >>>>>>> every IPI is requested for that each individual core. >>>>>> >>>>>> Just a general comment, without going into the details of this series: >>>>>> this issue is not only affecting IPIs, but also MSIs terminated at the >>>>>> GIC. Currently MSIs are terminated at the PCIe core, but terminating >>>>>> them at the GIC is clearly preferable, as this allows assigning CPU >>>>>> affinity to individual MSIs and lowers IRQ service overhead. >>>>>> >>>>>> I'm not sure what the consequences are for upstream Linux support yet, >>>>>> but we should keep in mind that having a workaround for IPIs is only >>>>>> solving part of the issue. >>>>> >>>>> If this erratum is affecting more than just IPIs, then indeed I don't >>>>> see how this patch series solves anything. >>>>> >>>>> But the erratum documentation seems to imply that only SGIs are >>>>> affected, and goes as far as suggesting to use an external interrupt >>>>> would solve it. How comes this is not the case? Or is it that anything >>>>> directly routed to a redistributor is also affected? This would break >>>>> LPIs (and thus MSIs) and PPIs (the CPU timer, among others). >>>> >>>> Anything that isn't visible to the GPC and requires the GIC >>>> wake_request signal to behave as specified is broken by this erratum. >>> >>> I really wonder how a timer interrupt (a PPI, hence not routed through >>> the GPC) can wake up the CPU in this case. It really feels like >>> something like "program CNTV_CVAL_EL0 to expire at some later point; >>> WFI" could result in the CPU going to a deep sleep state, and not >>> wake-up at all. >> >> This is already a common issue for cpuidle implementions handled by the >> "local-timer-stop" property. imx has other timer blocks in the SOC, >> they generate SPIs which are connected to GPC. > > It is not a common issue. The tick-broadcast mechanism relies on > IPIs that are sent to specific CPUs upon timer expiry. > > If IPIs don't work for CPUs in shutdown state (which is what this patch > is fixing AFAIU), the only reason I can see how a CPU can resume from > idle on a timer expiry is the GPC waking up all cores upon the global > timer SPI; if that's the case there is precious little point in > implementing CPUidle at all - too bad people worked hard to implement > NOHZ in a power efficient manner. > >>> This would indicate that not only cpuidle is broken with this, but >>> absolutely every interrupt that is not routed through the GPC. >> >> Yes, cpuidle is broken for irqs not routed through GPC. However: >> >> * All SPIs are connected to GPC in a 1:1 mapping >> * This series deals with SGIs >> * The timer PPIs are not required; covered by local-timer-stop >> * LPIs are currently unused (I understand imx-pci uses SPI by default >> from Lucas) >> >> Anything missing? > > Yes, LPIs must be able to wake up CPUs and only the CPU for which > an IRQ is actually pending. > >>From an architectural perspective, an ARM core executing the WFI > instruction must resume execution upon an IRQ occurrence targeted > at it and that's true regardless of the idle state entered. > > Anything deviating from this behaviour is not architecture compliant. What if you enter a deeper state than WFI, which leads to the power gating of your CPU core, and you are missing the necessary hardware that should be driven from the GIC's nIRQOUT/nFIQOUT signals to automatically bring the core back on upon the GIC seeing a pending interrupt targeting that core? Would it be acceptable in that case to "help" the platform by ensuring that there is at least one core that is not allowed to enter the deepest idle state and be able to help wake back up the others? I am asking because I am facing a similar issue to what Abel is trying to solve here with ARCH_BRCMSTB platforms which do not have the ability to have their CPU cores wake-up on their once power gated. > >> My understanding is that this wake request feature via GIC is new in v3 >> and this is maybe why HW team missed it during integration. Older >> imx6/7 has GICv2 and has deep idle states which always rely on GPC to >> wakeup so the approach can work. > > If HW designers really wanted to have sensible power management policy > in this SoC they would have paid attention, I am against patching the > kernel heavily to fix a platform bug. HW designers may not be aware of how the cpuifle framework operates or what its constraints are, so they may not understand that any interrupt, must be able to autonomously (with lack of a better name) wake-up a given core, given any idle state it has entered.
On 06.11.2019 22:15, Florian Fainelli wrote: > On 3/28/19 3:45 AM, Lorenzo Pieralisi wrote: >> On Wed, Mar 27, 2019 at 06:40:07PM +0000, Leonard Crestez wrote: >>> On Wed, 2019-03-27 at 17:45 +0000, Marc Zyngier wrote: >>>> On 27/03/2019 16:06, Lucas Stach wrote: >>>>> Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: >>>>>> On 27/03/2019 15:44, Lucas Stach wrote: >>>>>>> Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: >>>>>>>> This work is a workaround I'm looking into (more as a background task) >>>>>>>> in order to add support for cpuidle on i.MX8MQ based platforms. >>>>>>>> >>>>>>>> The main idea here is getting around the missing GIC wake_request signal >>>>>>>> (due to integration design issue) by waking up a each individual core through >>>>>>>> some dedicated SW power-up bits inside the power controller (GPC) right before >>>>>>>> every IPI is requested for that each individual core. >>>>>>> >>>>>>> Just a general comment, without going into the details of this series: >>>>>>> this issue is not only affecting IPIs, but also MSIs terminated at the >>>>>>> GIC. Currently MSIs are terminated at the PCIe core, but terminating >>>>>>> them at the GIC is clearly preferable, as this allows assigning CPU >>>>>>> affinity to individual MSIs and lowers IRQ service overhead. >>>>>>> >>>>>>> I'm not sure what the consequences are for upstream Linux support yet, >>>>>>> but we should keep in mind that having a workaround for IPIs is only >>>>>>> solving part of the issue. >>>>>> >>>>>> If this erratum is affecting more than just IPIs, then indeed I don't >>>>>> see how this patch series solves anything. >>>>>> >>>>>> But the erratum documentation seems to imply that only SGIs are >>>>>> affected, and goes as far as suggesting to use an external interrupt >>>>>> would solve it. How comes this is not the case? Or is it that anything >>>>>> directly routed to a redistributor is also affected? This would break >>>>>> LPIs (and thus MSIs) and PPIs (the CPU timer, among others). >>>>> >>>>> Anything that isn't visible to the GPC and requires the GIC >>>>> wake_request signal to behave as specified is broken by this erratum. >>>> >>>> I really wonder how a timer interrupt (a PPI, hence not routed through >>>> the GPC) can wake up the CPU in this case. It really feels like >>>> something like "program CNTV_CVAL_EL0 to expire at some later point; >>>> WFI" could result in the CPU going to a deep sleep state, and not >>>> wake-up at all. >>> >>> This is already a common issue for cpuidle implementions handled by the >>> "local-timer-stop" property. imx has other timer blocks in the SOC, >>> they generate SPIs which are connected to GPC. >> >> It is not a common issue. The tick-broadcast mechanism relies on >> IPIs that are sent to specific CPUs upon timer expiry. >> >> If IPIs don't work for CPUs in shutdown state (which is what this patch >> is fixing AFAIU), the only reason I can see how a CPU can resume from >> idle on a timer expiry is the GPC waking up all cores upon the global >> timer SPI; if that's the case there is precious little point in >> implementing CPUidle at all - too bad people worked hard to implement >> NOHZ in a power efficient manner. >> >>>> This would indicate that not only cpuidle is broken with this, but >>>> absolutely every interrupt that is not routed through the GPC. >>> >>> Yes, cpuidle is broken for irqs not routed through GPC. However: >>> >>> * All SPIs are connected to GPC in a 1:1 mapping >>> * This series deals with SGIs >>> * The timer PPIs are not required; covered by local-timer-stop >>> * LPIs are currently unused (I understand imx-pci uses SPI by default >>> from Lucas) >>> >>> Anything missing? >> >> Yes, LPIs must be able to wake up CPUs and only the CPU for which >> an IRQ is actually pending. >> >> >From an architectural perspective, an ARM core executing the WFI >> instruction must resume execution upon an IRQ occurrence targeted >> at it and that's true regardless of the idle state entered. >> >> Anything deviating from this behaviour is not architecture compliant. > > What if you enter a deeper state than WFI, which leads to the power > gating of your CPU core, and you are missing the necessary hardware that > should be driven from the GIC's nIRQOUT/nFIQOUT signals to automatically > bring the core back on upon the GIC seeing a pending interrupt targeting > that core? imx8mq has a secondary "GPC" block which receives SPIs and can wake the cores. Do you have something similar? Because if you only have the GIC then that sounds much worse: you'd have to ensure that all peripheral interrupts are routed away from sleeping cores. On IMX only SGIs need special treatment and a newer version just replaces __smp_cross_call in a platform-specific manner: https://lkml.org/lkml/2019/6/10/350 > Would it be acceptable in that case to "help" the platform by ensuring > that there is at least one core that is not allowed to enter the deepest > idle state and be able to help wake back up the others? I am asking > because I am facing a similar issue to what Abel is trying to solve here > with ARCH_BRCMSTB platforms which do not have the ability to have their > CPU cores wake-up on their once power gated. Maybe you can workaround in ATF: if (last_core) wfi(); else powerdown(); But you still need special treatment for interrupts targeted at gated cores. >>> My understanding is that this wake request feature via GIC is new in v3 >>> and this is maybe why HW team missed it during integration. Older >>> imx6/7 has GICv2 and has deep idle states which always rely on GPC to >>> wakeup so the approach can work. >> >> If HW designers really wanted to have sensible power management policy >> in this SoC they would have paid attention, I am against patching the >> kernel heavily to fix a platform bug. > HW designers may not be aware of how the cpuifle framework operates or > what its constraints are, so they may not understand that any interrupt, > must be able to autonomously (with lack of a better name) wake-up a > given core, given any idle state it has entered. My understanding is that this is a requirement of GICv3 architecture. -- Regards, Leonard
On 11/6/19 1:31 PM, Leonard Crestez wrote: > On 06.11.2019 22:15, Florian Fainelli wrote: >> On 3/28/19 3:45 AM, Lorenzo Pieralisi wrote: >>> On Wed, Mar 27, 2019 at 06:40:07PM +0000, Leonard Crestez wrote: >>>> On Wed, 2019-03-27 at 17:45 +0000, Marc Zyngier wrote: >>>>> On 27/03/2019 16:06, Lucas Stach wrote: >>>>>> Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: >>>>>>> On 27/03/2019 15:44, Lucas Stach wrote: >>>>>>>> Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: >>>>>>>>> This work is a workaround I'm looking into (more as a background task) >>>>>>>>> in order to add support for cpuidle on i.MX8MQ based platforms. >>>>>>>>> >>>>>>>>> The main idea here is getting around the missing GIC wake_request signal >>>>>>>>> (due to integration design issue) by waking up a each individual core through >>>>>>>>> some dedicated SW power-up bits inside the power controller (GPC) right before >>>>>>>>> every IPI is requested for that each individual core. >>>>>>>> >>>>>>>> Just a general comment, without going into the details of this series: >>>>>>>> this issue is not only affecting IPIs, but also MSIs terminated at the >>>>>>>> GIC. Currently MSIs are terminated at the PCIe core, but terminating >>>>>>>> them at the GIC is clearly preferable, as this allows assigning CPU >>>>>>>> affinity to individual MSIs and lowers IRQ service overhead. >>>>>>>> >>>>>>>> I'm not sure what the consequences are for upstream Linux support yet, >>>>>>>> but we should keep in mind that having a workaround for IPIs is only >>>>>>>> solving part of the issue. >>>>>>> >>>>>>> If this erratum is affecting more than just IPIs, then indeed I don't >>>>>>> see how this patch series solves anything. >>>>>>> >>>>>>> But the erratum documentation seems to imply that only SGIs are >>>>>>> affected, and goes as far as suggesting to use an external interrupt >>>>>>> would solve it. How comes this is not the case? Or is it that anything >>>>>>> directly routed to a redistributor is also affected? This would break >>>>>>> LPIs (and thus MSIs) and PPIs (the CPU timer, among others). >>>>>> >>>>>> Anything that isn't visible to the GPC and requires the GIC >>>>>> wake_request signal to behave as specified is broken by this erratum. >>>>> >>>>> I really wonder how a timer interrupt (a PPI, hence not routed through >>>>> the GPC) can wake up the CPU in this case. It really feels like >>>>> something like "program CNTV_CVAL_EL0 to expire at some later point; >>>>> WFI" could result in the CPU going to a deep sleep state, and not >>>>> wake-up at all. >>>> >>>> This is already a common issue for cpuidle implementions handled by the >>>> "local-timer-stop" property. imx has other timer blocks in the SOC, >>>> they generate SPIs which are connected to GPC. >>> >>> It is not a common issue. The tick-broadcast mechanism relies on >>> IPIs that are sent to specific CPUs upon timer expiry. >>> >>> If IPIs don't work for CPUs in shutdown state (which is what this patch >>> is fixing AFAIU), the only reason I can see how a CPU can resume from >>> idle on a timer expiry is the GPC waking up all cores upon the global >>> timer SPI; if that's the case there is precious little point in >>> implementing CPUidle at all - too bad people worked hard to implement >>> NOHZ in a power efficient manner. >>> >>>>> This would indicate that not only cpuidle is broken with this, but >>>>> absolutely every interrupt that is not routed through the GPC. >>>> >>>> Yes, cpuidle is broken for irqs not routed through GPC. However: >>>> >>>> * All SPIs are connected to GPC in a 1:1 mapping >>>> * This series deals with SGIs >>>> * The timer PPIs are not required; covered by local-timer-stop >>>> * LPIs are currently unused (I understand imx-pci uses SPI by default >>>> from Lucas) >>>> >>>> Anything missing? >>> >>> Yes, LPIs must be able to wake up CPUs and only the CPU for which >>> an IRQ is actually pending. >>> >>> >From an architectural perspective, an ARM core executing the WFI >>> instruction must resume execution upon an IRQ occurrence targeted >>> at it and that's true regardless of the idle state entered. >>> >>> Anything deviating from this behaviour is not architecture compliant. >> >> What if you enter a deeper state than WFI, which leads to the power >> gating of your CPU core, and you are missing the necessary hardware that >> should be driven from the GIC's nIRQOUT/nFIQOUT signals to automatically >> bring the core back on upon the GIC seeing a pending interrupt targeting >> that core? > > imx8mq has a secondary "GPC" block which receives SPIs and can wake the > cores. Do you have something similar? Because if you only have the GIC > then that sounds much worse: you'd have to ensure that all peripheral > interrupts are routed away from sleeping cores. We have a legacy interrupt controller that receives all SPIs as well, and it can be used as a full replacement for the GIC (with the loss of nVIRQ/nFIQ) but it cannot wake-up the cores unfortunately. This is all custom logic, so we could have done at least wake-up based on SPIs, but we missed that apparently, at least we were consistent. Out of curiosity, does your GPC somehow know the affinity of a given interrupt to a particular core? > > On IMX only SGIs need special treatment and a newer version just > replaces __smp_cross_call in a platform-specific manner: > > https://lkml.org/lkml/2019/6/10/350 Right, because for PPIs you leverage the timer broadcast and for SPIs you have that GPC, so all your left are the remaining "intra GIC" interrupts which are SGIs. > >> Would it be acceptable in that case to "help" the platform by ensuring >> that there is at least one core that is not allowed to enter the deepest >> idle state and be able to help wake back up the others? I am asking >> because I am facing a similar issue to what Abel is trying to solve here >> with ARCH_BRCMSTB platforms which do not have the ability to have their >> CPU cores wake-up on their once power gated. > > Maybe you can workaround in ATF: if (last_core) wfi(); else powerdown(); Yes, that would certainly work, the biggest problem in my case is dealing with SPIs, since we still have no way to wake-up from those, other than by getting the help of another CPU that is not power gated. Lovely, I know. > > But you still need special treatment for interrupts targeted at gated cores. > >>>> My understanding is that this wake request feature via GIC is new in v3 >>>> and this is maybe why HW team missed it during integration. Older >>>> imx6/7 has GICv2 and has deep idle states which always rely on GPC to >>>> wakeup so the approach can work. >>> >>> If HW designers really wanted to have sensible power management policy >>> in this SoC they would have paid attention, I am against patching the >>> kernel heavily to fix a platform bug. > >> HW designers may not be aware of how the cpuifle framework operates or >> what its constraints are, so they may not understand that any interrupt, >> must be able to autonomously (with lack of a better name) wake-up a >> given core, given any idle state it has entered. > > My understanding is that this is a requirement of GICv3 architecture. > The systems I use have a GICv2 architecture though this is still no excuse for not having hooked the nIRQOUT/nFIQOUT to a power management controller, this is clearly an oversight, and it should have been possible to automatically take a core out of power gating, since we did design our own power gating logic, but this was done that way. Hopefully future designs can remedy that, designers are aware of why this is a problem now. -- Florian
On 07.11.2019 00:10, Florian Fainelli wrote: > On 11/6/19 1:31 PM, Leonard Crestez wrote: >> On 06.11.2019 22:15, Florian Fainelli wrote: >>> On 3/28/19 3:45 AM, Lorenzo Pieralisi wrote: >>>> On Wed, Mar 27, 2019 at 06:40:07PM +0000, Leonard Crestez wrote: >>>>> On Wed, 2019-03-27 at 17:45 +0000, Marc Zyngier wrote: >>>>>> On 27/03/2019 16:06, Lucas Stach wrote: >>>>>>> Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: >>>>>>>> On 27/03/2019 15:44, Lucas Stach wrote: >>>>>>>>> Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: >>>>>>>>>> This work is a workaround I'm looking into (more as a background task) >>>>>>>>>> in order to add support for cpuidle on i.MX8MQ based platforms. >>>>>>>>>> >>>>>>>>>> The main idea here is getting around the missing GIC wake_request signal >>>>>>>>>> (due to integration design issue) by waking up a each individual core through >>>>>>>>>> some dedicated SW power-up bits inside the power controller (GPC) right before >>>>>>>>>> every IPI is requested for that each individual core. >>>>>>>>> >>>>>>>>> Just a general comment, without going into the details of this series: >>>>>>>>> this issue is not only affecting IPIs, but also MSIs terminated at the >>>>>>>>> GIC. Currently MSIs are terminated at the PCIe core, but terminating >>>>>>>>> them at the GIC is clearly preferable, as this allows assigning CPU >>>>>>>>> affinity to individual MSIs and lowers IRQ service overhead. >>>>>>>>> >>>>>>>>> I'm not sure what the consequences are for upstream Linux support yet, >>>>>>>>> but we should keep in mind that having a workaround for IPIs is only >>>>>>>>> solving part of the issue. >>>>>>>> >>>>>>>> If this erratum is affecting more than just IPIs, then indeed I don't >>>>>>>> see how this patch series solves anything. >>>>>>>> >>>>>>>> But the erratum documentation seems to imply that only SGIs are >>>>>>>> affected, and goes as far as suggesting to use an external interrupt >>>>>>>> would solve it. How comes this is not the case? Or is it that anything >>>>>>>> directly routed to a redistributor is also affected? This would break >>>>>>>> LPIs (and thus MSIs) and PPIs (the CPU timer, among others). >>>>>>> >>>>>>> Anything that isn't visible to the GPC and requires the GIC >>>>>>> wake_request signal to behave as specified is broken by this erratum. >>>>>> >>>>>> I really wonder how a timer interrupt (a PPI, hence not routed through >>>>>> the GPC) can wake up the CPU in this case. It really feels like >>>>>> something like "program CNTV_CVAL_EL0 to expire at some later point; >>>>>> WFI" could result in the CPU going to a deep sleep state, and not >>>>>> wake-up at all. >>>>> >>>>> This is already a common issue for cpuidle implementions handled by the >>>>> "local-timer-stop" property. imx has other timer blocks in the SOC, >>>>> they generate SPIs which are connected to GPC. >>>> >>>> It is not a common issue. The tick-broadcast mechanism relies on >>>> IPIs that are sent to specific CPUs upon timer expiry. >>>> >>>> If IPIs don't work for CPUs in shutdown state (which is what this patch >>>> is fixing AFAIU), the only reason I can see how a CPU can resume from >>>> idle on a timer expiry is the GPC waking up all cores upon the global >>>> timer SPI; if that's the case there is precious little point in >>>> implementing CPUidle at all - too bad people worked hard to implement >>>> NOHZ in a power efficient manner. >>>> >>>>>> This would indicate that not only cpuidle is broken with this, but >>>>>> absolutely every interrupt that is not routed through the GPC. >>>>> >>>>> Yes, cpuidle is broken for irqs not routed through GPC. However: >>>>> >>>>> * All SPIs are connected to GPC in a 1:1 mapping >>>>> * This series deals with SGIs >>>>> * The timer PPIs are not required; covered by local-timer-stop >>>>> * LPIs are currently unused (I understand imx-pci uses SPI by default >>>>> from Lucas) >>>>> >>>>> Anything missing? >>>> >>>> Yes, LPIs must be able to wake up CPUs and only the CPU for which >>>> an IRQ is actually pending. >>>> >>>> >From an architectural perspective, an ARM core executing the WFI >>>> instruction must resume execution upon an IRQ occurrence targeted >>>> at it and that's true regardless of the idle state entered. >>>> >>>> Anything deviating from this behaviour is not architecture compliant. >>> >>> What if you enter a deeper state than WFI, which leads to the power >>> gating of your CPU core, and you are missing the necessary hardware that >>> should be driven from the GIC's nIRQOUT/nFIQOUT signals to automatically >>> bring the core back on upon the GIC seeing a pending interrupt targeting >>> that core? >> >> imx8mq has a secondary "GPC" block which receives SPIs and can wake the >> cores. Do you have something similar? Because if you only have the GIC >> then that sounds much worse: you'd have to ensure that all peripheral >> interrupts are routed away from sleeping cores. > > We have a legacy interrupt controller that receives all SPIs as well, > and it can be used as a full replacement for the GIC (with the loss of > nVIRQ/nFIQ) but it cannot wake-up the cores unfortunately. This is all > custom logic, so we could have done at least wake-up based on SPIs, but > we missed that apparently, at least we were consistent. > > Out of curiosity, does your GPC somehow know the affinity of a given > interrupt to a particular core? Yes, if it's told by software. There are mask and status registers for each SPI for each core in GPC but AFAIK GIC bits are unrelated. >> On IMX only SGIs need special treatment and a newer version just >> replaces __smp_cross_call in a platform-specific manner: >> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml.org%2Flkml%2F2019%2F6%2F10%2F350&data=02%7C01%7Cleonard.crestez%40nxp.com%7Ce019c8afbfff487ef72208d7630629f3%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C1%7C637086750468897744&sdata=SPCLEZJU5bpTrs8vQNQ7CuBWmlF8f3uyPaNUB%2F%2BAm%2Fs%3D&reserved=0 > > Right, because for PPIs you leverage the timer broadcast and for SPIs > you have that GPC, so all your left are the remaining "intra GIC" > interrupts which are SGIs. > >>> Would it be acceptable in that case to "help" the platform by ensuring >>> that there is at least one core that is not allowed to enter the deepest >>> idle state and be able to help wake back up the others? I am asking >>> because I am facing a similar issue to what Abel is trying to solve here >>> with ARCH_BRCMSTB platforms which do not have the ability to have their >>> CPU cores wake-up on their once power gated. >> >> Maybe you can workaround in ATF: if (last_core) wfi(); else powerdown(); > > Yes, that would certainly work, the biggest problem in my case is > dealing with SPIs, since we still have no way to wake-up from those, > other than by getting the help of another CPU that is not power gated. > Lovely, I know. By default irqs are only routed to core0 so maybe you could only power down if your core has no irqs enabled? It might even be possible to do this by reading GIC registers in ATF but this might race with other GIC manipulation from kernel. Perhaps your workarounds could also be encapsulated into a platform-specific irqchip implementation which occasionally pokes at ATF. >> But you still need special treatment for interrupts targeted at gated cores. >> >>>>> My understanding is that this wake request feature via GIC is new in v3 >>>>> and this is maybe why HW team missed it during integration. Older >>>>> imx6/7 has GICv2 and has deep idle states which always rely on GPC to >>>>> wakeup so the approach can work. >>>> >>>> If HW designers really wanted to have sensible power management policy >>>> in this SoC they would have paid attention, I am against patching the >>>> kernel heavily to fix a platform bug. >> >>> HW designers may not be aware of how the cpuifle framework operates or >>> what its constraints are, so they may not understand that any interrupt, >>> must be able to autonomously (with lack of a better name) wake-up a >>> given core, given any idle state it has entered. >> >> My understanding is that this is a requirement of GICv3 architecture. >> > > The systems I use have a GICv2 architecture though this is still no > excuse for not having hooked the nIRQOUT/nFIQOUT to a power management > controller, this is clearly an oversight, and it should have been > possible to automatically take a core out of power gating, since we did > design our own power gating logic, but this was done that way. Hopefully > future designs can remedy that, designers are aware of why this is a > problem now. > -- > Florian >