| Message ID | 20220216090845.1278114-1-maz@kernel.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | net: mvpp2: Survive CPU hotplug events | expand |
Hi Marc, śr., 16 lut 2022 o 10:08 Marc Zyngier <maz@kernel.org> napisał(a): > > I recently realised that playing with CPU hotplug on a system equiped > with a set of MVPP2 devices (Marvell 8040) was fraught with danger and > would result in a rapid lockup or panic. > > As it turns out, the per-CPU nature of the MVPP2 interrupts are > getting in the way. A good solution for this seems to rely on the > kernel's managed interrupt approach, where the core kernel will not > move interrupts around as the CPUs for down, but will simply disable > the corresponding interrupt. > > Converting the driver to this requires a bit of refactoring in the IRQ > subsystem to expose the required primitive, as well as a bit of > surgery in the driver itself. > > Note that although the system now survives such event, the driver > seems to assume that all queues are always active and doesn't inform > the device that a CPU has gone away. Someout who actually understand > this driver should have a look at it. > > Patches on top of 5.17-rc3, lightly tested on a McBin. > Thank you for the patches. Can you, please, share the commands you used? I'd like to test it more. Best regards, Marcin > Marc Zyngier (2): > genirq: Extract irq_set_affinity_masks() from > devm_platform_get_irqs_affinity() > net: mvpp2: Convert to managed interrupts to fix CPU HP issues > > drivers/base/platform.c | 20 +----- > drivers/net/ethernet/marvell/mvpp2/mvpp2.h | 1 - > .../net/ethernet/marvell/mvpp2/mvpp2_main.c | 67 ++++++++++--------- > include/linux/interrupt.h | 8 +++ > kernel/irq/affinity.c | 27 ++++++++ > 5 files changed, 72 insertions(+), 51 deletions(-) > > -- > 2.30.2 >
On Wed, 16 Feb 2022 13:19:30 +0000, Marcin Wojtas <mw@semihalf.com> wrote: > > Hi Marc, > > śr., 16 lut 2022 o 10:08 Marc Zyngier <maz@kernel.org> napisał(a): > > > > I recently realised that playing with CPU hotplug on a system equiped > > with a set of MVPP2 devices (Marvell 8040) was fraught with danger and > > would result in a rapid lockup or panic. > > > > As it turns out, the per-CPU nature of the MVPP2 interrupts are > > getting in the way. A good solution for this seems to rely on the > > kernel's managed interrupt approach, where the core kernel will not > > move interrupts around as the CPUs for down, but will simply disable > > the corresponding interrupt. > > > > Converting the driver to this requires a bit of refactoring in the IRQ > > subsystem to expose the required primitive, as well as a bit of > > surgery in the driver itself. > > > > Note that although the system now survives such event, the driver > > seems to assume that all queues are always active and doesn't inform > > the device that a CPU has gone away. Someout who actually understand > > this driver should have a look at it. > > > > Patches on top of 5.17-rc3, lightly tested on a McBin. > > > > Thank you for the patches. Can you, please, share the commands you > used? I'd like to test it more. Offline CPU3: # echo 0 > /sys/devices/system/cpu/cpu3/online Online CPU3: # echo 1 > /sys/devices/system/cpu/cpu3/online Put that in a loop, using different CPUs. On my HW, turning off CPU0 leads to odd behaviours (I wouldn't be surprised if the firmware was broken in that respect, and also the fact that the device keeps trying to send stuff to that CPU...). M.
śr., 16 lut 2022 o 14:29 Marc Zyngier <maz@kernel.org> napisał(a): > > On Wed, 16 Feb 2022 13:19:30 +0000, > Marcin Wojtas <mw@semihalf.com> wrote: > > > > Hi Marc, > > > > śr., 16 lut 2022 o 10:08 Marc Zyngier <maz@kernel.org> napisał(a): > > > > > > I recently realised that playing with CPU hotplug on a system equiped > > > with a set of MVPP2 devices (Marvell 8040) was fraught with danger and > > > would result in a rapid lockup or panic. > > > > > > As it turns out, the per-CPU nature of the MVPP2 interrupts are > > > getting in the way. A good solution for this seems to rely on the > > > kernel's managed interrupt approach, where the core kernel will not > > > move interrupts around as the CPUs for down, but will simply disable > > > the corresponding interrupt. > > > > > > Converting the driver to this requires a bit of refactoring in the IRQ > > > subsystem to expose the required primitive, as well as a bit of > > > surgery in the driver itself. > > > > > > Note that although the system now survives such event, the driver > > > seems to assume that all queues are always active and doesn't inform > > > the device that a CPU has gone away. Someout who actually understand > > > this driver should have a look at it. > > > > > > Patches on top of 5.17-rc3, lightly tested on a McBin. > > > > > > > Thank you for the patches. Can you, please, share the commands you > > used? I'd like to test it more. > > Offline CPU3: > # echo 0 > /sys/devices/system/cpu/cpu3/online > > Online CPU3: > # echo 1 > /sys/devices/system/cpu/cpu3/online > > Put that in a loop, using different CPUs. > > On my HW, turning off CPU0 leads to odd behaviours (I wouldn't be > surprised if the firmware was broken in that respect, and also the > fact that the device keeps trying to send stuff to that CPU...). > Thanks, I think stressing DUT with traffic during CPU hotplug will be a good scenario - I'll try that. Marcin
I recently realised that playing with CPU hotplug on a system equiped with a set of MVPP2 devices (Marvell 8040) was fraught with danger and would result in a rapid lockup or panic. As it turns out, the per-CPU nature of the MVPP2 interrupts are getting in the way. A good solution for this seems to rely on the kernel's managed interrupt approach, where the core kernel will not move interrupts around as the CPUs for down, but will simply disable the corresponding interrupt. Converting the driver to this requires a bit of refactoring in the IRQ subsystem to expose the required primitive, as well as a bit of surgery in the driver itself. Note that although the system now survives such event, the driver seems to assume that all queues are always active and doesn't inform the device that a CPU has gone away. Someout who actually understand this driver should have a look at it. Patches on top of 5.17-rc3, lightly tested on a McBin. Marc Zyngier (2): genirq: Extract irq_set_affinity_masks() from devm_platform_get_irqs_affinity() net: mvpp2: Convert to managed interrupts to fix CPU HP issues drivers/base/platform.c | 20 +----- drivers/net/ethernet/marvell/mvpp2/mvpp2.h | 1 - .../net/ethernet/marvell/mvpp2/mvpp2_main.c | 67 ++++++++++--------- include/linux/interrupt.h | 8 +++ kernel/irq/affinity.c | 27 ++++++++ 5 files changed, 72 insertions(+), 51 deletions(-)