Message ID | 20250317-kvm_exit_fix-v1-1-aa5240c5dbd2@rivosinc.com (mailing list archive) |
---|---|
State | New |
Headers | show |
Series | RISC-V: KVM: Teardown riscv specific bits after kvm_exit | expand |
On Mon, Mar 17, 2025, Atish Patra wrote: > During a module removal, kvm_exit invokes arch specific disable > call which disables AIA. However, we invoke aia_exit before kvm_exit > resulting in the following warning. KVM kernel module can't be inserted > afterwards due to inconsistent state of IRQ. > > [25469.031389] percpu IRQ 31 still enabled on CPU0! > [25469.031732] WARNING: CPU: 3 PID: 943 at kernel/irq/manage.c:2476 __free_percpu_irq+0xa2/0x150 > [25469.031804] Modules linked in: kvm(-) > [25469.031848] CPU: 3 UID: 0 PID: 943 Comm: rmmod Not tainted 6.14.0-rc5-06947-g91c763118f47-dirty #2 > [25469.031905] Hardware name: riscv-virtio,qemu (DT) > [25469.031928] epc : __free_percpu_irq+0xa2/0x150 > [25469.031976] ra : __free_percpu_irq+0xa2/0x150 > [25469.032197] epc : ffffffff8007db1e ra : ffffffff8007db1e sp : ff2000000088bd50 > [25469.032241] gp : ffffffff8131cef8 tp : ff60000080b96400 t0 : ff2000000088baf8 > [25469.032285] t1 : fffffffffffffffc t2 : 5249207570637265 s0 : ff2000000088bd90 > [25469.032329] s1 : ff60000098b21080 a0 : 037d527a15eb4f00 a1 : 037d527a15eb4f00 > [25469.032372] a2 : 0000000000000023 a3 : 0000000000000001 a4 : ffffffff8122dbf8 > [25469.032410] a5 : 0000000000000fff a6 : 0000000000000000 a7 : ffffffff8122dc10 > [25469.032448] s2 : ff60000080c22eb0 s3 : 0000000200000022 s4 : 000000000000001f > [25469.032488] s5 : ff60000080c22e00 s6 : ffffffff80c351c0 s7 : 0000000000000000 > [25469.032582] s8 : 0000000000000003 s9 : 000055556b7fb490 s10: 00007ffff0e12fa0 > [25469.032621] s11: 00007ffff0e13e9a t3 : ffffffff81354ac7 t4 : ffffffff81354ac7 > [25469.032664] t5 : ffffffff81354ac8 t6 : ffffffff81354ac7 > [25469.032698] status: 0000000200000100 badaddr: ffffffff8007db1e cause: 0000000000000003 > [25469.032738] [<ffffffff8007db1e>] __free_percpu_irq+0xa2/0x150 > [25469.032797] [<ffffffff8007dbfc>] free_percpu_irq+0x30/0x5e > [25469.032856] [<ffffffff013a57dc>] kvm_riscv_aia_exit+0x40/0x42 [kvm] > [25469.033947] [<ffffffff013b4e82>] cleanup_module+0x10/0x32 [kvm] > [25469.035300] [<ffffffff8009b150>] __riscv_sys_delete_module+0x18e/0x1fc > [25469.035374] [<ffffffff8000c1ca>] syscall_handler+0x3a/0x46 > [25469.035456] [<ffffffff809ec9a4>] do_trap_ecall_u+0x72/0x134 > [25469.035536] [<ffffffff809f5e18>] handle_exception+0x148/0x156 > > Invoke aia_exit and other arch specific cleanup functions after kvm_exit > so that disable gets a chance to be called first before exit. > > Fixes: 54e43320c2ba ("RISC-V: KVM: Initial skeletal support for AIA") > > Signed-off-by: Atish Patra <atishp@rivosinc.com> > --- FWIW, Reviewed-by: Sean Christopherson <seanjc@google.com> > arch/riscv/kvm/main.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c > index 1fa8be5ee509..4b24705dc63a 100644 > --- a/arch/riscv/kvm/main.c > +++ b/arch/riscv/kvm/main.c > @@ -172,8 +172,8 @@ module_init(riscv_kvm_init); > > static void __exit riscv_kvm_exit(void) > { > - kvm_riscv_teardown(); > - > kvm_exit(); > + > + kvm_riscv_teardown(); I wonder if there's a way we can guard against kvm_init()/kvm_exit() being called too early/late. x86 had similar bugs for a very long time, e.g. see commit e32b120071ea ("KVM: VMX: Do _all_ initialization before exposing /dev/kvm to userspace"). E.g. maybe we do something like create+destroy a VM at the end of kvm_init() and the beginning of kvm_exit()? Not sure if that would work for kvm_exit(), but it should definitely be fine for kvm_init(). It wouldn't prevent bugs, but maybe it would help detect them during development?
diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c index 1fa8be5ee509..4b24705dc63a 100644 --- a/arch/riscv/kvm/main.c +++ b/arch/riscv/kvm/main.c @@ -172,8 +172,8 @@ module_init(riscv_kvm_init); static void __exit riscv_kvm_exit(void) { - kvm_riscv_teardown(); - kvm_exit(); + + kvm_riscv_teardown(); } module_exit(riscv_kvm_exit);
During a module removal, kvm_exit invokes arch specific disable call which disables AIA. However, we invoke aia_exit before kvm_exit resulting in the following warning. KVM kernel module can't be inserted afterwards due to inconsistent state of IRQ. [25469.031389] percpu IRQ 31 still enabled on CPU0! [25469.031732] WARNING: CPU: 3 PID: 943 at kernel/irq/manage.c:2476 __free_percpu_irq+0xa2/0x150 [25469.031804] Modules linked in: kvm(-) [25469.031848] CPU: 3 UID: 0 PID: 943 Comm: rmmod Not tainted 6.14.0-rc5-06947-g91c763118f47-dirty #2 [25469.031905] Hardware name: riscv-virtio,qemu (DT) [25469.031928] epc : __free_percpu_irq+0xa2/0x150 [25469.031976] ra : __free_percpu_irq+0xa2/0x150 [25469.032197] epc : ffffffff8007db1e ra : ffffffff8007db1e sp : ff2000000088bd50 [25469.032241] gp : ffffffff8131cef8 tp : ff60000080b96400 t0 : ff2000000088baf8 [25469.032285] t1 : fffffffffffffffc t2 : 5249207570637265 s0 : ff2000000088bd90 [25469.032329] s1 : ff60000098b21080 a0 : 037d527a15eb4f00 a1 : 037d527a15eb4f00 [25469.032372] a2 : 0000000000000023 a3 : 0000000000000001 a4 : ffffffff8122dbf8 [25469.032410] a5 : 0000000000000fff a6 : 0000000000000000 a7 : ffffffff8122dc10 [25469.032448] s2 : ff60000080c22eb0 s3 : 0000000200000022 s4 : 000000000000001f [25469.032488] s5 : ff60000080c22e00 s6 : ffffffff80c351c0 s7 : 0000000000000000 [25469.032582] s8 : 0000000000000003 s9 : 000055556b7fb490 s10: 00007ffff0e12fa0 [25469.032621] s11: 00007ffff0e13e9a t3 : ffffffff81354ac7 t4 : ffffffff81354ac7 [25469.032664] t5 : ffffffff81354ac8 t6 : ffffffff81354ac7 [25469.032698] status: 0000000200000100 badaddr: ffffffff8007db1e cause: 0000000000000003 [25469.032738] [<ffffffff8007db1e>] __free_percpu_irq+0xa2/0x150 [25469.032797] [<ffffffff8007dbfc>] free_percpu_irq+0x30/0x5e [25469.032856] [<ffffffff013a57dc>] kvm_riscv_aia_exit+0x40/0x42 [kvm] [25469.033947] [<ffffffff013b4e82>] cleanup_module+0x10/0x32 [kvm] [25469.035300] [<ffffffff8009b150>] __riscv_sys_delete_module+0x18e/0x1fc [25469.035374] [<ffffffff8000c1ca>] syscall_handler+0x3a/0x46 [25469.035456] [<ffffffff809ec9a4>] do_trap_ecall_u+0x72/0x134 [25469.035536] [<ffffffff809f5e18>] handle_exception+0x148/0x156 Invoke aia_exit and other arch specific cleanup functions after kvm_exit so that disable gets a chance to be called first before exit. Fixes: 54e43320c2ba ("RISC-V: KVM: Initial skeletal support for AIA") Signed-off-by: Atish Patra <atishp@rivosinc.com> --- arch/riscv/kvm/main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) --- base-commit: 4701f33a10702d5fc577c32434eb62adde0a1ae1 change-id: 20250316-kvm_exit_fix-77cd0632d740 -- Regards, Atish patra