[kvm-unit-tests] arch-run: Fix run_qemu return correct error code

Commit Message

任敏敏(联通集团联通数字科技有限公司本部) April 23, 2023, 4:34 a.m. UTC

From: rminmin <[e-mail]>

run_qemu should return 0 if logs doesn't
contain "warning" keyword.

Fixes: b2a2aa5d ("arch-run: reduce return code ambiguity")
Signed-off-by: rminmin <[e-mail]>
---
 scripts/arch-run.bash | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--
2.33.0

Èç¹ûÄú´íÎó½ÓÊÕÁ˸ÃÓʼþ£¬Çëͨ¹ýµç×ÓÓʼþÁ¢¼´Í¨ÖªÎÒÃÇ¡£Çë»Ø¸´Óʼþµ½ [e-mail]£¬¼´¿ÉÒÔÍ˶©´ËÓʼþ¡£ÎÒÃǽ«Á¢¼´½«ÄúµÄÐÅÏ¢´ÓÎÒÃǵķ¢ËÍĿ¼ÖÐɾ³ý¡£ If you have received this email in error please notify us immediately by e-mail. Please reply to [e-mail] ,you can unsubscribe from this mail. We will immediately remove your information from send catalogue of our.

Comments

Andrew Jones April 24, 2023, 7:48 a.m. UTC | #1

On Sun, Apr 23, 2023 at 12:34:36PM +0800, 任敏敏(联通集团联通数字科技有限公司本部) wrote:
> From: rminmin <[e-mail]>
> 
> run_qemu should return 0 if logs doesn't
> contain "warning" keyword.

Why? What are you trying to fix?

> 
> Fixes: b2a2aa5d ("arch-run: reduce return code ambiguity")
> Signed-off-by: rminmin <[e-mail]>
> ---
>  scripts/arch-run.bash | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/arch-run.bash b/scripts/arch-run.bash
> index 51e4b97..9878d32 100644
> --- a/scripts/arch-run.bash
> +++ b/scripts/arch-run.bash
> @@ -61,7 +61,7 @@ run_qemu ()
>                 # Even when ret==1 (unittest success) if we also got stderr
>                 # logs, then we assume a QEMU failure. Otherwise we translate
>                 # status of 1 to 0 (SUCCESS)
> -               if [ -z "$(echo "$errors" | grep -vi warning)" ]; then
> +               if [ -z "$(echo "$errors" | grep -i warning)" ]; then

This will now filter out all the errors, leaving only warnings or nothing.
If you want the check to include warnings, then it should be

 if [ -z "$(echo "$errors")" ]

>                         ret=0
>                 fi
>         fi
> --
> 2.33.0
>

Thanks,
drew

任敏敏(联通集团联通数字科技有限公司本部) April 24, 2023, 9:58 a.m. UTC | #2

>On Sun, Apr 23, 2023 at 12:34:36PM +0800, 任敏敏(联通集团联通数字科技有限公司本部) wrote:
>> From: rminmin <[e-mail]>
>>
>> run_qemu should return 0 if logs doesn't
>> contain "warning" keyword.
>
>Why? What are you trying to fix?
>

I encountered a problem that the differet results when I run the same
test case using standalone mode, "run_test.sh -t -g", "run_test.sh -g".
When I use "run_test.sh -g", it always returns a FAIL message.
In my env, qemu version is 6.2.

e.g. runnig debug test cases.

# without "-t"
./run_tests.sh -g debug
FAIL debug (22 tests)

# "-t"
./run_tests.sh -t -g debug
TAP version 13
ok 1 - debug: DR4==DR6 with CR4.DE == 0
ok 2 - debug: DR4 read got #UD with CR4.DE == 1
ok 3 - debug: #BP
ok 4 - debug: hw breakpoint (test that dr6.BS is not set)
ok 5 - debug: hw breakpoint (test that dr6.BS is not cleared)
ok 6 - debug: Single-step #DB basic test
ok 7 - debug: Usermode Single-step #DB basic test
ok 8 - debug: Single-step #DB on emulated instructions
ok 9 - debug: Usermode Single-step #DB on emulated instructions
ok 10 - debug: Single-step #DB w/ STI blocking
ok 11 - debug: Usermode Single-step #DB w/ STI blocking
ok 12 - debug: Single-step #DB w/ MOVSS blocking
ok 13 - debug: Usermode Single-step #DB w/ MOVSS blocking
ok 14 - debug: Single-Step + ICEBP #DB w/ MOVSS blocking
ok 15 - debug: Usermode Single-Step + ICEBP #DB w/ MOVSS blocking
ok 16 - debug: Single-step #DB w/ MOVSS blocking and DR7.GD=1
ok 17 - debug: hw watchpoint (test that dr6.BS is not cleared)
ok 18 - debug: hw watchpoint (test that dr6.BS is not set)
ok 19 - debug: icebp
ok 20 - debug: MOV SS + watchpoint + ICEBP
ok 21 - debug: MOV SS + watchpoint + int $1
ok 22 - debug: MOV SS + watchpoint + INT3
1..22

# standalone

tests/debug
BUILD_HEAD=02d8befe
timeout -k 1s --foreground 90s /usr/bin/qemu-kvm --no-reboot -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.1GdMZXhTTs -smp 1 # -initrd /tmp/tmp.LzHnmXchfO
configure accelerator pc-i440fx-6.2 start
machine init start
device init start
add qdev pc-testdev:none success
add qdev pc-testdev:none success
add qdev isa-debug-exit:none success
add qdev isa-debug-exit:none success
add qdev pci-testdev:none success
add qdev pci-testdev:none success
reset all devices
qmp cont is received and vm is started
qemu enter main_loop
enabling apic
smp: waiting for 0 APs
PASS: DR4==DR6 with CR4.DE == 0
PASS: DR4 read got #UD with CR4.DE == 1
PASS: #BP
PASS: hw breakpoint (test that dr6.BS is not set)
PASS: hw breakpoint (test that dr6.BS is not cleared)
PASS: Single-step #DB basic test
PASS: Usermode Single-step #DB basic test
PASS: Single-step #DB on emulated instructions
PASS: Usermode Single-step #DB on emulated instructions
PASS: Single-step #DB w/ STI blocking
PASS: Usermode Single-step #DB w/ STI blocking
PASS: Single-step #DB w/ MOVSS blocking
PASS: Usermode Single-step #DB w/ MOVSS blocking
PASS: Single-Step + ICEBP #DB w/ MOVSS blocking
PASS: Usermode Single-Step + ICEBP #DB w/ MOVSS blocking
PASS: Single-step #DB w/ MOVSS blocking and DR7.GD=1
PASS: hw watchpoint (test that dr6.BS is not cleared)
PASS: hw watchpoint (test that dr6.BS is not set)
PASS: icebp
PASS: MOV SS + watchpoint + ICEBP
PASS: MOV SS + watchpoint + int $1
PASS: MOV SS + watchpoint + INT3
SUMMARY: 22 tests
FAIL debug (22 tests)


>>
>> Fixes: b2a2aa5d ("arch-run: reduce return code ambiguity")
>> Signed-off-by: rminmin <[e-mail]>
>> ---
>>  scripts/arch-run.bash | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/scripts/arch-run.bash b/scripts/arch-run.bash
>> index 51e4b97..9878d32 100644
>> --- a/scripts/arch-run.bash
>> +++ b/scripts/arch-run.bash
>> @@ -61,7 +61,7 @@ run_qemu ()
>>                 # Even when ret==1 (unittest success) if we also got stderr
>>                 # logs, then we assume a QEMU failure. Otherwise we translate
>>                 # status of 1 to 0 (SUCCESS)
>> -               if [ -z "$(echo "$errors" | grep -vi warning)" ]; then
>> +               if [ -z "$(echo "$errors" | grep -i warning)" ]; then
>
>This will now filter out all the errors, leaving only warnings or nothing.
>If you want the check to include warnings, then it should be
>
> if [ -z "$(echo "$errors")" ]
>

I found the root case is that "$(echo "$errors" | grep -vi warning)"
is always not zero, because the $errors include qemu log(
configure accelerator pc-i440fx-6.2 start .......) without any
errors or warning keywords.

Did I misunderstand something or miss some information?

>>                         ret=0
>>                 fi
>>         fi
>> --
>> 2.33.0
>>
>
>Thanks,
>drew

如果您错误接收了该邮件，请通过电子邮件立即通知我们。请回复邮件到 [e-mail]，即可以退订此邮件。我们将立即将您的信息从我们的发送目录中删除。 If you have received this email in error please notify us immediately by e-mail. Please reply to [e-mail] ,you can unsubscribe from this mail. We will immediately remove your information from send catalogue of our.

Andrew Jones April 25, 2023, 8:38 a.m. UTC | #3

On Mon, Apr 24, 2023 at 05:58:16PM +0800, 任敏敏(联通集团联通数字科技有限公司本部) wrote:
> >On Sun, Apr 23, 2023 at 12:34:36PM +0800, 任敏敏(联通集团联通数字科技有限公司本部) wrote:
> >> From: rminmin <[e-mail]>
> >>
> >> run_qemu should return 0 if logs doesn't
> >> contain "warning" keyword.
> >
> >Why? What are you trying to fix?
> >
> 
> I encountered a problem that the differet results when I run the same
> test case using standalone mode, "run_test.sh -t -g", "run_test.sh -g".
> When I use "run_test.sh -g", it always returns a FAIL message.
> In my env, qemu version is 6.2.
> 
> e.g. runnig debug test cases.
> 
> # without "-t"
> ./run_tests.sh -g debug
> FAIL debug (22 tests)
> 
> # "-t"
> ./run_tests.sh -t -g debug
> TAP version 13
> ok 1 - debug: DR4==DR6 with CR4.DE == 0
> ok 2 - debug: DR4 read got #UD with CR4.DE == 1
> ok 3 - debug: #BP
> ok 4 - debug: hw breakpoint (test that dr6.BS is not set)
> ok 5 - debug: hw breakpoint (test that dr6.BS is not cleared)
> ok 6 - debug: Single-step #DB basic test
> ok 7 - debug: Usermode Single-step #DB basic test
> ok 8 - debug: Single-step #DB on emulated instructions
> ok 9 - debug: Usermode Single-step #DB on emulated instructions
> ok 10 - debug: Single-step #DB w/ STI blocking
> ok 11 - debug: Usermode Single-step #DB w/ STI blocking
> ok 12 - debug: Single-step #DB w/ MOVSS blocking
> ok 13 - debug: Usermode Single-step #DB w/ MOVSS blocking
> ok 14 - debug: Single-Step + ICEBP #DB w/ MOVSS blocking
> ok 15 - debug: Usermode Single-Step + ICEBP #DB w/ MOVSS blocking
> ok 16 - debug: Single-step #DB w/ MOVSS blocking and DR7.GD=1
> ok 17 - debug: hw watchpoint (test that dr6.BS is not cleared)
> ok 18 - debug: hw watchpoint (test that dr6.BS is not set)
> ok 19 - debug: icebp
> ok 20 - debug: MOV SS + watchpoint + ICEBP
> ok 21 - debug: MOV SS + watchpoint + int $1
> ok 22 - debug: MOV SS + watchpoint + INT3
> 1..22
> 
> # standalone
> 
> tests/debug
> BUILD_HEAD=02d8befe
> timeout -k 1s --foreground 90s /usr/bin/qemu-kvm --no-reboot -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.1GdMZXhTTs -smp 1 # -initrd /tmp/tmp.LzHnmXchfO
> configure accelerator pc-i440fx-6.2 start
> machine init start
> device init start
> add qdev pc-testdev:none success
> add qdev pc-testdev:none success
> add qdev isa-debug-exit:none success
> add qdev isa-debug-exit:none success
> add qdev pci-testdev:none success
> add qdev pci-testdev:none success
> reset all devices
> qmp cont is received and vm is started
> qemu enter main_loop

All the above messages are debug messages that QEMU doesn't output. So
you're running your own QEMU build with extra messages which
kvm-unit-tests has to assume are errors.

Please don't try to debug test frameworks without using known-good
software-under-test.

It also appears you modified kvm-unit-tests, because there is no 'debug'
group.

Thanks,
drew

任敏敏(联通集团联通数字科技有限公司本部) April 27, 2023, 3:13 a.m. UTC | #4

From: rminmin <[e-mail]>

> On Mon, Apr 24, 2023 at 05:58:16PM +0800, 任敏敏(联通集团联通数字科技有限公司本部) wrote:
> > >On Sun, Apr 23, 2023 at 12:34:36PM +0800, 任敏敏(联通集团联通数字科技有限公司本部) wrote:
> > >> From: rminmin <[e-mail]>
> > >>
> > >> run_qemu should return 0 if logs doesn't
> > >> contain "warning" keyword.
> > >
> > >Why? What are you trying to fix?
> > >
> >
> > I encountered a problem that the differet results when I run the same
> > test case using standalone mode, "run_test.sh -t -g", "run_test.sh -g".
> > When I use "run_test.sh -g", it always returns a FAIL message.
> > In my env, qemu version is 6.2.
> >
> > e.g. runnig debug test cases.
> >
> > # without "-t"
> > ./run_tests.sh -g debug
> > FAIL debug (22 tests)
> >
> > # "-t"
> > ./run_tests.sh -t -g debug
> > TAP version 13
> > ok 1 - debug: DR4==DR6 with CR4.DE == 0
> > ok 2 - debug: DR4 read got #UD with CR4.DE == 1
> > ok 3 - debug: #BP
> > ok 4 - debug: hw breakpoint (test that dr6.BS is not set)
> > ok 5 - debug: hw breakpoint (test that dr6.BS is not cleared)
> > ok 6 - debug: Single-step #DB basic test
> > ok 7 - debug: Usermode Single-step #DB basic test
> > ok 8 - debug: Single-step #DB on emulated instructions
> > ok 9 - debug: Usermode Single-step #DB on emulated instructions
> > ok 10 - debug: Single-step #DB w/ STI blocking
> > ok 11 - debug: Usermode Single-step #DB w/ STI blocking
> > ok 12 - debug: Single-step #DB w/ MOVSS blocking
> > ok 13 - debug: Usermode Single-step #DB w/ MOVSS blocking
> > ok 14 - debug: Single-Step + ICEBP #DB w/ MOVSS blocking
> > ok 15 - debug: Usermode Single-Step + ICEBP #DB w/ MOVSS blocking
> > ok 16 - debug: Single-step #DB w/ MOVSS blocking and DR7.GD=1
> > ok 17 - debug: hw watchpoint (test that dr6.BS is not cleared)
> > ok 18 - debug: hw watchpoint (test that dr6.BS is not set)
> > ok 19 - debug: icebp
> > ok 20 - debug: MOV SS + watchpoint + ICEBP
> > ok 21 - debug: MOV SS + watchpoint + int $1
> > ok 22 - debug: MOV SS + watchpoint + INT3
> > 1..22
> >
> > # standalone
> >
> > tests/debug
> > BUILD_HEAD=02d8befe
> > timeout -k 1s --foreground 90s /usr/bin/qemu-kvm --no-reboot -nodefaults -device pc-testdev -device isa-debug-exit,iobase=0xf4,iosize=0x4 -vnc none -serial stdio -device pci-testdev -machine accel=kvm -kernel /tmp/tmp.1GdMZXhTTs -smp 1 # -initrd /tmp/tmp.LzHnmXchfO
> > configure accelerator pc-i440fx-6.2 start
> > machine init start
> > device init start
> > add qdev pc-testdev:none success
> > add qdev pc-testdev:none success
> > add qdev isa-debug-exit:none success
> > add qdev isa-debug-exit:none success
> > add qdev pci-testdev:none success
> > add qdev pci-testdev:none success
> > reset all devices
> > qmp cont is received and vm is started
> > qemu enter main_loop
>
> All the above messages are debug messages that QEMU doesn't output. So
> you're running your own QEMU build with extra messages which
> kvm-unit-tests has to assume are errors.
>
> Please don't try to debug test frameworks without using known-good
> software-under-test.
>
> It also appears you modified kvm-unit-tests, because there is no 'debug'
> group.
>
> Thanks,
> drew

Thanks for your help. You're right. The qemu in my env is from Openeuler repo
and I found some patch files adding the extra qemu log info in src rpm.

如果您错误接收了该邮件，请通过电子邮件立即通知我们。请回复邮件到 [e-mail]，即可以退订此邮件。我们将立即将您的信息从我们的发送目录中删除。 If you have received this email in error please notify us immediately by e-mail. Please reply to [e-mail] ,you can unsubscribe from this mail. We will immediately remove your information from send catalogue of our.

Patch

diff --git a/scripts/arch-run.bash b/scripts/arch-run.bash
index 51e4b97..9878d32 100644
--- a/scripts/arch-run.bash
+++ b/scripts/arch-run.bash
@@ -61,7 +61,7 @@  run_qemu ()
                # Even when ret==1 (unittest success) if we also got stderr
                # logs, then we assume a QEMU failure. Otherwise we translate
                # status of 1 to 0 (SUCCESS)
-               if [ -z "$(echo "$errors" | grep -vi warning)" ]; then
+               if [ -z "$(echo "$errors" | grep -i warning)" ]; then
                        ret=0
                fi
        fi