| Message ID | 511b2f6009fb830b3f32b4be3dca99596c684fa3.1689759351.git.falcon@tinylab.org (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | selftests/nolibc: add minimal kernel config support - part1 | expand |
On Wed, Jul 19, 2023 at 09:27:08PM +0800, Zhangjin Wu wrote: > The kernel of some architectures can not poweroff qemu-system normally, > especially for tinyconfig. > > Some architectures may have no kernel poweroff support, the others may > require more kernel config options and therefore slow down the > tinyconfig build and test. and also, it's very hard (and some even not > possible) to find out the exact poweroff related kernel config options > for every architecture. > > Since the low-level poweroff support is heavily kernel & qemu dependent, > it is not that critical to both nolibc and nolibc-test, let's simply > ignore the poweroff required kernel config options for tinyconfig (and > even for defconfig) and quit qemu-system after a specified timeout or > with an expected system halt or poweroff string (these strings mean our > reboot() library routine is perfectly ok). > > QEMU_TIMEOUT value can be configured for every architecture based on > their time cost requirement of boot+test+poweroff. > > Signed-off-by: Zhangjin Wu <falcon@tinylab.org> > --- > tools/testing/selftests/nolibc/Makefile | 23 +++++++++++++++++++++-- > 1 file changed, 21 insertions(+), 2 deletions(-) > > diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile > index 541f3565e584..a03fab020ebe 100644 > --- a/tools/testing/selftests/nolibc/Makefile > +++ b/tools/testing/selftests/nolibc/Makefile > @@ -93,6 +93,9 @@ QEMU_ARGS_s390 = -M s390-ccw-virtio -m 1G -append "console=ttyS0 panic=-1 > QEMU_ARGS_loongarch = -M virt -append "console=ttyS0,115200 panic=-1 $(TEST:%=NOLIBC_TEST=%)" > QEMU_ARGS = $(QEMU_ARGS_$(XARCH)) $(QEMU_ARGS_EXTRA) > > +# QEMU_TIMEOUT: some architectures can not poweroff normally, especially for tinyconfig > +QEMU_TIMEOUT = $(QEMU_TIMEOUT_$(XARCH)) > + > # OUTPUT is only set when run from the main makefile, otherwise > # it defaults to this nolibc directory. > OUTPUT ?= $(CURDIR)/ > @@ -224,16 +227,32 @@ kernel: extconfig > # common macros for qemu run/rerun targets > QEMU_SYSTEM_RUN = qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(KERNEL_IMAGE)" -serial stdio $(QEMU_ARGS) > > +ifneq ($(QEMU_TIMEOUT),) > +TIMEOUT_CMD = t=$(QEMU_TIMEOUT); \ > + while [ $$t -gt 0 ]; do \ > + sleep 5; t=$$(expr $$t - 5); echo "detecting power off ..."; \ > + if grep -qE "reboot: System halted|reboot: Power down" "$(RUN_OUT)"; then \ > + pkill -9 qemu-system-$(QEMU_ARCH); \ > + echo "powered off, test finish"; t=1; break; \ > + fi; \ > + done; \ > + if [ $$t -le 0 ]; then pkill -9 qemu-system-$(QEMU_ARCH); echo "qemu-system-$(QEMU_ARCH) timeout"; fi Please have a look at the "timeout" command whichi makes all this much simpler. Also, please get used to never ever use kill -9 first. This is exactly the way to leave temporary files and IPCs wandering around while many programs that care about cleanups at least try to do that upon a regular TERM or INT signal. Willy
Hi, Willy > On Wed, Jul 19, 2023 at 09:27:08PM +0800, Zhangjin Wu wrote: > > The kernel of some architectures can not poweroff qemu-system normally, > > especially for tinyconfig. > > > > Some architectures may have no kernel poweroff support, the others may > > require more kernel config options and therefore slow down the > > tinyconfig build and test. and also, it's very hard (and some even not > > possible) to find out the exact poweroff related kernel config options > > for every architecture. > > > > Since the low-level poweroff support is heavily kernel & qemu dependent, > > it is not that critical to both nolibc and nolibc-test, let's simply > > ignore the poweroff required kernel config options for tinyconfig (and > > even for defconfig) and quit qemu-system after a specified timeout or > > with an expected system halt or poweroff string (these strings mean our > > reboot() library routine is perfectly ok). > > > > QEMU_TIMEOUT value can be configured for every architecture based on > > their time cost requirement of boot+test+poweroff. > > > > Signed-off-by: Zhangjin Wu <falcon@tinylab.org> > > --- > > tools/testing/selftests/nolibc/Makefile | 23 +++++++++++++++++++++-- > > 1 file changed, 21 insertions(+), 2 deletions(-) > > > > diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile > > index 541f3565e584..a03fab020ebe 100644 > > --- a/tools/testing/selftests/nolibc/Makefile > > +++ b/tools/testing/selftests/nolibc/Makefile > > @@ -93,6 +93,9 @@ QEMU_ARGS_s390 = -M s390-ccw-virtio -m 1G -append "console=ttyS0 panic=-1 > > QEMU_ARGS_loongarch = -M virt -append "console=ttyS0,115200 panic=-1 $(TEST:%=NOLIBC_TEST=%)" > > QEMU_ARGS = $(QEMU_ARGS_$(XARCH)) $(QEMU_ARGS_EXTRA) > > > > +# QEMU_TIMEOUT: some architectures can not poweroff normally, especially for tinyconfig > > +QEMU_TIMEOUT = $(QEMU_TIMEOUT_$(XARCH)) > > + > > # OUTPUT is only set when run from the main makefile, otherwise > > # it defaults to this nolibc directory. > > OUTPUT ?= $(CURDIR)/ > > @@ -224,16 +227,32 @@ kernel: extconfig > > # common macros for qemu run/rerun targets > > QEMU_SYSTEM_RUN = qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(KERNEL_IMAGE)" -serial stdio $(QEMU_ARGS) > > > > +ifneq ($(QEMU_TIMEOUT),) > > +TIMEOUT_CMD = t=$(QEMU_TIMEOUT); \ > > + while [ $$t -gt 0 ]; do \ > > + sleep 5; t=$$(expr $$t - 5); echo "detecting power off ..."; \ > > + if grep -qE "reboot: System halted|reboot: Power down" "$(RUN_OUT)"; then \ > > + pkill -9 qemu-system-$(QEMU_ARCH); \ > > + echo "powered off, test finish"; t=1; break; \ > > + fi; \ > > + done; \ > > + if [ $$t -le 0 ]; then pkill -9 qemu-system-$(QEMU_ARCH); echo "qemu-system-$(QEMU_ARCH) timeout"; fi > > Please have a look at the "timeout" command whichi makes all this much > simpler. Yeah, I used 'timeout --forgeground' before, but it is still a dead wait and it is very hard for us to configure a just right wait time. If the configured wait time is short, the qemu will be killed during the test or before running the test, If the configured wait time is long, we will need to dead wait there even if the test is finished, although during interactive running, we can press 'CTRL + A X', but for background running, it is just time waste. To fix up this issue, the above method used at last allow to detect the output string, when the test finish and print something lines like: reboot: System halted reboot: Power down. We will use pkill to send signal to tell qemu quit, so, it is ok for us to configure a even'big' timeout, if the kernel can normally poweroff or if nolibc can successfully send the poweroff syscall, the above message will be detected and qemu will quit as expected, it will completely avoid dead wait, the configured timeout will never happen, this is comfortable. The worst case is only when qemu or kernel reject to boot, for example, a qemu bios missing or mismatch issue or a kernel regression or a wrong kernel config option, for these cases, the real timeout logic will work for us. As a summary, our timeout logic here include two parts: one is 'poweroff' related string detection, another is the real timeout logic. Based on current implementation, I even plan to add the test finish string in the expected strings: Leaving init with final status And even further, when a hang detected (no normal poweroff or test finish string detected), print the whole or last part of running log to tell users what happens. > Also, please get used to never ever use kill -9 first. This > is exactly the way to leave temporary files and IPCs wandering around > while many programs that care about cleanups at least try to do that > upon a regular TERM or INT signal. > Ok, thanks, will use TERM or INT signal instead. > Willy
On Tue, Jul 25, 2023 at 10:59:55PM +0800, Zhangjin Wu wrote: > > On Wed, Jul 19, 2023 at 09:27:08PM +0800, Zhangjin Wu wrote: > > > The kernel of some architectures can not poweroff qemu-system normally, > > > especially for tinyconfig. > > > > > > Some architectures may have no kernel poweroff support, the others may > > > require more kernel config options and therefore slow down the > > > tinyconfig build and test. and also, it's very hard (and some even not > > > possible) to find out the exact poweroff related kernel config options > > > for every architecture. > > > > > > Since the low-level poweroff support is heavily kernel & qemu dependent, > > > it is not that critical to both nolibc and nolibc-test, let's simply > > > ignore the poweroff required kernel config options for tinyconfig (and > > > even for defconfig) and quit qemu-system after a specified timeout or > > > with an expected system halt or poweroff string (these strings mean our > > > reboot() library routine is perfectly ok). > > > > > > QEMU_TIMEOUT value can be configured for every architecture based on > > > their time cost requirement of boot+test+poweroff. > > > > > > Signed-off-by: Zhangjin Wu <falcon@tinylab.org> > > > --- > > > tools/testing/selftests/nolibc/Makefile | 23 +++++++++++++++++++++-- > > > 1 file changed, 21 insertions(+), 2 deletions(-) > > > > > > diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile > > > index 541f3565e584..a03fab020ebe 100644 > > > --- a/tools/testing/selftests/nolibc/Makefile > > > +++ b/tools/testing/selftests/nolibc/Makefile > > > @@ -93,6 +93,9 @@ QEMU_ARGS_s390 = -M s390-ccw-virtio -m 1G -append "console=ttyS0 panic=-1 > > > QEMU_ARGS_loongarch = -M virt -append "console=ttyS0,115200 panic=-1 $(TEST:%=NOLIBC_TEST=%)" > > > QEMU_ARGS = $(QEMU_ARGS_$(XARCH)) $(QEMU_ARGS_EXTRA) > > > > > > +# QEMU_TIMEOUT: some architectures can not poweroff normally, especially for tinyconfig > > > +QEMU_TIMEOUT = $(QEMU_TIMEOUT_$(XARCH)) > > > + > > > # OUTPUT is only set when run from the main makefile, otherwise > > > # it defaults to this nolibc directory. > > > OUTPUT ?= $(CURDIR)/ > > > @@ -224,16 +227,32 @@ kernel: extconfig > > > # common macros for qemu run/rerun targets > > > QEMU_SYSTEM_RUN = qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(KERNEL_IMAGE)" -serial stdio $(QEMU_ARGS) > > > > > > +ifneq ($(QEMU_TIMEOUT),) > > > +TIMEOUT_CMD = t=$(QEMU_TIMEOUT); \ > > > + while [ $$t -gt 0 ]; do \ > > > + sleep 5; t=$$(expr $$t - 5); echo "detecting power off ..."; \ > > > + if grep -qE "reboot: System halted|reboot: Power down" "$(RUN_OUT)"; then \ > > > + pkill -9 qemu-system-$(QEMU_ARCH); \ > > > + echo "powered off, test finish"; t=1; break; \ > > > + fi; \ > > > + done; \ > > > + if [ $$t -le 0 ]; then pkill -9 qemu-system-$(QEMU_ARCH); echo "qemu-system-$(QEMU_ARCH) timeout"; fi > > > > Please have a look at the "timeout" command whichi makes all this much > > simpler. > > Yeah, I used 'timeout --forgeground' before, but it is still a dead wait > and it is very hard for us to configure a just right wait time. > > If the configured wait time is short, the qemu will be killed during the > test or before running the test, If the configured wait time is long, we > will need to dead wait there even if the test is finished, although > during interactive running, we can press 'CTRL + A X', but for > background running, it is just time waste. > > To fix up this issue, the above method used at last allow to detect the > output string, when the test finish and print something lines like: > > reboot: System halted > reboot: Power down. > > We will use pkill to send signal to tell qemu quit, so, it is ok for us > to configure a even'big' timeout, if the kernel can normally poweroff or > if nolibc can successfully send the poweroff syscall, the above message > will be detected and qemu will quit as expected, it will completely > avoid dead wait, the configured timeout will never happen, this is > comfortable. > > The worst case is only when qemu or kernel reject to boot, for example, > a qemu bios missing or mismatch issue or a kernel regression or a wrong > kernel config option, for these cases, the real timeout logic will work > for us. > > As a summary, our timeout logic here include two parts: one is > 'poweroff' related string detection, another is the real timeout logic. > > Based on current implementation, I even plan to add the test finish > string in the expected strings: > > Leaving init with final status > > And even further, when a hang detected (no normal poweroff or test > finish string detected), print the whole or last part of running log to > tell users what happens. Again, all of this seems ridiculously complicated for what seems to be a very unlikely issue. You mentioned earlier: > > > The kernel of some architectures can not poweroff qemu-system > > > normally, especially for tinyconfig. Till now all those I tested do power off correctly. So if the problem is related to one specific arch, first it should be mentioned in the commit message, and maybe we need to look closer why it's doing that instead of papering over it, and if it's caused by tinyconfig, it just means we need to produce a more reasonable config. But I still don't like the idea of causing a problem on one side, and making the other side totally ugly just because of the problem. The timeout solution should be a last resort one which clearly explains why we can't do differently. Also, defconfig and tinyconfig are for people who work on nolibc. These ones should support working in batch mode, being called from a script iterating over multiple archs. For other config, we should not interfer with what the user does. Maybe some will consider that it's fine to run a test slowly on a simulated platform for example. Willy
diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile index 541f3565e584..a03fab020ebe 100644 --- a/tools/testing/selftests/nolibc/Makefile +++ b/tools/testing/selftests/nolibc/Makefile @@ -93,6 +93,9 @@ QEMU_ARGS_s390 = -M s390-ccw-virtio -m 1G -append "console=ttyS0 panic=-1 QEMU_ARGS_loongarch = -M virt -append "console=ttyS0,115200 panic=-1 $(TEST:%=NOLIBC_TEST=%)" QEMU_ARGS = $(QEMU_ARGS_$(XARCH)) $(QEMU_ARGS_EXTRA) +# QEMU_TIMEOUT: some architectures can not poweroff normally, especially for tinyconfig +QEMU_TIMEOUT = $(QEMU_TIMEOUT_$(XARCH)) + # OUTPUT is only set when run from the main makefile, otherwise # it defaults to this nolibc directory. OUTPUT ?= $(CURDIR)/ @@ -224,16 +227,32 @@ kernel: extconfig # common macros for qemu run/rerun targets QEMU_SYSTEM_RUN = qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(KERNEL_IMAGE)" -serial stdio $(QEMU_ARGS) +ifneq ($(QEMU_TIMEOUT),) +TIMEOUT_CMD = t=$(QEMU_TIMEOUT); \ + while [ $$t -gt 0 ]; do \ + sleep 5; t=$$(expr $$t - 5); echo "detecting power off ..."; \ + if grep -qE "reboot: System halted|reboot: Power down" "$(RUN_OUT)"; then \ + pkill -9 qemu-system-$(QEMU_ARCH); \ + echo "powered off, test finish"; t=1; break; \ + fi; \ + done; \ + if [ $$t -le 0 ]; then pkill -9 qemu-system-$(QEMU_ARCH); echo "qemu-system-$(QEMU_ARCH) timeout"; fi + +TIMEOUT_QEMU_RUN = ($(QEMU_SYSTEM_RUN) $(LOG_OUT) &); $(TIMEOUT_CMD) +else +TIMEOUT_QEMU_RUN = $(QEMU_SYSTEM_RUN) $(LOG_OUT) +endif + # run the tests after building the kernel PHONY += $(KERNEL_IMAGE) $(KERNEL_IMAGE): kernel run: $(KERNEL_IMAGE) - $(Q)$(QEMU_SYSTEM_RUN) $(LOG_OUT) + $(Q)$(TIMEOUT_QEMU_RUN) $(Q)$(REPORT_RUN_OUT) # re-run the tests from an existing kernel rerun: - $(Q)$(QEMU_SYSTEM_RUN) $(LOG_OUT) + $(Q)$(TIMEOUT_QEMU_RUN) $(Q)$(REPORT_RUN_OUT) # report with existing test log
The kernel of some architectures can not poweroff qemu-system normally, especially for tinyconfig. Some architectures may have no kernel poweroff support, the others may require more kernel config options and therefore slow down the tinyconfig build and test. and also, it's very hard (and some even not possible) to find out the exact poweroff related kernel config options for every architecture. Since the low-level poweroff support is heavily kernel & qemu dependent, it is not that critical to both nolibc and nolibc-test, let's simply ignore the poweroff required kernel config options for tinyconfig (and even for defconfig) and quit qemu-system after a specified timeout or with an expected system halt or poweroff string (these strings mean our reboot() library routine is perfectly ok). QEMU_TIMEOUT value can be configured for every architecture based on their time cost requirement of boot+test+poweroff. Signed-off-by: Zhangjin Wu <falcon@tinylab.org> --- tools/testing/selftests/nolibc/Makefile | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)