| Message ID | 20210809111621.54454-1-alxndr@bu.edu (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | fuzz: avoid building twice, when running on gitlab | expand |
On Monday, 2021-08-09 at 07:16:21 -04, Alexander Bulekov wrote: > On oss-fuzz, we build twice, to put together a build that is portable to > the runner containers. On gitlab ci, this is wasteful and contributes to > timeouts on the build-oss-fuzz job. Avoid building twice on gitlab, at > the remote cost of potentially missing some cases that break oss-fuzz > builds. > > Signed-off-by: Alexander Bulekov <alxndr@bu.edu> Reviewed-by: Darren Kenny <darren.kenny@oracle.com> > --- > > From a couple test runs it looks like this can shave off 15-20 minutes. > > scripts/oss-fuzz/build.sh | 24 +++++++++++++----------- > 1 file changed, 13 insertions(+), 11 deletions(-) > > diff --git a/scripts/oss-fuzz/build.sh b/scripts/oss-fuzz/build.sh > index c1af43fded..98b56e0521 100755 > --- a/scripts/oss-fuzz/build.sh > +++ b/scripts/oss-fuzz/build.sh > @@ -73,17 +73,19 @@ if ! make "-j$(nproc)" qemu-fuzz-i386; then > "\nFor example: CC=clang CXX=clang++ $0" > fi > > -for i in $(ldd ./qemu-fuzz-i386 | cut -f3 -d' '); do > - cp "$i" "$DEST_DIR/lib/" > -done > -rm qemu-fuzz-i386 > - > -# Build a second time to build the final binary with correct rpath > -../configure --disable-werror --cc="$CC" --cxx="$CXX" --enable-fuzzing \ > - --prefix="$DEST_DIR" --bindir="$DEST_DIR" --datadir="$DEST_DIR/data/" \ > - --extra-cflags="$EXTRA_CFLAGS" --extra-ldflags="-Wl,-rpath,\$ORIGIN/lib" \ > - --target-list="i386-softmmu" > -make "-j$(nproc)" qemu-fuzz-i386 V=1 > +if [ "$GITLAB_CI" != "true" ]; then > + for i in $(ldd ./qemu-fuzz-i386 | cut -f3 -d' '); do > + cp "$i" "$DEST_DIR/lib/" > + done > + rm qemu-fuzz-i386 > + > + # Build a second time to build the final binary with correct rpath > + ../configure --disable-werror --cc="$CC" --cxx="$CXX" --enable-fuzzing \ > + --prefix="$DEST_DIR" --bindir="$DEST_DIR" --datadir="$DEST_DIR/data/" \ > + --extra-cflags="$EXTRA_CFLAGS" --extra-ldflags="-Wl,-rpath,\$ORIGIN/lib" \ > + --target-list="i386-softmmu" > + make "-j$(nproc)" qemu-fuzz-i386 V=1 > +fi > > # Copy over the datadir > cp -r ../pc-bios/ "$DEST_DIR/pc-bios" > -- > 2.30.2
On Mon, 9 Aug 2021 at 12:18, Alexander Bulekov <alxndr@bu.edu> wrote: > > On oss-fuzz, we build twice, to put together a build that is portable to > the runner containers. On gitlab ci, this is wasteful and contributes to > timeouts on the build-oss-fuzz job. Avoid building twice on gitlab, at > the remote cost of potentially missing some cases that break oss-fuzz > builds. > > Signed-off-by: Alexander Bulekov <alxndr@bu.edu> > --- > > From a couple test runs it looks like this can shave off 15-20 minutes. > > scripts/oss-fuzz/build.sh | 24 +++++++++++++----------- > 1 file changed, 13 insertions(+), 11 deletions(-) I tried a test run with this, but it still hit the 1 hour timeout: https://gitlab.com/qemu-project/qemu/-/pipelines/350387482 -- PMM
On 210809 1925, Peter Maydell wrote: > On Mon, 9 Aug 2021 at 12:18, Alexander Bulekov <alxndr@bu.edu> wrote: > > > > On oss-fuzz, we build twice, to put together a build that is portable to > > the runner containers. On gitlab ci, this is wasteful and contributes to > > timeouts on the build-oss-fuzz job. Avoid building twice on gitlab, at > > the remote cost of potentially missing some cases that break oss-fuzz > > builds. > > > > Signed-off-by: Alexander Bulekov <alxndr@bu.edu> > > --- > > > > From a couple test runs it looks like this can shave off 15-20 minutes. > > > > scripts/oss-fuzz/build.sh | 24 +++++++++++++----------- > > 1 file changed, 13 insertions(+), 11 deletions(-) > > I tried a test run with this, but it still hit the 1 hour timeout: > > https://gitlab.com/qemu-project/qemu/-/pipelines/350387482 It also timed out for me with a 120 minute timeout: https://gitlab.com/a1xndr/qemu/-/jobs/1488160601 The log has almost exactly the same number of lines as yours, so I'm guessing one of the qtests is timing out with --enable-sanitizers . -Alex > > -- PMM
On 210809 1506, Alexander Bulekov wrote: > On 210809 1925, Peter Maydell wrote: > > On Mon, 9 Aug 2021 at 12:18, Alexander Bulekov <alxndr@bu.edu> wrote: > > > > > > On oss-fuzz, we build twice, to put together a build that is portable to > > > the runner containers. On gitlab ci, this is wasteful and contributes to > > > timeouts on the build-oss-fuzz job. Avoid building twice on gitlab, at > > > the remote cost of potentially missing some cases that break oss-fuzz > > > builds. > > > > > > Signed-off-by: Alexander Bulekov <alxndr@bu.edu> > > > --- > > > > > > From a couple test runs it looks like this can shave off 15-20 minutes. > > > > > > scripts/oss-fuzz/build.sh | 24 +++++++++++++----------- > > > 1 file changed, 13 insertions(+), 11 deletions(-) > > > > I tried a test run with this, but it still hit the 1 hour timeout: > > > > https://gitlab.com/qemu-project/qemu/-/pipelines/350387482 > > It also timed out for me with a 120 minute timeout: > https://gitlab.com/a1xndr/qemu/-/jobs/1488160601 > > The log has almost exactly the same number of lines as yours, so I'm > guessing one of the qtests is timing out with --enable-sanitizers . > > -Alex > Building locally: $ CC=clang-11 CXX=clang++-11 ../configure --enable-fuzzing \ --enable-debug --enable-sanitizers $ make check-qtest-i386 check-unit Same as on gitlab, this times out shortly after outputting "sh: 1: exec: ./storage-daemon/qemu-storage-daemon: not found" Manually running qos-test, the same way check-qtest-i386 invokes it: $ QTEST_QEMU_BINARY=./qemu-system-i386 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon tests/qtest/qos-test --tap -k -m quick < /dev/null # starting vhost-user backend: exec ./storage-daemon/qemu-storage-daemon --blockdev driver=file,node-name=disk0,filename=qtest.XRAzzu --export type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-94561-sock.NdKWpt,node-name=disk0,writable=on,num-queues=1 sh: 1: exec: ./storage-daemon/qemu-storage-daemon: not found # starting QEMU: exec ./qemu-system-i386 -qtest unix:/tmp/qtest-94561.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-94561.qmp,id=char0 -mon chardev=char0,mode=control -display none -M pc -device vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem -m 256M -chardev socket id=char1,path=/tmp/qtest-94561-sock.NdKWpt -accel qtest *timeout* Ok, lets try to manually build ./storage-daemon/qemu-storage-daemon $ make ./storage-daemon/qemu-storage-daemon And rerun the tests... $ QTEST_QEMU_BINARY=./qemu-system-i386 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon tests/qtest/qos-test --tap -k -m quick < /dev/null No timeout... Still not sure why ./storage-daemon/qemu-storage-daemon isn't being built by make check, and how to fix that. -Alex > > > > -- PMM
On Mon, 9 Aug 2021 at 20:30, Alexander Bulekov <alxndr@bu.edu> wrote: > > On 210809 1506, Alexander Bulekov wrote: > > On 210809 1925, Peter Maydell wrote: > > > On Mon, 9 Aug 2021 at 12:18, Alexander Bulekov <alxndr@bu.edu> wrote: > > > > > > > > On oss-fuzz, we build twice, to put together a build that is portable to > > > > the runner containers. On gitlab ci, this is wasteful and contributes to > > > > timeouts on the build-oss-fuzz job. Avoid building twice on gitlab, at > > > > the remote cost of potentially missing some cases that break oss-fuzz > > > > builds. > > > > > > > > Signed-off-by: Alexander Bulekov <alxndr@bu.edu> > > > > --- > > > > > > > > From a couple test runs it looks like this can shave off 15-20 minutes. > > > > > > > > scripts/oss-fuzz/build.sh | 24 +++++++++++++----------- > > > > 1 file changed, 13 insertions(+), 11 deletions(-) > > > > > > I tried a test run with this, but it still hit the 1 hour timeout: > > > > > > https://gitlab.com/qemu-project/qemu/-/pipelines/350387482 > > > > It also timed out for me with a 120 minute timeout: > > https://gitlab.com/a1xndr/qemu/-/jobs/1488160601 > > > > The log has almost exactly the same number of lines as yours, so I'm > > guessing one of the qtests is timing out with --enable-sanitizers . > > Building locally: > $ CC=clang-11 CXX=clang++-11 ../configure --enable-fuzzing \ > --enable-debug --enable-sanitizers > $ make check-qtest-i386 check-unit > > Same as on gitlab, this times out shortly after outputting > "sh: 1: exec: ./storage-daemon/qemu-storage-daemon: not found" > > Manually running qos-test, the same way check-qtest-i386 invokes it: > > $ QTEST_QEMU_BINARY=./qemu-system-i386 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon tests/qtest/qos-test --tap -k -m quick < /dev/null > > # starting vhost-user backend: exec ./storage-daemon/qemu-storage-daemon --blockdev driver=file,node-name=disk0,filename=qtest.XRAzzu --export type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-94561-sock.NdKWpt,node-name=disk0,writable=on,num-queues=1 > sh: 1: exec: ./storage-daemon/qemu-storage-daemon: not found > # starting QEMU: exec ./qemu-system-i386 -qtest unix:/tmp/qtest-94561.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-94561.qmp,id=char0 -mon chardev=char0,mode=control -display none -M pc -device vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem -m 256M -chardev socket id=char1,path=/tmp/qtest-94561-sock.NdKWpt -accel qtest > > *timeout* vhost-user timing out in realize I suspect. I see that as an intermittent hang in non-sanitizer configs. vhost-user folk: Can we either look at fixing this or else disable the test ? (Stack backtraces available in the other email thread.) thanks -- PMM
+Coiby Xu & qemu-block@ On 8/9/21 9:36 PM, Peter Maydell wrote: > On Mon, 9 Aug 2021 at 20:30, Alexander Bulekov <alxndr@bu.edu> wrote: >> >> On 210809 1506, Alexander Bulekov wrote: >>> On 210809 1925, Peter Maydell wrote: >>>> On Mon, 9 Aug 2021 at 12:18, Alexander Bulekov <alxndr@bu.edu> wrote: >>>>> >>>>> On oss-fuzz, we build twice, to put together a build that is portable to >>>>> the runner containers. On gitlab ci, this is wasteful and contributes to >>>>> timeouts on the build-oss-fuzz job. Avoid building twice on gitlab, at >>>>> the remote cost of potentially missing some cases that break oss-fuzz >>>>> builds. >>>>> >>>>> Signed-off-by: Alexander Bulekov <alxndr@bu.edu> >>>>> --- >>>>> >>>>> From a couple test runs it looks like this can shave off 15-20 minutes. >>>>> >>>>> scripts/oss-fuzz/build.sh | 24 +++++++++++++----------- >>>>> 1 file changed, 13 insertions(+), 11 deletions(-) >>>> >>>> I tried a test run with this, but it still hit the 1 hour timeout: >>>> >>>> https://gitlab.com/qemu-project/qemu/-/pipelines/350387482 >>> >>> It also timed out for me with a 120 minute timeout: >>> https://gitlab.com/a1xndr/qemu/-/jobs/1488160601 >>> >>> The log has almost exactly the same number of lines as yours, so I'm >>> guessing one of the qtests is timing out with --enable-sanitizers . > >> >> Building locally: >> $ CC=clang-11 CXX=clang++-11 ../configure --enable-fuzzing \ >> --enable-debug --enable-sanitizers >> $ make check-qtest-i386 check-unit >> >> Same as on gitlab, this times out shortly after outputting >> "sh: 1: exec: ./storage-daemon/qemu-storage-daemon: not found" >> >> Manually running qos-test, the same way check-qtest-i386 invokes it: >> >> $ QTEST_QEMU_BINARY=./qemu-system-i386 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon tests/qtest/qos-test --tap -k -m quick < /dev/null >> >> # starting vhost-user backend: exec ./storage-daemon/qemu-storage-daemon --blockdev driver=file,node-name=disk0,filename=qtest.XRAzzu --export type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-94561-sock.NdKWpt,node-name=disk0,writable=on,num-queues=1 >> sh: 1: exec: ./storage-daemon/qemu-storage-daemon: not found >> # starting QEMU: exec ./qemu-system-i386 -qtest unix:/tmp/qtest-94561.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-94561.qmp,id=char0 -mon chardev=char0,mode=control -display none -M pc -device vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem -m 256M -chardev socket id=char1,path=/tmp/qtest-94561-sock.NdKWpt -accel qtest >> >> *timeout* > > vhost-user timing out in realize I suspect. I see that as > an intermittent hang in non-sanitizer configs. > > vhost-user folk: Can we either look at fixing this or else disable > the test ? (Stack backtraces available in the other email thread.) > > thanks > -- PMM >
On Tue, Aug 10, 2021 at 07:01:45AM +0200, Philippe Mathieu-Daudé wrote: >+Coiby Xu & qemu-block@ Thanks for adding me to the Cc list. > >On 8/9/21 9:36 PM, Peter Maydell wrote: >> On Mon, 9 Aug 2021 at 20:30, Alexander Bulekov <alxndr@bu.edu> wrote: >>> >>> On 210809 1506, Alexander Bulekov wrote: >>>> On 210809 1925, Peter Maydell wrote: >>>>> On Mon, 9 Aug 2021 at 12:18, Alexander Bulekov <alxndr@bu.edu> wrote: >>>>>> >>>>>> On oss-fuzz, we build twice, to put together a build that is portable to >>>>>> the runner containers. On gitlab ci, this is wasteful and contributes to >>>>>> timeouts on the build-oss-fuzz job. Avoid building twice on gitlab, at >>>>>> the remote cost of potentially missing some cases that break oss-fuzz >>>>>> builds. >>>>>> >>>>>> Signed-off-by: Alexander Bulekov <alxndr@bu.edu> >>>>>> --- >>>>>> >>>>>> From a couple test runs it looks like this can shave off 15-20 minutes. >>>>>> >>>>>> scripts/oss-fuzz/build.sh | 24 +++++++++++++----------- >>>>>> 1 file changed, 13 insertions(+), 11 deletions(-) >>>>> >>>>> I tried a test run with this, but it still hit the 1 hour timeout: >>>>> >>>>> https://gitlab.com/qemu-project/qemu/-/pipelines/350387482 >>>> >>>> It also timed out for me with a 120 minute timeout: >>>> https://gitlab.com/a1xndr/qemu/-/jobs/1488160601 >>>> >>>> The log has almost exactly the same number of lines as yours, so I'm >>>> guessing one of the qtests is timing out with --enable-sanitizers . >> >>> >>> Building locally: >>> $ CC=clang-11 CXX=clang++-11 ../configure --enable-fuzzing \ >>> --enable-debug --enable-sanitizers >>> $ make check-qtest-i386 check-unit >>> >>> Same as on gitlab, this times out shortly after outputting >>> "sh: 1: exec: ./storage-daemon/qemu-storage-daemon: not found" >>> >>> Manually running qos-test, the same way check-qtest-i386 invokes it: >>> >>> $ QTEST_QEMU_BINARY=./qemu-system-i386 QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon tests/qtest/qos-test --tap -k -m quick < /dev/null >>> >>> # starting vhost-user backend: exec ./storage-daemon/qemu-storage-daemon --blockdev driver=file,node-name=disk0,filename=qtest.XRAzzu --export type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-94561-sock.NdKWpt,node-name=disk0,writable=on,num-queues=1 >>> sh: 1: exec: ./storage-daemon/qemu-storage-daemon: not found The error is different from the previous issue of intermittent hang. This time the hang is caused by missing qemu-storage-daemon and I guess the hang could happen reliably each time. The reason of missing qemu-storage-daemon is the test doesn't add qemu-storage-daemon as dependency. If we run `make`, qemu-storage-daemon would be built. But if `make check-qtest-i386` is run directly, qemu-storage-daemon wouldn't be built. I'll send a patch to make vhost-user-blk test depends on emu-storage-daemon. >>> # starting QEMU: exec ./qemu-system-i386 -qtest unix:/tmp/qtest-94561.sock -qtest-log /dev/null -chardev socket,path=/tmp/qtest-94561.qmp,id=char0 -mon chardev=char0,mode=control -display none -M pc -device vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem -m 256M -chardev socket id=char1,path=/tmp/qtest-94561-sock.NdKWpt -accel qtest >>> >>> *timeout* >> >> vhost-user timing out in realize I suspect. I see that as >> an intermittent hang in non-sanitizer configs. >> >> vhost-user folk: Can we either look at fixing this or else disable >> the test ? (Stack backtraces available in the other email thread.) >> >> thanks >> -- PMM >> >
diff --git a/scripts/oss-fuzz/build.sh b/scripts/oss-fuzz/build.sh index c1af43fded..98b56e0521 100755 --- a/scripts/oss-fuzz/build.sh +++ b/scripts/oss-fuzz/build.sh @@ -73,17 +73,19 @@ if ! make "-j$(nproc)" qemu-fuzz-i386; then "\nFor example: CC=clang CXX=clang++ $0" fi -for i in $(ldd ./qemu-fuzz-i386 | cut -f3 -d' '); do - cp "$i" "$DEST_DIR/lib/" -done -rm qemu-fuzz-i386 - -# Build a second time to build the final binary with correct rpath -../configure --disable-werror --cc="$CC" --cxx="$CXX" --enable-fuzzing \ - --prefix="$DEST_DIR" --bindir="$DEST_DIR" --datadir="$DEST_DIR/data/" \ - --extra-cflags="$EXTRA_CFLAGS" --extra-ldflags="-Wl,-rpath,\$ORIGIN/lib" \ - --target-list="i386-softmmu" -make "-j$(nproc)" qemu-fuzz-i386 V=1 +if [ "$GITLAB_CI" != "true" ]; then + for i in $(ldd ./qemu-fuzz-i386 | cut -f3 -d' '); do + cp "$i" "$DEST_DIR/lib/" + done + rm qemu-fuzz-i386 + + # Build a second time to build the final binary with correct rpath + ../configure --disable-werror --cc="$CC" --cxx="$CXX" --enable-fuzzing \ + --prefix="$DEST_DIR" --bindir="$DEST_DIR" --datadir="$DEST_DIR/data/" \ + --extra-cflags="$EXTRA_CFLAGS" --extra-ldflags="-Wl,-rpath,\$ORIGIN/lib" \ + --target-list="i386-softmmu" + make "-j$(nproc)" qemu-fuzz-i386 V=1 +fi # Copy over the datadir cp -r ../pc-bios/ "$DEST_DIR/pc-bios"
On oss-fuzz, we build twice, to put together a build that is portable to the runner containers. On gitlab ci, this is wasteful and contributes to timeouts on the build-oss-fuzz job. Avoid building twice on gitlab, at the remote cost of potentially missing some cases that break oss-fuzz builds. Signed-off-by: Alexander Bulekov <alxndr@bu.edu> --- From a couple test runs it looks like this can shave off 15-20 minutes. scripts/oss-fuzz/build.sh | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-)