| Message ID | 20200716163330.29141-1-alxndr@bu.edu (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | gitlab-ci.yml: Add oss-fuzz build tests | expand |
On 16/07/2020 18.33, Alexander Bulekov wrote: > This tries to build and run the fuzzers with the same build-script used > by oss-fuzz. This doesn't guarantee that the builds on oss-fuzz will > also succeed, since oss-fuzz provides its own compiler and fuzzer vars, > but it can catch changes that are not compatible with the the > ./scripts/oss-fuzz/build.sh script. > The strange way of finding fuzzer binaries stems from the method used by > oss-fuzz: > https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/targets_list > > Signed-off-by: Alexander Bulekov <alxndr@bu.edu> > --- > > Similar to Thomas' patch: > >> Note: This patch needs two other patches merged first to work correctly: > >> - 'fuzz: Expect the cmdline in a freeable GString' from Alexander > >> - 'qom: Plug memory leak in "info qom-tree"' from Markus > > Otherwise the test will fail due to detected memory leaks. > > Fair warning: I haven't been able to trigger this new job yet. I tried > to run the pipeline with these changes on my forked repo on gitlab, but > did not reach the build-oss-fuzz. I think this is due to some failures > in the Containers-layer-2 stage: > > ... > Error response from daemon: manifest for > registry.gitlab.com/a1xndr/qemu/qemu/debian-all-test-cross:latest not > found: manifest unknown: manifest unknown > #2 [internal] load .dockerignore > #2 transferring context: > #2 transferring context: 2B 0.1s done > #2 DONE 0.1s > #1 [internal] load build definition from tmpg8j4xoop.docker > #1 transferring dockerfile: 2.21kB 0.1s done > #1 DONE 0.2s > #3 [internal] load metadata for docker.io/qemu/debian10:latest > #3 ERROR: pull access denied, repository does not exist or may require > authorization: server message: insufficient_scope: authorization failed These look like the problems that we've seen with the main repo until two days ago, too, e.g.: https://gitlab.com/qemu-project/qemu/-/jobs/640410842 Maybe Alex (Bennée) can comment on how to resolve them? > > .gitlab-ci.yml | 14 ++++++++++++++ > 1 file changed, 14 insertions(+) > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index e96f8794b9..a50df420c9 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -182,6 +182,20 @@ build-fuzzer: > || exit 1 ; > done As mentioned in my other mail, I think you can replace my build-fuzzer job once this is working. > +build-oss-fuzz: > + <<: *native_build_job_definition > + variables: > + IMAGE: fedora > + script: > + - OUT_DIR="./build" CC=clang-9 CXX=clang++-9 CFLAGS="-fsanitize=address" > + LIB_FUZZING_ENGINE="-fsanitize=fuzzer" CFL That "CFL" at the end seems to be a typo (leftover from "CFLAGS")? Also the fedora container does not have clang-9 : https://gitlab.com/huth/qemu/-/jobs/643383032#L28 I think it is at clang 10 already, so maybe just use CC=clang (without version number)? > + ./scripts/oss-fuzz/build.sh > + - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f); do > + grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ; > + echo Testing ${fuzzer} ... ; > + "${fuzzer}" -runs=1000 || exit 1 ; > + done Should we exclude the virtio-net tests, since they could leak network traffic to the host? Thomas
On 17/07/2020 07.40, Thomas Huth wrote: > On 16/07/2020 18.33, Alexander Bulekov wrote: >> This tries to build and run the fuzzers with the same build-script used >> by oss-fuzz. This doesn't guarantee that the builds on oss-fuzz will >> also succeed, since oss-fuzz provides its own compiler and fuzzer vars, >> but it can catch changes that are not compatible with the the >> ./scripts/oss-fuzz/build.sh script. >> The strange way of finding fuzzer binaries stems from the method used by >> oss-fuzz: >> https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/targets_list >> >> Signed-off-by: Alexander Bulekov <alxndr@bu.edu> >> --- >> >> Similar to Thomas' patch: >> >>> Note: This patch needs two other patches merged first to work correctly: >> >>> - 'fuzz: Expect the cmdline in a freeable GString' from Alexander >> >>> - 'qom: Plug memory leak in "info qom-tree"' from Markus >> >> Otherwise the test will fail due to detected memory leaks. >> >> Fair warning: I haven't been able to trigger this new job yet. I tried >> to run the pipeline with these changes on my forked repo on gitlab, but >> did not reach the build-oss-fuzz. I think this is due to some failures >> in the Containers-layer-2 stage: I think I've got it basically working like this: build-oss-fuzz: <<: *native_build_job_definition variables: IMAGE: fedora script: - mkdir build-oss-fuzz - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address" ./scripts/oss-fuzz/build.sh - for fuzzer in $(find build-oss-fuzz/DEST_DIR/ -executable -type f | grep -v slirp); do grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ; echo Testing ${fuzzer} ... ; "${fuzzer}" -runs=1000 || exit 1 ; done However, it still triggered a memory leak and thus the pipeline failed: https://gitlab.com/huth/qemu/-/jobs/643472695 ... and this was with all the other "leak fix" patches already applied. Is there an easy way to debug that issue? That information from the LeakSanitizer looks pretty sparse... Thomas
Thomas Huth <thuth@redhat.com> writes: > On 16/07/2020 18.33, Alexander Bulekov wrote: >> This tries to build and run the fuzzers with the same build-script used >> by oss-fuzz. This doesn't guarantee that the builds on oss-fuzz will >> also succeed, since oss-fuzz provides its own compiler and fuzzer vars, >> but it can catch changes that are not compatible with the the >> ./scripts/oss-fuzz/build.sh script. >> The strange way of finding fuzzer binaries stems from the method used by >> oss-fuzz: >> https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/targets_list >> >> Signed-off-by: Alexander Bulekov <alxndr@bu.edu> >> --- >> >> Similar to Thomas' patch: >> >>> Note: This patch needs two other patches merged first to work correctly: >> >>> - 'fuzz: Expect the cmdline in a freeable GString' from Alexander >> >>> - 'qom: Plug memory leak in "info qom-tree"' from Markus >> >> Otherwise the test will fail due to detected memory leaks. >> >> Fair warning: I haven't been able to trigger this new job yet. I tried >> to run the pipeline with these changes on my forked repo on gitlab, but >> did not reach the build-oss-fuzz. I think this is due to some failures >> in the Containers-layer-2 stage: >> >> ... >> Error response from daemon: manifest for >> registry.gitlab.com/a1xndr/qemu/qemu/debian-all-test-cross:latest not >> found: manifest unknown: manifest unknown >> #2 [internal] load .dockerignore >> #2 transferring context: >> #2 transferring context: 2B 0.1s done >> #2 DONE 0.1s >> #1 [internal] load build definition from tmpg8j4xoop.docker >> #1 transferring dockerfile: 2.21kB 0.1s done >> #1 DONE 0.2s >> #3 [internal] load metadata for docker.io/qemu/debian10:latest >> #3 ERROR: pull access denied, repository does not exist or may require >> authorization: server message: insufficient_scope: authorization failed > > These look like the problems that we've seen with the main repo until > two days ago, too, e.g.: > > https://gitlab.com/qemu-project/qemu/-/jobs/640410842 > > Maybe Alex (Bennée) can comment on how to resolve them? It all should be working now the qemu-project container repository has been properly seeded: https://gitlab.com/qemu-project/qemu/container_registry > >> >> .gitlab-ci.yml | 14 ++++++++++++++ >> 1 file changed, 14 insertions(+) >> >> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml >> index e96f8794b9..a50df420c9 100644 >> --- a/.gitlab-ci.yml >> +++ b/.gitlab-ci.yml >> @@ -182,6 +182,20 @@ build-fuzzer: >> || exit 1 ; >> done > > As mentioned in my other mail, I think you can replace my build-fuzzer > job once this is working. > >> +build-oss-fuzz: >> + <<: *native_build_job_definition >> + variables: >> + IMAGE: fedora >> + script: >> + - OUT_DIR="./build" CC=clang-9 CXX=clang++-9 CFLAGS="-fsanitize=address" >> + LIB_FUZZING_ENGINE="-fsanitize=fuzzer" CFL > > That "CFL" at the end seems to be a typo (leftover from "CFLAGS")? > > Also the fedora container does not have clang-9 : > > https://gitlab.com/huth/qemu/-/jobs/643383032#L28 > > I think it is at clang 10 already, so maybe just use CC=clang (without > version number)? I think all the clang-10 fixes are in now so yes. > >> + ./scripts/oss-fuzz/build.sh >> + - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f); do >> + grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ; >> + echo Testing ${fuzzer} ... ; >> + "${fuzzer}" -runs=1000 || exit 1 ; >> + done > > Should we exclude the virtio-net tests, since they could leak network > traffic to the host? > > Thomas
On 200717 0740, Thomas Huth wrote: > On 16/07/2020 18.33, Alexander Bulekov wrote: > > This tries to build and run the fuzzers with the same build-script used > > by oss-fuzz. This doesn't guarantee that the builds on oss-fuzz will > > also succeed, since oss-fuzz provides its own compiler and fuzzer vars, > > but it can catch changes that are not compatible with the the > > ./scripts/oss-fuzz/build.sh script. > > The strange way of finding fuzzer binaries stems from the method used by > > oss-fuzz: > > https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/targets_list > > > > Signed-off-by: Alexander Bulekov <alxndr@bu.edu> > > --- > > > > Similar to Thomas' patch: > > > >> Note: This patch needs two other patches merged first to work correctly: > > > >> - 'fuzz: Expect the cmdline in a freeable GString' from Alexander > > > >> - 'qom: Plug memory leak in "info qom-tree"' from Markus > > > > Otherwise the test will fail due to detected memory leaks. > > > > Fair warning: I haven't been able to trigger this new job yet. I tried > > to run the pipeline with these changes on my forked repo on gitlab, but > > did not reach the build-oss-fuzz. I think this is due to some failures > > in the Containers-layer-2 stage: > > > > ... > > Error response from daemon: manifest for > > registry.gitlab.com/a1xndr/qemu/qemu/debian-all-test-cross:latest not > > found: manifest unknown: manifest unknown > > #2 [internal] load .dockerignore > > #2 transferring context: > > #2 transferring context: 2B 0.1s done > > #2 DONE 0.1s > > #1 [internal] load build definition from tmpg8j4xoop.docker > > #1 transferring dockerfile: 2.21kB 0.1s done > > #1 DONE 0.2s > > #3 [internal] load metadata for docker.io/qemu/debian10:latest > > #3 ERROR: pull access denied, repository does not exist or may require > > authorization: server message: insufficient_scope: authorization failed > > These look like the problems that we've seen with the main repo until > two days ago, too, e.g.: > > https://gitlab.com/qemu-project/qemu/-/jobs/640410842 > > Maybe Alex (Bennée) can comment on how to resolve them? > > > > > .gitlab-ci.yml | 14 ++++++++++++++ > > 1 file changed, 14 insertions(+) > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > > index e96f8794b9..a50df420c9 100644 > > --- a/.gitlab-ci.yml > > +++ b/.gitlab-ci.yml > > @@ -182,6 +182,20 @@ build-fuzzer: > > || exit 1 ; > > done > > As mentioned in my other mail, I think you can replace my build-fuzzer > job once this is working. > > > +build-oss-fuzz: > > + <<: *native_build_job_definition > > + variables: > > + IMAGE: fedora > > + script: > > + - OUT_DIR="./build" CC=clang-9 CXX=clang++-9 CFLAGS="-fsanitize=address" > > + LIB_FUZZING_ENGINE="-fsanitize=fuzzer" CFL > > That "CFL" at the end seems to be a typo (leftover from "CFLAGS")? oops... > Also the fedora container does not have clang-9 : > > https://gitlab.com/huth/qemu/-/jobs/643383032#L28 > > I think it is at clang 10 already, so maybe just use CC=clang (without > version number)? > For some reason my local machine doesn't have symlinks for clang and clang++, and I forgot to remove the versions when I copied the command. > > + ./scripts/oss-fuzz/build.sh > > + - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f); do > > + grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ; > > + echo Testing ${fuzzer} ... ; > > + "${fuzzer}" -runs=1000 || exit 1 ; > > + done > > Should we exclude the virtio-net tests, since they could leak network > traffic to the host? Ah good point. I doubt that 1000 runs is enough to generate something that slirp will forward, but we should probably skip over these targets just to be on the safe side. -Alex > Thomas >
On 200717 0951, Thomas Huth wrote: > On 17/07/2020 07.40, Thomas Huth wrote: > > On 16/07/2020 18.33, Alexander Bulekov wrote: > >> This tries to build and run the fuzzers with the same build-script used > >> by oss-fuzz. This doesn't guarantee that the builds on oss-fuzz will > >> also succeed, since oss-fuzz provides its own compiler and fuzzer vars, > >> but it can catch changes that are not compatible with the the > >> ./scripts/oss-fuzz/build.sh script. > >> The strange way of finding fuzzer binaries stems from the method used by > >> oss-fuzz: > >> https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-runner/targets_list > >> > >> Signed-off-by: Alexander Bulekov <alxndr@bu.edu> > >> --- > >> > >> Similar to Thomas' patch: > >> > >>> Note: This patch needs two other patches merged first to work correctly: > >> > >>> - 'fuzz: Expect the cmdline in a freeable GString' from Alexander > >> > >>> - 'qom: Plug memory leak in "info qom-tree"' from Markus > >> > >> Otherwise the test will fail due to detected memory leaks. > >> > >> Fair warning: I haven't been able to trigger this new job yet. I tried > >> to run the pipeline with these changes on my forked repo on gitlab, but > >> did not reach the build-oss-fuzz. I think this is due to some failures > >> in the Containers-layer-2 stage: > > I think I've got it basically working like this: > > build-oss-fuzz: > <<: *native_build_job_definition > variables: > IMAGE: fedora > script: > - mkdir build-oss-fuzz > - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address" > ./scripts/oss-fuzz/build.sh > - for fuzzer in $(find build-oss-fuzz/DEST_DIR/ -executable -type f > | grep -v slirp); do > grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || > continue ; > echo Testing ${fuzzer} ... ; > "${fuzzer}" -runs=1000 || exit 1 ; > done > > However, it still triggered a memory leak and thus the pipeline failed: > > https://gitlab.com/huth/qemu/-/jobs/643472695 > > ... and this was with all the other "leak fix" patches already applied. > Is there an easy way to debug that issue? That information from the > LeakSanitizer looks pretty sparse... Strange... Especially since Philippe didn't get the same error. We should probably add -seed=1 after -runs=1000, to make sure the fuzzer is using the same rng seed. That said, I just ran the fuzzer for 50k iterations, and still could not reproduce the leak... This environment variable should fix the partial stacktrace, so we don't have to guess next time: ASAN_OPTIONS="fast_unwind_on_malloc=0" -Alex > Thomas >
On 17/07/2020 15.20, Alexander Bulekov wrote: > On 200717 0951, Thomas Huth wrote: >> On 17/07/2020 07.40, Thomas Huth wrote: [...] >> I think I've got it basically working like this: >> >> build-oss-fuzz: >> <<: *native_build_job_definition >> variables: >> IMAGE: fedora >> script: >> - mkdir build-oss-fuzz >> - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address" >> ./scripts/oss-fuzz/build.sh >> - for fuzzer in $(find build-oss-fuzz/DEST_DIR/ -executable -type f >> | grep -v slirp); do >> grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || >> continue ; >> echo Testing ${fuzzer} ... ; >> "${fuzzer}" -runs=1000 || exit 1 ; >> done >> >> However, it still triggered a memory leak and thus the pipeline failed: >> >> https://gitlab.com/huth/qemu/-/jobs/643472695 >> >> ... and this was with all the other "leak fix" patches already applied. >> Is there an easy way to debug that issue? That information from the >> LeakSanitizer looks pretty sparse... > > Strange... Especially since Philippe didn't get the same error. We > should probably add -seed=1 after -runs=1000, to make sure the fuzzer is > using the same rng seed. That said, I just ran the fuzzer for 50k > iterations, and still could not reproduce the leak... > > This environment variable should fix the partial stacktrace, so we don't > have to guess next time: > ASAN_OPTIONS="fast_unwind_on_malloc=0" Thanks, that did the trick: https://gitlab.com/huth/qemu/-/jobs/644354706#L3616 ... that also explains why I haven't seen it in my other tests where I am using the --fuzz-target parameter : The leak only happens if the fuzz-target is encoded in the program name - looks like we have to free the memory returned by g_path_get_dirname() there... Thomas
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index e96f8794b9..a50df420c9 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -182,6 +182,20 @@ build-fuzzer: || exit 1 ; done +build-oss-fuzz: + <<: *native_build_job_definition + variables: + IMAGE: fedora + script: + - OUT_DIR="./build" CC=clang-9 CXX=clang++-9 CFLAGS="-fsanitize=address" + LIB_FUZZING_ENGINE="-fsanitize=fuzzer" CFL + ./scripts/oss-fuzz/build.sh + - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f); do + grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ; + echo Testing ${fuzzer} ... ; + "${fuzzer}" -runs=1000 || exit 1 ; + done + build-tci: <<: *native_build_job_definition variables: