| Message ID | 20200605175028.5626-1-alxndr@bu.edu (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v2] fuzz: add oss-fuzz build.sh script | expand |
Hi Alex, From looking at another OSS Fuzz project recently (a coincidence) I wonder if we could make this script work so that it can be run outside of the OSS-Fuzz environment? Specifically, for example, if $OUT is not set, then creating a subdir in the build directory, and setting it to be that. Similarly for some other things like $LIB_FUZZING_ENGINE? I'm just thinking that it might help someone that is not familiar with OSS-Fuzz to validate that the script still works without having to go through setting up the containers, etc that would be required to validate it. Also, I would definitely recommend running ShellCheck against any script to ensure that you're catching any mistakes that can so easily be put in to shell scripts - speaking from experience here ;) Thanks, Darren. On Friday, 2020-06-05 at 13:50:28 -04, Alexander Bulekov wrote: > It is neater to keep this in the QEMU repo, since any change that > requires an update to the oss-fuzz build configuration, can make the > necessary changes in the same series. > > Suggested-by: Philippe Mathieu-Daude <f4bug@amsat.org> > Signed-off-by: Alexander Bulekov <alxndr@bu.edu> > --- > > v2 updates the script header comment. > > scripts/oss-fuzz/build.sh | 50 +++++++++++++++++++++++++++++++++++++++ > 1 file changed, 50 insertions(+) > create mode 100755 scripts/oss-fuzz/build.sh > > diff --git a/scripts/oss-fuzz/build.sh b/scripts/oss-fuzz/build.sh > new file mode 100755 > index 0000000000..e93d6f2e03 > --- /dev/null > +++ b/scripts/oss-fuzz/build.sh > @@ -0,0 +1,50 @@ > +#!/bin/sh > +# > +# OSS-Fuzz build script. See: > +# https://google.github.io/oss-fuzz/getting-started/new-project-guide/#buildsh > +# > +# This code is licensed under the GPL version 2 or later. See > +# the COPYING file in the top-level directory. > +# > + > +# build project > +# e.g. > +# ./autogen.sh > +# ./configure > +# make -j$(nproc) all > + > +# build fuzzers > +# e.g. > +# $CXX $CXXFLAGS -std=c++11 -Iinclude \ > +# /path/to/name_of_fuzzer.cc -o $OUT/name_of_fuzzer \ > +# $LIB_FUZZING_ENGINE /path/to/library.a > + > +mkdir -p $OUT/lib/ # Shared libraries > + > +# Build once to get the list of dynamic lib paths, and copy them over > +./configure --datadir="./data/" --disable-werror --cc="$CC" --cxx="$CXX" \ > + --extra-cflags="$CFLAGS -U __OPTIMIZE__ " > +make CONFIG_FUZZ=y CFLAGS="$LIB_FUZZING_ENGINE" -j$(nproc) i386-softmmu/fuzz > + > +for i in $(ldd ./i386-softmmu/qemu-fuzz-i386 | cut -f3 -d' '); do > + cp $i $OUT/lib/ > +done > +rm ./i386-softmmu/qemu-fuzz-i386 > + > +# Build a second time to build the final binary with correct rpath > +./configure --datadir="./data/" --disable-werror --cc="$CC" --cxx="$CXX" \ > + --extra-cflags="$CFLAGS -U __OPTIMIZE__" \ > + --extra-ldflags="-Wl,-rpath,'\$\$ORIGIN/lib'" > +make CONFIG_FUZZ=y CFLAGS="$LIB_FUZZING_ENGINE" -j$(nproc) i386-softmmu/fuzz > + > +# Copy over the datadir > +cp -r ./pc-bios/ $OUT/pc-bios > + > +# Run the fuzzer with no arguments, to print the help-string and get the list > +# of available fuzz-targets. Copy over the qemu-fuzz-i386, naming it according > +# to each available fuzz target (See 05509c8e6d fuzz: select fuzz target using > +# executable name) > +for target in $(./i386-softmmu/qemu-fuzz-i386 | awk '$1 ~ /\*/ {print $2}'); > +do > + cp ./i386-softmmu/qemu-fuzz-i386 $OUT/qemu-fuzz-i386-target-$target > +done > -- > 2.26.2
Hi Darren, On 200605 1858, Darren Kenny wrote: > Hi Alex, > > From looking at another OSS Fuzz project recently (a coincidence) I > wonder if we could make this script work so that it can be run outside > of the OSS-Fuzz environment? > > Specifically, for example, if $OUT is not set, then creating a subdir in > the build directory, and setting it to be that. > For $OUT, do you think it would be better to require it as a user-configurable environment variable? My concern is that making it a subdirectory of the build dir would mean that the pc-bios files exist located in $OUT/../pc-bios. This doesn't reflect OSS-Fuzz, where we specifically have to copy them to $OUT/pc-bios/ > Similarly for some other things like $LIB_FUZZING_ENGINE? Will do. > I'm just thinking that it might help someone that is not familiar with > OSS-Fuzz to validate that the script still works without having to go > through setting up the containers, etc that would be required to > validate it. > > Also, I would definitely recommend running ShellCheck against any script > to ensure that you're catching any mistakes that can so easily be put in > to shell scripts - speaking from experience here ;) I will :) > Thanks, > > Darren. Thanks for bringing these up! -Alex > > On Friday, 2020-06-05 at 13:50:28 -04, Alexander Bulekov wrote: > > It is neater to keep this in the QEMU repo, since any change that > > requires an update to the oss-fuzz build configuration, can make the > > necessary changes in the same series. > > > > Suggested-by: Philippe Mathieu-Daude <f4bug@amsat.org> > > Signed-off-by: Alexander Bulekov <alxndr@bu.edu> > > --- > > > > v2 updates the script header comment. > > > > scripts/oss-fuzz/build.sh | 50 +++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 50 insertions(+) > > create mode 100755 scripts/oss-fuzz/build.sh > > > > diff --git a/scripts/oss-fuzz/build.sh b/scripts/oss-fuzz/build.sh > > new file mode 100755 > > index 0000000000..e93d6f2e03 > > --- /dev/null > > +++ b/scripts/oss-fuzz/build.sh > > @@ -0,0 +1,50 @@ > > +#!/bin/sh > > +# > > +# OSS-Fuzz build script. See: > > +# https://google.github.io/oss-fuzz/getting-started/new-project-guide/#buildsh > > +# > > +# This code is licensed under the GPL version 2 or later. See > > +# the COPYING file in the top-level directory. > > +# > > + > > +# build project > > +# e.g. > > +# ./autogen.sh > > +# ./configure > > +# make -j$(nproc) all > > + > > +# build fuzzers > > +# e.g. > > +# $CXX $CXXFLAGS -std=c++11 -Iinclude \ > > +# /path/to/name_of_fuzzer.cc -o $OUT/name_of_fuzzer \ > > +# $LIB_FUZZING_ENGINE /path/to/library.a > > + > > +mkdir -p $OUT/lib/ # Shared libraries > > + > > +# Build once to get the list of dynamic lib paths, and copy them over > > +./configure --datadir="./data/" --disable-werror --cc="$CC" --cxx="$CXX" \ > > + --extra-cflags="$CFLAGS -U __OPTIMIZE__ " > > +make CONFIG_FUZZ=y CFLAGS="$LIB_FUZZING_ENGINE" -j$(nproc) i386-softmmu/fuzz > > + > > +for i in $(ldd ./i386-softmmu/qemu-fuzz-i386 | cut -f3 -d' '); do > > + cp $i $OUT/lib/ > > +done > > +rm ./i386-softmmu/qemu-fuzz-i386 > > + > > +# Build a second time to build the final binary with correct rpath > > +./configure --datadir="./data/" --disable-werror --cc="$CC" --cxx="$CXX" \ > > + --extra-cflags="$CFLAGS -U __OPTIMIZE__" \ > > + --extra-ldflags="-Wl,-rpath,'\$\$ORIGIN/lib'" > > +make CONFIG_FUZZ=y CFLAGS="$LIB_FUZZING_ENGINE" -j$(nproc) i386-softmmu/fuzz > > + > > +# Copy over the datadir > > +cp -r ./pc-bios/ $OUT/pc-bios > > + > > +# Run the fuzzer with no arguments, to print the help-string and get the list > > +# of available fuzz-targets. Copy over the qemu-fuzz-i386, naming it according > > +# to each available fuzz target (See 05509c8e6d fuzz: select fuzz target using > > +# executable name) > > +for target in $(./i386-softmmu/qemu-fuzz-i386 | awk '$1 ~ /\*/ {print $2}'); > > +do > > + cp ./i386-softmmu/qemu-fuzz-i386 $OUT/qemu-fuzz-i386-target-$target > > +done > > -- > > 2.26.2
On 6/5/20 7:58 PM, Darren Kenny wrote: > Hi Alex, > > From looking at another OSS Fuzz project recently (a coincidence) I > wonder if we could make this script work so that it can be run outside > of the OSS-Fuzz environment? Yes, we want to have this feature to reproduce/debug what oss-fuzz does. > > Specifically, for example, if $OUT is not set, then creating a subdir in > the build directory, and setting it to be that. > > Similarly for some other things like $LIB_FUZZING_ENGINE? > > I'm just thinking that it might help someone that is not familiar with > OSS-Fuzz to validate that the script still works without having to go > through setting up the containers, etc that would be required to > validate it. Exactly. > > Also, I would definitely recommend running ShellCheck against any script > to ensure that you're catching any mistakes that can so easily be put in > to shell scripts - speaking from experience here ;) > > Thanks, > > Darren. [...]
On Friday, 2020-06-05 at 14:24:59 -04, Alexander Bulekov wrote: > Hi Darren, > > On 200605 1858, Darren Kenny wrote: >> Hi Alex, >> >> From looking at another OSS Fuzz project recently (a coincidence) I >> wonder if we could make this script work so that it can be run outside >> of the OSS-Fuzz environment? >> >> Specifically, for example, if $OUT is not set, then creating a subdir in >> the build directory, and setting it to be that. >> > For $OUT, do you think it would be better to require it as > a user-configurable environment variable? My concern is that making it > a subdirectory of the build dir would mean that the pc-bios files exist > located in $OUT/../pc-bios. This doesn't reflect OSS-Fuzz, where we > specifically have to copy them to $OUT/pc-bios/ > The script is copying them in to $OUT near the end still, isn't it? That should be fine if it is, shouldn't it? Or am I missing something? Thanks, Darren.
Patchew URL: https://patchew.org/QEMU/20200605175028.5626-1-alxndr@bu.edu/ Hi, This series seems to have some coding style problems. See output below for more information: Message-id: 20200605175028.5626-1-alxndr@bu.edu Subject: [PATCH v2] fuzz: add oss-fuzz build.sh script Type: series === TEST SCRIPT BEGIN === #!/bin/bash git rev-parse base > /dev/null || exit 0 git config --local diff.renamelimit 0 git config --local diff.renames True git config --local diff.algorithm histogram ./scripts/checkpatch.pl --mailback base.. === TEST SCRIPT END === Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384 Switched to a new branch 'test' c9ada67 fuzz: add oss-fuzz build.sh script === OUTPUT BEGIN === WARNING: added, moved or deleted file(s), does MAINTAINERS need updating? #16: new file mode 100755 ERROR: trailing whitespace #49: FILE: scripts/oss-fuzz/build.sh:29: +for i in $(ldd ./i386-softmmu/qemu-fuzz-i386 | cut -f3 -d' '); do $ total: 1 errors, 1 warnings, 50 lines checked Commit c9ada6738325 (fuzz: add oss-fuzz build.sh script) has style problems, please review. If any of these errors are false positives report them to the maintainer, see CHECKPATCH in MAINTAINERS. === OUTPUT END === Test command exited with code: 1 The full log is available at http://patchew.org/logs/20200605175028.5626-1-alxndr@bu.edu/testing.checkpatch/?type=message. --- Email generated automatically by Patchew [https://patchew.org/]. Please send your feedback to patchew-devel@redhat.com
On Fri, Jun 05, 2020 at 01:50:28PM -0400, Alexander Bulekov wrote: > It is neater to keep this in the QEMU repo, since any change that > requires an update to the oss-fuzz build configuration, can make the > necessary changes in the same series. > > Suggested-by: Philippe Mathieu-Daude <f4bug@amsat.org> > Signed-off-by: Alexander Bulekov <alxndr@bu.edu> > --- > > v2 updates the script header comment. > > scripts/oss-fuzz/build.sh | 50 +++++++++++++++++++++++++++++++++++++++ > 1 file changed, 50 insertions(+) > create mode 100755 scripts/oss-fuzz/build.sh Please add a MAINTAINERS file entry. > diff --git a/scripts/oss-fuzz/build.sh b/scripts/oss-fuzz/build.sh > new file mode 100755 > index 0000000000..e93d6f2e03 > --- /dev/null > +++ b/scripts/oss-fuzz/build.sh > @@ -0,0 +1,50 @@ > +#!/bin/sh > +# > +# OSS-Fuzz build script. See: > +# https://google.github.io/oss-fuzz/getting-started/new-project-guide/#buildsh > +# > +# This code is licensed under the GPL version 2 or later. See > +# the COPYING file in the top-level directory. > +# > + > +# build project > +# e.g. > +# ./autogen.sh > +# ./configure > +# make -j$(nproc) all > + > +# build fuzzers > +# e.g. > +# $CXX $CXXFLAGS -std=c++11 -Iinclude \ > +# /path/to/name_of_fuzzer.cc -o $OUT/name_of_fuzzer \ > +# $LIB_FUZZING_ENGINE /path/to/library.a > + > +mkdir -p $OUT/lib/ # Shared libraries > + > +# Build once to get the list of dynamic lib paths, and copy them over > +./configure --datadir="./data/" --disable-werror --cc="$CC" --cxx="$CXX" \ > + --extra-cflags="$CFLAGS -U __OPTIMIZE__ " If you respin, please extract -U __OPTIMIZE__ into a separate variable with a doc comment. # Explanation for why __OPTIMIZE__ needs to be undefined goes here... EXTRA_CFLAGS="$CFLAGS -U __OPTIMIZE__" This will help others understand why this build flag is needed.
diff --git a/scripts/oss-fuzz/build.sh b/scripts/oss-fuzz/build.sh new file mode 100755 index 0000000000..e93d6f2e03 --- /dev/null +++ b/scripts/oss-fuzz/build.sh @@ -0,0 +1,50 @@ +#!/bin/sh +# +# OSS-Fuzz build script. See: +# https://google.github.io/oss-fuzz/getting-started/new-project-guide/#buildsh +# +# This code is licensed under the GPL version 2 or later. See +# the COPYING file in the top-level directory. +# + +# build project +# e.g. +# ./autogen.sh +# ./configure +# make -j$(nproc) all + +# build fuzzers +# e.g. +# $CXX $CXXFLAGS -std=c++11 -Iinclude \ +# /path/to/name_of_fuzzer.cc -o $OUT/name_of_fuzzer \ +# $LIB_FUZZING_ENGINE /path/to/library.a + +mkdir -p $OUT/lib/ # Shared libraries + +# Build once to get the list of dynamic lib paths, and copy them over +./configure --datadir="./data/" --disable-werror --cc="$CC" --cxx="$CXX" \ + --extra-cflags="$CFLAGS -U __OPTIMIZE__ " +make CONFIG_FUZZ=y CFLAGS="$LIB_FUZZING_ENGINE" -j$(nproc) i386-softmmu/fuzz + +for i in $(ldd ./i386-softmmu/qemu-fuzz-i386 | cut -f3 -d' '); do + cp $i $OUT/lib/ +done +rm ./i386-softmmu/qemu-fuzz-i386 + +# Build a second time to build the final binary with correct rpath +./configure --datadir="./data/" --disable-werror --cc="$CC" --cxx="$CXX" \ + --extra-cflags="$CFLAGS -U __OPTIMIZE__" \ + --extra-ldflags="-Wl,-rpath,'\$\$ORIGIN/lib'" +make CONFIG_FUZZ=y CFLAGS="$LIB_FUZZING_ENGINE" -j$(nproc) i386-softmmu/fuzz + +# Copy over the datadir +cp -r ./pc-bios/ $OUT/pc-bios + +# Run the fuzzer with no arguments, to print the help-string and get the list +# of available fuzz-targets. Copy over the qemu-fuzz-i386, naming it according +# to each available fuzz target (See 05509c8e6d fuzz: select fuzz target using +# executable name) +for target in $(./i386-softmmu/qemu-fuzz-i386 | awk '$1 ~ /\*/ {print $2}'); +do + cp ./i386-softmmu/qemu-fuzz-i386 $OUT/qemu-fuzz-i386-target-$target +done
It is neater to keep this in the QEMU repo, since any change that requires an update to the oss-fuzz build configuration, can make the necessary changes in the same series. Suggested-by: Philippe Mathieu-Daude <f4bug@amsat.org> Signed-off-by: Alexander Bulekov <alxndr@bu.edu> --- v2 updates the script header comment. scripts/oss-fuzz/build.sh | 50 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100755 scripts/oss-fuzz/build.sh