| Message ID | 20230214153945.15719-1-michal.orzel@amd.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [v2] automation: Add container and build jobs to run cppcheck analysis | expand |
On 14/02/2023 3:39 pm, Michal Orzel wrote: > diff --git a/automation/build/debian/unstable-cppcheck.dockerfile b/automation/build/debian/unstable-cppcheck.dockerfile > new file mode 100644 > index 000000000000..54b99f87dfec > --- /dev/null > +++ b/automation/build/debian/unstable-cppcheck.dockerfile > @@ -0,0 +1,37 @@ > +FROM arm64v8/debian:unstable > +LABEL maintainer.name="The Xen Project" \ > + maintainer.email="xen-devel@lists.xenproject.org" > + > +ENV DEBIAN_FRONTEND=noninteractive > +ENV CPPCHECK_VERSION=2.7 > +ENV USER root > + > +RUN mkdir /build > +WORKDIR /build > + > +# dependencies for cppcheck and Xen-only build/cross-build > +RUN apt-get update && \ > + apt-get --quiet --yes install \ > + build-essential \ > + curl \ > + python-is-python3 \ > + libpcre3-dev \ > + flex \ > + bison \ > + gcc-arm-linux-gnueabihf \ > + gcc-x86-64-linux-gnu > + > +# cppcheck release build (see cppcheck readme.md) > +RUN curl -fsSLO https://github.com/danmar/cppcheck/archive/"$CPPCHECK_VERSION".tar.gz && \ > + tar xvzf "$CPPCHECK_VERSION".tar.gz && \ > + cd cppcheck-"$CPPCHECK_VERSION" && \ > + make install -j$(nproc) \ > + MATCHCOMPILER=yes \ > + FILESDIR=/usr/share/cppcheck \ > + HAVE_RULES=yes CXXFLAGS="-O2 -DNDEBUG -Wall -Wno-sign-compare -Wno-unused-function" I think you want to be using a mutli-FROM dockerfile here, otherwise you're including all the intermediate build artefacts in the final image. See debian/buster-gcc-ibt.dockerfile for an example. That said, I'm not sure we want to be making custom containers for every minor tweak we have on a build environment. What's wrong with just putting CPPCHECK in the normal container? ~Andrew
On Tue, 14 Feb 2023, Andrew Cooper wrote: > On 14/02/2023 3:39 pm, Michal Orzel wrote: > > diff --git a/automation/build/debian/unstable-cppcheck.dockerfile b/automation/build/debian/unstable-cppcheck.dockerfile > > new file mode 100644 > > index 000000000000..54b99f87dfec > > --- /dev/null > > +++ b/automation/build/debian/unstable-cppcheck.dockerfile > > @@ -0,0 +1,37 @@ > > +FROM arm64v8/debian:unstable > > +LABEL maintainer.name="The Xen Project" \ > > + maintainer.email="xen-devel@lists.xenproject.org" > > + > > +ENV DEBIAN_FRONTEND=noninteractive > > +ENV CPPCHECK_VERSION=2.7 > > +ENV USER root > > + > > +RUN mkdir /build > > +WORKDIR /build > > + > > +# dependencies for cppcheck and Xen-only build/cross-build > > +RUN apt-get update && \ > > + apt-get --quiet --yes install \ > > + build-essential \ > > + curl \ > > + python-is-python3 \ > > + libpcre3-dev \ > > + flex \ > > + bison \ > > + gcc-arm-linux-gnueabihf \ > > + gcc-x86-64-linux-gnu > > + > > +# cppcheck release build (see cppcheck readme.md) > > +RUN curl -fsSLO https://github.com/danmar/cppcheck/archive/"$CPPCHECK_VERSION".tar.gz && \ > > + tar xvzf "$CPPCHECK_VERSION".tar.gz && \ > > + cd cppcheck-"$CPPCHECK_VERSION" && \ > > + make install -j$(nproc) \ > > + MATCHCOMPILER=yes \ > > + FILESDIR=/usr/share/cppcheck \ > > + HAVE_RULES=yes CXXFLAGS="-O2 -DNDEBUG -Wall -Wno-sign-compare -Wno-unused-function" > > I think you want to be using a mutli-FROM dockerfile here, otherwise > you're including all the intermediate build artefacts in the final image. > > See debian/buster-gcc-ibt.dockerfile for an example. > > That said, I'm not sure we want to be making custom containers for every > minor tweak we have on a build environment. What's wrong with just > putting CPPCHECK in the normal container? cppcheck is not large but needs to be built from source (as part of the Dockerfile). So we thought it would be best to keep it separate from the regular containers. I don't foresee another case like cppcheck at the moment. Also by having it separate it is clearer that this container is "special". I think it would be preferable to keep it in its own separate container but it would be OK either way.
On Tue, 14 Feb 2023, Michal Orzel wrote: > Add a debian container with cppcheck installation routine inside, > capable of performing cppcheck analysis on Xen-only build including > cross-builds for arm32 and x86_64. > > Populate build jobs making use of that container to run cppcheck > analysis to produce a text report (xen-cppcheck.txt) containing the list > of all the findings. > > This patch does not aim at performing any sort of bisection. Cppcheck is > imperfect and for now, our goal is to at least be aware of its reports, > so that we can compare them with the ones produced by better tools and > to be able to see how these reports change as a result of further > infrastructure improvements (e.g. exception list, rules exclusion). > > Signed-off-by: Michal Orzel <michal.orzel@amd.com> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org> > --- > Changes in v2: > - use arm64 container instead of x86 to make pipeline faster > - explicitly set HYPERVISOR_ONLY=y for cppcheck jobs > > For convenience and own testing, I built and pushed the new container > to registry. CI pipeline including dom0less series: > https://gitlab.com/xen-project/people/morzel/xen-orzelmichal/-/pipelines/777181033 > --- > .../build/debian/unstable-cppcheck.dockerfile | 37 ++++++++++++++++ > automation/gitlab-ci/build.yaml | 43 +++++++++++++++++++ > automation/scripts/build | 11 ++++- > 3 files changed, 90 insertions(+), 1 deletion(-) > create mode 100644 automation/build/debian/unstable-cppcheck.dockerfile > > diff --git a/automation/build/debian/unstable-cppcheck.dockerfile b/automation/build/debian/unstable-cppcheck.dockerfile > new file mode 100644 > index 000000000000..54b99f87dfec > --- /dev/null > +++ b/automation/build/debian/unstable-cppcheck.dockerfile > @@ -0,0 +1,37 @@ > +FROM arm64v8/debian:unstable > +LABEL maintainer.name="The Xen Project" \ > + maintainer.email="xen-devel@lists.xenproject.org" > + > +ENV DEBIAN_FRONTEND=noninteractive > +ENV CPPCHECK_VERSION=2.7 > +ENV USER root > + > +RUN mkdir /build > +WORKDIR /build > + > +# dependencies for cppcheck and Xen-only build/cross-build > +RUN apt-get update && \ > + apt-get --quiet --yes install \ > + build-essential \ > + curl \ > + python-is-python3 \ > + libpcre3-dev \ > + flex \ > + bison \ > + gcc-arm-linux-gnueabihf \ > + gcc-x86-64-linux-gnu > + > +# cppcheck release build (see cppcheck readme.md) > +RUN curl -fsSLO https://github.com/danmar/cppcheck/archive/"$CPPCHECK_VERSION".tar.gz && \ > + tar xvzf "$CPPCHECK_VERSION".tar.gz && \ > + cd cppcheck-"$CPPCHECK_VERSION" && \ > + make install -j$(nproc) \ > + MATCHCOMPILER=yes \ > + FILESDIR=/usr/share/cppcheck \ > + HAVE_RULES=yes CXXFLAGS="-O2 -DNDEBUG -Wall -Wno-sign-compare -Wno-unused-function" > + > +# clean > +RUN apt-get autoremove -y && \ > + apt-get clean && \ > + rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/* && \ > + rm -rf cppcheck-"$CPPCHECK_VERSION"* "$CPPCHECK_VERSION".tar.gz > diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml > index 079e9b73f659..1441b7dbb6fa 100644 > --- a/automation/gitlab-ci/build.yaml > +++ b/automation/gitlab-ci/build.yaml > @@ -7,6 +7,7 @@ > paths: > - binaries/ > - xen-config > + - xen-cppcheck.txt > - '*.log' > - '*/*.log' > when: always > @@ -199,6 +200,23 @@ > variables: > <<: *gcc > > +.x86-64-cross-build-tmpl: > + <<: *build > + variables: > + XEN_TARGET_ARCH: x86_64 > + tags: > + - arm64 > + > +.x86-64-cross-build: > + extends: .x86-64-cross-build-tmpl > + variables: > + debug: n > + > +.gcc-x86-64-cross-build: > + extends: .x86-64-cross-build > + variables: > + <<: *gcc > + > # Jobs below this line > > archlinux-gcc: > @@ -699,6 +717,31 @@ archlinux-current-gcc-riscv64-debug-randconfig: > EXTRA_FIXED_RANDCONFIG: > CONFIG_COVERAGE=n > > +# Cppcheck analysis jobs > + > +debian-unstable-gcc-cppcheck: > + extends: .gcc-x86-64-cross-build > + variables: > + CONTAINER: debian:unstable-cppcheck > + CROSS_COMPILE: /usr/bin/x86_64-linux-gnu- > + CPPCHECK: y > + HYPERVISOR_ONLY: y > + > +debian-unstable-gcc-arm32-cppcheck: > + extends: .gcc-arm32-cross-build > + variables: > + CONTAINER: debian:unstable-cppcheck > + CROSS_COMPILE: /usr/bin/arm-linux-gnueabihf- > + CPPCHECK: y > + HYPERVISOR_ONLY: y > + > +debian-unstable-gcc-arm64-cppcheck: > + extends: .gcc-arm64-build > + variables: > + CONTAINER: debian:unstable-cppcheck > + CPPCHECK: y > + HYPERVISOR_ONLY: y > + > ## Test artifacts common > > .test-jobs-artifact-common: > diff --git a/automation/scripts/build b/automation/scripts/build > index f2f5e55bc04f..7d1b19c4250d 100755 > --- a/automation/scripts/build > +++ b/automation/scripts/build > @@ -38,7 +38,16 @@ cp xen/.config xen-config > # Directory for the artefacts to be dumped into > mkdir binaries > > -if [[ "${HYPERVISOR_ONLY}" == "y" ]]; then > +if [[ "${CPPCHECK}" == "y" ]] && [[ "${HYPERVISOR_ONLY}" == "y" ]]; then > + # Cppcheck analysis invokes Xen-only build. > + # Known limitation: cppcheck generates inconsistent reports when running > + # in parallel mode, therefore do not specify -j<n>. > + xen/scripts/xen-analysis.py --run-cppcheck --cppcheck-misra > + > + # Preserve artefacts > + cp xen/xen binaries/xen > + cp xen/cppcheck-report/xen-cppcheck.txt xen-cppcheck.txt > +elif [[ "${HYPERVISOR_ONLY}" == "y" ]]; then > # Xen-only build > make -j$(nproc) xen > > -- > 2.25.1 >
diff --git a/automation/build/debian/unstable-cppcheck.dockerfile b/automation/build/debian/unstable-cppcheck.dockerfile new file mode 100644 index 000000000000..54b99f87dfec --- /dev/null +++ b/automation/build/debian/unstable-cppcheck.dockerfile @@ -0,0 +1,37 @@ +FROM arm64v8/debian:unstable +LABEL maintainer.name="The Xen Project" \ + maintainer.email="xen-devel@lists.xenproject.org" + +ENV DEBIAN_FRONTEND=noninteractive +ENV CPPCHECK_VERSION=2.7 +ENV USER root + +RUN mkdir /build +WORKDIR /build + +# dependencies for cppcheck and Xen-only build/cross-build +RUN apt-get update && \ + apt-get --quiet --yes install \ + build-essential \ + curl \ + python-is-python3 \ + libpcre3-dev \ + flex \ + bison \ + gcc-arm-linux-gnueabihf \ + gcc-x86-64-linux-gnu + +# cppcheck release build (see cppcheck readme.md) +RUN curl -fsSLO https://github.com/danmar/cppcheck/archive/"$CPPCHECK_VERSION".tar.gz && \ + tar xvzf "$CPPCHECK_VERSION".tar.gz && \ + cd cppcheck-"$CPPCHECK_VERSION" && \ + make install -j$(nproc) \ + MATCHCOMPILER=yes \ + FILESDIR=/usr/share/cppcheck \ + HAVE_RULES=yes CXXFLAGS="-O2 -DNDEBUG -Wall -Wno-sign-compare -Wno-unused-function" + +# clean +RUN apt-get autoremove -y && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/* && \ + rm -rf cppcheck-"$CPPCHECK_VERSION"* "$CPPCHECK_VERSION".tar.gz diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml index 079e9b73f659..1441b7dbb6fa 100644 --- a/automation/gitlab-ci/build.yaml +++ b/automation/gitlab-ci/build.yaml @@ -7,6 +7,7 @@ paths: - binaries/ - xen-config + - xen-cppcheck.txt - '*.log' - '*/*.log' when: always @@ -199,6 +200,23 @@ variables: <<: *gcc +.x86-64-cross-build-tmpl: + <<: *build + variables: + XEN_TARGET_ARCH: x86_64 + tags: + - arm64 + +.x86-64-cross-build: + extends: .x86-64-cross-build-tmpl + variables: + debug: n + +.gcc-x86-64-cross-build: + extends: .x86-64-cross-build + variables: + <<: *gcc + # Jobs below this line archlinux-gcc: @@ -699,6 +717,31 @@ archlinux-current-gcc-riscv64-debug-randconfig: EXTRA_FIXED_RANDCONFIG: CONFIG_COVERAGE=n +# Cppcheck analysis jobs + +debian-unstable-gcc-cppcheck: + extends: .gcc-x86-64-cross-build + variables: + CONTAINER: debian:unstable-cppcheck + CROSS_COMPILE: /usr/bin/x86_64-linux-gnu- + CPPCHECK: y + HYPERVISOR_ONLY: y + +debian-unstable-gcc-arm32-cppcheck: + extends: .gcc-arm32-cross-build + variables: + CONTAINER: debian:unstable-cppcheck + CROSS_COMPILE: /usr/bin/arm-linux-gnueabihf- + CPPCHECK: y + HYPERVISOR_ONLY: y + +debian-unstable-gcc-arm64-cppcheck: + extends: .gcc-arm64-build + variables: + CONTAINER: debian:unstable-cppcheck + CPPCHECK: y + HYPERVISOR_ONLY: y + ## Test artifacts common .test-jobs-artifact-common: diff --git a/automation/scripts/build b/automation/scripts/build index f2f5e55bc04f..7d1b19c4250d 100755 --- a/automation/scripts/build +++ b/automation/scripts/build @@ -38,7 +38,16 @@ cp xen/.config xen-config # Directory for the artefacts to be dumped into mkdir binaries -if [[ "${HYPERVISOR_ONLY}" == "y" ]]; then +if [[ "${CPPCHECK}" == "y" ]] && [[ "${HYPERVISOR_ONLY}" == "y" ]]; then + # Cppcheck analysis invokes Xen-only build. + # Known limitation: cppcheck generates inconsistent reports when running + # in parallel mode, therefore do not specify -j<n>. + xen/scripts/xen-analysis.py --run-cppcheck --cppcheck-misra + + # Preserve artefacts + cp xen/xen binaries/xen + cp xen/cppcheck-report/xen-cppcheck.txt xen-cppcheck.txt +elif [[ "${HYPERVISOR_ONLY}" == "y" ]]; then # Xen-only build make -j$(nproc) xen
Add a debian container with cppcheck installation routine inside, capable of performing cppcheck analysis on Xen-only build including cross-builds for arm32 and x86_64. Populate build jobs making use of that container to run cppcheck analysis to produce a text report (xen-cppcheck.txt) containing the list of all the findings. This patch does not aim at performing any sort of bisection. Cppcheck is imperfect and for now, our goal is to at least be aware of its reports, so that we can compare them with the ones produced by better tools and to be able to see how these reports change as a result of further infrastructure improvements (e.g. exception list, rules exclusion). Signed-off-by: Michal Orzel <michal.orzel@amd.com> --- Changes in v2: - use arm64 container instead of x86 to make pipeline faster - explicitly set HYPERVISOR_ONLY=y for cppcheck jobs For convenience and own testing, I built and pushed the new container to registry. CI pipeline including dom0less series: https://gitlab.com/xen-project/people/morzel/xen-orzelmichal/-/pipelines/777181033 --- .../build/debian/unstable-cppcheck.dockerfile | 37 ++++++++++++++++ automation/gitlab-ci/build.yaml | 43 +++++++++++++++++++ automation/scripts/build | 11 ++++- 3 files changed, 90 insertions(+), 1 deletion(-) create mode 100644 automation/build/debian/unstable-cppcheck.dockerfile