| Message ID | 20240228225527.1052240-2-helen.koike@collabora.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | kci-gitlab: Introducing GitLab-CI Pipeline for Kernel Testing | expand |
> -----Original Message----- > From: Helen Koike <helen.koike@collabora.com> > .... > Hey all, > > You can check the validation of this patchset on: > https://gitlab.collabora.com/koike/linux/-/pipelines/87035 > > I would appreciate your feedback on this work, what do you think? > > If you would rate from 0 to 5, where: > > [ ] 0. I don't think this is useful at all, and I doubt it will ever be. It doesn't seem worthwhile. > [ ] 1. I don't find it useful in its current form. > [ ] 2. It might be useful to others, but not for me. > [ ] 3. It has potential, but it's not yet something I can incorporate into my workflow. > [ ] 4. This is useful, but it needs some adjustments before I can include it in my workflow. > [ ] 5. This is really useful! I'm eager to start using it right away. Why didn't you send this earlier? :) > > Which rating would you select? For me, this is a "5". I don't currently use gitlab, but I might install it just to test this out. It looks like there are a large number of YAML files which define the integration between the test scripts and gitlab. Also, there are a number of shell scripts to perform some of the setup and tests. Do you have any idea how difficult it would be to use the shell scripts outside of the CI system (e.g. manually)? That is, are there dependencies between the CI system and the shell scripts? I think the convention, of putting this kind of stuff under a "ci" directory, makes sense. And it appears that the sub-dir structure allows for other CI systems to add their own config and files also. I have a few minor comments below. > > --- > .gitlab-ci.yml | 2 + > MAINTAINERS | 8 ++ > ci/gitlab-ci/bootstrap-gitlab-runner.sh | 55 +++++++++ > ci/gitlab-ci/ci-scripts/build-docs.sh | 35 ++++++ > ci/gitlab-ci/ci-scripts/build-kernel.sh | 35 ++++++ > ci/gitlab-ci/ci-scripts/ici-functions.sh | 104 ++++++++++++++++++ > ci/gitlab-ci/ci-scripts/install-smatch.sh | 13 +++ > .../ci-scripts/parse_commit_message.sh | 27 +++++ > ci/gitlab-ci/ci-scripts/run-checkpatch.sh | 20 ++++ > ci/gitlab-ci/ci-scripts/run-smatch.sh | 45 ++++++++ > ci/gitlab-ci/docker-compose.yaml | 18 +++ > ci/gitlab-ci/linux.code-workspace | 11 ++ > ci/gitlab-ci/yml/build.yml | 43 ++++++++ > ci/gitlab-ci/yml/cache.yml | 26 +++++ > ci/gitlab-ci/yml/container.yml | 36 ++++++ > ci/gitlab-ci/yml/gitlab-ci.yml | 71 ++++++++++++ > ci/gitlab-ci/yml/kernel-combinations.yml | 18 +++ > ci/gitlab-ci/yml/scenarios.yml | 12 ++ > ci/gitlab-ci/yml/scenarios/file-systems.yml | 21 ++++ > ci/gitlab-ci/yml/scenarios/media.yml | 21 ++++ > ci/gitlab-ci/yml/scenarios/network.yml | 21 ++++ > ci/gitlab-ci/yml/static-checks.yml | 21 ++++ > 22 files changed, 663 insertions(+) > create mode 100644 .gitlab-ci.yml > create mode 100755 ci/gitlab-ci/bootstrap-gitlab-runner.sh > create mode 100755 ci/gitlab-ci/ci-scripts/build-docs.sh > create mode 100755 ci/gitlab-ci/ci-scripts/build-kernel.sh > create mode 100644 ci/gitlab-ci/ci-scripts/ici-functions.sh > create mode 100755 ci/gitlab-ci/ci-scripts/install-smatch.sh > create mode 100755 ci/gitlab-ci/ci-scripts/parse_commit_message.sh > create mode 100755 ci/gitlab-ci/ci-scripts/run-checkpatch.sh > create mode 100755 ci/gitlab-ci/ci-scripts/run-smatch.sh > create mode 100644 ci/gitlab-ci/docker-compose.yaml > create mode 100644 ci/gitlab-ci/linux.code-workspace > create mode 100644 ci/gitlab-ci/yml/build.yml > create mode 100644 ci/gitlab-ci/yml/cache.yml > create mode 100644 ci/gitlab-ci/yml/container.yml > create mode 100644 ci/gitlab-ci/yml/gitlab-ci.yml > create mode 100644 ci/gitlab-ci/yml/kernel-combinations.yml > create mode 100644 ci/gitlab-ci/yml/scenarios.yml > create mode 100644 ci/gitlab-ci/yml/scenarios/file-systems.yml > create mode 100644 ci/gitlab-ci/yml/scenarios/media.yml > create mode 100644 ci/gitlab-ci/yml/scenarios/network.yml > create mode 100644 ci/gitlab-ci/yml/static-checks.yml > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > new file mode 100644 > index 0000000000000..fac523bb86866 > --- /dev/null > +++ b/.gitlab-ci.yml > @@ -0,0 +1,2 @@ > +include: > + - ci/gitlab-ci/yml/gitlab-ci.yml > diff --git a/MAINTAINERS b/MAINTAINERS > index 716b2e22598c8..aa0f65791c2ee 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -4998,6 +4998,14 @@ T: git git://linuxtv.org/media_tree.git > F: Documentation/devicetree/bindings/media/i2c/chrontel,ch7322.yaml > F: drivers/media/cec/i2c/ch7322.c > > +CI AUTOMATED TESTING > +M: Helen Koike <helen.koike@collabora.com> > +L: linux-kernel@vger.kernel.org > +S: Maintained > +T: git https://gitlab.collabora.com/koike/linux.git > +F: .gitlab-ci.yml > +F: ci/ > + > CIRRUS LOGIC AUDIO CODEC DRIVERS > M: James Schulman <james.schulman@cirrus.com> > M: David Rhodes <david.rhodes@cirrus.com> > diff --git a/ci/gitlab-ci/bootstrap-gitlab-runner.sh b/ci/gitlab-ci/bootstrap-gitlab-runner.sh > new file mode 100755 > index 0000000000000..73238960d0880 > --- /dev/null > +++ b/ci/gitlab-ci/bootstrap-gitlab-runner.sh > @@ -0,0 +1,55 @@ > +#!/bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +set -eo pipefail > + > +# Define variables > +CONFIG_VOLUME="/srv/gitlab-runner/config" # Path to your GitLab Runner config > + > +# Check if RUNNER_REGISTRATION_TOKEN is set > +if [ -z "${RUNNER_REGISTRATION_TOKEN}" ]; then > + echo "Error: RUNNER_REGISTRATION_TOKEN is not set." > + echo "Please set the RUNNER_REGISTRATION_TOKEN environment variable and try again." > + exit 1 > +fi > + > +# Check if GITLAB_URL is set > +if [ -z "${GITLAB_URL}" ]; then > + GITLAB_URL="https://gitlab.com/" > + echo "Info: GITLAB_URL is not set. Using the default $GITLAB_URL" > + echo "Please set the RUNNER_REGISTRATION_TOKEN environment variable and try again." > +fi > + > +# Check if docker-compose is installed > +if ! command -v docker-compose &> /dev/null > +then > + echo "docker-compose could not be found. Please install it first." > + exit 1 > +fi > + > +# Start the GitLab Runner using Docker Compose > +echo "Starting GitLab Runner..." > +docker-compose up -d > + > +# Wait for a few seconds to ensure the service is up > +sleep 5 > + > +# Register the GitLab Runner > +echo "Registering GitLab Runner..." > +docker run --rm -v ${CONFIG_VOLUME}:/etc/gitlab-runner gitlab/gitlab-runner register \ > + --non-interactive \ > + --url ${GITLAB_URL} \ > + --token ${RUNNER_REGISTRATION_TOKEN} \ > + --executor docker \ > + --docker-image "alpine:latest" \ > + --description "Docker Runner" \ > + --docker-privileged > + > +echo "" > +echo "INFO: To configure the number of concurrent jobs, edit the value of" > +echo "INFO: concurrent in ${CONFIG_VOLUME}/config.toml, than restart the GitLab" > +echo "INFO: Runner using docker-compose restart" > +echo "" > +echo "GitLab Runner setup complete." > diff --git a/ci/gitlab-ci/ci-scripts/build-docs.sh b/ci/gitlab-ci/ci-scripts/build-docs.sh > new file mode 100755 > index 0000000000000..2053cb8794acf > --- /dev/null > +++ b/ci/gitlab-ci/ci-scripts/build-docs.sh > @@ -0,0 +1,35 @@ > +#!/bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +set -exo pipefail > + > +source ci/gitlab-ci/ci-scripts/ici-functions.sh > + > +ici_get_patch_series_size > + > +# Get the list of modified files in the last $ICI_PATCH_SERIES_SIZE commits > +MODIFIED_DOC_FILES=$(git diff HEAD~$ICI_PATCH_SERIES_SIZE --name-only -- Documentation/) > + > +make -j$(nproc) "$ICI_DOC_TYPE" 2>&1 | tee output.txt > + > +mkdir -p "${CI_PROJECT_DIR}/artifacts" > +mv Documentation/output "${CI_PROJECT_DIR}/artifacts/Documentation-output" > + > +# Check if any of the MODIFIED_DOC_FILES generated a warning > +# NOTE: the alternative solution was to touch the modified files and run make > +# again, but too much warnings still appears > +for file in $MODIFIED_DOC_FILES; do > + if grep -qi "warning" output.txt && grep -q "$file" output.txt; then > + echo "Warning found in $file" > + exit 101 > + fi > +done > + > +if [ -n "$ICI_UNABLE_TO_DETECT_PATCH_SERIES_SIZE" ]; then > + # If the patch series size was not detected, exit with a warning > + echo -n "The patch series size was not detected, we probably didn't check the" > + echo " whole series. Exiting with a warning." > + exit 101 > +fi > diff --git a/ci/gitlab-ci/ci-scripts/build-kernel.sh b/ci/gitlab-ci/ci-scripts/build-kernel.sh > new file mode 100755 > index 0000000000000..361826368a573 > --- /dev/null > +++ b/ci/gitlab-ci/ci-scripts/build-kernel.sh > @@ -0,0 +1,35 @@ > +#!/bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +set -exo pipefail > + > +source ci/gitlab-ci/ci-scripts/ici-functions.sh > + > +ici_prepare_build > + > +pushd build > + > +# compile the entire kernel > +make CF=-D__CHECK_ENDIAN__ -C "$ICI_KERNEL_DIR" O=$(pwd) -j$(nproc) 2>&1 | tee output.txt > + > +export INSTALL_PATH="${CI_PROJECT_DIR}/artifacts/" > +INSTALL_PATH+="kernel-install-${KCI_KERNEL_ARCH}-${KCI_DEFCONFIG}_config" > +mkdir -p "$INSTALL_PATH" > + > +# install the kernel image to artifacts/kernel-install > +make -C "$ICI_KERNEL_DIR" O=$(pwd) install INSTALL_PATH="$INSTALL_PATH" > + > +# install kernel modules to artifacts/kernel-install > +make -C "$ICI_KERNEL_DIR" O=$(pwd) modules_install INSTALL_MOD_PATH="$INSTALL_PATH" > + > +# export config as artifact > +cp .config "${CI_PROJECT_DIR}/artifacts/${KCI_KERNEL_ARCH}-${KCI_DEFCONFIG}_config" > + > +# if the compilation has warnings, exit with the warning code > +if grep -iq "warning" output.txt; then I'm not sure, but do you need a popd here in this early exit path? > + exit 101 > +fi > + > +popd > diff --git a/ci/gitlab-ci/ci-scripts/ici-functions.sh b/ci/gitlab-ci/ci-scripts/ici-functions.sh > new file mode 100644 > index 0000000000000..df133b4d00103 > --- /dev/null > +++ b/ci/gitlab-ci/ci-scripts/ici-functions.sh > @@ -0,0 +1,104 @@ > +#!/bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +set -exo pipefail > + > +# internal CI bash functions > + > +# convention: > +# KCI_<VARIABLE_NAME> for variables defined by the user (outside of this script) > +# ICI_<VARIABLE_NAME> for variables defined internally for usage between scripts > +# CI_<VARIABLE_NAME> for variables defined by GitLab CI > + > + > +ici_prepare_build() { > + BUILD_DIR="${1:-build}" > + > + echo "" > + echo "Architecture: $KCI_KERNEL_ARCH" > + echo "Defconfig: $KCI_DEFCONFIG" > + echo "" > + > + # Get the current directory if KCI_KERNEL_DIR is not set > + ICI_KERNEL_DIR="${KCI_KERNEL_DIR:-$(pwd)}" > + > + cd "$ICI_KERNEL_DIR" || { echo "Kernel directory not found"; exit 1; } > + > + # Clean up stale rebases that GitLab might not have removed when reusing a checkout dir > + rm -rf .git/rebase-apply > + > + if [[ "$KCI_KERNEL_ARCH" = "arm64" ]]; then > + GCC_ARCH="aarch64-linux-gnu" > + elif [[ "$KCI_KERNEL_ARCH" = "arm" ]]; then > + GCC_ARCH="arm-linux-gnueabihf" > + else > + GCC_ARCH="x86_64-linux-gnu" > + fi > + > + # do not set ARCH and CROSS_COMPILE if KCI_KERNEL_ARCH is not set, useful for local run > + if [ -n "$KCI_KERNEL_ARCH" ]; then > + export ARCH=${KCI_KERNEL_ARCH} > + export CROSS_COMPILE="${GCC_ARCH}-" > + fi > + > + mkdir -p "$BUILD_DIR" > + > + pushd "$BUILD_DIR" || { echo "Failed to create $BUILD_DIR directory"; exit 1; } > + > + # generate defconfig > + make -C "$ICI_KERNEL_DIR" O=$(pwd) $(basename ${KCI_DEFCONFIG-"defconfig"}) > + > + # add extra configs from variable KCI_KCONFIGS_{ENABLE,DISABLE,MODULE} > + for opt in $KCI_KCONFIGS_ENABLE; do > + ../scripts/config --file .config --enable CONFIG_$opt > + done > + for opt in $KCI_KCONFIGS_DISABLE; do > + ../scripts/config --file .config --disable CONFIG_$opt > + done > + for opt in $KCI_KCONFIGS_MODULE; do > + ../scripts/config --file .config --module CONFIG_$opt > + done > + > + if [ -n "$KCI_KCONFIGS_DISABLE" ] || [ -n "$KCI_KCONFIGS_ENABLE" ] || > + [ -n "$KCI_KCONFIGS_MODULE" ]; then > + # execude olddefconfig only if we changed the default config, otherwise, > + # let it raise warnings if any > + make -C "$ICI_KERNEL_DIR" O=$(pwd) olddefconfig > + fi > + > + popd > +} > + > +ici_get_patch_series_size() > +{ > + local CLONE_DEPTH > + CLONE_DEPTH=$(git rev-list --count HEAD) > + echo "The depth of the clone is $CLONE_DEPTH" > + > + # If this is in the context of a merge request, calculate the patch series > + # size comparing to the target branch > + if [ -n "$CI_MERGE_REQUEST_IID" ]; then > + git fetch origin "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" --depth $CLONE_DEPTH > + BASE_COMMIT="origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" > + ICI_PATCH_SERIES_SIZE=$(git rev-list --count ${BASE_COMMIT}.."$CI_COMMIT_SHA") > + > + # if KCI_PATCH_SERIES_SIZE is set, use it > + elif [ -n "$KCI_PATCH_SERIES_SIZE" ]; then > + ICI_PATCH_SERIES_SIZE="$KCI_PATCH_SERIES_SIZE" > + else > + ICI_PATCH_SERIES_SIZE=1 > + echo "WARNING: unable to detect the patch series size, using the default value of 1." > + # shellcheck disable=SC2034 > + ICI_UNABLE_TO_DETECT_PATCH_SERIES_SIZE=true > + fi > + > + # Check if the clone depth is smaller than or equal to KCI_PATCH_SERIES_SIZE, > + # otherwise the checkpatch.pl hangs > + if [ "$ICI_PATCH_SERIES_SIZE" -ge "$CLONE_DEPTH" ]; then > + echo -n "ERROR: the depth of the clone is $CLONE_DEPTH, smaller than or equal to the patch" > + echo " series size. Update your GitLab configuration to increase the size of the clone." > + return 1 > + fi > +} > diff --git a/ci/gitlab-ci/ci-scripts/install-smatch.sh b/ci/gitlab-ci/ci-scripts/install-smatch.sh > new file mode 100755 > index 0000000000000..cae16a0e90fa8 > --- /dev/null > +++ b/ci/gitlab-ci/ci-scripts/install-smatch.sh > @@ -0,0 +1,13 @@ > +#!/bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +set -exo pipefail > + > +pushd / At the root? really? Could this be /usr/local/src or $HOME/src or something? or is this inside a container and doesn't matter? > +git clone --depth 1 https://repo.or.cz/smatch.git > +pushd smatch > +make > +popd > +popd > diff --git a/ci/gitlab-ci/ci-scripts/parse_commit_message.sh b/ci/gitlab-ci/ci-scripts/parse_commit_message.sh > new file mode 100755 > index 0000000000000..c9792c64ad51e > --- /dev/null > +++ b/ci/gitlab-ci/ci-scripts/parse_commit_message.sh > @@ -0,0 +1,27 @@ > +#!/bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +set -exo pipefail > + > +# Get the last commit message > +commit_message=$(git log -1 --pretty=%B) > + > +pattern='(KCI_[A-Za-z_]+)=("[^"]*"|[^ ]+)' > + > +while read -r line; do > + if [[ $line =~ $pattern ]]; then > + variable_name="${BASH_REMATCH[1]}" > + variable_value="${BASH_REMATCH[2]}" > + > + # Remove quotes if present > + variable_value="${variable_value%\"}" > + variable_value="${variable_value#\"}" > + > + # Export the variable > + export "$variable_name=$variable_value" > + > + echo "Exported $variable_name=$variable_value" > + fi > +done <<< "$commit_message" > diff --git a/ci/gitlab-ci/ci-scripts/run-checkpatch.sh b/ci/gitlab-ci/ci-scripts/run-checkpatch.sh > new file mode 100755 > index 0000000000000..98585ec18be2b > --- /dev/null > +++ b/ci/gitlab-ci/ci-scripts/run-checkpatch.sh > @@ -0,0 +1,20 @@ > +#!/bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +set -exo pipefail > + > +source ci/gitlab-ci/ci-scripts/ici-functions.sh > + > +ici_get_patch_series_size > + > +# shellcheck disable=SC2086 > +scripts/checkpatch.pl $KCI_CHECKPATCH_OPTIONS --git HEAD-"$ICI_PATCH_SERIES_SIZE" > + > +if [ -n "$ICI_UNABLE_TO_DETECT_PATCH_SERIES_SIZE" ]; then > + # If the patch series size was not detected, exit with a warning > + echo -n "The patch series size was not detected, we probably didn't check the whole series." > + echo " Exiting with a warning." > + exit 101 > +fi > diff --git a/ci/gitlab-ci/ci-scripts/run-smatch.sh b/ci/gitlab-ci/ci-scripts/run-smatch.sh > new file mode 100755 > index 0000000000000..ec054df9ef179 > --- /dev/null > +++ b/ci/gitlab-ci/ci-scripts/run-smatch.sh > @@ -0,0 +1,45 @@ > +#!/bin/bash > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +set -exo pipefail > + > +source ci/gitlab-ci/ci-scripts/ici-functions.sh > + > +ls -l > +pwd > + > +# generate config file > +ici_prepare_build > + > +ici_get_patch_series_size > + > +cp build/.config . > + > +# Get a list of modified .c files in the last ICI_PATCH_SERIES_SIZE commits > +MODIFIED_C_FILES=$(git diff --name-only HEAD~$ICI_PATCH_SERIES_SIZE HEAD | grep '\.c$' || true) > + > +# Check if any .c files were modified > +if [ -z "$MODIFIED_C_FILES" ]; then > + echo "No .c files were modified in the last $ICI_PATCH_SERIES_SIZE commits." > +else > + echo "Running kchecker on modified .c files..." > + mkdir -p "$CI_PROJECT_DIR"/artifacts > +fi > + > +# Run kchecker on each modified .c file > +for file in $MODIFIED_C_FILES; do > + if [ -f "$file" ]; then > + /smatch/smatch_scripts/kchecker "$file" | tee "$CI_PROJECT_DIR"/artifacts/smatch_warns.txt > + else > + echo "File not found: $file" > + fi > +done > + > +if [ -n "$ICI_UNABLE_TO_DETECT_PATCH_SERIES_SIZE" ]; then > + # If the patch series size was not detected, exit with a warning > + echo -n "The patch series size was not detected, we probably didn't check the whole series." > + echo " Exiting with a warning." > + exit 101 > +fi > diff --git a/ci/gitlab-ci/docker-compose.yaml b/ci/gitlab-ci/docker-compose.yaml > new file mode 100644 > index 0000000000000..7edf8055ca375 > --- /dev/null > +++ b/ci/gitlab-ci/docker-compose.yaml > @@ -0,0 +1,18 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +version: '3.8' > + > +services: > + gitlab-runner: > + image: gitlab/gitlab-runner:latest > + container_name: gitlab-runner > + restart: always > + privileged: true > + volumes: > + - /srv/gitlab-runner/config:/etc/gitlab-runner > + - /var/run/docker.sock:/var/run/docker.sock > + > +# To register the GitLab Runner, run the following command: > +# docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register --url https://gitlab.com --token > YOUR_REGISTRATION_TOKEN > diff --git a/ci/gitlab-ci/linux.code-workspace b/ci/gitlab-ci/linux.code-workspace > new file mode 100644 > index 0000000000000..dd76698e38c2c > --- /dev/null > +++ b/ci/gitlab-ci/linux.code-workspace > @@ -0,0 +1,11 @@ > +{ > + "folders": [ > + { > + "path": "../../../.." > + }, > + { > + "path": "../../../../../mesa" > + } > + ], > + "settings": {} > +} > \ No newline at end of file > diff --git a/ci/gitlab-ci/yml/build.yml b/ci/gitlab-ci/yml/build.yml > new file mode 100644 > index 0000000000000..f81410c293f35 > --- /dev/null > +++ b/ci/gitlab-ci/yml/build.yml > @@ -0,0 +1,43 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +.build: > + extends: .use-debian/x86_64_build > + stage: build > + > +build-kernel: > + extends: > + - .build > + - .kernel-combinations > + variables: > + # ccache in gitlab-runner to speed up builds > + CCACHE_BASEDIR: $CI_PROJECT_DIR > + CCACHE_DIR: $CI_PROJECT_DIR/ccache > + CCACHE_COMPILERCHECK: content > + cache: > + - key: ccache-$CI_JOB_NAME > + paths: > + - $CCACHE_DIR > + before_script: > + - export PATH="/usr/lib/ccache:$PATH" > + - ccache --zero-stats || true > + - ccache --show-stats || true > + after_script: > + - ccache --show-stats > + script: > + - ./ci/gitlab-ci/ci-scripts/build-kernel.sh > + > +build-docs: > + extends: > + - .build > + parallel: > + matrix: > + - ICI_DOC_TYPE: "htmldocs" > + # TODO: re-add pdfdocs once build errors are fixed > + # - ICI_DOC_TYPE: "pdfdocs" > + - ICI_DOC_TYPE: "latexdocs" > + - ICI_DOC_TYPE: "epubdocs" > + script: > + - ./ci/gitlab-ci/ci-scripts/build-docs.sh > + when: manual > \ No newline at end of file > diff --git a/ci/gitlab-ci/yml/cache.yml b/ci/gitlab-ci/yml/cache.yml > new file mode 100644 > index 0000000000000..efe4fc69b11ca > --- /dev/null > +++ b/ci/gitlab-ci/yml/cache.yml > @@ -0,0 +1,26 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +# Smatch db is saved to a cached folder, so it can be used by other jobs and pipelines. > +# It is set to manual so it can be run when needed > + > +.use-cache-smatch-db: > + cache: > + # TODO: check if this cache shouldn't be per architecture > + - key: smatch-db > + paths: > + - /smatch/smatch_data > + > +smatch-db-generate: > + stage: cache > + extends: > + - .kernel-combinations > + - .use-debian/x86_64_build > + - .use-cache-smatch-db > + script: > + - source ci/gitlab-ci/ci-scripts/ici-functions.sh > + - ici_prepare_build > + - cp build/.config . > + - /smatch/smatch_scripts/build_kernel_data.sh > + when: manual > \ No newline at end of file > diff --git a/ci/gitlab-ci/yml/container.yml b/ci/gitlab-ci/yml/container.yml > new file mode 100644 > index 0000000000000..eecca95caca0e > --- /dev/null > +++ b/ci/gitlab-ci/yml/container.yml > @@ -0,0 +1,36 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +.prep-debian/x86_64_build: > + variables: > + FDO_DISTRIBUTION_VERSION: bookworm-slim > + FDO_DISTRIBUTION_TAG: "2024-02-28-ci-test-1" > + > + > +debian/x86_64_build: > + extends: > + - ".fdo.container-build@debian" > + - ".prep-debian/x86_64_build" > + variables: > + FDO_DISTRIBUTION_PACKAGES: > > + make bc flex bison pahole mount build-essential > + libcairo-dev libdw-dev libjson-c-dev libkmod2 > + libkmod-dev libpciaccess-dev libproc2-dev libudev-dev > + libunwind-dev python3-docutils bc python3-ply > + libssl-dev ccache > + gcc-aarch64-linux-gnu binutils-aarch64-linux-gnu > + gcc-arm-linux-gnueabihf binutils-arm-linux-gnueabihf > + perl git kmod python3-git python3-yaml python3-dulwich > + sqlite3 libsqlite3-dev libdbd-sqlite3-perl libssl-dev libtry-tiny-perl > + python3-sphinx imagemagick graphviz dvipng python3-venv fonts-noto-cjk > + latexmk librsvg2-bin texlive-lang-chinese texlive-xetex > + FDO_DISTRIBUTION_EXEC: ./ci/gitlab-ci/ci-scripts/install-smatch.sh > + stage: container > + > + > +.use-debian/x86_64_build: > + extends: > + - ".fdo.distribution-image@debian" > + - ".prep-debian/x86_64_build" > + needs: [debian/x86_64_build] > \ No newline at end of file > diff --git a/ci/gitlab-ci/yml/gitlab-ci.yml b/ci/gitlab-ci/yml/gitlab-ci.yml > new file mode 100644 > index 0000000000000..57b9c02904713 > --- /dev/null > +++ b/ci/gitlab-ci/yml/gitlab-ci.yml > @@ -0,0 +1,71 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +workflow: > + name: $PIPELINE_NAME > + rules: > + # when triggered as a multi-project pipeline for an MR > + - if: $CI_PIPELINE_SOURCE == 'pipeline' && $PARENT_MERGE_REQUEST_IID != null && $PARENT_MERGE_REQUEST_IID != "" > + variables: > + PIPELINE_NAME: 'Downstream pipeline for $PARENT_PROJECT_PATH!$PARENT_MERGE_REQUEST_IID' > + # when triggered as a multi-project pipeline > + - if: $CI_PIPELINE_SOURCE == 'pipeline' > + variables: > + PIPELINE_NAME: 'Downstream pipeline for $PARENT_PROJECT_PATH' > + # when triggered via a schedule > + - if: $CI_PIPELINE_SOURCE == 'schedule' > + variables: > + PIPELINE_NAME: 'Scheduled pipeline for $ONLY_JOB_NAME' > + # for merge requests > + - if: $CI_MERGE_REQUEST_ID > + # when triggered via the REST api > + - if: $CI_PIPELINE_SOURCE == 'api' > + # for the tip of the default branch > + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH > + # when triggered via a trigger token > + - if: $CI_PIPELINE_SOURCE == 'trigger' > + # when triggered from a button press in the web interface > + - if: $CI_PIPELINE_SOURCE == 'web' > + # for branch tips without open MRs, ignoring special branches > + - if: $CI_PIPELINE_SOURCE == 'push' && $CI_OPEN_MERGE_REQUESTS == null > + # when forced via '-o ci.variable="FORCE_CI=true"' during pushing > + - if: $FORCE_CI == 'true' > + > +variables: > + FDO_UPSTREAM_REPO: helen.fornazier/linux # The repo where to look for cached images > + # ccache builds in gitlab-runner to speed up builds > + SMATCH_DB_DIR: /smatch/smatch_data > + # exit code of bash script on `script` will be the exit code of the job > + FF_USE_NEW_BASH_EVAL_STRATEGY: "true" > + > +default: > + artifacts: > + paths: > + - artifacts/ > + when: always > + > +include: > + - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/16bc29078de5e0a067ff84a1a199a3760d3b3811/templates/ci- > fairy.yml' > + - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/- > /raw/16bc29078de5e0a067ff84a1a199a3760d3b3811/templates/debian.yml' > + > + - ci/gitlab-ci/yml/kernel-combinations.yml > + - ci/gitlab-ci/yml/container.yml > + - ci/gitlab-ci/yml/cache.yml > + - ci/gitlab-ci/yml/build.yml > + - ci/gitlab-ci/yml/static-checks.yml > + - ci/gitlab-ci/yml/scenarios.yml > + > +before_script: > + - source ci/gitlab-ci/ci-scripts/parse_commit_message.sh > + > +.use-debian/x86_64_build: > + allow_failure: > + # Code to exit with a warning > + exit_codes: 101 > + > +stages: > + - container > + - static-checks > + - build > + - cache > \ No newline at end of file > diff --git a/ci/gitlab-ci/yml/kernel-combinations.yml b/ci/gitlab-ci/yml/kernel-combinations.yml > new file mode 100644 > index 0000000000000..87f08579ffd62 > --- /dev/null > +++ b/ci/gitlab-ci/yml/kernel-combinations.yml > @@ -0,0 +1,18 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +.kernel-combinations: > + parallel: > + matrix: > + - KCI_KERNEL_ARCH: "x86_64" > + KCI_DEFCONFIG: "x86_64_defconfig" > + KCI_KCONFIGS_ENABLE: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT CONFIG_DEBUG_SECTION_MISMATCH" > + > + - KCI_KERNEL_ARCH: "arm64" > + KCI_DEFCONFIG: "defconfig" > + KCI_KCONFIGS_ENABLE: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT CONFIG_DEBUG_SECTION_MISMATCH" > + > + - KCI_KERNEL_ARCH: "arm" > + KCI_DEFCONFIG: "multi_v7_defconfig" > + KCI_KCONFIGS_ENABLE: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT CONFIG_DEBUG_SECTION_MISMATCH" > \ No newline at end of file > diff --git a/ci/gitlab-ci/yml/scenarios.yml b/ci/gitlab-ci/yml/scenarios.yml > new file mode 100644 > index 0000000000000..11598c3b6f2a9 > --- /dev/null > +++ b/ci/gitlab-ci/yml/scenarios.yml > @@ -0,0 +1,12 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +# Extend the CI by including a test scenario here. The scenario will be > +# activated if KCI_SCENARIO is set to the scenario name. KCI_SCENARIO can be > +# defined in the gitlab-ci.yml file, or through GitLab UI. > + > +include: > + - local: 'ci/gitlab-ci/yml/scenarios/$KCI_SCENARIO.yml' > + rules: > + - if: '$KCI_SCENARIO' > \ No newline at end of file > diff --git a/ci/gitlab-ci/yml/scenarios/file-systems.yml b/ci/gitlab-ci/yml/scenarios/file-systems.yml > new file mode 100644 > index 0000000000000..66456600a9dd3 > --- /dev/null > +++ b/ci/gitlab-ci/yml/scenarios/file-systems.yml > @@ -0,0 +1,21 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +# Write here specific configurations and extensions for the given scenario > + > +# Example - overwrite kernel combinations in the pipeline > +# .kernel-combinations: > +# parallel: > +# matrix: > +# - KCI_KERNEL_ARCH: "x86_64" > +# KCI_DEFCONFIG: "path/to/my/config/file_config" > +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." > +# > +# - KCI_KERNEL_ARCH: "arm64" > +# KCI_DEFCONFIG: "path/to/my/config/file_config" > +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." > +# > +# - KCI_KERNEL_ARCH: "arm" > +# KCI_DEFCONFIG: "path/to/my/config/file_config" > +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." > \ No newline at end of file > diff --git a/ci/gitlab-ci/yml/scenarios/media.yml b/ci/gitlab-ci/yml/scenarios/media.yml > new file mode 100644 > index 0000000000000..66456600a9dd3 > --- /dev/null > +++ b/ci/gitlab-ci/yml/scenarios/media.yml > @@ -0,0 +1,21 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +# Write here specific configurations and extensions for the given scenario > + > +# Example - overwrite kernel combinations in the pipeline > +# .kernel-combinations: > +# parallel: > +# matrix: > +# - KCI_KERNEL_ARCH: "x86_64" > +# KCI_DEFCONFIG: "path/to/my/config/file_config" > +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." > +# > +# - KCI_KERNEL_ARCH: "arm64" > +# KCI_DEFCONFIG: "path/to/my/config/file_config" > +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." > +# > +# - KCI_KERNEL_ARCH: "arm" > +# KCI_DEFCONFIG: "path/to/my/config/file_config" > +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." > \ No newline at end of file > diff --git a/ci/gitlab-ci/yml/scenarios/network.yml b/ci/gitlab-ci/yml/scenarios/network.yml > new file mode 100644 > index 0000000000000..66456600a9dd3 > --- /dev/null > +++ b/ci/gitlab-ci/yml/scenarios/network.yml > @@ -0,0 +1,21 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +# Write here specific configurations and extensions for the given scenario > + > +# Example - overwrite kernel combinations in the pipeline > +# .kernel-combinations: > +# parallel: > +# matrix: > +# - KCI_KERNEL_ARCH: "x86_64" > +# KCI_DEFCONFIG: "path/to/my/config/file_config" > +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." > +# > +# - KCI_KERNEL_ARCH: "arm64" > +# KCI_DEFCONFIG: "path/to/my/config/file_config" > +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." > +# > +# - KCI_KERNEL_ARCH: "arm" > +# KCI_DEFCONFIG: "path/to/my/config/file_config" > +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." > \ No newline at end of file Do you have any real examples of scenario files? These all seem like placeholders. > diff --git a/ci/gitlab-ci/yml/static-checks.yml b/ci/gitlab-ci/yml/static-checks.yml > new file mode 100644 > index 0000000000000..a99a1ea910f40 > --- /dev/null > +++ b/ci/gitlab-ci/yml/static-checks.yml > @@ -0,0 +1,21 @@ > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> > + > +.static-checks: > + stage: static-checks > + extends: > + - .use-debian/x86_64_build > + > +checkpatch: > + extends: .static-checks > + script: > + - ci/gitlab-ci/ci-scripts/run-checkpatch.sh > + > +smatch: > + extends: > + - .static-checks > + - .kernel-combinations > + - .use-cache-smatch-db > + script: > + - ci/gitlab-ci/ci-scripts/run-smatch.sh > \ No newline at end of file > -- > 2.40.1 > Looks very interesting! -- Tim
Hi Helen, Thanks for working on this On Wed, Feb 28, 2024 at 07:55:25PM -0300, Helen Koike wrote: > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, > defininga basic test pipeline triggered by code pushes to a GitLab-CI > instance. This initial version includes static checks (checkpatch and > smatch for now) and build tests across various architectures and > configurations. It leverages an integrated cache for efficient build > times and introduces a flexible 'scenarios' mechanism for > subsystem-specific extensions. > > [ci: add prerequisites to run check-patch on MRs] > Co-developed-by: Tales Aparecida <tales.aparecida@redhat.com> > Signed-off-by: Tales Aparecida <tales.aparecida@redhat.com> > Signed-off-by: Helen Koike <helen.koike@collabora.com> > > --- > > Hey all, > > You can check the validation of this patchset on: > https://gitlab.collabora.com/koike/linux/-/pipelines/87035 > > I would appreciate your feedback on this work, what do you think? > > If you would rate from 0 to 5, where: > > [ ] 0. I don't think this is useful at all, and I doubt it will ever be. It doesn't seem worthwhile. > [ ] 1. I don't find it useful in its current form. > [ ] 2. It might be useful to others, but not for me. > [ ] 3. It has potential, but it's not yet something I can incorporate into my workflow. > [ ] 4. This is useful, but it needs some adjustments before I can include it in my workflow. > [ ] 5. This is really useful! I'm eager to start using it right away. Why didn't you send this earlier? :) > > Which rating would you select? 4.5 :) One thing I'm wondering here is how we're going to cope with the different requirements each user / framework has. Like, Linus probably want to have a different set of CI before merging a PR than (say) linux-next does, or stable, or before doing an actual release. Similarly, DRM probably has a different set of requirements than drm-misc, drm-amd or nouveau. I don't see how the current architecture could accomodate for that. I know that Gitlab allows to store issues template in a separate repo, maybe we could ask them to provide a feature where the actions would be separate from the main repo? That way, any gitlab project could provide its own set of tests, without conflicting with each others (and we could still share them if we wanted to) I know some of use had good relationship with Gitlab, so maybe it would be worth asking? Maxime
Hi everyone, On 2/29/24 11:02, Maxime Ripard wrote: > On Wed, Feb 28, 2024 at 07:55:25PM -0300, Helen Koike wrote: >> Which rating would you select? > > 4.5 :) > > One thing I'm wondering here is how we're going to cope with the > different requirements each user / framework has. > > Like, Linus probably want to have a different set of CI before merging a > PR than (say) linux-next does, or stable, or before doing an actual > release. > > Similarly, DRM probably has a different set of requirements than > drm-misc, drm-amd or nouveau. > > I don't see how the current architecture could accomodate for that. I > know that Gitlab allows to store issues template in a separate repo, > maybe we could ask them to provide a feature where the actions would be > separate from the main repo? That way, any gitlab project could provide > its own set of tests, without conflicting with each others (and we could > still share them if we wanted to) > > I know some of use had good relationship with Gitlab, so maybe it would > be worth asking? GitLab already supports getting the CI YAML from other repos. You can change that in the repo settings. However, I think a better approach would be *not* to add the .gitlab-ci.yaml file in the root of the source tree, but instead change the very same repo setting to point to a particular entry YAML, *inside* the repo (somewhere under "ci" directory) instead. This way all the different subtrees can have completely different setup, but some could still use Helen's work and employ the "scenarios" she implemented. Nick
Hi! On Thu, Feb 29, 2024 at 11:23:22AM +0200, Nikolai Kondrashov wrote: > Hi everyone, > > On 2/29/24 11:02, Maxime Ripard wrote: > > On Wed, Feb 28, 2024 at 07:55:25PM -0300, Helen Koike wrote: > > > Which rating would you select? > > > > 4.5 :) > > > > One thing I'm wondering here is how we're going to cope with the > > different requirements each user / framework has. > > > > Like, Linus probably want to have a different set of CI before merging a > > PR than (say) linux-next does, or stable, or before doing an actual > > release. > > > > Similarly, DRM probably has a different set of requirements than > > drm-misc, drm-amd or nouveau. > > > > I don't see how the current architecture could accomodate for that. I > > know that Gitlab allows to store issues template in a separate repo, > > maybe we could ask them to provide a feature where the actions would be > > separate from the main repo? That way, any gitlab project could provide > > its own set of tests, without conflicting with each others (and we could > > still share them if we wanted to) > > > > I know some of use had good relationship with Gitlab, so maybe it would > > be worth asking? > > GitLab already supports getting the CI YAML from other repos. You can change > that in the repo settings. I'm interested but couldn't find it in the doc, do you have a link to the right section? > However, I think a better approach would be *not* to add the .gitlab-ci.yaml > file in the root of the source tree, but instead change the very same repo > setting to point to a particular entry YAML, *inside* the repo (somewhere > under "ci" directory) instead. > > This way all the different subtrees can have completely different setup, but > some could still use Helen's work and employ the "scenarios" she > implemented. I'm worried that this kind of setup will just create duplicated YAML that will be developped in complete silos and will become difficult to maintain. But that's definitely an opinion :) Maxime
On Thu, Feb 29, 2024 at 10:56:58AM +0100, Maxime Ripard wrote: > Hi! > > On Thu, Feb 29, 2024 at 11:23:22AM +0200, Nikolai Kondrashov wrote: > > Hi everyone, > > > > On 2/29/24 11:02, Maxime Ripard wrote: > > > On Wed, Feb 28, 2024 at 07:55:25PM -0300, Helen Koike wrote: > > > > Which rating would you select? > > > > > > 4.5 :) > > > > > > One thing I'm wondering here is how we're going to cope with the > > > different requirements each user / framework has. > > > > > > Like, Linus probably want to have a different set of CI before merging a > > > PR than (say) linux-next does, or stable, or before doing an actual > > > release. > > > > > > Similarly, DRM probably has a different set of requirements than > > > drm-misc, drm-amd or nouveau. > > > > > > I don't see how the current architecture could accomodate for that. I > > > know that Gitlab allows to store issues template in a separate repo, > > > maybe we could ask them to provide a feature where the actions would be > > > separate from the main repo? That way, any gitlab project could provide > > > its own set of tests, without conflicting with each others (and we could > > > still share them if we wanted to) > > > > > > I know some of use had good relationship with Gitlab, so maybe it would > > > be worth asking? > > > > GitLab already supports getting the CI YAML from other repos. You can change > > that in the repo settings. > > I'm interested but couldn't find it in the doc, do you have a link to > the right section? e.g. https://gitlab.freedesktop.org/camera/libcamera/-/settings/ci_cd Expand "General pipelines", the setting is "CI/CD configuration file". You can specify the path to a file in the local repository, or in a separate repository. See https://gitlab.freedesktop.org/help/ci/pipelines/settings#specify-a-custom-cicd-configuration-file for the syntax. > > However, I think a better approach would be *not* to add the .gitlab-ci.yaml > > file in the root of the source tree, but instead change the very same repo > > setting to point to a particular entry YAML, *inside* the repo (somewhere > > under "ci" directory) instead. > > > > This way all the different subtrees can have completely different setup, but > > some could still use Helen's work and employ the "scenarios" she > > implemented. > > I'm worried that this kind of setup will just create duplicated YAML > that will be developped in complete silos and will become difficult to > maintain. But that's definitely an opinion :)
Hi Tim, just replying below to one of your comment which I happen to be involved in, but I'll let others reply for the more specific comments. Le jeudi 29 février 2024 à 02:44 +0000, Bird, Tim a écrit : > > -----Original Message----- > > From: Helen Koike <helen.koike@collabora.com> > > > .... > > > Hey all, > > > > You can check the validation of this patchset on: > > https://gitlab.collabora.com/koike/linux/-/pipelines/87035 > > > > I would appreciate your feedback on this work, what do you think? > > > > If you would rate from 0 to 5, where: > > > > [ ] 0. I don't think this is useful at all, and I doubt it will ever be. It doesn't seem worthwhile. > > [ ] 1. I don't find it useful in its current form. > > [ ] 2. It might be useful to others, but not for me. > > [ ] 3. It has potential, but it's not yet something I can incorporate into my workflow. > > [ ] 4. This is useful, but it needs some adjustments before I can include it in my workflow. > > [ ] 5. This is really useful! I'm eager to start using it right away. Why didn't you send this earlier? :) > > > > Which rating would you select? > > For me, this is a "5". I don't currently use gitlab, but I might install it just to test this out. > > It looks like there are a large number of YAML files which define the integration between the > test scripts and gitlab. Also, there are a number of shell scripts to perform some of the setup > and tests. Do you have any idea how difficult it would be to use the shell scripts outside of > the CI system (e.g. manually)? That is, are there dependencies between the CI system > and the shell scripts? You are effectively the second person I'm aware to provide similar feedback. We agreed to conduct an effort to remove the gitlab specifics from the script. Avoid using gitlab CI shell environment in favour of command line arguments. Also ensure scripts have a "-h" option. This should ease local reproduction and allow for other CI integration. After all, the Linux kernel is a large community and gitlab is just one option for managing CI. It is a big system, so we rarely "just install it" ourself. DRM and Linux Media community are using the Freedesktop instance, sharing resources and cost within that instance. In Linux Media we are developing out of tree scripts with similar purpose, but we also believe a common set of tool, directly in the kernel tree would be a better long term solution. > > I think the convention, of putting this kind of stuff under a "ci" directory, makes sense. > And it appears that the sub-dir structure allows for other CI systems to add their > own config and files also. CI scripts have the particularity of being very granular, which is very unlike what a human user would prefer. But when CI fails, you really want to know which small step failed, which can sometimes be hidden by more en-globing scripts. We also care a lot about parallelism, since we have hundreds of runners available to execute these tests. Short answer, I also like that this is under a CI directory, its makes ensure the purpose and intention of this work is clear. regards, Nicolas
On Thu, 29 Feb 2024 at 01:23, Nikolai Kondrashov <spbnick@gmail.com> wrote: > > However, I think a better approach would be *not* to add the .gitlab-ci.yaml > file in the root of the source tree, but instead change the very same repo > setting to point to a particular entry YAML, *inside* the repo (somewhere > under "ci" directory) instead. I really don't want some kind of top-level CI for the base kernel project. We already have the situation that the drm people have their own ci model. II'm ok with that, partly because then at least the maintainers of that subsystem can agree on the rules for that one subsystem. I'm not at all interested in having something that people will then either fight about, or - more likely - ignore, at the top level because there isn't some global agreement about what the rules are. For example, even just running checkpatch is often a stylistic thing, and not everybody agrees about all the checkpatch warnings. I would suggest the CI project be separate from the kernel. And having that slack channel that is restricted to particular companies is just another sign of this whole disease. If you want to make a google/microsoft project to do kernel CI, then more power to you, but don't expect it to be some kind of agreed-upon kernel project when it's a closed system. Linus
Thanks for stopping by, Linus! On 2/29/24 10:21 PM, Linus Torvalds wrote: > On Thu, 29 Feb 2024 at 01:23, Nikolai Kondrashov <spbnick@gmail.com> wrote: >> >> However, I think a better approach would be *not* to add the .gitlab-ci.yaml >> file in the root of the source tree, but instead change the very same repo >> setting to point to a particular entry YAML, *inside* the repo (somewhere >> under "ci" directory) instead. > > I really don't want some kind of top-level CI for the base kernel project. > > We already have the situation that the drm people have their own ci > model. II'm ok with that, partly because then at least the maintainers > of that subsystem can agree on the rules for that one subsystem. > > I'm not at all interested in having something that people will then > either fight about, or - more likely - ignore, at the top level > because there isn't some global agreement about what the rules are. > > For example, even just running checkpatch is often a stylistic thing, > and not everybody agrees about all the checkpatch warnings. I agree, it's hard to imagine even a simple majority agreeing on how GitLab CI should be done. Still, we would like to help people, who are interested in this kind of thing, to set it up. How about we reframe this contribution as a sort of template, or a reference for people to start their setup with, assuming that most maintainers would want to tweak it? We would also be glad to stand by for questions and help, as people try to use it. > I would suggest the CI project be separate from the kernel. It is possible to have a GitLab CI setup with the YAML files in a separate repository. And we can start with that. However, ultimately I think it's better to have it in the same repo with the code being tested. This way you could submit code changes together with the required tweaks to the CI to keep it passing, making development smoother and faster. With that in mind, and if you agree, where else would you say we could put it? Under "scripts"? Or "Documentation"? And where it would be best for the various subsystems to put theirs? Or could we have the top-level "ci" dir and pile all the different setups there? Or would you like to wait and see how adoption goes, and then decide? > And having that slack channel that is restricted to particular > companies is just another sign of this whole disease. Regarding the Slack channel, we're also on #kernelci on libera.chat, and we have some people there, but if more people start showing up, we'll be spending more time there too. > If you want to make a google/microsoft project to do kernel CI, then > more power to you, but don't expect it to be some kind of agreed-upon > kernel project when it's a closed system. Yes, our Slack is a closed system, unfortunately. However, it's not a part of the proposed CI setup *at all*, and it doesn't depend on Slack. We just like using it, but we're also fine with IRC. GitLab is open-core, and no closed-source software is required for the proposed setup. There is the public gitlab.com where you can deploy your CI, there's also the freedesktop.org instance, and it's possible to set up your own (albeit not easily). Nick
On Fri, Mar 01, 2024 at 12:27:13PM +0200, Nikolai Kondrashov wrote: > On 2/29/24 10:21 PM, Linus Torvalds wrote: > > We already have the situation that the drm people have their own ci > > model. II'm ok with that, partly because then at least the maintainers > > of that subsystem can agree on the rules for that one subsystem. > > I'm not at all interested in having something that people will then > > either fight about, or - more likely - ignore, at the top level > > because there isn't some global agreement about what the rules are. > > For example, even just running checkpatch is often a stylistic thing, > > and not everybody agrees about all the checkpatch warnings. ... > > I would suggest the CI project be separate from the kernel. > It is possible to have a GitLab CI setup with the YAML files in a separate > repository. And we can start with that. However, ultimately I think it's > better to have it in the same repo with the code being tested. This way you > could submit code changes together with the required tweaks to the CI to keep > it passing, making development smoother and faster. > With that in mind, and if you agree, where else would you say we could put it? > Under "scripts"? Or "Documentation"? And where it would be best for the > various subsystems to put theirs? Or could we have the top-level "ci" dir and > pile all the different setups there? Or would you like to wait and see how > adoption goes, and then decide? If we were going to put bits of this in tree how about something like tools/testing/forges? I'd hope that things could be shared by multiple services, if not we could always have subdirs I guess. We could put glue bits like defining how to run kunit, checkpatch or whatever with these systems in there so people can share figuring that bit out. Individual trees or CI systems using these forge based systems could then reference these files when defining what specific tests they want to run when which seems more like where the differences will be. I'm not super familiar with this stuff, the above is based on it looking like there's an OK degree of separation between the "what to run" and "how to run" bits. I might be misreading things, and it's not clear to me how often it'll be useful to be able to update things in tree.
On 3/1/24 4:07 PM, Mark Brown wrote: > On Fri, Mar 01, 2024 at 12:27:13PM +0200, Nikolai Kondrashov wrote: >> On 2/29/24 10:21 PM, Linus Torvalds wrote: >>> I would suggest the CI project be separate from the kernel. > >> It is possible to have a GitLab CI setup with the YAML files in a separate >> repository. And we can start with that. However, ultimately I think it's >> better to have it in the same repo with the code being tested. This way you >> could submit code changes together with the required tweaks to the CI to keep >> it passing, making development smoother and faster. > >> With that in mind, and if you agree, where else would you say we could put it? >> Under "scripts"? Or "Documentation"? And where it would be best for the >> various subsystems to put theirs? Or could we have the top-level "ci" dir and >> pile all the different setups there? Or would you like to wait and see how >> adoption goes, and then decide? > > If we were going to put bits of this in tree how about something like > tools/testing/forges? I'd hope that things could be shared by multiple > services, if not we could always have subdirs I guess. We could put > glue bits like defining how to run kunit, checkpatch or whatever with > these systems in there so people can share figuring that bit out. > Individual trees or CI systems using these forge based systems could > then reference these files when defining what specific tests they want > to run when which seems more like where the differences will be. > > I'm not super familiar with this stuff, the above is based on it looking > like there's an OK degree of separation between the "what to run" and > "how to run" bits. I might be misreading things, and it's not clear to > me how often it'll be useful to be able to update things in tree. Yes, facilitating reuse and collaboration between CI setups, even if they're largely different, is another good reason to have it inside the kernel repo. Nick
On Fri, 1 Mar 2024 at 02:27, Nikolai Kondrashov <spbnick@gmail.com> wrote: > > I agree, it's hard to imagine even a simple majority agreeing on how GitLab CI > should be done. Still, we would like to help people, who are interested in > this kind of thing, to set it up. How about we reframe this contribution as a > sort of template, or a reference for people to start their setup with, > assuming that most maintainers would want to tweak it? We would also be glad > to stand by for questions and help, as people try to use it. Ack. I think seeing it as a library for various gitlab CI models would be a lot more palatable. Particularly if you can then show that yes, it is also relevant to our currently existing drm case. So I'm not objecting to having (for example) some kind of CI helper templates - I think a logical place would be in tools/ci/ which is kind of alongside our tools/testing subdirectory. (And then perhaps have a 'gitlab' directory under that. I'm not sure whether - and how much - commonality there might be between the different CI models of different hosts). Just to clarify: when I say "a logical place", I very much want to emphasize the "a" - maybe there are better places, and I'm not saying that is the only possible place. But it sounds more logical to me than some. Linus
On Thu, Feb 29, 2024 at 12:21 PM Linus Torvalds <torvalds@linuxfoundation.org> wrote: > > On Thu, 29 Feb 2024 at 01:23, Nikolai Kondrashov <spbnick@gmail.com> wrote: > > > > However, I think a better approach would be *not* to add the .gitlab-ci.yaml > > file in the root of the source tree, but instead change the very same repo > > setting to point to a particular entry YAML, *inside* the repo (somewhere > > under "ci" directory) instead. > > I really don't want some kind of top-level CI for the base kernel project. > > We already have the situation that the drm people have their own ci > model. II'm ok with that, partly because then at least the maintainers > of that subsystem can agree on the rules for that one subsystem. > > I'm not at all interested in having something that people will then > either fight about, or - more likely - ignore, at the top level > because there isn't some global agreement about what the rules are. > > For example, even just running checkpatch is often a stylistic thing, > and not everybody agrees about all the checkpatch warnings. > While checkpatch is indeed of arguable value, I think it would help a lot not having to bother about the persistent _build_ failures on 32-bit systems. You mentioned the fancy drm CI system above, but they don't run tests and not even test builds on 32-bit targets, which has repeatedly caused (and currently does cause) build failures in drm code when trying to build, say, arm:allmodconfig in linux-next. Most trivial build failures in linux-next (and, yes, sometimes mainline) could be prevented with a simple generic CI. Sure, argue against checkpatch as much as you like, but the code should at least _build_, and it should not be necessary for random people to report build failures to the submitters. Guenter > I would suggest the CI project be separate from the kernel. > > And having that slack channel that is restricted to particular > companies is just another sign of this whole disease. > > If you want to make a google/microsoft project to do kernel CI, then > more power to you, but don't expect it to be some kind of agreed-upon > kernel project when it's a closed system. > > Linus >
On 3/2/24 14:10, Guenter Roeck wrote: > On Thu, Feb 29, 2024 at 12:21 PM Linus Torvalds > <torvalds@linuxfoundation.org> wrote: >> >> On Thu, 29 Feb 2024 at 01:23, Nikolai Kondrashov <spbnick@gmail.com> wrote: >>> >>> However, I think a better approach would be *not* to add the .gitlab-ci.yaml >>> file in the root of the source tree, but instead change the very same repo >>> setting to point to a particular entry YAML, *inside* the repo (somewhere >>> under "ci" directory) instead. >> >> I really don't want some kind of top-level CI for the base kernel project. >> >> We already have the situation that the drm people have their own ci >> model. II'm ok with that, partly because then at least the maintainers >> of that subsystem can agree on the rules for that one subsystem. >> >> I'm not at all interested in having something that people will then >> either fight about, or - more likely - ignore, at the top level >> because there isn't some global agreement about what the rules are. >> >> For example, even just running checkpatch is often a stylistic thing, >> and not everybody agrees about all the checkpatch warnings. >> > > While checkpatch is indeed of arguable value, I think it would help a > lot not having to bother about the persistent _build_ failures on > 32-bit systems. You mentioned the fancy drm CI system above, but they > don't run tests and not even test builds on 32-bit targets, which has > repeatedly caused (and currently does cause) build failures in drm > code when trying to build, say, arm:allmodconfig in linux-next. Most > trivial build failures in linux-next (and, yes, sometimes mainline) > could be prevented with a simple generic CI. Yes, definitely. Thanks for bringing that up. > Sure, argue against checkpatch as much as you like, but the code > should at least _build_, and it should not be necessary for random > people to report build failures to the submitters. I do 110 randconfig builds nightly (10 each of 11 $ARCH/$BITS). That's about all the horsepower that I have. and I am not a CI. :) So I see quite a bit of what you are saying. It seems that Arnd is in the same boat.
On Sun, Mar 3, 2024 at 3:30 AM Randy Dunlap <rdunlap@infradead.org> wrote: > On 3/2/24 14:10, Guenter Roeck wrote: > > On Thu, Feb 29, 2024 at 12:21 PM Linus Torvalds > > <torvalds@linuxfoundation.org> wrote: > >> On Thu, 29 Feb 2024 at 01:23, Nikolai Kondrashov <spbnick@gmail.com> wrote: > >>> > >>> However, I think a better approach would be *not* to add the .gitlab-ci.yaml > >>> file in the root of the source tree, but instead change the very same repo > >>> setting to point to a particular entry YAML, *inside* the repo (somewhere > >>> under "ci" directory) instead. > >> > >> I really don't want some kind of top-level CI for the base kernel project. > >> > >> We already have the situation that the drm people have their own ci > >> model. II'm ok with that, partly because then at least the maintainers > >> of that subsystem can agree on the rules for that one subsystem. > >> > >> I'm not at all interested in having something that people will then > >> either fight about, or - more likely - ignore, at the top level > >> because there isn't some global agreement about what the rules are. > >> > >> For example, even just running checkpatch is often a stylistic thing, > >> and not everybody agrees about all the checkpatch warnings. > > > > While checkpatch is indeed of arguable value, I think it would help a > > lot not having to bother about the persistent _build_ failures on > > 32-bit systems. You mentioned the fancy drm CI system above, but they > > don't run tests and not even test builds on 32-bit targets, which has > > repeatedly caused (and currently does cause) build failures in drm > > code when trying to build, say, arm:allmodconfig in linux-next. Most > > trivial build failures in linux-next (and, yes, sometimes mainline) > > could be prevented with a simple generic CI. > > Yes, definitely. Thanks for bringing that up. +1 > > Sure, argue against checkpatch as much as you like, but the code > > should at least _build_, and it should not be necessary for random > > people to report build failures to the submitters. > > I do 110 randconfig builds nightly (10 each of 11 $ARCH/$BITS). > That's about all the horsepower that I have. and I am not a CI. :) > > So I see quite a bit of what you are saying. It seems that Arnd is > in the same boat. You don't even have to do your own builds (although it does help), and can look at e.g. http://kisskb.ellerman.id.au/kisskb/ Kisskb can send out email when builds get broken, and when they get fixed again. I receive such emails for the m68k builds. I have the feeling this is not used up to its full potential yet. My initial plan with the "Build regressions/improvements in ..." emails [1] was to fully automate this, and enable it for the other daily builds (e.g. linux-next), too, but there are only so many hours in a day... [1] https://lore.kernel.org/all/20240226081253.3688538-1-geert@linux-m68k.org/ Gr{oetje,eeting}s, Geert
On Sun, Mar 3, 2024 at 10:30 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > On Sun, Mar 3, 2024 at 3:30 AM Randy Dunlap <rdunlap@infradead.org> wrote: > > On 3/2/24 14:10, Guenter Roeck wrote: > > > While checkpatch is indeed of arguable value, I think it would help a > > > lot not having to bother about the persistent _build_ failures on > > > 32-bit systems. You mentioned the fancy drm CI system above, but they > > > don't run tests and not even test builds on 32-bit targets, which has > > > repeatedly caused (and currently does cause) build failures in drm > > > code when trying to build, say, arm:allmodconfig in linux-next. Most > > > trivial build failures in linux-next (and, yes, sometimes mainline) > > > could be prevented with a simple generic CI. > > > > Yes, definitely. Thanks for bringing that up. > > +1 > Kisskb can send out email when builds get broken, and when they get > fixed again. I receive such emails for the m68k builds. Like this (yes, one more in DRM; sometimes I wonder if DRM is meant only for 64-bit little-endian platforms with +200 GiB/s memory bandwidth): ---8<------------------------------------------------------------------- Subject: kisskb: FAILED linux-next/m68k-allmodconfig/m68k-gcc8 Mon Mar 04, 06:35 To: geert@linux-m68k.org Date: Mon, 04 Mar 2024 08:05:14 -0000 FAILED linux-next/m68k-allmodconfig/m68k-gcc8 Mon Mar 04, 06:35 http://kisskb.ellerman.id.au/kisskb/buildresult/15135537/ Commit: Add linux-next specific files for 20240304 67908bf6954b7635d33760ff6dfc189fc26ccc89 Compiler: m68k-linux-gcc (GCC) 8.5.0 / GNU ld (GNU Binutils) 2.36.1 Possible errors --------------- ERROR: modpost: "__udivdi3" [drivers/gpu/drm/sun4i/sun4i-drm-hdmi.ko] undefined! make[3]: *** [scripts/Makefile.modpost:145: Module.symvers] Error 1 make[2]: *** [Makefile:1871: modpost] Error 2 make[1]: *** [Makefile:240: __sub-make] Error 2 make: *** [Makefile:240: __sub-make] Error 2 No warnings found in log. ------------------------------------------------------------------->8--- Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
On Mon, Mar 04, 2024 at 09:12:38AM +0100, Geert Uytterhoeven wrote: > On Sun, Mar 3, 2024 at 10:30 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > On Sun, Mar 3, 2024 at 3:30 AM Randy Dunlap <rdunlap@infradead.org> wrote: > > > On 3/2/24 14:10, Guenter Roeck wrote: > > > > While checkpatch is indeed of arguable value, I think it would help a > > > > lot not having to bother about the persistent _build_ failures on > > > > 32-bit systems. You mentioned the fancy drm CI system above, but they > > > > don't run tests and not even test builds on 32-bit targets, which has > > > > repeatedly caused (and currently does cause) build failures in drm > > > > code when trying to build, say, arm:allmodconfig in linux-next. Most > > > > trivial build failures in linux-next (and, yes, sometimes mainline) > > > > could be prevented with a simple generic CI. > > > > > > Yes, definitely. Thanks for bringing that up. > > > > +1 > > > Kisskb can send out email when builds get broken, and when they get > > fixed again. I receive such emails for the m68k builds. > > Like this (yes, one more in DRM; sometimes I wonder if DRM is meant only > for 64-bit little-endian platforms with +200 GiB/s memory bandwidth): > > ---8<------------------------------------------------------------------- > Subject: kisskb: FAILED linux-next/m68k-allmodconfig/m68k-gcc8 Mon Mar 04, 06:35 > To: geert@linux-m68k.org > Date: Mon, 04 Mar 2024 08:05:14 -0000 > > FAILED linux-next/m68k-allmodconfig/m68k-gcc8 Mon Mar 04, 06:35 > > http://kisskb.ellerman.id.au/kisskb/buildresult/15135537/ > > Commit: Add linux-next specific files for 20240304 > 67908bf6954b7635d33760ff6dfc189fc26ccc89 > Compiler: m68k-linux-gcc (GCC) 8.5.0 / GNU ld (GNU Binutils) 2.36.1 > > Possible errors > --------------- > > ERROR: modpost: "__udivdi3" [drivers/gpu/drm/sun4i/sun4i-drm-hdmi.ko] undefined! > make[3]: *** [scripts/Makefile.modpost:145: Module.symvers] Error 1 > make[2]: *** [Makefile:1871: modpost] Error 2 > make[1]: *** [Makefile:240: __sub-make] Error 2 > make: *** [Makefile:240: __sub-make] Error 2 > > No warnings found in log. > ------------------------------------------------------------------->8--- The driver is meant for a controller featured in an SoC with a Cortex-A8 ARM CPU and less than a GiB/s memory bandwidth. And I just sent a fix for that one, thanks for the report. Maxime
On Sat, Mar 02, 2024 at 02:10:51PM -0800, Guenter Roeck wrote: > On Thu, Feb 29, 2024 at 12:21 PM Linus Torvalds > <torvalds@linuxfoundation.org> wrote: > > > > On Thu, 29 Feb 2024 at 01:23, Nikolai Kondrashov <spbnick@gmail.com> wrote: > > > > > > However, I think a better approach would be *not* to add the .gitlab-ci.yaml > > > file in the root of the source tree, but instead change the very same repo > > > setting to point to a particular entry YAML, *inside* the repo (somewhere > > > under "ci" directory) instead. > > > > I really don't want some kind of top-level CI for the base kernel project. > > > > We already have the situation that the drm people have their own ci > > model. II'm ok with that, partly because then at least the maintainers > > of that subsystem can agree on the rules for that one subsystem. > > > > I'm not at all interested in having something that people will then > > either fight about, or - more likely - ignore, at the top level > > because there isn't some global agreement about what the rules are. > > > > For example, even just running checkpatch is often a stylistic thing, > > and not everybody agrees about all the checkpatch warnings. > > > > While checkpatch is indeed of arguable value, I think it would help a > lot not having to bother about the persistent _build_ failures on > 32-bit systems. You mentioned the fancy drm CI system above, but they > don't run tests and not even test builds on 32-bit targets, which has > repeatedly caused (and currently does cause) build failures in drm > code when trying to build, say, arm:allmodconfig in linux-next. Most > trivial build failures in linux-next (and, yes, sometimes mainline) > could be prevented with a simple generic CI. Ultimately, the question here is about funding. Can we expect DRM CI to pay for builders out of the X.org foundation pocket to build kernel configurations that are seeing close to no development (arm), or not having any driver for (xtensa, m68k)? And if we would turn the argument around, I don't really expect, say, the clock framework, to validate that all DRM kunit tests pass for each commit they merge, even though one of them could totally break some of the DRM tests. If anything, it's more of a side-effect to the push for COMPILE_TEST than anything. Maxime
Hi Maxime, On Mon, Mar 4, 2024 at 10:15 AM Maxime Ripard <mripard@kernel.org> wrote: > On Mon, Mar 04, 2024 at 09:12:38AM +0100, Geert Uytterhoeven wrote: > > On Sun, Mar 3, 2024 at 10:30 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > > On Sun, Mar 3, 2024 at 3:30 AM Randy Dunlap <rdunlap@infradead.org> wrote: > > > > On 3/2/24 14:10, Guenter Roeck wrote: > > > > > While checkpatch is indeed of arguable value, I think it would help a > > > > > lot not having to bother about the persistent _build_ failures on > > > > > 32-bit systems. You mentioned the fancy drm CI system above, but they > > > > > don't run tests and not even test builds on 32-bit targets, which has > > > > > repeatedly caused (and currently does cause) build failures in drm > > > > > code when trying to build, say, arm:allmodconfig in linux-next. Most > > > > > trivial build failures in linux-next (and, yes, sometimes mainline) > > > > > could be prevented with a simple generic CI. > > > > > > > > Yes, definitely. Thanks for bringing that up. > > > > > > +1 > > > > > Kisskb can send out email when builds get broken, and when they get > > > fixed again. I receive such emails for the m68k builds. > > > > Like this (yes, one more in DRM; sometimes I wonder if DRM is meant only > > for 64-bit little-endian platforms with +200 GiB/s memory bandwidth): > > > > ---8<------------------------------------------------------------------- > > Subject: kisskb: FAILED linux-next/m68k-allmodconfig/m68k-gcc8 Mon Mar 04, 06:35 > > To: geert@linux-m68k.org > > Date: Mon, 04 Mar 2024 08:05:14 -0000 > > > > FAILED linux-next/m68k-allmodconfig/m68k-gcc8 Mon Mar 04, 06:35 > > > > http://kisskb.ellerman.id.au/kisskb/buildresult/15135537/ > > > > Commit: Add linux-next specific files for 20240304 > > 67908bf6954b7635d33760ff6dfc189fc26ccc89 > > Compiler: m68k-linux-gcc (GCC) 8.5.0 / GNU ld (GNU Binutils) 2.36.1 > > > > Possible errors > > --------------- > > > > ERROR: modpost: "__udivdi3" [drivers/gpu/drm/sun4i/sun4i-drm-hdmi.ko] undefined! > > make[3]: *** [scripts/Makefile.modpost:145: Module.symvers] Error 1 > > make[2]: *** [Makefile:1871: modpost] Error 2 > > make[1]: *** [Makefile:240: __sub-make] Error 2 > > make: *** [Makefile:240: __sub-make] Error 2 > > > > No warnings found in log. > > ------------------------------------------------------------------->8--- > > The driver is meant for a controller featured in an SoC with a Cortex-A8 > ARM CPU and less than a GiB/s memory bandwidth. Good, so the hardware cannot possibly need 64-bit pixel clock values ;-) BTW, doesn't the build fail on arm32, too? > And I just sent a fix for that one, thanks for the report. Thanks! Gr{oetje,eeting}s, Geert
On Mon, Mar 04, 2024 at 11:07:22AM +0100, Geert Uytterhoeven wrote: > Hi Maxime, > > On Mon, Mar 4, 2024 at 10:15 AM Maxime Ripard <mripard@kernel.org> wrote: > > On Mon, Mar 04, 2024 at 09:12:38AM +0100, Geert Uytterhoeven wrote: > > > On Sun, Mar 3, 2024 at 10:30 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > > > On Sun, Mar 3, 2024 at 3:30 AM Randy Dunlap <rdunlap@infradead.org> wrote: > > > > > On 3/2/24 14:10, Guenter Roeck wrote: > > > > > > While checkpatch is indeed of arguable value, I think it would help a > > > > > > lot not having to bother about the persistent _build_ failures on > > > > > > 32-bit systems. You mentioned the fancy drm CI system above, but they > > > > > > don't run tests and not even test builds on 32-bit targets, which has > > > > > > repeatedly caused (and currently does cause) build failures in drm > > > > > > code when trying to build, say, arm:allmodconfig in linux-next. Most > > > > > > trivial build failures in linux-next (and, yes, sometimes mainline) > > > > > > could be prevented with a simple generic CI. > > > > > > > > > > Yes, definitely. Thanks for bringing that up. > > > > > > > > +1 > > > > > > > Kisskb can send out email when builds get broken, and when they get > > > > fixed again. I receive such emails for the m68k builds. > > > > > > Like this (yes, one more in DRM; sometimes I wonder if DRM is meant only > > > for 64-bit little-endian platforms with +200 GiB/s memory bandwidth): > > > > > > ---8<------------------------------------------------------------------- > > > Subject: kisskb: FAILED linux-next/m68k-allmodconfig/m68k-gcc8 Mon Mar 04, 06:35 > > > To: geert@linux-m68k.org > > > Date: Mon, 04 Mar 2024 08:05:14 -0000 > > > > > > FAILED linux-next/m68k-allmodconfig/m68k-gcc8 Mon Mar 04, 06:35 > > > > > > http://kisskb.ellerman.id.au/kisskb/buildresult/15135537/ > > > > > > Commit: Add linux-next specific files for 20240304 > > > 67908bf6954b7635d33760ff6dfc189fc26ccc89 > > > Compiler: m68k-linux-gcc (GCC) 8.5.0 / GNU ld (GNU Binutils) 2.36.1 > > > > > > Possible errors > > > --------------- > > > > > > ERROR: modpost: "__udivdi3" [drivers/gpu/drm/sun4i/sun4i-drm-hdmi.ko] undefined! > > > make[3]: *** [scripts/Makefile.modpost:145: Module.symvers] Error 1 > > > make[2]: *** [Makefile:1871: modpost] Error 2 > > > make[1]: *** [Makefile:240: __sub-make] Error 2 > > > make: *** [Makefile:240: __sub-make] Error 2 > > > > > > No warnings found in log. > > > ------------------------------------------------------------------->8--- > > > > The driver is meant for a controller featured in an SoC with a Cortex-A8 > > ARM CPU and less than a GiB/s memory bandwidth. > > Good, so the hardware cannot possibly need 64-bit pixel clock values ;-) This is an early patch to convert that function into a framework hook implementation. HDMI 2.1 has a max TMDS character rate of slightly less than 6GHz, so larger than 2^32 - 1. So yes, this driver doesn't need to. The framework does however. > BTW, doesn't the build fail on arm32, too? It seems like gcc vs clang plays a role too. I had the same defconfig building for arm with gcc and reporting the error above with clang. I didn't look further because there was something to fix indeed. Maxime
Hi Maxime, On Mon, Mar 4, 2024 at 11:20 AM Maxime Ripard <mripard@kernel.org> wrote: > On Mon, Mar 04, 2024 at 11:07:22AM +0100, Geert Uytterhoeven wrote: > > On Mon, Mar 4, 2024 at 10:15 AM Maxime Ripard <mripard@kernel.org> wrote: > > > On Mon, Mar 04, 2024 at 09:12:38AM +0100, Geert Uytterhoeven wrote: > > > > On Sun, Mar 3, 2024 at 10:30 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > > > ERROR: modpost: "__udivdi3" [drivers/gpu/drm/sun4i/sun4i-drm-hdmi.ko] undefined! > > > > make[3]: *** [scripts/Makefile.modpost:145: Module.symvers] Error 1 > > > > make[2]: *** [Makefile:1871: modpost] Error 2 > > > > make[1]: *** [Makefile:240: __sub-make] Error 2 > > > > make: *** [Makefile:240: __sub-make] Error 2 > > > > > > > > No warnings found in log. > > > > ------------------------------------------------------------------->8--- > > > > > > The driver is meant for a controller featured in an SoC with a Cortex-A8 > > > ARM CPU and less than a GiB/s memory bandwidth. > > > > Good, so the hardware cannot possibly need 64-bit pixel clock values ;-) > > This is an early patch to convert that function into a framework hook > implementation. HDMI 2.1 has a max TMDS character rate of slightly less > than 6GHz, so larger than 2^32 - 1. > > So yes, this driver doesn't need to. The framework does however. That's gonna be interesting, as the Common Clock Framework does not support 64-bit clock rates on 32-bit platforms yet... Gr{oetje,eeting}s, Geert
On Mon, Mar 04, 2024 at 12:12:47PM +0100, Geert Uytterhoeven wrote: > Hi Maxime, > > On Mon, Mar 4, 2024 at 11:20 AM Maxime Ripard <mripard@kernel.org> wrote: > > On Mon, Mar 04, 2024 at 11:07:22AM +0100, Geert Uytterhoeven wrote: > > > On Mon, Mar 4, 2024 at 10:15 AM Maxime Ripard <mripard@kernel.org> wrote: > > > > On Mon, Mar 04, 2024 at 09:12:38AM +0100, Geert Uytterhoeven wrote: > > > > > On Sun, Mar 3, 2024 at 10:30 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote: > > > > > ERROR: modpost: "__udivdi3" [drivers/gpu/drm/sun4i/sun4i-drm-hdmi.ko] undefined! > > > > > make[3]: *** [scripts/Makefile.modpost:145: Module.symvers] Error 1 > > > > > make[2]: *** [Makefile:1871: modpost] Error 2 > > > > > make[1]: *** [Makefile:240: __sub-make] Error 2 > > > > > make: *** [Makefile:240: __sub-make] Error 2 > > > > > > > > > > No warnings found in log. > > > > > ------------------------------------------------------------------->8--- > > > > > > > > The driver is meant for a controller featured in an SoC with a Cortex-A8 > > > > ARM CPU and less than a GiB/s memory bandwidth. > > > > > > Good, so the hardware cannot possibly need 64-bit pixel clock values ;-) > > > > This is an early patch to convert that function into a framework hook > > implementation. HDMI 2.1 has a max TMDS character rate of slightly less > > than 6GHz, so larger than 2^32 - 1. > > > > So yes, this driver doesn't need to. The framework does however. > > That's gonna be interesting, as the Common Clock Framework does not > support 64-bit clock rates on 32-bit platforms yet... As far as KMS goes, it doesn't really matter. The TMDS character rate doesn't need to be backed by a CCF clock. And realistically speaking, I don't expect any 32 bit platform to support HDMI 2.1. Maxime
On Mon, Mar 4, 2024 at 1:24 AM Maxime Ripard <mripard@kernel.org> wrote: [ ... ] > > If anything, it's more of a side-effect to the push for COMPILE_TEST > than anything. > If the drm subsystem maintainers don't want people to build it with COMPILE_TEST while at the same time not limiting it to platforms where it doesn't even build, I'd suggest making it dependent on !COMPILE_TEST. The same applies to all other subsystems where maintainers don't want build tests to run but also don't want to add restrictions such as "64-bit only". After all, this was just one example. Guenter
On Mon, Mar 04, 2024 at 07:46:34AM -0800, Guenter Roeck wrote: > On Mon, Mar 4, 2024 at 1:24 AM Maxime Ripard <mripard@kernel.org> wrote: > [ ... ] > > > > If anything, it's more of a side-effect to the push for COMPILE_TEST > > than anything. > > > > If the drm subsystem maintainers don't want people to build it with > COMPILE_TEST while at the same time not limiting it to platforms where > it doesn't even build, I'd suggest making it dependent on > !COMPILE_TEST. I don't think we want anything. My point was that you can't have an option that is meant to explore for bad practices and expose drivers that don't go through the proper abstraction, and at the same time complain that things gets broken. It's the whole point of it. > The same applies to all other subsystems where maintainers don't want > build tests to run but also don't want to add restrictions such as > "64-bit only". After all, this was just one example. We have drivers for some 32 bits platforms. Maxime
On Mon, Mar 4, 2024 at 8:05 AM Maxime Ripard <mripard@kernel.org> wrote: > > On Mon, Mar 04, 2024 at 07:46:34AM -0800, Guenter Roeck wrote: > > On Mon, Mar 4, 2024 at 1:24 AM Maxime Ripard <mripard@kernel.org> wrote: > > [ ... ] > > > > > > If anything, it's more of a side-effect to the push for COMPILE_TEST > > > than anything. > > > > > > > If the drm subsystem maintainers don't want people to build it with > > COMPILE_TEST while at the same time not limiting it to platforms where > > it doesn't even build, I'd suggest making it dependent on > > !COMPILE_TEST. > > I don't think we want anything. My point was that you can't have an > option that is meant to explore for bad practices and expose drivers > that don't go through the proper abstraction, and at the same time > complain that things gets broken. It's the whole point of it. > Can we get back to the original problem, please ? Build errors such as failed 32-bit builds are a nuisance for those running build tests. It seems to me that an automated infrastructure to prevent such build errors from making it into the kernel should be desirable. You seem to disagree, and at least it looked like you complained about the existence of COMPILE_TEST, or that people are doing COMPILE_TEST builds. What is your suggested alternative ? Disabling build tests on drm doesn't seem to be it, and it seems you don't like the idea of a basic generic CI either, but what is it ? > > The same applies to all other subsystems where maintainers don't want > > build tests to run but also don't want to add restrictions such as > > "64-bit only". After all, this was just one example. > > We have drivers for some 32 bits platforms. > I said "such as". Again, that was an example. In this case it would obviously only apply to parts of drm which are not supported on 32-bit systems (and, presumably, the parts of drm which are supposed to be supported on 32-bit systems should build on those). Thanks, Guenter
On Mon, Mar 04, 2024 at 08:17:22AM -0800, Guenter Roeck wrote: > On Mon, Mar 4, 2024 at 8:05 AM Maxime Ripard <mripard@kernel.org> wrote: > > > > On Mon, Mar 04, 2024 at 07:46:34AM -0800, Guenter Roeck wrote: > > > On Mon, Mar 4, 2024 at 1:24 AM Maxime Ripard <mripard@kernel.org> wrote: > > > [ ... ] > > > > > > > > If anything, it's more of a side-effect to the push for COMPILE_TEST > > > > than anything. > > > > > > > > > > If the drm subsystem maintainers don't want people to build it with > > > COMPILE_TEST while at the same time not limiting it to platforms where > > > it doesn't even build, I'd suggest making it dependent on > > > !COMPILE_TEST. > > > > I don't think we want anything. My point was that you can't have an > > option that is meant to explore for bad practices and expose drivers > > that don't go through the proper abstraction, and at the same time > > complain that things gets broken. It's the whole point of it. > > > Can we get back to the original problem, please ? Sure. > Build errors such as failed 32-bit builds are a nuisance for those > running build tests. It seems to me that an automated infrastructure > to prevent such build errors from making it into the kernel should be > desirable. You seem to disagree, and at least it looked like you > complained about the existence of COMPILE_TEST, or that people are > doing COMPILE_TEST builds. What is your suggested alternative ? > Disabling build tests on drm doesn't seem to be it, and it seems you > don't like the idea of a basic generic CI either, but what is it ? You don't have to be aggressive about it though. Anyway. The original problem I pointed out was funding. You can't expect everyone to pay for builders running things they fundamentally don't care about. That's it. I'm all for CI, I'm all for automated tests and builds, I don't think COMPILE_TEST is a bad idea, I also think doing those kind of builds is worth it and useful. But the point of those exploratory kind of builds is precisely to look for breakages, so is something we should expect, not complain about. There's nothing to fix, or nothing to improve to me, except the general discourse. And singling out DRM because it regularly allegedly breaks things on xtensa or m68k and claiming we're not taking CI seriously because of it is completely ridiculous. If the all the subsystems were taking CI as seriously as DRM, we would be in a much better place. Maxime
On Mon, Mar 4, 2024 at 9:09 AM Maxime Ripard <mripard@kernel.org> wrote: > And singling out DRM because it regularly allegedly breaks things on > xtensa or m68k and claiming we're not taking CI seriously because of it > is completely ridiculous. If the all the subsystems were taking CI as > seriously as DRM, we would be in a much better place. > The failure I reported as an example was on arm, not on xtensa or m68k I'll disable CONFIG_DRM in my build tests for xtensa and m68k going forward. Thanks, Guenter
On Mon, Mar 4, 2024 at 9:09 AM Maxime Ripard <mripard@kernel.org> wrote: [ ...] > > And singling out DRM because it regularly allegedly breaks things on > xtensa or m68k and claiming we're not taking CI seriously because of it > is completely ridiculous. If the all the subsystems were taking CI as > seriously as DRM, we would be in a much better place. > FWIW: $ git grep CONFIG_DRM arch/xtensa/ arch/m68k/ arch/m68k/configs/virt_defconfig:CONFIG_DRM=y arch/m68k/configs/virt_defconfig:CONFIG_DRM_FBDEV_EMULATION=y arch/m68k/configs/virt_defconfig:CONFIG_DRM_VIRTIO_GPU=y arch/xtensa/configs/virt_defconfig:CONFIG_DRM=y arch/xtensa/configs/virt_defconfig:CONFIG_DRM_VGEM=y arch/xtensa/configs/virt_defconfig:CONFIG_DRM_VIRTIO_GPU=y If that isn't supported, it might really make sense to declare that CONFIG_DRM depends on !xtensa and !m68k. Guenter
Hi Linus, Thank you for your reply and valuable inputs. On 01/03/2024 17:10, Linus Torvalds wrote: > On Fri, 1 Mar 2024 at 02:27, Nikolai Kondrashov <spbnick@gmail.com> wrote: >> >> I agree, it's hard to imagine even a simple majority agreeing on how GitLab CI >> should be done. Still, we would like to help people, who are interested in >> this kind of thing, to set it up. How about we reframe this contribution as a >> sort of template, or a reference for people to start their setup with, >> assuming that most maintainers would want to tweak it? We would also be glad >> to stand by for questions and help, as people try to use it. > > Ack. I think seeing it as a library for various gitlab CI models would > be a lot more palatable. Particularly if you can then show that yes, > it is also relevant to our currently existing drm case. Having it as a library would certainly make my work as the DRM-CI maintainer easier and also simplify the process whenever we consider integrating tests into other subsystems. > > So I'm not objecting to having (for example) some kind of CI helper > templates - I think a logical place would be in tools/ci/ which is > kind of alongside our tools/testing subdirectory. Works for me. We can skip having a default .gitlab-ci.yml in the root directory and instead include clear instructions in our documentation for using these templates. Thanks, Helen Koike > > (And then perhaps have a 'gitlab' directory under that. I'm not sure > whether - and how much - commonality there might be between the > different CI models of different hosts). > > Just to clarify: when I say "a logical place", I very much want to > emphasize the "a" - maybe there are better places, and I'm not saying > that is the only possible place. But it sounds more logical to me than > some. > > Linus
On 2024-02-29 21:21, Linus Torvalds wrote: > On Thu, 29 Feb 2024 at 01:23, Nikolai Kondrashov <spbnick@gmail.com> wrote: >> >> However, I think a better approach would be *not* to add the .gitlab-ci.yaml >> file in the root of the source tree, but instead change the very same repo >> setting to point to a particular entry YAML, *inside* the repo (somewhere >> under "ci" directory) instead. > > I really don't want some kind of top-level CI for the base kernel project. > > We already have the situation that the drm people have their own ci > model. II'm ok with that, partly because then at least the maintainers > of that subsystem can agree on the rules for that one subsystem. > > I'm not at all interested in having something that people will then > either fight about, or - more likely - ignore, at the top level > because there isn't some global agreement about what the rules are. > > For example, even just running checkpatch is often a stylistic thing, > and not everybody agrees about all the checkpatch warnings. > > I would suggest the CI project be separate from the kernel. That would be missing a lot of the point / benefit of CI. A CI system which is separate from the kernel will tend to be out of sync, so it can't gate the merging of changes and thus can't prevent regressions from propagating.
Le jeudi 29 février 2024 à 10:02 +0100, Maxime Ripard a écrit : > Hi Helen, > > Thanks for working on this > > On Wed, Feb 28, 2024 at 07:55:25PM -0300, Helen Koike wrote: > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, > > defininga basic test pipeline triggered by code pushes to a GitLab-CI > > instance. This initial version includes static checks (checkpatch and > > smatch for now) and build tests across various architectures and > > configurations. It leverages an integrated cache for efficient build > > times and introduces a flexible 'scenarios' mechanism for > > subsystem-specific extensions. > > > > [ci: add prerequisites to run check-patch on MRs] > > Co-developed-by: Tales Aparecida <tales.aparecida@redhat.com> > > Signed-off-by: Tales Aparecida <tales.aparecida@redhat.com> > > Signed-off-by: Helen Koike <helen.koike@collabora.com> > > > > --- > > > > Hey all, > > > > You can check the validation of this patchset on: > > https://gitlab.collabora.com/koike/linux/-/pipelines/87035 > > > > I would appreciate your feedback on this work, what do you think? > > > > If you would rate from 0 to 5, where: > > > > [ ] 0. I don't think this is useful at all, and I doubt it will ever be. It doesn't seem worthwhile. > > [ ] 1. I don't find it useful in its current form. > > [ ] 2. It might be useful to others, but not for me. > > [ ] 3. It has potential, but it's not yet something I can incorporate into my workflow. > > [ ] 4. This is useful, but it needs some adjustments before I can include it in my workflow. > > [ ] 5. This is really useful! I'm eager to start using it right away. Why didn't you send this earlier? :) > > > > Which rating would you select? > > 4.5 :) > > One thing I'm wondering here is how we're going to cope with the > different requirements each user / framework has. > > Like, Linus probably want to have a different set of CI before merging a > PR than (say) linux-next does, or stable, or before doing an actual > release. > > Similarly, DRM probably has a different set of requirements than > drm-misc, drm-amd or nouveau. > > I don't see how the current architecture could accomodate for that. I > know that Gitlab allows to store issues template in a separate repo, > maybe we could ask them to provide a feature where the actions would be > separate from the main repo? That way, any gitlab project could provide > its own set of tests, without conflicting with each others (and we could > still share them if we wanted to) > > I know some of use had good relationship with Gitlab, so maybe it would > be worth asking? As agreed, the .gitlab-ci.yaml file at the list will go away. Its a default location, but not a required location. This way, each sub-system can have their own (or not have one). The different sub-system forks will have to be configured to point to their respective CI main configuration. Of course nothing prevents having common set of configuration for jobs and jobs template. As an example, we could have a job template common for checkpatch, and allow each subsystem adding their own sauce on top. It can save the duplicate effort of parsing the tool results and reporting it in a format gitlab understand. This also allow subsystems to offer different coverage, e.g. a fast vs a full coverage. Or perhaps a configuration to focus on specific devices. But in general, just not having a central config is enough to support the idea. What we have now could be entirely drm specific and "commonized" later when other subsystem wanting to use gitlab joins (Linux Media is among those). Nicolas
On Mon, 2024-03-04 at 18:45 -0300, Helen Koike wrote: > Hi Linus, > > Thank you for your reply and valuable inputs. > > On 01/03/2024 17:10, Linus Torvalds wrote: > > On Fri, 1 Mar 2024 at 02:27, Nikolai Kondrashov <spbnick@gmail.com> wrote: > > > > > > I agree, it's hard to imagine even a simple majority agreeing on how GitLab CI > > > should be done. Still, we would like to help people, who are interested in > > > this kind of thing, to set it up. How about we reframe this contribution as a > > > sort of template, or a reference for people to start their setup with, > > > assuming that most maintainers would want to tweak it? We would also be glad > > > to stand by for questions and help, as people try to use it. > > > > Ack. I think seeing it as a library for various gitlab CI models would > > be a lot more palatable. Particularly if you can then show that yes, > > it is also relevant to our currently existing drm case. > > Having it as a library would certainly make my work as the DRM-CI > maintainer easier and also simplify the process whenever we consider > integrating tests into other subsystems. > > > > > So I'm not objecting to having (for example) some kind of CI helper > > templates - I think a logical place would be in tools/ci/ which is > > kind of alongside our tools/testing subdirectory. > > Works for me. > > We can skip having a default .gitlab-ci.yml in the root directory and > instead include clear instructions in our documentation for using these > templates. From previous experience[1], I recommend this approach. This way it does not bother current Gitlab mirrors / personal repos, while allowing anyone to setup the CI from Gitlab menus just by changing: Repo -> Settings -> CI/CD -> General Pipelines -> CI/CD configuration file Thanks! Leo [1] Last year I implemented Gitlab-CI for the perfbook repo, and I came across some issues, including the disruption of .gitlab-ci.yml in the root of a repo. https://lore.kernel.org/perfbook/20230201201529.901316-1-leobras.c@gmail.com/ > > Thanks, > Helen Koike > > > > > (And then perhaps have a 'gitlab' directory under that. I'm not sure > > whether - and how much - commonality there might be between the > > different CI models of different hosts). > > > > Just to clarify: when I say "a logical place", I very much want to > > emphasize the "a" - maybe there are better places, and I'm not saying > > that is the only possible place. But it sounds more logical to me than > > some. > > > > Linus
Hi Nicolas, On Thu, Mar 07, 2024 at 01:05:12PM -0500, Nicolas Dufresne wrote: > Le jeudi 29 février 2024 à 10:02 +0100, Maxime Ripard a écrit : > > On Wed, Feb 28, 2024 at 07:55:25PM -0300, Helen Koike wrote: > > > This patch introduces a `.gitlab-ci` file along with a `ci/` folder, > > > defininga basic test pipeline triggered by code pushes to a GitLab-CI > > > instance. This initial version includes static checks (checkpatch and > > > smatch for now) and build tests across various architectures and > > > configurations. It leverages an integrated cache for efficient build > > > times and introduces a flexible 'scenarios' mechanism for > > > subsystem-specific extensions. > > > > > > [ci: add prerequisites to run check-patch on MRs] > > > Co-developed-by: Tales Aparecida <tales.aparecida@redhat.com> > > > Signed-off-by: Tales Aparecida <tales.aparecida@redhat.com> > > > Signed-off-by: Helen Koike <helen.koike@collabora.com> > > > > > > --- > > > > > > Hey all, > > > > > > You can check the validation of this patchset on: > > > https://gitlab.collabora.com/koike/linux/-/pipelines/87035 > > > > > > I would appreciate your feedback on this work, what do you think? > > > > > > If you would rate from 0 to 5, where: > > > > > > [ ] 0. I don't think this is useful at all, and I doubt it will ever be. It doesn't seem worthwhile. > > > [ ] 1. I don't find it useful in its current form. > > > [ ] 2. It might be useful to others, but not for me. > > > [ ] 3. It has potential, but it's not yet something I can incorporate into my workflow. > > > [ ] 4. This is useful, but it needs some adjustments before I can include it in my workflow. > > > [ ] 5. This is really useful! I'm eager to start using it right away. Why didn't you send this earlier? :) > > > > > > Which rating would you select? > > > > 4.5 :) > > > > One thing I'm wondering here is how we're going to cope with the > > different requirements each user / framework has. > > > > Like, Linus probably want to have a different set of CI before merging a > > PR than (say) linux-next does, or stable, or before doing an actual > > release. > > > > Similarly, DRM probably has a different set of requirements than > > drm-misc, drm-amd or nouveau. > > > > I don't see how the current architecture could accomodate for that. I > > know that Gitlab allows to store issues template in a separate repo, > > maybe we could ask them to provide a feature where the actions would be > > separate from the main repo? That way, any gitlab project could provide > > its own set of tests, without conflicting with each others (and we could > > still share them if we wanted to) > > > > I know some of use had good relationship with Gitlab, so maybe it would > > be worth asking? > > As agreed, the .gitlab-ci.yaml file at the list will go away. Its a default > location, but not a required location. This way, each sub-system can have their > own (or not have one). The different sub-system forks will have to be configured > to point to their respective CI main configuration. > > Of course nothing prevents having common set of configuration for jobs and jobs > template. As an example, we could have a job template common for checkpatch, and > allow each subsystem adding their own sauce on top. It can save the duplicate > effort of parsing the tool results and reporting it in a format gitlab > understand. That makes total sense to me and would be incredibly useful indeed. Maxime
On Mon, Mar 04, 2024 at 06:45:33PM -0300, Helen Koike wrote: > Hi Linus, > > Thank you for your reply and valuable inputs. > > On 01/03/2024 17:10, Linus Torvalds wrote: > > On Fri, 1 Mar 2024 at 02:27, Nikolai Kondrashov <spbnick@gmail.com> wrote: > > > > > > I agree, it's hard to imagine even a simple majority agreeing on how GitLab CI > > > should be done. Still, we would like to help people, who are interested in > > > this kind of thing, to set it up. How about we reframe this contribution as a > > > sort of template, or a reference for people to start their setup with, > > > assuming that most maintainers would want to tweak it? We would also be glad > > > to stand by for questions and help, as people try to use it. > > > > Ack. I think seeing it as a library for various gitlab CI models would > > be a lot more palatable. Particularly if you can then show that yes, > > it is also relevant to our currently existing drm case. > > Having it as a library would certainly make my work as the DRM-CI maintainer > easier and also simplify the process whenever we consider integrating tests > into other subsystems. Kinda ignored this thread, just wanted to throw my +1 in here. To spin it positively, the kernel CI space is wide open (more negatively, it's a fractured mess). And I think there's just no way to force top-down unification. Imo the only way is to land subsystem CI support in upstream, figure out what exactly that should look like (I sketched a lot of open questions in the DRM CI PR around what should and should not be in upstream). Then, once we have a few of those, extract common scripts and tools into tools/ci/ or scripts/ci or whatever. And only then, best case years down the road, dare to have some common top-level CI, once it's clear what the actual common pieces and test stages even are. > > So I'm not objecting to having (for example) some kind of CI helper > > templates - I think a logical place would be in tools/ci/ which is > > kind of alongside our tools/testing subdirectory. > > Works for me. > > We can skip having a default .gitlab-ci.yml in the root directory and > instead include clear instructions in our documentation for using these > templates. I'd go a few steps more back and start with trying to get more subsystem CI into upstream. And then once that dust has settled, figure out what the common pieces actually are. Because I'm pretty sure that what we have for drm ci or kernelci right now won't be it, but likely just a local optimum. Cheers, Sima > > Thanks, > Helen Koike > > > > > (And then perhaps have a 'gitlab' directory under that. I'm not sure > > whether - and how much - commonality there might be between the > > different CI models of different hosts). > > > > Just to clarify: when I say "a logical place", I very much want to > > emphasize the "a" - maybe there are better places, and I'm not saying > > that is the only possible place. But it sounds more logical to me than > > some. > > > > Linus
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000000000..fac523bb86866 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,2 @@ +include: + - ci/gitlab-ci/yml/gitlab-ci.yml diff --git a/MAINTAINERS b/MAINTAINERS index 716b2e22598c8..aa0f65791c2ee 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4998,6 +4998,14 @@ T: git git://linuxtv.org/media_tree.git F: Documentation/devicetree/bindings/media/i2c/chrontel,ch7322.yaml F: drivers/media/cec/i2c/ch7322.c +CI AUTOMATED TESTING +M: Helen Koike <helen.koike@collabora.com> +L: linux-kernel@vger.kernel.org +S: Maintained +T: git https://gitlab.collabora.com/koike/linux.git +F: .gitlab-ci.yml +F: ci/ + CIRRUS LOGIC AUDIO CODEC DRIVERS M: James Schulman <james.schulman@cirrus.com> M: David Rhodes <david.rhodes@cirrus.com> diff --git a/ci/gitlab-ci/bootstrap-gitlab-runner.sh b/ci/gitlab-ci/bootstrap-gitlab-runner.sh new file mode 100755 index 0000000000000..73238960d0880 --- /dev/null +++ b/ci/gitlab-ci/bootstrap-gitlab-runner.sh @@ -0,0 +1,55 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +set -eo pipefail + +# Define variables +CONFIG_VOLUME="/srv/gitlab-runner/config" # Path to your GitLab Runner config + +# Check if RUNNER_REGISTRATION_TOKEN is set +if [ -z "${RUNNER_REGISTRATION_TOKEN}" ]; then + echo "Error: RUNNER_REGISTRATION_TOKEN is not set." + echo "Please set the RUNNER_REGISTRATION_TOKEN environment variable and try again." + exit 1 +fi + +# Check if GITLAB_URL is set +if [ -z "${GITLAB_URL}" ]; then + GITLAB_URL="https://gitlab.com/" + echo "Info: GITLAB_URL is not set. Using the default $GITLAB_URL" + echo "Please set the RUNNER_REGISTRATION_TOKEN environment variable and try again." +fi + +# Check if docker-compose is installed +if ! command -v docker-compose &> /dev/null +then + echo "docker-compose could not be found. Please install it first." + exit 1 +fi + +# Start the GitLab Runner using Docker Compose +echo "Starting GitLab Runner..." +docker-compose up -d + +# Wait for a few seconds to ensure the service is up +sleep 5 + +# Register the GitLab Runner +echo "Registering GitLab Runner..." +docker run --rm -v ${CONFIG_VOLUME}:/etc/gitlab-runner gitlab/gitlab-runner register \ + --non-interactive \ + --url ${GITLAB_URL} \ + --token ${RUNNER_REGISTRATION_TOKEN} \ + --executor docker \ + --docker-image "alpine:latest" \ + --description "Docker Runner" \ + --docker-privileged + +echo "" +echo "INFO: To configure the number of concurrent jobs, edit the value of" +echo "INFO: concurrent in ${CONFIG_VOLUME}/config.toml, than restart the GitLab" +echo "INFO: Runner using docker-compose restart" +echo "" +echo "GitLab Runner setup complete." diff --git a/ci/gitlab-ci/ci-scripts/build-docs.sh b/ci/gitlab-ci/ci-scripts/build-docs.sh new file mode 100755 index 0000000000000..2053cb8794acf --- /dev/null +++ b/ci/gitlab-ci/ci-scripts/build-docs.sh @@ -0,0 +1,35 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +set -exo pipefail + +source ci/gitlab-ci/ci-scripts/ici-functions.sh + +ici_get_patch_series_size + +# Get the list of modified files in the last $ICI_PATCH_SERIES_SIZE commits +MODIFIED_DOC_FILES=$(git diff HEAD~$ICI_PATCH_SERIES_SIZE --name-only -- Documentation/) + +make -j$(nproc) "$ICI_DOC_TYPE" 2>&1 | tee output.txt + +mkdir -p "${CI_PROJECT_DIR}/artifacts" +mv Documentation/output "${CI_PROJECT_DIR}/artifacts/Documentation-output" + +# Check if any of the MODIFIED_DOC_FILES generated a warning +# NOTE: the alternative solution was to touch the modified files and run make +# again, but too much warnings still appears +for file in $MODIFIED_DOC_FILES; do + if grep -qi "warning" output.txt && grep -q "$file" output.txt; then + echo "Warning found in $file" + exit 101 + fi +done + +if [ -n "$ICI_UNABLE_TO_DETECT_PATCH_SERIES_SIZE" ]; then + # If the patch series size was not detected, exit with a warning + echo -n "The patch series size was not detected, we probably didn't check the" + echo " whole series. Exiting with a warning." + exit 101 +fi diff --git a/ci/gitlab-ci/ci-scripts/build-kernel.sh b/ci/gitlab-ci/ci-scripts/build-kernel.sh new file mode 100755 index 0000000000000..361826368a573 --- /dev/null +++ b/ci/gitlab-ci/ci-scripts/build-kernel.sh @@ -0,0 +1,35 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +set -exo pipefail + +source ci/gitlab-ci/ci-scripts/ici-functions.sh + +ici_prepare_build + +pushd build + +# compile the entire kernel +make CF=-D__CHECK_ENDIAN__ -C "$ICI_KERNEL_DIR" O=$(pwd) -j$(nproc) 2>&1 | tee output.txt + +export INSTALL_PATH="${CI_PROJECT_DIR}/artifacts/" +INSTALL_PATH+="kernel-install-${KCI_KERNEL_ARCH}-${KCI_DEFCONFIG}_config" +mkdir -p "$INSTALL_PATH" + +# install the kernel image to artifacts/kernel-install +make -C "$ICI_KERNEL_DIR" O=$(pwd) install INSTALL_PATH="$INSTALL_PATH" + +# install kernel modules to artifacts/kernel-install +make -C "$ICI_KERNEL_DIR" O=$(pwd) modules_install INSTALL_MOD_PATH="$INSTALL_PATH" + +# export config as artifact +cp .config "${CI_PROJECT_DIR}/artifacts/${KCI_KERNEL_ARCH}-${KCI_DEFCONFIG}_config" + +# if the compilation has warnings, exit with the warning code +if grep -iq "warning" output.txt; then + exit 101 +fi + +popd diff --git a/ci/gitlab-ci/ci-scripts/ici-functions.sh b/ci/gitlab-ci/ci-scripts/ici-functions.sh new file mode 100644 index 0000000000000..df133b4d00103 --- /dev/null +++ b/ci/gitlab-ci/ci-scripts/ici-functions.sh @@ -0,0 +1,104 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +set -exo pipefail + +# internal CI bash functions + +# convention: +# KCI_<VARIABLE_NAME> for variables defined by the user (outside of this script) +# ICI_<VARIABLE_NAME> for variables defined internally for usage between scripts +# CI_<VARIABLE_NAME> for variables defined by GitLab CI + + +ici_prepare_build() { + BUILD_DIR="${1:-build}" + + echo "" + echo "Architecture: $KCI_KERNEL_ARCH" + echo "Defconfig: $KCI_DEFCONFIG" + echo "" + + # Get the current directory if KCI_KERNEL_DIR is not set + ICI_KERNEL_DIR="${KCI_KERNEL_DIR:-$(pwd)}" + + cd "$ICI_KERNEL_DIR" || { echo "Kernel directory not found"; exit 1; } + + # Clean up stale rebases that GitLab might not have removed when reusing a checkout dir + rm -rf .git/rebase-apply + + if [[ "$KCI_KERNEL_ARCH" = "arm64" ]]; then + GCC_ARCH="aarch64-linux-gnu" + elif [[ "$KCI_KERNEL_ARCH" = "arm" ]]; then + GCC_ARCH="arm-linux-gnueabihf" + else + GCC_ARCH="x86_64-linux-gnu" + fi + + # do not set ARCH and CROSS_COMPILE if KCI_KERNEL_ARCH is not set, useful for local run + if [ -n "$KCI_KERNEL_ARCH" ]; then + export ARCH=${KCI_KERNEL_ARCH} + export CROSS_COMPILE="${GCC_ARCH}-" + fi + + mkdir -p "$BUILD_DIR" + + pushd "$BUILD_DIR" || { echo "Failed to create $BUILD_DIR directory"; exit 1; } + + # generate defconfig + make -C "$ICI_KERNEL_DIR" O=$(pwd) $(basename ${KCI_DEFCONFIG-"defconfig"}) + + # add extra configs from variable KCI_KCONFIGS_{ENABLE,DISABLE,MODULE} + for opt in $KCI_KCONFIGS_ENABLE; do + ../scripts/config --file .config --enable CONFIG_$opt + done + for opt in $KCI_KCONFIGS_DISABLE; do + ../scripts/config --file .config --disable CONFIG_$opt + done + for opt in $KCI_KCONFIGS_MODULE; do + ../scripts/config --file .config --module CONFIG_$opt + done + + if [ -n "$KCI_KCONFIGS_DISABLE" ] || [ -n "$KCI_KCONFIGS_ENABLE" ] || + [ -n "$KCI_KCONFIGS_MODULE" ]; then + # execude olddefconfig only if we changed the default config, otherwise, + # let it raise warnings if any + make -C "$ICI_KERNEL_DIR" O=$(pwd) olddefconfig + fi + + popd +} + +ici_get_patch_series_size() +{ + local CLONE_DEPTH + CLONE_DEPTH=$(git rev-list --count HEAD) + echo "The depth of the clone is $CLONE_DEPTH" + + # If this is in the context of a merge request, calculate the patch series + # size comparing to the target branch + if [ -n "$CI_MERGE_REQUEST_IID" ]; then + git fetch origin "$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" --depth $CLONE_DEPTH + BASE_COMMIT="origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" + ICI_PATCH_SERIES_SIZE=$(git rev-list --count ${BASE_COMMIT}.."$CI_COMMIT_SHA") + + # if KCI_PATCH_SERIES_SIZE is set, use it + elif [ -n "$KCI_PATCH_SERIES_SIZE" ]; then + ICI_PATCH_SERIES_SIZE="$KCI_PATCH_SERIES_SIZE" + else + ICI_PATCH_SERIES_SIZE=1 + echo "WARNING: unable to detect the patch series size, using the default value of 1." + # shellcheck disable=SC2034 + ICI_UNABLE_TO_DETECT_PATCH_SERIES_SIZE=true + fi + + # Check if the clone depth is smaller than or equal to KCI_PATCH_SERIES_SIZE, + # otherwise the checkpatch.pl hangs + if [ "$ICI_PATCH_SERIES_SIZE" -ge "$CLONE_DEPTH" ]; then + echo -n "ERROR: the depth of the clone is $CLONE_DEPTH, smaller than or equal to the patch" + echo " series size. Update your GitLab configuration to increase the size of the clone." + return 1 + fi +} diff --git a/ci/gitlab-ci/ci-scripts/install-smatch.sh b/ci/gitlab-ci/ci-scripts/install-smatch.sh new file mode 100755 index 0000000000000..cae16a0e90fa8 --- /dev/null +++ b/ci/gitlab-ci/ci-scripts/install-smatch.sh @@ -0,0 +1,13 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +set -exo pipefail + +pushd / +git clone --depth 1 https://repo.or.cz/smatch.git +pushd smatch +make +popd +popd diff --git a/ci/gitlab-ci/ci-scripts/parse_commit_message.sh b/ci/gitlab-ci/ci-scripts/parse_commit_message.sh new file mode 100755 index 0000000000000..c9792c64ad51e --- /dev/null +++ b/ci/gitlab-ci/ci-scripts/parse_commit_message.sh @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +set -exo pipefail + +# Get the last commit message +commit_message=$(git log -1 --pretty=%B) + +pattern='(KCI_[A-Za-z_]+)=("[^"]*"|[^ ]+)' + +while read -r line; do + if [[ $line =~ $pattern ]]; then + variable_name="${BASH_REMATCH[1]}" + variable_value="${BASH_REMATCH[2]}" + + # Remove quotes if present + variable_value="${variable_value%\"}" + variable_value="${variable_value#\"}" + + # Export the variable + export "$variable_name=$variable_value" + + echo "Exported $variable_name=$variable_value" + fi +done <<< "$commit_message" diff --git a/ci/gitlab-ci/ci-scripts/run-checkpatch.sh b/ci/gitlab-ci/ci-scripts/run-checkpatch.sh new file mode 100755 index 0000000000000..98585ec18be2b --- /dev/null +++ b/ci/gitlab-ci/ci-scripts/run-checkpatch.sh @@ -0,0 +1,20 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +set -exo pipefail + +source ci/gitlab-ci/ci-scripts/ici-functions.sh + +ici_get_patch_series_size + +# shellcheck disable=SC2086 +scripts/checkpatch.pl $KCI_CHECKPATCH_OPTIONS --git HEAD-"$ICI_PATCH_SERIES_SIZE" + +if [ -n "$ICI_UNABLE_TO_DETECT_PATCH_SERIES_SIZE" ]; then + # If the patch series size was not detected, exit with a warning + echo -n "The patch series size was not detected, we probably didn't check the whole series." + echo " Exiting with a warning." + exit 101 +fi diff --git a/ci/gitlab-ci/ci-scripts/run-smatch.sh b/ci/gitlab-ci/ci-scripts/run-smatch.sh new file mode 100755 index 0000000000000..ec054df9ef179 --- /dev/null +++ b/ci/gitlab-ci/ci-scripts/run-smatch.sh @@ -0,0 +1,45 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +set -exo pipefail + +source ci/gitlab-ci/ci-scripts/ici-functions.sh + +ls -l +pwd + +# generate config file +ici_prepare_build + +ici_get_patch_series_size + +cp build/.config . + +# Get a list of modified .c files in the last ICI_PATCH_SERIES_SIZE commits +MODIFIED_C_FILES=$(git diff --name-only HEAD~$ICI_PATCH_SERIES_SIZE HEAD | grep '\.c$' || true) + +# Check if any .c files were modified +if [ -z "$MODIFIED_C_FILES" ]; then + echo "No .c files were modified in the last $ICI_PATCH_SERIES_SIZE commits." +else + echo "Running kchecker on modified .c files..." + mkdir -p "$CI_PROJECT_DIR"/artifacts +fi + +# Run kchecker on each modified .c file +for file in $MODIFIED_C_FILES; do + if [ -f "$file" ]; then + /smatch/smatch_scripts/kchecker "$file" | tee "$CI_PROJECT_DIR"/artifacts/smatch_warns.txt + else + echo "File not found: $file" + fi +done + +if [ -n "$ICI_UNABLE_TO_DETECT_PATCH_SERIES_SIZE" ]; then + # If the patch series size was not detected, exit with a warning + echo -n "The patch series size was not detected, we probably didn't check the whole series." + echo " Exiting with a warning." + exit 101 +fi diff --git a/ci/gitlab-ci/docker-compose.yaml b/ci/gitlab-ci/docker-compose.yaml new file mode 100644 index 0000000000000..7edf8055ca375 --- /dev/null +++ b/ci/gitlab-ci/docker-compose.yaml @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +version: '3.8' + +services: + gitlab-runner: + image: gitlab/gitlab-runner:latest + container_name: gitlab-runner + restart: always + privileged: true + volumes: + - /srv/gitlab-runner/config:/etc/gitlab-runner + - /var/run/docker.sock:/var/run/docker.sock + +# To register the GitLab Runner, run the following command: +# docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register --url https://gitlab.com --token YOUR_REGISTRATION_TOKEN diff --git a/ci/gitlab-ci/linux.code-workspace b/ci/gitlab-ci/linux.code-workspace new file mode 100644 index 0000000000000..dd76698e38c2c --- /dev/null +++ b/ci/gitlab-ci/linux.code-workspace @@ -0,0 +1,11 @@ +{ + "folders": [ + { + "path": "../../../.." + }, + { + "path": "../../../../../mesa" + } + ], + "settings": {} +} \ No newline at end of file diff --git a/ci/gitlab-ci/yml/build.yml b/ci/gitlab-ci/yml/build.yml new file mode 100644 index 0000000000000..f81410c293f35 --- /dev/null +++ b/ci/gitlab-ci/yml/build.yml @@ -0,0 +1,43 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +.build: + extends: .use-debian/x86_64_build + stage: build + +build-kernel: + extends: + - .build + - .kernel-combinations + variables: + # ccache in gitlab-runner to speed up builds + CCACHE_BASEDIR: $CI_PROJECT_DIR + CCACHE_DIR: $CI_PROJECT_DIR/ccache + CCACHE_COMPILERCHECK: content + cache: + - key: ccache-$CI_JOB_NAME + paths: + - $CCACHE_DIR + before_script: + - export PATH="/usr/lib/ccache:$PATH" + - ccache --zero-stats || true + - ccache --show-stats || true + after_script: + - ccache --show-stats + script: + - ./ci/gitlab-ci/ci-scripts/build-kernel.sh + +build-docs: + extends: + - .build + parallel: + matrix: + - ICI_DOC_TYPE: "htmldocs" + # TODO: re-add pdfdocs once build errors are fixed + # - ICI_DOC_TYPE: "pdfdocs" + - ICI_DOC_TYPE: "latexdocs" + - ICI_DOC_TYPE: "epubdocs" + script: + - ./ci/gitlab-ci/ci-scripts/build-docs.sh + when: manual \ No newline at end of file diff --git a/ci/gitlab-ci/yml/cache.yml b/ci/gitlab-ci/yml/cache.yml new file mode 100644 index 0000000000000..efe4fc69b11ca --- /dev/null +++ b/ci/gitlab-ci/yml/cache.yml @@ -0,0 +1,26 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +# Smatch db is saved to a cached folder, so it can be used by other jobs and pipelines. +# It is set to manual so it can be run when needed + +.use-cache-smatch-db: + cache: + # TODO: check if this cache shouldn't be per architecture + - key: smatch-db + paths: + - /smatch/smatch_data + +smatch-db-generate: + stage: cache + extends: + - .kernel-combinations + - .use-debian/x86_64_build + - .use-cache-smatch-db + script: + - source ci/gitlab-ci/ci-scripts/ici-functions.sh + - ici_prepare_build + - cp build/.config . + - /smatch/smatch_scripts/build_kernel_data.sh + when: manual \ No newline at end of file diff --git a/ci/gitlab-ci/yml/container.yml b/ci/gitlab-ci/yml/container.yml new file mode 100644 index 0000000000000..eecca95caca0e --- /dev/null +++ b/ci/gitlab-ci/yml/container.yml @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +.prep-debian/x86_64_build: + variables: + FDO_DISTRIBUTION_VERSION: bookworm-slim + FDO_DISTRIBUTION_TAG: "2024-02-28-ci-test-1" + + +debian/x86_64_build: + extends: + - ".fdo.container-build@debian" + - ".prep-debian/x86_64_build" + variables: + FDO_DISTRIBUTION_PACKAGES: > + make bc flex bison pahole mount build-essential + libcairo-dev libdw-dev libjson-c-dev libkmod2 + libkmod-dev libpciaccess-dev libproc2-dev libudev-dev + libunwind-dev python3-docutils bc python3-ply + libssl-dev ccache + gcc-aarch64-linux-gnu binutils-aarch64-linux-gnu + gcc-arm-linux-gnueabihf binutils-arm-linux-gnueabihf + perl git kmod python3-git python3-yaml python3-dulwich + sqlite3 libsqlite3-dev libdbd-sqlite3-perl libssl-dev libtry-tiny-perl + python3-sphinx imagemagick graphviz dvipng python3-venv fonts-noto-cjk + latexmk librsvg2-bin texlive-lang-chinese texlive-xetex + FDO_DISTRIBUTION_EXEC: ./ci/gitlab-ci/ci-scripts/install-smatch.sh + stage: container + + +.use-debian/x86_64_build: + extends: + - ".fdo.distribution-image@debian" + - ".prep-debian/x86_64_build" + needs: [debian/x86_64_build] \ No newline at end of file diff --git a/ci/gitlab-ci/yml/gitlab-ci.yml b/ci/gitlab-ci/yml/gitlab-ci.yml new file mode 100644 index 0000000000000..57b9c02904713 --- /dev/null +++ b/ci/gitlab-ci/yml/gitlab-ci.yml @@ -0,0 +1,71 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +workflow: + name: $PIPELINE_NAME + rules: + # when triggered as a multi-project pipeline for an MR + - if: $CI_PIPELINE_SOURCE == 'pipeline' && $PARENT_MERGE_REQUEST_IID != null && $PARENT_MERGE_REQUEST_IID != "" + variables: + PIPELINE_NAME: 'Downstream pipeline for $PARENT_PROJECT_PATH!$PARENT_MERGE_REQUEST_IID' + # when triggered as a multi-project pipeline + - if: $CI_PIPELINE_SOURCE == 'pipeline' + variables: + PIPELINE_NAME: 'Downstream pipeline for $PARENT_PROJECT_PATH' + # when triggered via a schedule + - if: $CI_PIPELINE_SOURCE == 'schedule' + variables: + PIPELINE_NAME: 'Scheduled pipeline for $ONLY_JOB_NAME' + # for merge requests + - if: $CI_MERGE_REQUEST_ID + # when triggered via the REST api + - if: $CI_PIPELINE_SOURCE == 'api' + # for the tip of the default branch + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + # when triggered via a trigger token + - if: $CI_PIPELINE_SOURCE == 'trigger' + # when triggered from a button press in the web interface + - if: $CI_PIPELINE_SOURCE == 'web' + # for branch tips without open MRs, ignoring special branches + - if: $CI_PIPELINE_SOURCE == 'push' && $CI_OPEN_MERGE_REQUESTS == null + # when forced via '-o ci.variable="FORCE_CI=true"' during pushing + - if: $FORCE_CI == 'true' + +variables: + FDO_UPSTREAM_REPO: helen.fornazier/linux # The repo where to look for cached images + # ccache builds in gitlab-runner to speed up builds + SMATCH_DB_DIR: /smatch/smatch_data + # exit code of bash script on `script` will be the exit code of the job + FF_USE_NEW_BASH_EVAL_STRATEGY: "true" + +default: + artifacts: + paths: + - artifacts/ + when: always + +include: + - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/16bc29078de5e0a067ff84a1a199a3760d3b3811/templates/ci-fairy.yml' + - remote: 'https://gitlab.freedesktop.org/freedesktop/ci-templates/-/raw/16bc29078de5e0a067ff84a1a199a3760d3b3811/templates/debian.yml' + + - ci/gitlab-ci/yml/kernel-combinations.yml + - ci/gitlab-ci/yml/container.yml + - ci/gitlab-ci/yml/cache.yml + - ci/gitlab-ci/yml/build.yml + - ci/gitlab-ci/yml/static-checks.yml + - ci/gitlab-ci/yml/scenarios.yml + +before_script: + - source ci/gitlab-ci/ci-scripts/parse_commit_message.sh + +.use-debian/x86_64_build: + allow_failure: + # Code to exit with a warning + exit_codes: 101 + +stages: + - container + - static-checks + - build + - cache \ No newline at end of file diff --git a/ci/gitlab-ci/yml/kernel-combinations.yml b/ci/gitlab-ci/yml/kernel-combinations.yml new file mode 100644 index 0000000000000..87f08579ffd62 --- /dev/null +++ b/ci/gitlab-ci/yml/kernel-combinations.yml @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +.kernel-combinations: + parallel: + matrix: + - KCI_KERNEL_ARCH: "x86_64" + KCI_DEFCONFIG: "x86_64_defconfig" + KCI_KCONFIGS_ENABLE: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT CONFIG_DEBUG_SECTION_MISMATCH" + + - KCI_KERNEL_ARCH: "arm64" + KCI_DEFCONFIG: "defconfig" + KCI_KCONFIGS_ENABLE: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT CONFIG_DEBUG_SECTION_MISMATCH" + + - KCI_KERNEL_ARCH: "arm" + KCI_DEFCONFIG: "multi_v7_defconfig" + KCI_KCONFIGS_ENABLE: "PROVE_LOCKING DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT CONFIG_DEBUG_SECTION_MISMATCH" \ No newline at end of file diff --git a/ci/gitlab-ci/yml/scenarios.yml b/ci/gitlab-ci/yml/scenarios.yml new file mode 100644 index 0000000000000..11598c3b6f2a9 --- /dev/null +++ b/ci/gitlab-ci/yml/scenarios.yml @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +# Extend the CI by including a test scenario here. The scenario will be +# activated if KCI_SCENARIO is set to the scenario name. KCI_SCENARIO can be +# defined in the gitlab-ci.yml file, or through GitLab UI. + +include: + - local: 'ci/gitlab-ci/yml/scenarios/$KCI_SCENARIO.yml' + rules: + - if: '$KCI_SCENARIO' \ No newline at end of file diff --git a/ci/gitlab-ci/yml/scenarios/file-systems.yml b/ci/gitlab-ci/yml/scenarios/file-systems.yml new file mode 100644 index 0000000000000..66456600a9dd3 --- /dev/null +++ b/ci/gitlab-ci/yml/scenarios/file-systems.yml @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +# Write here specific configurations and extensions for the given scenario + +# Example - overwrite kernel combinations in the pipeline +# .kernel-combinations: +# parallel: +# matrix: +# - KCI_KERNEL_ARCH: "x86_64" +# KCI_DEFCONFIG: "path/to/my/config/file_config" +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." +# +# - KCI_KERNEL_ARCH: "arm64" +# KCI_DEFCONFIG: "path/to/my/config/file_config" +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." +# +# - KCI_KERNEL_ARCH: "arm" +# KCI_DEFCONFIG: "path/to/my/config/file_config" +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." \ No newline at end of file diff --git a/ci/gitlab-ci/yml/scenarios/media.yml b/ci/gitlab-ci/yml/scenarios/media.yml new file mode 100644 index 0000000000000..66456600a9dd3 --- /dev/null +++ b/ci/gitlab-ci/yml/scenarios/media.yml @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +# Write here specific configurations and extensions for the given scenario + +# Example - overwrite kernel combinations in the pipeline +# .kernel-combinations: +# parallel: +# matrix: +# - KCI_KERNEL_ARCH: "x86_64" +# KCI_DEFCONFIG: "path/to/my/config/file_config" +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." +# +# - KCI_KERNEL_ARCH: "arm64" +# KCI_DEFCONFIG: "path/to/my/config/file_config" +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." +# +# - KCI_KERNEL_ARCH: "arm" +# KCI_DEFCONFIG: "path/to/my/config/file_config" +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." \ No newline at end of file diff --git a/ci/gitlab-ci/yml/scenarios/network.yml b/ci/gitlab-ci/yml/scenarios/network.yml new file mode 100644 index 0000000000000..66456600a9dd3 --- /dev/null +++ b/ci/gitlab-ci/yml/scenarios/network.yml @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +# Write here specific configurations and extensions for the given scenario + +# Example - overwrite kernel combinations in the pipeline +# .kernel-combinations: +# parallel: +# matrix: +# - KCI_KERNEL_ARCH: "x86_64" +# KCI_DEFCONFIG: "path/to/my/config/file_config" +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." +# +# - KCI_KERNEL_ARCH: "arm64" +# KCI_DEFCONFIG: "path/to/my/config/file_config" +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." +# +# - KCI_KERNEL_ARCH: "arm" +# KCI_DEFCONFIG: "path/to/my/config/file_config" +# KCI_ENABLE_KCONFIGS: "CONFIG1 CONFIG2 CONFIG3 ..." \ No newline at end of file diff --git a/ci/gitlab-ci/yml/static-checks.yml b/ci/gitlab-ci/yml/static-checks.yml new file mode 100644 index 0000000000000..a99a1ea910f40 --- /dev/null +++ b/ci/gitlab-ci/yml/static-checks.yml @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2024 Collabora, Helen Koike <helen.koike@collabora.com> + +.static-checks: + stage: static-checks + extends: + - .use-debian/x86_64_build + +checkpatch: + extends: .static-checks + script: + - ci/gitlab-ci/ci-scripts/run-checkpatch.sh + +smatch: + extends: + - .static-checks + - .kernel-combinations + - .use-cache-smatch-db + script: + - ci/gitlab-ci/ci-scripts/run-smatch.sh \ No newline at end of file