| Message ID | 20230302175332.56052-1-anthony.perard@citrix.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | automation, RFC prototype, Have GitLab CI built its own containers | expand |
On Thu, 2 Mar 2023, Anthony PERARD wrote: > Patch series available in this git branch: > https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-auto-rebuild-v1 > > Hi, > > I have done some research to be able to build containers in the CI. This works > only for x86 containers as I've setup only a single x86 gitlab-runner to be > able to run docker-in-docker. > > The runner is setup to only accept jobs from a branch that is "protected" in > gitlab. Also, one need credential to push to the container register, those are > called "Group deploy tokens", and I've set the variables CI_DEPLOY_USER and > CI_DEPLOY_PASSWORD in the project "xen-project/xen" (variables only visible on > a pipeline running on a protected branch). > > These patch introduce quite a lot of redundancies in jobs, 2 new jobs per > containers which build/push containers, and duplicate most of build.yaml. > This mean that if we go with this, we have to duplicate and keep in sync many > jobs. > > To avoid having to do the duplicated jobs by hand, I could look at > creating a script that use "build.yaml" as input and produce the 3 > stages needed to update a container, but that script would need to be > run by hand, as gitlab can't really use it, unless .. > > I've look at generated pipeline, and they look like this in gitlab: > https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/777665738 > But the result of the generated/child pipeline doesn't seems to be taken into > account in the original pipeline, which make me think that we can't use them to > generate "build.yaml". But maybe the could be use for generating the pipeline > that will update a container. > Doc: > https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines > > So, with all of this, is it reasonable to test containers before > pushing them to production? Or is it to much work? We could simply have jobs > tacking care of rebuilding a container and pushing them to production without > testing. I don't think it is a good idea to duplicate build.yaml, also because some of the containers are used in the testing stage too, so an updated container could be OK during the build phase and break the testing phase. We would need to duplicate both build.yaml and test.yaml, which is not feasible. In practice today people either: 1) re-build a container locally & test it locally before pushing 2) re-build a container locally, docker push it, then run a private gitlab pipeline, if it passes send out a patch to xen-devel 1) is not affected by this series 2) is also not affected because by the time the pipeline is created, the container is already updated However, there are cases where it would definitely be nice to have a "button" to press to update a container. For instance, when a pipeline failis due to a Debian unstable apt-get failure, which can be easily fixed by updating the Debian unstable container. So I think it would be nice to have jobs that can automatically update the build containers but I would set them to manually trigger instead of automatically (when: manual). Alternatively, we could move the "containers.yaml" stage to be the first step, rebuild the containers and push them to a "staging" area (registry.gitlab.com/xen-project/xen/staging), run the build and test steps fetching from the staging area instead of the regular, if all tests pass, then push the containers to registry.gitlab.com/xen-project/xen as last step. > An example with the variable DO_REBUILD_CONTAINER and PUSH_CONTAINER set (and > existing build/test jobs disabled): > https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/791711467 > > Cheers, > > Anthony PERARD (7): > automation: Automatically rebuild debian:unstable container > automation: Introduce test-containers stage > automation: Add a template per container for build jobs. > automation: Adding containers build jobs and test of thoses > automation: Introduce DO_REBUILD_CONTAINER, to allow to rebuild a > container > automation: Push container been tested > automation: Add some more push containers jobs > > .gitlab-ci.yml | 6 + > automation/build/Makefile | 14 +- > automation/gitlab-ci/build.yaml | 327 ++++++++------ > automation/gitlab-ci/containers.yaml | 98 +++++ > automation/gitlab-ci/push-containers.yaml | 79 ++++ > automation/gitlab-ci/test-containers.yaml | 496 ++++++++++++++++++++++ > 6 files changed, 894 insertions(+), 126 deletions(-) > create mode 100644 automation/gitlab-ci/containers.yaml > create mode 100644 automation/gitlab-ci/push-containers.yaml > create mode 100644 automation/gitlab-ci/test-containers.yaml > > -- > Anthony PERARD >
On Thu, Mar 02, 2023 at 06:48:35PM -0800, Stefano Stabellini wrote: > On Thu, 2 Mar 2023, Anthony PERARD wrote: > > Patch series available in this git branch: > > https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-auto-rebuild-v1 > > > > Hi, > > > > I have done some research to be able to build containers in the CI. This works > > only for x86 containers as I've setup only a single x86 gitlab-runner to be > > able to run docker-in-docker. > > > > The runner is setup to only accept jobs from a branch that is "protected" in > > gitlab. Also, one need credential to push to the container register, those are > > called "Group deploy tokens", and I've set the variables CI_DEPLOY_USER and > > CI_DEPLOY_PASSWORD in the project "xen-project/xen" (variables only visible on > > a pipeline running on a protected branch). > > > > These patch introduce quite a lot of redundancies in jobs, 2 new jobs per > > containers which build/push containers, and duplicate most of build.yaml. > > This mean that if we go with this, we have to duplicate and keep in sync many > > jobs. > > > > To avoid having to do the duplicated jobs by hand, I could look at > > creating a script that use "build.yaml" as input and produce the 3 > > stages needed to update a container, but that script would need to be > > run by hand, as gitlab can't really use it, unless .. > > > > I've look at generated pipeline, and they look like this in gitlab: > > https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/777665738 > > But the result of the generated/child pipeline doesn't seems to be taken into > > account in the original pipeline, which make me think that we can't use them to > > generate "build.yaml". But maybe the could be use for generating the pipeline > > that will update a container. > > Doc: > > https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines > > > > So, with all of this, is it reasonable to test containers before > > pushing them to production? Or is it to much work? We could simply have jobs > > tacking care of rebuilding a container and pushing them to production without > > testing. > > I don't think it is a good idea to duplicate build.yaml, also because > some of the containers are used in the testing stage too, so an updated > container could be OK during the build phase and break the testing > phase. We would need to duplicate both build.yaml and test.yaml, which > is not feasible. > > In practice today people either: > 1) re-build a container locally & test it locally before pushing > 2) re-build a container locally, docker push it, then run a private > gitlab pipeline, if it passes send out a patch to xen-devel > > 1) is not affected by this series > 2) is also not affected because by the time the pipeline is created, the > container is already updated > > However, there are cases where it would definitely be nice to have a > "button" to press to update a container. For instance, when a pipeline > failis due to a Debian unstable apt-get failure, which can be easily fixed > by updating the Debian unstable container. > > So I think it would be nice to have jobs that can automatically update > the build containers but I would set them to manually trigger instead of > automatically (when: manual). What I was looking at with this work was to be able to have container been rebuild automatically on a schedule. We have/had containers that were 3yr old, and when it's a container that supposed to test the lasted version of a distro, or a rolling release distro, they are kind of useless if they aren't rebuild regularly. So I was looking to take the human out of the loop and have computers the tedious work of rebuilding a container every month or two. Containers that needs to be rebuilt regularly to stay relevant are archlinux, debian/unstable, fedora/latest, propably opensuse/leap and opensuse/tumbleweed. I don't know if they are others. > Alternatively, we could move the "containers.yaml" stage to be the first > step, rebuild the containers and push them to a "staging" area > (registry.gitlab.com/xen-project/xen/staging), run the build and test > steps fetching from the staging area instead of the regular, if all > tests pass, then push the containers to > registry.gitlab.com/xen-project/xen as last step. Sounds good, I can look into how easily it would be to use a different registry to run a pipeline. Cheers,
On Fri, 3 Mar 2023, Anthony PERARD wrote: > On Thu, Mar 02, 2023 at 06:48:35PM -0800, Stefano Stabellini wrote: > > On Thu, 2 Mar 2023, Anthony PERARD wrote: > > > Patch series available in this git branch: > > > https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-auto-rebuild-v1 > > > > > > Hi, > > > > > > I have done some research to be able to build containers in the CI. This works > > > only for x86 containers as I've setup only a single x86 gitlab-runner to be > > > able to run docker-in-docker. > > > > > > The runner is setup to only accept jobs from a branch that is "protected" in > > > gitlab. Also, one need credential to push to the container register, those are > > > called "Group deploy tokens", and I've set the variables CI_DEPLOY_USER and > > > CI_DEPLOY_PASSWORD in the project "xen-project/xen" (variables only visible on > > > a pipeline running on a protected branch). > > > > > > These patch introduce quite a lot of redundancies in jobs, 2 new jobs per > > > containers which build/push containers, and duplicate most of build.yaml. > > > This mean that if we go with this, we have to duplicate and keep in sync many > > > jobs. > > > > > > To avoid having to do the duplicated jobs by hand, I could look at > > > creating a script that use "build.yaml" as input and produce the 3 > > > stages needed to update a container, but that script would need to be > > > run by hand, as gitlab can't really use it, unless .. > > > > > > I've look at generated pipeline, and they look like this in gitlab: > > > https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/777665738 > > > But the result of the generated/child pipeline doesn't seems to be taken into > > > account in the original pipeline, which make me think that we can't use them to > > > generate "build.yaml". But maybe the could be use for generating the pipeline > > > that will update a container. > > > Doc: > > > https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines > > > > > > So, with all of this, is it reasonable to test containers before > > > pushing them to production? Or is it to much work? We could simply have jobs > > > tacking care of rebuilding a container and pushing them to production without > > > testing. > > > > I don't think it is a good idea to duplicate build.yaml, also because > > some of the containers are used in the testing stage too, so an updated > > container could be OK during the build phase and break the testing > > phase. We would need to duplicate both build.yaml and test.yaml, which > > is not feasible. > > > > In practice today people either: > > 1) re-build a container locally & test it locally before pushing > > 2) re-build a container locally, docker push it, then run a private > > gitlab pipeline, if it passes send out a patch to xen-devel > > > > 1) is not affected by this series > > 2) is also not affected because by the time the pipeline is created, the > > container is already updated > > > > However, there are cases where it would definitely be nice to have a > > "button" to press to update a container. For instance, when a pipeline > > failis due to a Debian unstable apt-get failure, which can be easily fixed > > by updating the Debian unstable container. > > > > So I think it would be nice to have jobs that can automatically update > > the build containers but I would set them to manually trigger instead of > > automatically (when: manual). > > What I was looking at with this work was to be able to have container > been rebuild automatically on a schedule. We have/had containers that > were 3yr old, and when it's a container that supposed to test the lasted > version of a distro, or a rolling release distro, they are kind of > useless if they aren't rebuild regularly. So I was looking to take the > human out of the loop and have computers the tedious work of rebuilding a > container every month or two. > > Containers that needs to be rebuilt regularly to stay relevant are > archlinux, debian/unstable, fedora/latest, propably opensuse/leap and > opensuse/tumbleweed. I don't know if they are others. OK. This use-case is very different from 1) and 2) below. In this case the need to rebuild the container does not come from a change in the Dockerfile made by a human (e.g. a contributor adding a package to apt-get) but it comes from the fact that the underlying distro is updated continuously so the containers should also be updated continuously. For this use-case specifically, I think it would be better to trigger the jobs to do the containers update based on schedule, rather than a change to the Dockerfile? And also, given that there are supposed to be no breaking changes, it would be more acceptable to do the containers update without testing it. But it would still be nicer to run the tests as suggested below of course. > > Alternatively, we could move the "containers.yaml" stage to be the first > > step, rebuild the containers and push them to a "staging" area > > (registry.gitlab.com/xen-project/xen/staging), run the build and test > > steps fetching from the staging area instead of the regular, if all > > tests pass, then push the containers to > > registry.gitlab.com/xen-project/xen as last step. > > Sounds good, I can look into how easily it would be to use a different > registry to run a pipeline. Thanks!
Patch series available in this git branch: https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.gitlab-containers-auto-rebuild-v1 Hi, I have done some research to be able to build containers in the CI. This works only for x86 containers as I've setup only a single x86 gitlab-runner to be able to run docker-in-docker. The runner is setup to only accept jobs from a branch that is "protected" in gitlab. Also, one need credential to push to the container register, those are called "Group deploy tokens", and I've set the variables CI_DEPLOY_USER and CI_DEPLOY_PASSWORD in the project "xen-project/xen" (variables only visible on a pipeline running on a protected branch). These patch introduce quite a lot of redundancies in jobs, 2 new jobs per containers which build/push containers, and duplicate most of build.yaml. This mean that if we go with this, we have to duplicate and keep in sync many jobs. To avoid having to do the duplicated jobs by hand, I could look at creating a script that use "build.yaml" as input and produce the 3 stages needed to update a container, but that script would need to be run by hand, as gitlab can't really use it, unless .. I've look at generated pipeline, and they look like this in gitlab: https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/777665738 But the result of the generated/child pipeline doesn't seems to be taken into account in the original pipeline, which make me think that we can't use them to generate "build.yaml". But maybe the could be use for generating the pipeline that will update a container. Doc: https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#dynamic-child-pipelines So, with all of this, is it reasonable to test containers before pushing them to production? Or is it to much work? We could simply have jobs tacking care of rebuilding a container and pushing them to production without testing. An example with the variable DO_REBUILD_CONTAINER and PUSH_CONTAINER set (and existing build/test jobs disabled): https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/791711467 Cheers, Anthony PERARD (7): automation: Automatically rebuild debian:unstable container automation: Introduce test-containers stage automation: Add a template per container for build jobs. automation: Adding containers build jobs and test of thoses automation: Introduce DO_REBUILD_CONTAINER, to allow to rebuild a container automation: Push container been tested automation: Add some more push containers jobs .gitlab-ci.yml | 6 + automation/build/Makefile | 14 +- automation/gitlab-ci/build.yaml | 327 ++++++++------ automation/gitlab-ci/containers.yaml | 98 +++++ automation/gitlab-ci/push-containers.yaml | 79 ++++ automation/gitlab-ci/test-containers.yaml | 496 ++++++++++++++++++++++ 6 files changed, 894 insertions(+), 126 deletions(-) create mode 100644 automation/gitlab-ci/containers.yaml create mode 100644 automation/gitlab-ci/push-containers.yaml create mode 100644 automation/gitlab-ci/test-containers.yaml