| Message ID | cover.1729183051.git.victorm.lira@amd.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | automation: add x86_64 test (linux argo) | expand |
On 17/10/2024 6:18 pm, victorm.lira@amd.com wrote: > From: Victor Lira <victorm.lira@amd.com> > > Add x86_64 hardware test that creates a Xen Argo communication > connection between two PVH domains. > > To accomplish this, add new build artifacts for Linux and Argo, and > update the xilinx x86_64 test script. > > Victor Lira (3): > automation: add linux 6.6.56 artifact > automation: add linux argo test artifacts > automation: add x86_64 test (linux argo) > > automation/gitlab-ci/build.yaml | 34 +++++++ > automation/gitlab-ci/test.yaml | 10 ++ > .../scripts/xilinx-smoke-dom0-x86_64.sh | 76 ++++++++++----- > .../tests-artifacts/argo/6.6.56.dockerfile | 95 +++++++++++++++++++ > .../tests-artifacts/kernel/6.6.56.dockerfile | 54 +++++++++++ > 5 files changed, 244 insertions(+), 25 deletions(-) > create mode 100644 automation/tests-artifacts/argo/6.6.56.dockerfile > create mode 100644 automation/tests-artifacts/kernel/6.6.56.dockerfile I'm afraid that we need to stop doing this test-artefacts like this. The *-export pattern is nonsense (wasting lots of runner network&disk and time just to copy data back into Github's artefact store), and the main reason why we're at ~2T of data total for Xen-project. We need a project wide expires_in: setting, which will prune all of our not-most-recent content, and probably make most of that 2T disappear. But, the test jobs stating what artefacts they want are perfectly capable of pulling artefacts from somewhere other than an earlier nop build job, meaning that we could get away with having one artefact shared by everything, not one artefact copied per user per pipeline into the artefact store. I was thinking of experimenting with a separate top-level repo that does nothing but has a few manual runs to populate artefacts, and having the Xen tests pull artefacts from here rather than from earlier build jobs. But, I've not had a chance to play yet, so I don't know for sure if this is a workable direction. Alternatively, if anyone has a better idea, I'm all ears. But, it is very clear that the *-export pattern is a giant bodge and there are better ways. Other notes. Your dockerfiles use syntax:1 but not heredocs, and they're also root containers. Please follow the pattern in https://lore.kernel.org/xen-devel/178560672106e37648304f5e597cc0b16c8db6db.1729170005.git.javi.merino@cloud.com/T/#u or one of plenty other containers I've already converted in the past few months. Please do not add a new build job just to get a new minor variation of Xen. Just enable ARGO in general Xen build. Although, given that you are now putting in testing from an unsupported feature, we really need to bump the suggestion to move it to at least Tech Preview. (CC Dan/Chris). ~Andrew
On Thu, 17 Oct 2024, Andrew Cooper wrote: > On 17/10/2024 6:18 pm, victorm.lira@amd.com wrote: > > From: Victor Lira <victorm.lira@amd.com> > > > > Add x86_64 hardware test that creates a Xen Argo communication > > connection between two PVH domains. > > > > To accomplish this, add new build artifacts for Linux and Argo, and > > update the xilinx x86_64 test script. > > > > Victor Lira (3): > > automation: add linux 6.6.56 artifact > > automation: add linux argo test artifacts > > automation: add x86_64 test (linux argo) > > > > automation/gitlab-ci/build.yaml | 34 +++++++ > > automation/gitlab-ci/test.yaml | 10 ++ > > .../scripts/xilinx-smoke-dom0-x86_64.sh | 76 ++++++++++----- > > .../tests-artifacts/argo/6.6.56.dockerfile | 95 +++++++++++++++++++ > > .../tests-artifacts/kernel/6.6.56.dockerfile | 54 +++++++++++ > > 5 files changed, 244 insertions(+), 25 deletions(-) > > create mode 100644 automation/tests-artifacts/argo/6.6.56.dockerfile > > create mode 100644 automation/tests-artifacts/kernel/6.6.56.dockerfile > > I'm afraid that we need to stop doing this test-artefacts like this. > > The *-export pattern is nonsense (wasting lots of runner network&disk > and time just to copy data back into Github's artefact store), and the > main reason why we're at ~2T of data total for Xen-project. > > We need a project wide expires_in: setting, which will prune all of our > not-most-recent content, and probably make most of that 2T disappear. I looked at this for an hour and couldn't find a way to do it unfortunately. expires_in can only be changed if you are an Admin of the Gitlab instance, meaning an Admin of gitlab.com. However, there is a setting in each repository (not one global at the gitlab.com/xen-project level) called "Keep artifacts from most recent successful jobs" that can be disabled. I did it manually for my repo. The big offender is patchew. I went and disabled "Keep artifacts from most recent successful jobs" but I didn't see any immediate changes in storage utilization. But it is expected to take time to propagate. > But, the test jobs stating what artefacts they want are perfectly > capable of pulling artefacts from somewhere other than an earlier nop > build job, meaning that we could get away with having one artefact > shared by everything, not one artefact copied per user per pipeline into > the artefact store. > > > I was thinking of experimenting with a separate top-level repo that does > nothing but has a few manual runs to populate artefacts, and having the > Xen tests pull artefacts from here rather than from earlier build jobs. > > But, I've not had a chance to play yet, so I don't know for sure if this > is a workable direction. > > Alternatively, if anyone has a better idea, I'm all ears. But, it is > very clear that the *-export pattern is a giant bodge and there are > better ways. > > Other notes. Your dockerfiles use syntax:1 but not heredocs, and > they're also root containers. > > Please follow the pattern in > https://lore.kernel.org/xen-devel/178560672106e37648304f5e597cc0b16c8db6db.1729170005.git.javi.merino@cloud.com/T/#u > or one of plenty other containers I've already converted in the past few > months. > > Please do not add a new build job just to get a new minor variation of > Xen. Just enable ARGO in general Xen build. +1
On Thu, Oct 17, 2024 at 08:25:57PM +0100, Andrew Cooper wrote: > I was thinking of experimenting with a separate top-level repo that does > nothing but has a few manual runs to populate artefacts, and having the > Xen tests pull artefacts from here rather than from earlier build jobs. This sounds like a good idea. It may also help with rebuilding containers when needed?
From: Victor Lira <victorm.lira@amd.com> Add x86_64 hardware test that creates a Xen Argo communication connection between two PVH domains. To accomplish this, add new build artifacts for Linux and Argo, and update the xilinx x86_64 test script. Victor Lira (3): automation: add linux 6.6.56 artifact automation: add linux argo test artifacts automation: add x86_64 test (linux argo) automation/gitlab-ci/build.yaml | 34 +++++++ automation/gitlab-ci/test.yaml | 10 ++ .../scripts/xilinx-smoke-dom0-x86_64.sh | 76 ++++++++++----- .../tests-artifacts/argo/6.6.56.dockerfile | 95 +++++++++++++++++++ .../tests-artifacts/kernel/6.6.56.dockerfile | 54 +++++++++++ 5 files changed, 244 insertions(+), 25 deletions(-) create mode 100644 automation/tests-artifacts/argo/6.6.56.dockerfile create mode 100644 automation/tests-artifacts/kernel/6.6.56.dockerfile -- 2.25.1