| Message ID | 20250310183208.999430-4-cel@kernel.org (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | Fixes for git mirroring on Fedora | expand |
On Mon, Mar 10, 2025 at 02:32:07PM -0400, cel@kernel.org wrote: > From: Chuck Lever <chuck.lever@oracle.com> > > Each run takes longer than 10 seconds, which generates a "git > process is still running" warning in the system journal every > minute. > > Signed-off-by: Chuck Lever <chuck.lever@oracle.com> > --- > playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 > index dabbadcc6528..45e79349a3d4 100644 > --- a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 > +++ b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 > @@ -36,7 +36,7 @@ mirrors: > - short_name: "kdevops-results-archive" > url: "{{ defaults_kdevops_results_archive_git }}" > target: "kdevops-results-archive.git" > - refresh: "10s" > + refresh: "10m" We don't poke kernel.org here ever, this is a github tree. The question here is how often we want this updated to make CI updates take effect. It is why I went so low here. I think such a low value only makes sense for mirrors which won't need to leverage leverage kdevops-results-archive to push updates to that tree, as otherwise their tree will be out of sync. So, I think a bool to lower this default might be nice, disabled by default, and a new defconfig for mirror-ci might be useful which has that bool enabled. This way folks who wish to help with CI infra and use kdevops-results-archive to push results there, can just have their mirror setup use: make defconfig-mirror-ci Without this we risk that a CI run, once it has test results to upload, won't be able to push the results. Because it will git fetch prior, but its using the mirror. Luis
On 3/10/25 11:39 PM, Luis Chamberlain wrote: > On Mon, Mar 10, 2025 at 02:32:07PM -0400, cel@kernel.org wrote: >> From: Chuck Lever <chuck.lever@oracle.com> >> >> Each run takes longer than 10 seconds, which generates a "git >> process is still running" warning in the system journal every >> minute. >> >> Signed-off-by: Chuck Lever <chuck.lever@oracle.com> >> --- >> playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >> diff --git a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 >> index dabbadcc6528..45e79349a3d4 100644 >> --- a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 >> +++ b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 >> @@ -36,7 +36,7 @@ mirrors: >> - short_name: "kdevops-results-archive" >> url: "{{ defaults_kdevops_results_archive_git }}" >> target: "kdevops-results-archive.git" >> - refresh: "10s" >> + refresh: "10m" > > We don't poke kernel.org here ever, this is a github tree. Hi Luis, thanks for the review. The issue I see is local, on the mirror. This one's not an issue about unnecessary workload on the remote repo. Basically the poll for new commits takes a while on my system, so every minute I get a verbose system journal warning that there is a git process left over from the last timer run. The log messages are noise, and because they come out so often, they will fill up the controller's root filesystem. Also, from a code legibility standpoint, "10s" looks like an anomaly, a typo. IMO it could be clearer if it were documented (with what you said here, below) in a brief comment. Maybe it is already and I missed it. On my system, systemd timer jobs don't seem to run more often than once a minute anyway... 10s might be impossibly short. I'm open to finding another way to address this, and I am OK not pushing this one until we agree on a consensus solution. > The question here is how often we want this updated to make CI updates > take effect. It is why I went so low here. I think such a low value only > makes sense for mirrors which won't need to leverage leverage > kdevops-results-archive to push updates to that tree, as otherwise their > tree will be out of sync. So, I think a bool to lower this default might > be nice, disabled by default, and a new defconfig for mirror-ci might be > useful which has that bool enabled. This way folks who wish to help with > CI infra and use kdevops-results-archive to push results there, can just > have their mirror setup use: > > make defconfig-mirror-ci > > Without this we risk that a CI run, once it has test results to upload, > won't be able to push the results. Because it will git fetch prior, but > its using the mirror. Perhaps another choice would be to not mirror this repo at all, since it is also the target of local updates.
On 3/11/25 9:38 AM, Chuck Lever wrote: > On 3/10/25 11:39 PM, Luis Chamberlain wrote: >> On Mon, Mar 10, 2025 at 02:32:07PM -0400, cel@kernel.org wrote: >>> From: Chuck Lever <chuck.lever@oracle.com> >>> >>> Each run takes longer than 10 seconds, which generates a "git >>> process is still running" warning in the system journal every >>> minute. >>> >>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com> >>> --- >>> playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 | 2 +- >>> 1 file changed, 1 insertion(+), 1 deletion(-) >>> >>> diff --git a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 >>> index dabbadcc6528..45e79349a3d4 100644 >>> --- a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 >>> +++ b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 >>> @@ -36,7 +36,7 @@ mirrors: >>> - short_name: "kdevops-results-archive" >>> url: "{{ defaults_kdevops_results_archive_git }}" >>> target: "kdevops-results-archive.git" >>> - refresh: "10s" >>> + refresh: "10m" >> >> We don't poke kernel.org here ever, this is a github tree. > > Hi Luis, thanks for the review. > > The issue I see is local, on the mirror. This one's not an issue about > unnecessary workload on the remote repo. > > Basically the poll for new commits takes a while on my system, so every > minute I get a verbose system journal warning that there is a git > process left over from the last timer run. The log messages are noise, > and because they come out so often, they will fill up the controller's > root filesystem. In fact I see this for most of the mirrored repos, even on a kdevops controller system I set up a while ago: qemu-jic23-mirror.service: Found left-over process 2176 (git) in control group while starting unit. Ignoring. qemu-jic23-mirror.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies. qemu-mirror.service: Found left-over process 2180 (git) in control group while starting unit. Ignoring. qemu-mirror.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies. I'm not sure what to make of it. > Also, from a code legibility standpoint, "10s" looks like an anomaly, > a typo. IMO it could be clearer if it were documented (with what you > said here, below) in a brief comment. Maybe it is already and I missed > it. > > On my system, systemd timer jobs don't seem to run more often than once > a minute anyway... 10s might be impossibly short. > > I'm open to finding another way to address this, and I am OK not > pushing this one until we agree on a consensus solution. > > >> The question here is how often we want this updated to make CI updates >> take effect. It is why I went so low here. I think such a low value only >> makes sense for mirrors which won't need to leverage leverage >> kdevops-results-archive to push updates to that tree, as otherwise their >> tree will be out of sync. So, I think a bool to lower this default might >> be nice, disabled by default, and a new defconfig for mirror-ci might be >> useful which has that bool enabled. This way folks who wish to help with >> CI infra and use kdevops-results-archive to push results there, can just >> have their mirror setup use: >> >> make defconfig-mirror-ci >> >> Without this we risk that a CI run, once it has test results to upload, >> won't be able to push the results. Because it will git fetch prior, but >> its using the mirror. > > Perhaps another choice would be to not mirror this repo at all, since it > is also the target of local updates. > >
On Tue, Mar 11, 2025 at 09:38:40AM -0400, Chuck Lever wrote: > Perhaps another choice would be to not mirror this repo at all, since it > is also the target of local updates. I think that's a good idea, and just enable it for a new mirror-ci defconfig, ie, only silly CI setups should mirror everything. I hadn't added variability to mirroring, so adding some with sensible developer defaults makes sense, while pushing all the odd ball trees to the mirror-ci. Luis
On Tue, Mar 11, 2025 at 02:01:00PM -0400, Chuck Lever wrote: > On 3/11/25 9:38 AM, Chuck Lever wrote: > > On 3/10/25 11:39 PM, Luis Chamberlain wrote: > >> On Mon, Mar 10, 2025 at 02:32:07PM -0400, cel@kernel.org wrote: > >>> From: Chuck Lever <chuck.lever@oracle.com> > >>> > >>> Each run takes longer than 10 seconds, which generates a "git > >>> process is still running" warning in the system journal every > >>> minute. > >>> > >>> Signed-off-by: Chuck Lever <chuck.lever@oracle.com> > >>> --- > >>> playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 | 2 +- > >>> 1 file changed, 1 insertion(+), 1 deletion(-) > >>> > >>> diff --git a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 > >>> index dabbadcc6528..45e79349a3d4 100644 > >>> --- a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 > >>> +++ b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 > >>> @@ -36,7 +36,7 @@ mirrors: > >>> - short_name: "kdevops-results-archive" > >>> url: "{{ defaults_kdevops_results_archive_git }}" > >>> target: "kdevops-results-archive.git" > >>> - refresh: "10s" > >>> + refresh: "10m" > >> > >> We don't poke kernel.org here ever, this is a github tree. > > > > Hi Luis, thanks for the review. > > > > The issue I see is local, on the mirror. This one's not an issue about > > unnecessary workload on the remote repo. > > > > Basically the poll for new commits takes a while on my system, so every > > minute I get a verbose system journal warning that there is a git > > process left over from the last timer run. The log messages are noise, > > and because they come out so often, they will fill up the controller's > > root filesystem. > > In fact I see this for most of the mirrored repos, even on a kdevops > controller system I set up a while ago: > > qemu-jic23-mirror.service: Found left-over process 2176 (git) in control > group while starting unit. Ignoring. > qemu-jic23-mirror.service: This usually indicates unclean termination of > a previous run, or service implementation deficiencies. > qemu-mirror.service: Found left-over process 2180 (git) in control group > while starting unit. Ignoring. > qemu-mirror.service: This usually indicates unclean termination of a > previous run, or service implementation deficiencies. > > I'm not sure what to make of it. Just feel free to toss it to mirror-ci defconfig, it just means adding a bit of variability. Luis
diff --git a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 index dabbadcc6528..45e79349a3d4 100644 --- a/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 +++ b/playbooks/roles/linux-mirror/templates/mirrors.yaml.j2 @@ -36,7 +36,7 @@ mirrors: - short_name: "kdevops-results-archive" url: "{{ defaults_kdevops_results_archive_git }}" target: "kdevops-results-archive.git" - refresh: "10s" + refresh: "10m" - short_name: "kdevops/fstests" url: "{{ mirror_kdevops_fstests_url }}" target: "kdevops/fstests.git"