| Message ID | 20190827003709.26950-1-skhan@linuxfoundation.org (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | [v2] doc: kselftest: update for clarity on running kselftests in CI rings | expand |
On 8/26/19 6:37 PM, Shuah Khan wrote: > Update to add clarity and recommendations on running newer kselftests > on older kernels vs. matching the kernel and kselftest revisions. > > The recommendation is "Match kernel revision and kselftest." > > Signed-off-by: Shuah Khan <skhan@linuxfoundation.org> > --- > Changes since v1: Fixed "WARNING: Title underline too short." I have a few more changes and would like to make and send a v3 after the LPC's Testing and Fuzzing kselftest discussion. Holding off on this patch for now. thanks, -- Shuah
On Mon, Aug 26, 2019 at 5:37 PM Shuah Khan <skhan@linuxfoundation.org> wrote: > > Update to add clarity and recommendations on running newer kselftests > on older kernels vs. matching the kernel and kselftest revisions. > > The recommendation is "Match kernel revision and kselftest." > > Signed-off-by: Shuah Khan <skhan@linuxfoundation.org> Reviewed-by: Brendan Higgins <brendanhiggins@google.com> One minor comment below. > --- > Changes since v1: Fixed "WARNING: Title underline too short." > > Documentation/dev-tools/kselftest.rst | 25 +++++++++++++++++++++++++ > 1 file changed, 25 insertions(+) > > diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst > index 25604904fa6e..308506c5e8fa 100644 > --- a/Documentation/dev-tools/kselftest.rst > +++ b/Documentation/dev-tools/kselftest.rst > @@ -12,6 +12,31 @@ write new tests using the framework on Kselftest wiki: > > https://kselftest.wiki.kernel.org/ > > +Recommendations on running kselftests in Continuous Integration test rings > +========================================================================== > + > +It is recommended that users run Kselftest from the same release. Running > +newer Kselftest on older kernels isn't recommended for the following > +reasons: > + > +- Kselftest from mainline and linux-next might not be stable enough to run > + on stable kernels. > +- Kselftests detect feature dependencies at run-time and skip tests if a > + feature and/or configuration they test aren't enabled. Running newer > + tests on older kernels could result in a few too many skipped/failed > + conditions. It becomes difficult to evaluate the results. > +- Newer tests provide better coverage. However, users should make a judgement > + call on coverage vs. run to run consistency and being able to compare > + run to run results on older kernels. > + > +Recommendations: I think this should be the "note" directive. You already have a list of recommendations above; this seems like a TLDR section, or an important note or tip. > + > +Match kernel revision and kselftest. Especially important for LTS and > +Stable kernel Continuous Integration test rings. > + > +Hot-plug tests > +============== > + > On some systems, hot-plug tests could hang forever waiting for cpu and > memory to be ready to be offlined. A special hot-plug target is created > to run the full range of hot-plug tests. In default mode, hot-plug tests run > -- > 2.20.1 >
On Wed, Aug 28, 2019 at 12:18 PM shuah <shuah@kernel.org> wrote: > > On 8/26/19 6:37 PM, Shuah Khan wrote: > > Update to add clarity and recommendations on running newer kselftests > > on older kernels vs. matching the kernel and kselftest revisions. > > > > The recommendation is "Match kernel revision and kselftest." > > > > Signed-off-by: Shuah Khan <skhan@linuxfoundation.org> > > --- > > Changes since v1: Fixed "WARNING: Title underline too short." > > I have a few more changes and would like to make and send a v3 after > the LPC's Testing and Fuzzing kselftest discussion. > > Holding off on this patch for now. Is this just because you are busy, or because you expect what you want to say to change after the discussion? If it is because you expect what you want to say here to change, I am surprised. This seems like pretty good, straightforward advice. From where I stand this seems like it makes the documentation better without making anything worse, and so this change should probably be included. In anycase, your call. I just don't think anyone is going to dispute what you are saying here :-) Cheers!
On 8/29/19 11:30 AM, Brendan Higgins wrote: > On Mon, Aug 26, 2019 at 5:37 PM Shuah Khan <skhan@linuxfoundation.org> wrote: >> >> Update to add clarity and recommendations on running newer kselftests >> on older kernels vs. matching the kernel and kselftest revisions. >> >> The recommendation is "Match kernel revision and kselftest." >> >> Signed-off-by: Shuah Khan <skhan@linuxfoundation.org> > > Reviewed-by: Brendan Higgins <brendanhiggins@google.com> > > One minor comment below. > >> --- >> Changes since v1: Fixed "WARNING: Title underline too short." >> >> Documentation/dev-tools/kselftest.rst | 25 +++++++++++++++++++++++++ >> 1 file changed, 25 insertions(+) >> >> diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst >> index 25604904fa6e..308506c5e8fa 100644 >> --- a/Documentation/dev-tools/kselftest.rst >> +++ b/Documentation/dev-tools/kselftest.rst >> @@ -12,6 +12,31 @@ write new tests using the framework on Kselftest wiki: >> >> https://kselftest.wiki.kernel.org/ >> >> +Recommendations on running kselftests in Continuous Integration test rings >> +========================================================================== >> + >> +It is recommended that users run Kselftest from the same release. Running >> +newer Kselftest on older kernels isn't recommended for the following >> +reasons: >> + >> +- Kselftest from mainline and linux-next might not be stable enough to run >> + on stable kernels. >> +- Kselftests detect feature dependencies at run-time and skip tests if a >> + feature and/or configuration they test aren't enabled. Running newer >> + tests on older kernels could result in a few too many skipped/failed >> + conditions. It becomes difficult to evaluate the results. >> +- Newer tests provide better coverage. However, users should make a judgement >> + call on coverage vs. run to run consistency and being able to compare >> + run to run results on older kernels. >> + >> +Recommendations: > > I think this should be the "note" directive. You already have a list > of recommendations above; this seems like a TLDR section, or an > important note or tip. > >> + >> +Match kernel revision and kselftest. Especially important for LTS and >> +Stable kernel Continuous Integration test rings. >> + >> +Hot-plug tests >> +============== >> + >> On some systems, hot-plug tests could hang forever waiting for cpu and >> memory to be ready to be offlined. A special hot-plug target is created >> to run the full range of hot-plug tests. In default mode, hot-plug tests run >> -- >> 2.20.1 >> > Thanks Brendan. I am holding off on this patch for now. I will have to make some updates. I will fold your comment in when I send an updated patch. thanks, -- Shuah
diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst index 25604904fa6e..308506c5e8fa 100644 --- a/Documentation/dev-tools/kselftest.rst +++ b/Documentation/dev-tools/kselftest.rst @@ -12,6 +12,31 @@ write new tests using the framework on Kselftest wiki: https://kselftest.wiki.kernel.org/ +Recommendations on running kselftests in Continuous Integration test rings +========================================================================== + +It is recommended that users run Kselftest from the same release. Running +newer Kselftest on older kernels isn't recommended for the following +reasons: + +- Kselftest from mainline and linux-next might not be stable enough to run + on stable kernels. +- Kselftests detect feature dependencies at run-time and skip tests if a + feature and/or configuration they test aren't enabled. Running newer + tests on older kernels could result in a few too many skipped/failed + conditions. It becomes difficult to evaluate the results. +- Newer tests provide better coverage. However, users should make a judgement + call on coverage vs. run to run consistency and being able to compare + run to run results on older kernels. + +Recommendations: + +Match kernel revision and kselftest. Especially important for LTS and +Stable kernel Continuous Integration test rings. + +Hot-plug tests +============== + On some systems, hot-plug tests could hang forever waiting for cpu and memory to be ready to be offlined. A special hot-plug target is created to run the full range of hot-plug tests. In default mode, hot-plug tests run
Update to add clarity and recommendations on running newer kselftests on older kernels vs. matching the kernel and kselftest revisions. The recommendation is "Match kernel revision and kselftest." Signed-off-by: Shuah Khan <skhan@linuxfoundation.org> --- Changes since v1: Fixed "WARNING: Title underline too short." Documentation/dev-tools/kselftest.rst | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)