| Message ID | 20230831214847.209-1-michal.wajdeczko@intel.com (mailing list archive) |
|---|---|
| State | Accepted |
| Commit | ee5f8cc2770b2f0f7cfefb5bf7534e2859e39485 |
| Delegated to: | Brendan Higgins |
| Headers | show |
| Series | kunit: Reset test status on each param iteration | expand |
On Fri, 1 Sept 2023 at 05:49, Michal Wajdeczko <michal.wajdeczko@intel.com> wrote: > > If we skip one parametrized test case then test status remains > SKIP for all subsequent test params leading to wrong reports: > > $ ./tools/testing/kunit/kunit.py run \ > --kunitconfig ./lib/kunit/.kunitconfig *.example_params* > --raw_output \ > > [ ] Starting KUnit Kernel (1/1)... > KTAP version 1 > 1..1 > # example: initializing suite > KTAP version 1 > # Subtest: example > # module: kunit_example_test > 1..1 > KTAP version 1 > # Subtest: example_params_test > # example_params_test: initializing > # example_params_test: cleaning up > ok 1 example value 3 # SKIP unsupported param value 3 > # example_params_test: initializing > # example_params_test: cleaning up > ok 2 example value 2 # SKIP unsupported param value 3 > # example_params_test: initializing > # example_params_test: cleaning up > ok 3 example value 1 # SKIP unsupported param value 3 > # example_params_test: initializing > # example_params_test: cleaning up > ok 4 example value 0 # SKIP unsupported param value 0 > # example_params_test: pass:0 fail:0 skip:4 total:4 > ok 1 example_params_test # SKIP unsupported param value 0 > # example: exiting suite > ok 1 example # SKIP > > Reset test status and status comment after each param iteration > to avoid using stale results. > > Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com> > Cc: David Gow <davidgow@google.com> > Cc: Rae Moar <rmoar@google.com> > --- Nice catch, thanks! This looks good to me. Reviewed-by: David Gow <davidgow@google.com> Cheers, -- David > lib/kunit/kunit-example-test.c | 5 +++-- > lib/kunit/test.c | 6 ++++-- > 2 files changed, 7 insertions(+), 4 deletions(-) > > diff --git a/lib/kunit/kunit-example-test.c b/lib/kunit/kunit-example-test.c > index 01a769f35e1d..6bb5c2ef6696 100644 > --- a/lib/kunit/kunit-example-test.c > +++ b/lib/kunit/kunit-example-test.c > @@ -190,6 +190,7 @@ static void example_static_stub_test(struct kunit *test) > static const struct example_param { > int value; > } example_params_array[] = { > + { .value = 3, }, > { .value = 2, }, > { .value = 1, }, > { .value = 0, }, > @@ -213,8 +214,8 @@ static void example_params_test(struct kunit *test) > KUNIT_ASSERT_NOT_NULL(test, param); > > /* Test can be skipped on unsupported param values */ > - if (!param->value) > - kunit_skip(test, "unsupported param value"); > + if (!is_power_of_2(param->value)) > + kunit_skip(test, "unsupported param value %d", param->value); > > /* You can use param values for parameterized testing */ > KUNIT_EXPECT_EQ(test, param->value % param->value, 0); I'm a little tempted to change this to something more power-of-two specific now, like param->value & (param->value - 1). Thoughts? > diff --git a/lib/kunit/test.c b/lib/kunit/test.c > index 49698a168437..a53fd7e6d5bf 100644 > --- a/lib/kunit/test.c > +++ b/lib/kunit/test.c > @@ -648,12 +648,14 @@ int kunit_run_tests(struct kunit_suite *suite) > param_desc, > test.status_comment); > > + kunit_update_stats(¶m_stats, test.status); > + > /* Get next param. */ > param_desc[0] = '\0'; > test.param_value = test_case->generate_params(test.param_value, param_desc); > test.param_index++; > - > - kunit_update_stats(¶m_stats, test.status); > + test.status = KUNIT_SUCCESS; > + test.status_comment[0] = '\0'; > } > } > > -- > 2.25.1 >
diff --git a/lib/kunit/kunit-example-test.c b/lib/kunit/kunit-example-test.c index 01a769f35e1d..6bb5c2ef6696 100644 --- a/lib/kunit/kunit-example-test.c +++ b/lib/kunit/kunit-example-test.c @@ -190,6 +190,7 @@ static void example_static_stub_test(struct kunit *test) static const struct example_param { int value; } example_params_array[] = { + { .value = 3, }, { .value = 2, }, { .value = 1, }, { .value = 0, }, @@ -213,8 +214,8 @@ static void example_params_test(struct kunit *test) KUNIT_ASSERT_NOT_NULL(test, param); /* Test can be skipped on unsupported param values */ - if (!param->value) - kunit_skip(test, "unsupported param value"); + if (!is_power_of_2(param->value)) + kunit_skip(test, "unsupported param value %d", param->value); /* You can use param values for parameterized testing */ KUNIT_EXPECT_EQ(test, param->value % param->value, 0); diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 49698a168437..a53fd7e6d5bf 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -648,12 +648,14 @@ int kunit_run_tests(struct kunit_suite *suite) param_desc, test.status_comment); + kunit_update_stats(¶m_stats, test.status); + /* Get next param. */ param_desc[0] = '\0'; test.param_value = test_case->generate_params(test.param_value, param_desc); test.param_index++; - - kunit_update_stats(¶m_stats, test.status); + test.status = KUNIT_SUCCESS; + test.status_comment[0] = '\0'; } }
If we skip one parametrized test case then test status remains SKIP for all subsequent test params leading to wrong reports: $ ./tools/testing/kunit/kunit.py run \ --kunitconfig ./lib/kunit/.kunitconfig *.example_params* --raw_output \ [ ] Starting KUnit Kernel (1/1)... KTAP version 1 1..1 # example: initializing suite KTAP version 1 # Subtest: example # module: kunit_example_test 1..1 KTAP version 1 # Subtest: example_params_test # example_params_test: initializing # example_params_test: cleaning up ok 1 example value 3 # SKIP unsupported param value 3 # example_params_test: initializing # example_params_test: cleaning up ok 2 example value 2 # SKIP unsupported param value 3 # example_params_test: initializing # example_params_test: cleaning up ok 3 example value 1 # SKIP unsupported param value 3 # example_params_test: initializing # example_params_test: cleaning up ok 4 example value 0 # SKIP unsupported param value 0 # example_params_test: pass:0 fail:0 skip:4 total:4 ok 1 example_params_test # SKIP unsupported param value 0 # example: exiting suite ok 1 example # SKIP Reset test status and status comment after each param iteration to avoid using stale results. Signed-off-by: Michal Wajdeczko <michal.wajdeczko@intel.com> Cc: David Gow <davidgow@google.com> Cc: Rae Moar <rmoar@google.com> --- lib/kunit/kunit-example-test.c | 5 +++-- lib/kunit/test.c | 6 ++++-- 2 files changed, 7 insertions(+), 4 deletions(-)