| Message ID | 20201020233219.4146059-1-dlatypov@google.com (mailing list archive) |
|---|---|
| State | Superseded, archived |
| Delegated to: | Brendan Higgins |
| Headers | show |
| Series | kunit: tool: fix extra trailing \n in parsed test output | expand |
On Wed, Oct 21, 2020 at 7:32 AM Daniel Latypov <dlatypov@google.com> wrote: > > For simplcity, strip all trailing whitespace from parsed output. > I imagine no one is printing out meaningful trailing whitespace via > KUNIT_FAIL() or similar, and that if they are, they really shouldn't. > > At some point, the lines from `isolate_kunit_output()` started having > trailing \n, which results in artifacty output like this: > > $ ./tools/testing/kunit/kunit.py run > [16:16:46] [FAILED] example_simple_test > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 > > [16:16:46] Expected 1 + 1 == 3, but > > [16:16:46] 1 + 1 == 2 > > [16:16:46] 3 == 3 > > [16:16:46] not ok 1 - example_simple_test > > [16:16:46] > > After this change: > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 > [16:16:46] Expected 1 + 1 == 3, but > [16:16:46] 1 + 1 == 2 > [16:16:46] 3 == 3 > [16:16:46] not ok 1 - example_simple_test > [16:16:46] > > Signed-off-by: Daniel Latypov <dlatypov@google.com> > --- Thanks! This is a long-overdue fix, and it worked well for me. Tested-by: David Gow <davidgow@google.com> One comment below: > tools/testing/kunit/kunit_parser.py | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py > index 8019e3dd4c32..e68b1c66a73f 100644 > --- a/tools/testing/kunit/kunit_parser.py > +++ b/tools/testing/kunit/kunit_parser.py > @@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult: > total_tests = 0 > failed_tests = 0 > crashed_tests = 0 > - test_result = parse_test_result(list(isolate_kunit_output(kernel_output))) > + test_result = parse_test_result(list( > + l.rstrip() for l in isolate_kunit_output(kernel_output))) Could we do this inside isolate_kunit_output() instead? That seems like it'd be a more logical place for it (removing the newline is a sort of isolating the output), and it'd avoid making this line quite as horrifyingly nested. > if test_result.status == TestStatus.NO_TESTS: > print(red('[ERROR] ') + yellow('no tests run!')) > elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS: > > base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93 > -- > 2.29.0.rc1.297.gfa9743e501-goog >
On Thu, Oct 29, 2020 at 7:34 PM David Gow <davidgow@google.com> wrote: > > On Wed, Oct 21, 2020 at 7:32 AM Daniel Latypov <dlatypov@google.com> wrote: > > > > For simplcity, strip all trailing whitespace from parsed output. > > I imagine no one is printing out meaningful trailing whitespace via > > KUNIT_FAIL() or similar, and that if they are, they really shouldn't. > > > > At some point, the lines from `isolate_kunit_output()` started having > > trailing \n, which results in artifacty output like this: > > > > $ ./tools/testing/kunit/kunit.py run > > [16:16:46] [FAILED] example_simple_test > > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 > > > > [16:16:46] Expected 1 + 1 == 3, but > > > > [16:16:46] 1 + 1 == 2 > > > > [16:16:46] 3 == 3 > > > > [16:16:46] not ok 1 - example_simple_test > > > > [16:16:46] > > > > After this change: > > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 > > [16:16:46] Expected 1 + 1 == 3, but > > [16:16:46] 1 + 1 == 2 > > [16:16:46] 3 == 3 > > [16:16:46] not ok 1 - example_simple_test > > [16:16:46] > > > > Signed-off-by: Daniel Latypov <dlatypov@google.com> > > --- > > Thanks! This is a long-overdue fix, and it worked well for me. > > Tested-by: David Gow <davidgow@google.com> > > One comment below: > > > tools/testing/kunit/kunit_parser.py | 3 ++- > > 1 file changed, 2 insertions(+), 1 deletion(-) > > > > diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py > > index 8019e3dd4c32..e68b1c66a73f 100644 > > --- a/tools/testing/kunit/kunit_parser.py > > +++ b/tools/testing/kunit/kunit_parser.py > > @@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult: > > total_tests = 0 > > failed_tests = 0 > > crashed_tests = 0 > > - test_result = parse_test_result(list(isolate_kunit_output(kernel_output))) > > + test_result = parse_test_result(list( > > + l.rstrip() for l in isolate_kunit_output(kernel_output))) > > Could we do this inside isolate_kunit_output() instead? That seems > like it'd be a more logical place for it (removing the newline is a > sort of isolating the output), and it'd avoid making this line quite > as horrifyingly nested. Good point. We could either do it on each yield (messy), or before, i.e. diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 8019e3dd4c32..14d35deb96cd 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -54,6 +54,7 @@ kunit_end_re = re.compile('(List of all partitions:|' def isolate_kunit_output(kernel_output): started = False for line in kernel_output: + line = line.rstrip() # line always has a trailing \n if kunit_start_re.search(line): prefix_len = len(line.split('TAP version')[0]) started = True I had some vague concerns about this as kunit_start_re = re.compile(r'TAP version [0-9]+$') has that anchor at the end. This could ostensibly make it match more things than before. Since I'm using rstrip() out of laziness, that means strings like '<prefix we allow for some reason>TAP version 42\t\n' will now also match. I don't really think that's an issue, but I'd sent this as a more conservative change initially. I can send the diff above as a replacement for this patch. > > > if test_result.status == TestStatus.NO_TESTS: > > print(red('[ERROR] ') + yellow('no tests run!')) > > elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS: > > > > base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93 > > -- > > 2.29.0.rc1.297.gfa9743e501-goog > >
On Fri, Oct 30, 2020 at 1:41 PM Daniel Latypov <dlatypov@google.com> wrote: > > On Thu, Oct 29, 2020 at 7:34 PM David Gow <davidgow@google.com> wrote: > > > > On Wed, Oct 21, 2020 at 7:32 AM Daniel Latypov <dlatypov@google.com> wrote: > > > > > > For simplcity, strip all trailing whitespace from parsed output. > > > I imagine no one is printing out meaningful trailing whitespace via > > > KUNIT_FAIL() or similar, and that if they are, they really shouldn't. > > > > > > At some point, the lines from `isolate_kunit_output()` started having > > > trailing \n, which results in artifacty output like this: > > > > > > $ ./tools/testing/kunit/kunit.py run > > > [16:16:46] [FAILED] example_simple_test > > > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 > > > > > > [16:16:46] Expected 1 + 1 == 3, but > > > > > > [16:16:46] 1 + 1 == 2 > > > > > > [16:16:46] 3 == 3 > > > > > > [16:16:46] not ok 1 - example_simple_test > > > > > > [16:16:46] > > > > > > After this change: > > > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 > > > [16:16:46] Expected 1 + 1 == 3, but > > > [16:16:46] 1 + 1 == 2 > > > [16:16:46] 3 == 3 > > > [16:16:46] not ok 1 - example_simple_test > > > [16:16:46] > > > > > > Signed-off-by: Daniel Latypov <dlatypov@google.com> > > > --- > > > > Thanks! This is a long-overdue fix, and it worked well for me. > > > > Tested-by: David Gow <davidgow@google.com> > > > > One comment below: > > > > > tools/testing/kunit/kunit_parser.py | 3 ++- > > > 1 file changed, 2 insertions(+), 1 deletion(-) > > > > > > diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py > > > index 8019e3dd4c32..e68b1c66a73f 100644 > > > --- a/tools/testing/kunit/kunit_parser.py > > > +++ b/tools/testing/kunit/kunit_parser.py > > > @@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult: > > > total_tests = 0 > > > failed_tests = 0 > > > crashed_tests = 0 > > > - test_result = parse_test_result(list(isolate_kunit_output(kernel_output))) > > > + test_result = parse_test_result(list( > > > + l.rstrip() for l in isolate_kunit_output(kernel_output))) > > > > Could we do this inside isolate_kunit_output() instead? That seems > > like it'd be a more logical place for it (removing the newline is a > > sort of isolating the output), and it'd avoid making this line quite > > as horrifyingly nested. > > Good point. > We could either do it on each yield (messy), or before, i.e. > > diff --git a/tools/testing/kunit/kunit_parser.py > b/tools/testing/kunit/kunit_parser.py > index 8019e3dd4c32..14d35deb96cd 100644 > --- a/tools/testing/kunit/kunit_parser.py > +++ b/tools/testing/kunit/kunit_parser.py > @@ -54,6 +54,7 @@ kunit_end_re = re.compile('(List of all partitions:|' > def isolate_kunit_output(kernel_output): > started = False > for line in kernel_output: > + line = line.rstrip() # line always has a trailing \n > if kunit_start_re.search(line): > prefix_len = len(line.split('TAP version')[0]) > started = True > > I had some vague concerns about this as > kunit_start_re = re.compile(r'TAP version [0-9]+$') > has that anchor at the end. > > This could ostensibly make it match more things than before. > Since I'm using rstrip() out of laziness, that means strings like > '<prefix we allow for some reason>TAP version 42\t\n' > will now also match. > > I don't really think that's an issue, but I'd sent this as a more > conservative change initially. > I can send the diff above as a replacement for this patch. I prefer this if it works. From my cursory testing, it does (though the kunt_tool_tests.py tests will need updating). At the very least, I'm able to get it to work with --alltests / allyesconfig (with a few options tactically disabled), which was the main reason we needed isolate_kunit_output() in the first place. So, unless anyone can find a real-world case where this breaks something, let's go with this. Cheers, -- David > > > > > > if test_result.status == TestStatus.NO_TESTS: > > > print(red('[ERROR] ') + yellow('no tests run!')) > > > elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS: > > > > > > base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93 > > > -- > > > 2.29.0.rc1.297.gfa9743e501-goog > > >
On Fri, Oct 30, 2020 at 12:22 AM David Gow <davidgow@google.com> wrote: > > On Fri, Oct 30, 2020 at 1:41 PM Daniel Latypov <dlatypov@google.com> wrote: > > > > On Thu, Oct 29, 2020 at 7:34 PM David Gow <davidgow@google.com> wrote: > > > > > > On Wed, Oct 21, 2020 at 7:32 AM Daniel Latypov <dlatypov@google.com> wrote: > > > > > > > > For simplcity, strip all trailing whitespace from parsed output. > > > > I imagine no one is printing out meaningful trailing whitespace via > > > > KUNIT_FAIL() or similar, and that if they are, they really shouldn't. > > > > > > > > At some point, the lines from `isolate_kunit_output()` started having > > > > trailing \n, which results in artifacty output like this: > > > > > > > > $ ./tools/testing/kunit/kunit.py run > > > > [16:16:46] [FAILED] example_simple_test > > > > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 > > > > > > > > [16:16:46] Expected 1 + 1 == 3, but > > > > > > > > [16:16:46] 1 + 1 == 2 > > > > > > > > [16:16:46] 3 == 3 > > > > > > > > [16:16:46] not ok 1 - example_simple_test > > > > > > > > [16:16:46] > > > > > > > > After this change: > > > > [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 > > > > [16:16:46] Expected 1 + 1 == 3, but > > > > [16:16:46] 1 + 1 == 2 > > > > [16:16:46] 3 == 3 > > > > [16:16:46] not ok 1 - example_simple_test > > > > [16:16:46] > > > > > > > > Signed-off-by: Daniel Latypov <dlatypov@google.com> > > > > --- > > > > > > Thanks! This is a long-overdue fix, and it worked well for me. > > > > > > Tested-by: David Gow <davidgow@google.com> > > > > > > One comment below: > > > > > > > tools/testing/kunit/kunit_parser.py | 3 ++- > > > > 1 file changed, 2 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py > > > > index 8019e3dd4c32..e68b1c66a73f 100644 > > > > --- a/tools/testing/kunit/kunit_parser.py > > > > +++ b/tools/testing/kunit/kunit_parser.py > > > > @@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult: > > > > total_tests = 0 > > > > failed_tests = 0 > > > > crashed_tests = 0 > > > > - test_result = parse_test_result(list(isolate_kunit_output(kernel_output))) > > > > + test_result = parse_test_result(list( > > > > + l.rstrip() for l in isolate_kunit_output(kernel_output))) > > > > > > Could we do this inside isolate_kunit_output() instead? That seems > > > like it'd be a more logical place for it (removing the newline is a > > > sort of isolating the output), and it'd avoid making this line quite > > > as horrifyingly nested. > > > > Good point. > > We could either do it on each yield (messy), or before, i.e. > > > > diff --git a/tools/testing/kunit/kunit_parser.py > > b/tools/testing/kunit/kunit_parser.py > > index 8019e3dd4c32..14d35deb96cd 100644 > > --- a/tools/testing/kunit/kunit_parser.py > > +++ b/tools/testing/kunit/kunit_parser.py > > @@ -54,6 +54,7 @@ kunit_end_re = re.compile('(List of all partitions:|' > > def isolate_kunit_output(kernel_output): > > started = False > > for line in kernel_output: > > + line = line.rstrip() # line always has a trailing \n > > if kunit_start_re.search(line): > > prefix_len = len(line.split('TAP version')[0]) > > started = True > > > > I had some vague concerns about this as > > kunit_start_re = re.compile(r'TAP version [0-9]+$') > > has that anchor at the end. > > > > This could ostensibly make it match more things than before. > > Since I'm using rstrip() out of laziness, that means strings like > > '<prefix we allow for some reason>TAP version 42\t\n' > > will now also match. > > > > I don't really think that's an issue, but I'd sent this as a more > > conservative change initially. > > I can send the diff above as a replacement for this patch. > > I prefer this if it works. From my cursory testing, it does (though > the kunt_tool_tests.py tests will need updating). At the very least, > I'm able to get it to work with --alltests / allyesconfig (with a few > options tactically disabled), which was the main reason we needed > isolate_kunit_output() in the first place. > > So, unless anyone can find a real-world case where this breaks > something, let's go with this. Sounds good. Sent https://lore.kernel.org/linux-kselftest/20201030223853.554597-1-dlatypov@google.com > > Cheers, > -- David > > > > > > > > > > if test_result.status == TestStatus.NO_TESTS: > > > > print(red('[ERROR] ') + yellow('no tests run!')) > > > > elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS: > > > > > > > > base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93 > > > > -- > > > > 2.29.0.rc1.297.gfa9743e501-goog > > > >
diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 8019e3dd4c32..e68b1c66a73f 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -342,7 +342,8 @@ def parse_run_tests(kernel_output) -> TestResult: total_tests = 0 failed_tests = 0 crashed_tests = 0 - test_result = parse_test_result(list(isolate_kunit_output(kernel_output))) + test_result = parse_test_result(list( + l.rstrip() for l in isolate_kunit_output(kernel_output))) if test_result.status == TestStatus.NO_TESTS: print(red('[ERROR] ') + yellow('no tests run!')) elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:
For simplcity, strip all trailing whitespace from parsed output. I imagine no one is printing out meaningful trailing whitespace via KUNIT_FAIL() or similar, and that if they are, they really shouldn't. At some point, the lines from `isolate_kunit_output()` started having trailing \n, which results in artifacty output like this: $ ./tools/testing/kunit/kunit.py run [16:16:46] [FAILED] example_simple_test [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 [16:16:46] Expected 1 + 1 == 3, but [16:16:46] 1 + 1 == 2 [16:16:46] 3 == 3 [16:16:46] not ok 1 - example_simple_test [16:16:46] After this change: [16:16:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 [16:16:46] Expected 1 + 1 == 3, but [16:16:46] 1 + 1 == 2 [16:16:46] 3 == 3 [16:16:46] not ok 1 - example_simple_test [16:16:46] Signed-off-by: Daniel Latypov <dlatypov@google.com> --- tools/testing/kunit/kunit_parser.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) base-commit: c4d6fe7311762f2e03b3c27ad38df7c40c80cc93