diff mbox series

[2/2] kunit: add coverage_uml.config to enable GCOV on UML

Message ID 20220624001247.3255978-2-dlatypov@google.com (mailing list archive)
State Accepted
Commit 1d202d1496a0be94100d8cbc2b658dcd980a3edf
Delegated to: Brendan Higgins
Headers show
Series [1/2] kunit: tool: make --kunitconfig repeatable, blindly concat | expand

Commit Message

Daniel Latypov June 24, 2022, 12:12 a.m. UTC 
Now that kunit.py's --kunitconfig is repeatable, let's create a file to
hold the various options needed to enable coverage under UML.

This can be used like so:
$ ./tools/testing/kunit/kunit.py run \
  --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config \
  --kunitconfig=tools/testing/kunit/configs/coverage_uml.config \
  --make_options=CC=/usr/bin/gcc-6

which on my system is enough to get coverage working [1].

This is still a clunky command, but far better than before.

[1] at the time of this commit, I get:
  Overall coverage rate:
    lines......: 11.6% (34112 of 295033 lines)
    functions..: 15.3% (3721 of 24368 functions)

Signed-off-by: Daniel Latypov <dlatypov@google.com>
---
 Documentation/dev-tools/kunit/running_tips.rst  |  3 +--
 tools/testing/kunit/configs/coverage_uml.config | 11 +++++++++++
 2 files changed, 12 insertions(+), 2 deletions(-)
 create mode 100644 tools/testing/kunit/configs/coverage_uml.config

Comments

David Gow June 24, 2022, 7:55 a.m. UTC | #1 
On Fri, Jun 24, 2022 at 8:12 AM 'Daniel Latypov' via KUnit Development
<kunit-dev@googlegroups.com> wrote:
>
> Now that kunit.py's --kunitconfig is repeatable, let's create a file to
> hold the various options needed to enable coverage under UML.
>
> This can be used like so:
> $ ./tools/testing/kunit/kunit.py run \
>   --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config \
>   --kunitconfig=tools/testing/kunit/configs/coverage_uml.config \
>   --make_options=CC=/usr/bin/gcc-6
>
> which on my system is enough to get coverage working [1].
>
> This is still a clunky command, but far better than before.
>
> [1] at the time of this commit, I get:
>   Overall coverage rate:
>     lines......: 11.6% (34112 of 295033 lines)
>     functions..: 15.3% (3721 of 24368 functions)
>
> Signed-off-by: Daniel Latypov <dlatypov@google.com>
> ---

Looks good to me: depending on whether or not we do something like
[1], we should add a pci_uml.config as well.

Alas, I don't have a working gcc 6 install anymore, so I wasn't able
to test the coverage here, but I at least got an encouraging-looking
error, and the change looks very sensible, so:

Reviewed-by: David Gow <davidgow@google.com>

Cheers,
-- David

[1] https://lore.kernel.org/linux-kselftest/20220622035326.759935-1-davidgow@google.com/


>  Documentation/dev-tools/kunit/running_tips.rst  |  3 +--
>  tools/testing/kunit/configs/coverage_uml.config | 11 +++++++++++
>  2 files changed, 12 insertions(+), 2 deletions(-)
>  create mode 100644 tools/testing/kunit/configs/coverage_uml.config
>
> diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst
> index c36f6760087d..205ea21c9cca 100644
> --- a/Documentation/dev-tools/kunit/running_tips.rst
> +++ b/Documentation/dev-tools/kunit/running_tips.rst
> @@ -123,8 +123,7 @@ Putting it together into a copy-pastable sequence of commands:
>  .. code-block:: bash
>
>         # Append coverage options to the current config
> -       $ echo -e "CONFIG_DEBUG_KERNEL=y\nCONFIG_DEBUG_INFO=y\nCONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y\nCONFIG_GCOV=y" >> .kunit/.kunitconfig
> -       $ ./tools/testing/kunit/kunit.py run
> +       $ ./tools/testing/kunit/kunit.py run --kunitconfig=.kunit/ --kunitconfig=tools/testing/kunit/configs/coverage_uml.config
>         # Extract the coverage information from the build dir (.kunit/)
>         $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/
>
> diff --git a/tools/testing/kunit/configs/coverage_uml.config b/tools/testing/kunit/configs/coverage_uml.config
> new file mode 100644
> index 000000000000..bacb77664fa8
> --- /dev/null
> +++ b/tools/testing/kunit/configs/coverage_uml.config
> @@ -0,0 +1,11 @@
> +# This config fragment enables coverage on UML, which is different from the
> +# normal gcov used in other arches (no debugfs).
> +# Example usage:
> +# ./tools/testing/kunit/kunit.py run \
> +#   --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config \
> +#   --kunitconfig=tools/testing/kunit/configs/coverage_uml.config
> +
> +CONFIG_DEBUG_KERNEL=y
> +CONFIG_DEBUG_INFO=y
> +CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> +CONFIG_GCOV=y
> --
> 2.37.0.rc0.104.g0611611a94-goog
>
> --
> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/20220624001247.3255978-2-dlatypov%40google.com.
Maíra Canal June 24, 2022, 2:12 p.m. UTC | #2 
On 6/24/22 04:55, 'David Gow' via KUnit Development wrote:
> On Fri, Jun 24, 2022 at 8:12 AM 'Daniel Latypov' via KUnit Development
> <kunit-dev@googlegroups.com> wrote:
>>
>> Now that kunit.py's --kunitconfig is repeatable, let's create a file to
>> hold the various options needed to enable coverage under UML.
>>
>> This can be used like so:
>> $ ./tools/testing/kunit/kunit.py run \
>>    --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config \
>>    --kunitconfig=tools/testing/kunit/configs/coverage_uml.config \
>>    --make_options=CC=/usr/bin/gcc-6
>>
>> which on my system is enough to get coverage working [1].
>>

It's great to see this coming to KUnit! As I was testing this series, I 
wasn't able to generate the coverage stats with GCC11. I got a linking 
error from ld:

ERROR:root:/usr/bin/ld: 
/usr/lib/gcc/x86_64-redhat-linux/11/libgcov.a(_gcov.o): in function 
`mangle_path':
(.text+0x19f0): multiple definition of `mangle_path'; 
fs/seq_file.o:/home/mairacanal/linux/.kunit/../fs/seq_file.c:441: first 
defined here
collect2: error: ld returned 1 exit status

By changing the name of the function to `seq_mangle_path`, it is 
possible to solve the linking error and run the tests. But, anyway, no 
.gcda files are generated.

I checked out, and this is a known issue that was already discussed in 
this series [1] (but I guess it didn't move on).

Is there any hope to see better UM support for gcov as using GCC6 is 
quite uncommon nowadays?

[1] 
https://lore.kernel.org/all/20210312095526.197739-1-johannes@sipsolutions.net/

Best regards,
- Maíra Canal

>> This is still a clunky command, but far better than before.
>>
>> [1] at the time of this commit, I get:
>>    Overall coverage rate:
>>      lines......: 11.6% (34112 of 295033 lines)
>>      functions..: 15.3% (3721 of 24368 functions)
>>
>> Signed-off-by: Daniel Latypov <dlatypov@google.com>
>> ---
> 
> Looks good to me: depending on whether or not we do something like
> [1], we should add a pci_uml.config as well.
> 
> Alas, I don't have a working gcc 6 install anymore, so I wasn't able
> to test the coverage here, but I at least got an encouraging-looking
> error, and the change looks very sensible, so:
> 
> Reviewed-by: David Gow <davidgow@google.com>
> 
> Cheers,
> -- David
> 
> [1] https://lore.kernel.org/linux-kselftest/20220622035326.759935-1-davidgow@google.com/
> 
> 
>>   Documentation/dev-tools/kunit/running_tips.rst  |  3 +--
>>   tools/testing/kunit/configs/coverage_uml.config | 11 +++++++++++
>>   2 files changed, 12 insertions(+), 2 deletions(-)
>>   create mode 100644 tools/testing/kunit/configs/coverage_uml.config
>>
>> diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst
>> index c36f6760087d..205ea21c9cca 100644
>> --- a/Documentation/dev-tools/kunit/running_tips.rst
>> +++ b/Documentation/dev-tools/kunit/running_tips.rst
>> @@ -123,8 +123,7 @@ Putting it together into a copy-pastable sequence of commands:
>>   .. code-block:: bash
>>
>>          # Append coverage options to the current config
>> -       $ echo -e "CONFIG_DEBUG_KERNEL=y\nCONFIG_DEBUG_INFO=y\nCONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y\nCONFIG_GCOV=y" >> .kunit/.kunitconfig
>> -       $ ./tools/testing/kunit/kunit.py run
>> +       $ ./tools/testing/kunit/kunit.py run --kunitconfig=.kunit/ --kunitconfig=tools/testing/kunit/configs/coverage_uml.config
>>          # Extract the coverage information from the build dir (.kunit/)
>>          $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/
>>
>> diff --git a/tools/testing/kunit/configs/coverage_uml.config b/tools/testing/kunit/configs/coverage_uml.config
>> new file mode 100644
>> index 000000000000..bacb77664fa8
>> --- /dev/null
>> +++ b/tools/testing/kunit/configs/coverage_uml.config
>> @@ -0,0 +1,11 @@
>> +# This config fragment enables coverage on UML, which is different from the
>> +# normal gcov used in other arches (no debugfs).
>> +# Example usage:
>> +# ./tools/testing/kunit/kunit.py run \
>> +#   --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config \
>> +#   --kunitconfig=tools/testing/kunit/configs/coverage_uml.config
>> +
>> +CONFIG_DEBUG_KERNEL=y
>> +CONFIG_DEBUG_INFO=y
>> +CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
>> +CONFIG_GCOV=y
>> --
>> 2.37.0.rc0.104.g0611611a94-goog
>>
>> --
>> You received this message because you are subscribed to the Google Groups "KUnit Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to kunit-dev+unsubscribe@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/kunit-dev/20220624001247.3255978-2-dlatypov%40google.com.
>
Daniel Latypov June 24, 2022, 4:13 p.m. UTC | #3 
On Fri, Jun 24, 2022 at 7:12 AM Maíra Canal <maira.canal@usp.br> wrote:
>
> On 6/24/22 04:55, 'David Gow' via KUnit Development wrote:
> > On Fri, Jun 24, 2022 at 8:12 AM 'Daniel Latypov' via KUnit Development
> > <kunit-dev@googlegroups.com> wrote:
> >>
> >> Now that kunit.py's --kunitconfig is repeatable, let's create a file to
> >> hold the various options needed to enable coverage under UML.
> >>
> >> This can be used like so:
> >> $ ./tools/testing/kunit/kunit.py run \
> >>    --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config \
> >>    --kunitconfig=tools/testing/kunit/configs/coverage_uml.config \
> >>    --make_options=CC=/usr/bin/gcc-6
> >>
> >> which on my system is enough to get coverage working [1].
> >>
>
> It's great to see this coming to KUnit! As I was testing this series, I
> wasn't able to generate the coverage stats with GCC11. I got a linking
> error from ld:
>
> ERROR:root:/usr/bin/ld:
> /usr/lib/gcc/x86_64-redhat-linux/11/libgcov.a(_gcov.o): in function
> `mangle_path':
> (.text+0x19f0): multiple definition of `mangle_path';
> fs/seq_file.o:/home/mairacanal/linux/.kunit/../fs/seq_file.c:441: first
> defined here
> collect2: error: ld returned 1 exit status
>
> By changing the name of the function to `seq_mangle_path`, it is
> possible to solve the linking error and run the tests. But, anyway, no
> .gcda files are generated.
>
> I checked out, and this is a known issue that was already discussed in
> this series [1] (but I guess it didn't move on).

There's the two main issues I knew about and that you noted.
* no gcda files produced (gcc 7+)
* seq_mangle_path (I don't remember, gcc 8 or 9+?)

The latter can be worked around locally fairly easily, as you also found.
The former is due to how the exit handlers work, iirc.

I didn't know about [1], but it also mentions the exit handlers issue.
However, from the sounds of [2], it seems like it might be a dead end for now :\
So I wouldn't hold onto hope of using modern GCC versions with this.

[2] https://lore.kernel.org/all/d36ea54d8c0a8dd706826ba844a6f27691f45d55.camel@sipsolutions.net/

Daniel
diff mbox series

Patch

diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst
index c36f6760087d..205ea21c9cca 100644
--- a/Documentation/dev-tools/kunit/running_tips.rst
+++ b/Documentation/dev-tools/kunit/running_tips.rst
@@ -123,8 +123,7 @@  Putting it together into a copy-pastable sequence of commands:
 .. code-block:: bash
 
 	# Append coverage options to the current config
-	$ echo -e "CONFIG_DEBUG_KERNEL=y\nCONFIG_DEBUG_INFO=y\nCONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y\nCONFIG_GCOV=y" >> .kunit/.kunitconfig
-	$ ./tools/testing/kunit/kunit.py run
+	$ ./tools/testing/kunit/kunit.py run --kunitconfig=.kunit/ --kunitconfig=tools/testing/kunit/configs/coverage_uml.config
 	# Extract the coverage information from the build dir (.kunit/)
 	$ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/
 
diff --git a/tools/testing/kunit/configs/coverage_uml.config b/tools/testing/kunit/configs/coverage_uml.config
new file mode 100644
index 000000000000..bacb77664fa8
--- /dev/null
+++ b/tools/testing/kunit/configs/coverage_uml.config
@@ -0,0 +1,11 @@ 
+# This config fragment enables coverage on UML, which is different from the
+# normal gcov used in other arches (no debugfs).
+# Example usage:
+# ./tools/testing/kunit/kunit.py run \
+#   --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config \
+#   --kunitconfig=tools/testing/kunit/configs/coverage_uml.config
+
+CONFIG_DEBUG_KERNEL=y
+CONFIG_DEBUG_INFO=y
+CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
+CONFIG_GCOV=y