[v5,0/6] Allow Valgrind checking all QEMU processes
mbox series

Message ID 1563553816-148827-1-git-send-email-andrey.shinkevich@virtuozzo.com
Headers show
Series
  • Allow Valgrind checking all QEMU processes
Related show

Message

Andrey Shinkevich July 19, 2019, 4:30 p.m. UTC
In the current implementation of the QEMU bash iotests, only qemu-io
processes may be run under the Valgrind, which is a useful tool for
finding memory usage issues. Let's allow the common.rc bash script
runing all the QEMU processes, such as qemu-kvm, qemu-img, qemu-ndb
and qemu-vxhs, under the Valgrind tool.

v5:
  01: The patch "block/nbd: NBDReply is used being uninitialized" was detached
      and taken into account in the patch "nbd: Initialize reply on failure"
      by Eric Blake.

v4:
  01: The patch "iotests: Set read-zeroes on in null block driver for Valgrind"
      was extended with new cases and issued as a separate series.
  02: The new patch "block/nbd: NBDReply is used being uninitialized" was
      added to resolve the failure of the iotest 083 run under Valgrind.

v3:
  01: The new function _casenotrun() was added to the common.rc bash
      script to notify the user of test cases dropped for some reason.
      Suggested by Kevin.
      Particularly, the notification about the nonexistent TMPDIR in
      the test 051 was added (noticed by Vladimir).
  02: The timeout in some test cases was extended for Valgrind because
      it differs when running on the ramdisk.
  03: Due to the common.nbd script has been changed with the commit
      b28f582c, the patch "iotests: amend QEMU NBD process synchronization"
      is actual no more. Note that QEMU_NBD is launched in the bash nested
      shell in the _qemu_nbd_wrapper() as it was before in common.rc.
  04: The patch "iotests: new file to suppress Valgrind errors" was dropped
      due to my superficial understanding of the work of the function
      blk_pread_unthrottled(). Special thanks to Kevin who shed the light
      on the null block driver involved. Now, the parameter 'read-zeroes=on'
      is passed to the null block driver to initialize the buffer in the
      function guess_disk_lchs() that the Valgrind was complaining to.

v2:
  01: The patch 2/7 of v1 was merged into the patch 1/7, suggested by Daniel.
  02: Another patch 7/7 was added to introduce the Valgrind error suppression
      file into the QEMU project.
  Discussed in the email thread with the message ID:
  <1560276131-683243-1-git-send-email-andrey.shinkevich@virtuozzo.com>

Andrey Shinkevich (6):
  iotests: allow Valgrind checking all QEMU processes
  iotests: exclude killed processes from running under  Valgrind
  iotests: Add casenotrun report to bash tests
  iotests: Valgrind fails with nonexistent directory
  iotests: extended timeout under Valgrind
  iotests: extend sleeping time under Valgrind

 tests/qemu-iotests/028       |  6 +++-
 tests/qemu-iotests/039       |  5 +++
 tests/qemu-iotests/039.out   | 30 +++--------------
 tests/qemu-iotests/051       |  4 +++
 tests/qemu-iotests/061       |  2 ++
 tests/qemu-iotests/061.out   | 12 ++-----
 tests/qemu-iotests/137       |  1 +
 tests/qemu-iotests/137.out   |  6 +---
 tests/qemu-iotests/183       |  9 +++++-
 tests/qemu-iotests/192       |  6 +++-
 tests/qemu-iotests/247       |  6 +++-
 tests/qemu-iotests/common.rc | 76 +++++++++++++++++++++++++++++++++-----------
 12 files changed, 101 insertions(+), 62 deletions(-)

Comments

Andrey Shinkevich Aug. 6, 2019, 4:24 p.m. UTC | #1
PINGING...

On 19/07/2019 19:30, Andrey Shinkevich wrote:
> In the current implementation of the QEMU bash iotests, only qemu-io
> processes may be run under the Valgrind, which is a useful tool for
> finding memory usage issues. Let's allow the common.rc bash script
> runing all the QEMU processes, such as qemu-kvm, qemu-img, qemu-ndb
> and qemu-vxhs, under the Valgrind tool.
> 
> v5:
>    01: The patch "block/nbd: NBDReply is used being uninitialized" was detached
>        and taken into account in the patch "nbd: Initialize reply on failure"
>        by Eric Blake.
> 
> v4:
>    01: The patch "iotests: Set read-zeroes on in null block driver for Valgrind"
>        was extended with new cases and issued as a separate series.
>    02: The new patch "block/nbd: NBDReply is used being uninitialized" was
>        added to resolve the failure of the iotest 083 run under Valgrind.
> 
> v3:
>    01: The new function _casenotrun() was added to the common.rc bash
>        script to notify the user of test cases dropped for some reason.
>        Suggested by Kevin.
>        Particularly, the notification about the nonexistent TMPDIR in
>        the test 051 was added (noticed by Vladimir).
>    02: The timeout in some test cases was extended for Valgrind because
>        it differs when running on the ramdisk.
>    03: Due to the common.nbd script has been changed with the commit
>        b28f582c, the patch "iotests: amend QEMU NBD process synchronization"
>        is actual no more. Note that QEMU_NBD is launched in the bash nested
>        shell in the _qemu_nbd_wrapper() as it was before in common.rc.
>    04: The patch "iotests: new file to suppress Valgrind errors" was dropped
>        due to my superficial understanding of the work of the function
>        blk_pread_unthrottled(). Special thanks to Kevin who shed the light
>        on the null block driver involved. Now, the parameter 'read-zeroes=on'
>        is passed to the null block driver to initialize the buffer in the
>        function guess_disk_lchs() that the Valgrind was complaining to.
> 
> v2:
>    01: The patch 2/7 of v1 was merged into the patch 1/7, suggested by Daniel.
>    02: Another patch 7/7 was added to introduce the Valgrind error suppression
>        file into the QEMU project.
>    Discussed in the email thread with the message ID:
>    <1560276131-683243-1-git-send-email-andrey.shinkevich@virtuozzo.com>
> 
> Andrey Shinkevich (6):
>    iotests: allow Valgrind checking all QEMU processes
>    iotests: exclude killed processes from running under  Valgrind
>    iotests: Add casenotrun report to bash tests
>    iotests: Valgrind fails with nonexistent directory
>    iotests: extended timeout under Valgrind
>    iotests: extend sleeping time under Valgrind
> 
>   tests/qemu-iotests/028       |  6 +++-
>   tests/qemu-iotests/039       |  5 +++
>   tests/qemu-iotests/039.out   | 30 +++--------------
>   tests/qemu-iotests/051       |  4 +++
>   tests/qemu-iotests/061       |  2 ++
>   tests/qemu-iotests/061.out   | 12 ++-----
>   tests/qemu-iotests/137       |  1 +
>   tests/qemu-iotests/137.out   |  6 +---
>   tests/qemu-iotests/183       |  9 +++++-
>   tests/qemu-iotests/192       |  6 +++-
>   tests/qemu-iotests/247       |  6 +++-
>   tests/qemu-iotests/common.rc | 76 +++++++++++++++++++++++++++++++++-----------
>   12 files changed, 101 insertions(+), 62 deletions(-)
>
Cleber Rosa Aug. 16, 2019, 8:05 p.m. UTC | #2
On Fri, Jul 19, 2019 at 07:30:10PM +0300, Andrey Shinkevich wrote:
> In the current implementation of the QEMU bash iotests, only qemu-io
> processes may be run under the Valgrind, which is a useful tool for
> finding memory usage issues. Let's allow the common.rc bash script
> runing all the QEMU processes, such as qemu-kvm, qemu-img, qemu-ndb
> and qemu-vxhs, under the Valgrind tool.
>

FIY, this looks very similar (in purpose) to:

   https://avocado-framework.readthedocs.io/en/71.0/WrapProcess.html

And in fact Valgrind was one of the original motivations:

   https://github.com/avocado-framework/avocado/blob/master/examples/wrappers/valgrind.sh

Maybe this can be helpful for the Python based iotests.

- Cleber.

> v5:
>   01: The patch "block/nbd: NBDReply is used being uninitialized" was detached
>       and taken into account in the patch "nbd: Initialize reply on failure"
>       by Eric Blake.
> 
> v4:
>   01: The patch "iotests: Set read-zeroes on in null block driver for Valgrind"
>       was extended with new cases and issued as a separate series.
>   02: The new patch "block/nbd: NBDReply is used being uninitialized" was
>       added to resolve the failure of the iotest 083 run under Valgrind.
> 
> v3:
>   01: The new function _casenotrun() was added to the common.rc bash
>       script to notify the user of test cases dropped for some reason.
>       Suggested by Kevin.
>       Particularly, the notification about the nonexistent TMPDIR in
>       the test 051 was added (noticed by Vladimir).
>   02: The timeout in some test cases was extended for Valgrind because
>       it differs when running on the ramdisk.
>   03: Due to the common.nbd script has been changed with the commit
>       b28f582c, the patch "iotests: amend QEMU NBD process synchronization"
>       is actual no more. Note that QEMU_NBD is launched in the bash nested
>       shell in the _qemu_nbd_wrapper() as it was before in common.rc.
>   04: The patch "iotests: new file to suppress Valgrind errors" was dropped
>       due to my superficial understanding of the work of the function
>       blk_pread_unthrottled(). Special thanks to Kevin who shed the light
>       on the null block driver involved. Now, the parameter 'read-zeroes=on'
>       is passed to the null block driver to initialize the buffer in the
>       function guess_disk_lchs() that the Valgrind was complaining to.
> 
> v2:
>   01: The patch 2/7 of v1 was merged into the patch 1/7, suggested by Daniel.
>   02: Another patch 7/7 was added to introduce the Valgrind error suppression
>       file into the QEMU project.
>   Discussed in the email thread with the message ID:
>   <1560276131-683243-1-git-send-email-andrey.shinkevich@virtuozzo.com>
> 
> Andrey Shinkevich (6):
>   iotests: allow Valgrind checking all QEMU processes
>   iotests: exclude killed processes from running under  Valgrind
>   iotests: Add casenotrun report to bash tests
>   iotests: Valgrind fails with nonexistent directory
>   iotests: extended timeout under Valgrind
>   iotests: extend sleeping time under Valgrind
> 
>  tests/qemu-iotests/028       |  6 +++-
>  tests/qemu-iotests/039       |  5 +++
>  tests/qemu-iotests/039.out   | 30 +++--------------
>  tests/qemu-iotests/051       |  4 +++
>  tests/qemu-iotests/061       |  2 ++
>  tests/qemu-iotests/061.out   | 12 ++-----
>  tests/qemu-iotests/137       |  1 +
>  tests/qemu-iotests/137.out   |  6 +---
>  tests/qemu-iotests/183       |  9 +++++-
>  tests/qemu-iotests/192       |  6 +++-
>  tests/qemu-iotests/247       |  6 +++-
>  tests/qemu-iotests/common.rc | 76 +++++++++++++++++++++++++++++++++-----------
>  12 files changed, 101 insertions(+), 62 deletions(-)
> 
> -- 
> 1.8.3.1
> 
>
Andrey Shinkevich Aug. 25, 2019, 10:30 a.m. UTC | #3
On 16/08/2019 23:05, Cleber Rosa wrote:
> On Fri, Jul 19, 2019 at 07:30:10PM +0300, Andrey Shinkevich wrote:
>> In the current implementation of the QEMU bash iotests, only qemu-io
>> processes may be run under the Valgrind, which is a useful tool for
>> finding memory usage issues. Let's allow the common.rc bash script
>> runing all the QEMU processes, such as qemu-kvm, qemu-img, qemu-ndb
>> and qemu-vxhs, under the Valgrind tool.
>>
> 
> FIY, this looks very similar (in purpose) to:
> 
>     https://avocado-framework.readthedocs.io/en/71.0/WrapProcess.html
> 
> And in fact Valgrind was one of the original motivations:
> 
>     https://github.com/avocado-framework/avocado/blob/master/examples/wrappers/valgrind.sh
> 
> Maybe this can be helpful for the Python based iotests.
> 
> - Cleber.
> 

Thank you Cleber for the advice. That is the way I actually ran Python 
iotests under Valgrind on my host and discovered some issues with them 
already.

Andrey

>> v5:
>>    01: The patch "block/nbd: NBDReply is used being uninitialized" was detached
>>        and taken into account in the patch "nbd: Initialize reply on failure"
>>        by Eric Blake.
>>
>> v4:
>>    01: The patch "iotests: Set read-zeroes on in null block driver for Valgrind"
>>        was extended with new cases and issued as a separate series.
>>    02: The new patch "block/nbd: NBDReply is used being uninitialized" was
>>        added to resolve the failure of the iotest 083 run under Valgrind.
>>
>> v3:
>>    01: The new function _casenotrun() was added to the common.rc bash
>>        script to notify the user of test cases dropped for some reason.
>>        Suggested by Kevin.
>>        Particularly, the notification about the nonexistent TMPDIR in
>>        the test 051 was added (noticed by Vladimir).
>>    02: The timeout in some test cases was extended for Valgrind because
>>        it differs when running on the ramdisk.
>>    03: Due to the common.nbd script has been changed with the commit
>>        b28f582c, the patch "iotests: amend QEMU NBD process synchronization"
>>        is actual no more. Note that QEMU_NBD is launched in the bash nested
>>        shell in the _qemu_nbd_wrapper() as it was before in common.rc.
>>    04: The patch "iotests: new file to suppress Valgrind errors" was dropped
>>        due to my superficial understanding of the work of the function
>>        blk_pread_unthrottled(). Special thanks to Kevin who shed the light
>>        on the null block driver involved. Now, the parameter 'read-zeroes=on'
>>        is passed to the null block driver to initialize the buffer in the
>>        function guess_disk_lchs() that the Valgrind was complaining to.
>>
>> v2:
>>    01: The patch 2/7 of v1 was merged into the patch 1/7, suggested by Daniel.
>>    02: Another patch 7/7 was added to introduce the Valgrind error suppression
>>        file into the QEMU project.
>>    Discussed in the email thread with the message ID:
>>    <1560276131-683243-1-git-send-email-andrey.shinkevich@virtuozzo.com>
>>
>> Andrey Shinkevich (6):
>>    iotests: allow Valgrind checking all QEMU processes
>>    iotests: exclude killed processes from running under  Valgrind
>>    iotests: Add casenotrun report to bash tests
>>    iotests: Valgrind fails with nonexistent directory
>>    iotests: extended timeout under Valgrind
>>    iotests: extend sleeping time under Valgrind
>>
>>   tests/qemu-iotests/028       |  6 +++-
>>   tests/qemu-iotests/039       |  5 +++
>>   tests/qemu-iotests/039.out   | 30 +++--------------
>>   tests/qemu-iotests/051       |  4 +++
>>   tests/qemu-iotests/061       |  2 ++
>>   tests/qemu-iotests/061.out   | 12 ++-----
>>   tests/qemu-iotests/137       |  1 +
>>   tests/qemu-iotests/137.out   |  6 +---
>>   tests/qemu-iotests/183       |  9 +++++-
>>   tests/qemu-iotests/192       |  6 +++-
>>   tests/qemu-iotests/247       |  6 +++-
>>   tests/qemu-iotests/common.rc | 76 +++++++++++++++++++++++++++++++++-----------
>>   12 files changed, 101 insertions(+), 62 deletions(-)
>>
>> -- 
>> 1.8.3.1
>>
>>