diff mbox series

[4/4] lib/test_printf.c: use deterministic sequence of random numbers

Message ID 20201025214842.5924-5-linux@rasmusvillemoes.dk
State New
Headers show
Series deterministic random testing | expand

Commit Message

Rasmus Villemoes Oct. 25, 2020, 9:48 p.m. UTC
The printf test suite does each test with a few different buffer sizes
to ensure vsnprintf() behaves correctly with respect to truncation and
size reporting. It calls vsnprintf() with a buffer size that is
guaranteed to be big enough, a buffer size of 0 to ensure that nothing
gets written to the buffer, but it also calls vsnprintf() with a
buffer size chosen to guarantee the output gets truncated somewhere in
the middle.

That buffer size is chosen randomly to increase the chance of finding
some corner case bug (for example, there used to be some %p<foo>
extension that would fail to produce any output if there wasn't room
enough for it all, despite the requirement of producing as much as
there's room for). I'm not aware of that having found anything yet,
but should it happen, it's annoying not to be able to repeat the
test with the same sequence of truncated lengths.

For demonstration purposes, if we break one of the test cases
deliberately, we still get different buffer sizes if we don't pass the
seed parameter:

root@(none):/# modprobe test_printf
[   15.317783] test_printf: vsnprintf(buf, 18, "%piS|%pIS", ...) wrote '127.000.000.001|1', expected '127-000.000.001|1'
[   15.323182] test_printf: failed 3 out of 388 tests
[   15.324034] test_printf: random seed used was 0x278bb9311979cc91
modprobe: ERROR: could not insert 'test_printf': Invalid argument

root@(none):/# modprobe test_printf
[   13.940909] test_printf: vsnprintf(buf, 22, "%piS|%pIS", ...) wrote '127.000.000.001|127.0', expected '127-000.000.001|127.0'
[   13.944744] test_printf: failed 3 out of 388 tests
[   13.945607] test_printf: random seed used was 0x9f72eee1c9dc02e5
modprobe: ERROR: could not insert 'test_printf': Invalid argument

but to repeat a specific sequence of tests, we can do

root@(none):/# modprobe test_printf seed=0x9f72eee1c9dc02e5
[  448.328685] test_printf: vsnprintf(buf, 22, "%piS|%pIS", ...) wrote '127.000.000.001|127.0', expected '127-000.000.001|127.0'
[  448.331650] test_printf: failed 3 out of 388 tests
[  448.332295] test_printf: random seed used was 0x9f72eee1c9dc02e5
modprobe: ERROR: could not insert 'test_printf': Invalid argument

Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
---
 lib/test_printf.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

Comments

Petr Mladek Oct. 30, 2020, 4:26 p.m. UTC | #1
On Sun 2020-10-25 22:48:42, Rasmus Villemoes wrote:
> The printf test suite does each test with a few different buffer sizes
> to ensure vsnprintf() behaves correctly with respect to truncation and
> size reporting. It calls vsnprintf() with a buffer size that is
> guaranteed to be big enough, a buffer size of 0 to ensure that nothing
> gets written to the buffer, but it also calls vsnprintf() with a
> buffer size chosen to guarantee the output gets truncated somewhere in
> the middle.
> 
> That buffer size is chosen randomly to increase the chance of finding
> some corner case bug (for example, there used to be some %p<foo>
> extension that would fail to produce any output if there wasn't room
> enough for it all, despite the requirement of producing as much as
> there's room for). I'm not aware of that having found anything yet,
> but should it happen, it's annoying not to be able to repeat the
> test with the same sequence of truncated lengths.
> 
> For demonstration purposes, if we break one of the test cases
> deliberately, we still get different buffer sizes if we don't pass the
> seed parameter:
> 
> root@(none):/# modprobe test_printf
> [   15.317783] test_printf: vsnprintf(buf, 18, "%piS|%pIS", ...) wrote '127.000.000.001|1', expected '127-000.000.001|1'
> [   15.323182] test_printf: failed 3 out of 388 tests
> [   15.324034] test_printf: random seed used was 0x278bb9311979cc91
> modprobe: ERROR: could not insert 'test_printf': Invalid argument
> 
> root@(none):/# modprobe test_printf
> [   13.940909] test_printf: vsnprintf(buf, 22, "%piS|%pIS", ...) wrote '127.000.000.001|127.0', expected '127-000.000.001|127.0'
> [   13.944744] test_printf: failed 3 out of 388 tests
> [   13.945607] test_printf: random seed used was 0x9f72eee1c9dc02e5
> modprobe: ERROR: could not insert 'test_printf': Invalid argument
> 
> but to repeat a specific sequence of tests, we can do
> 
> root@(none):/# modprobe test_printf seed=0x9f72eee1c9dc02e5
> [  448.328685] test_printf: vsnprintf(buf, 22, "%piS|%pIS", ...) wrote '127.000.000.001|127.0', expected '127-000.000.001|127.0'
> [  448.331650] test_printf: failed 3 out of 388 tests
> [  448.332295] test_printf: random seed used was 0x9f72eee1c9dc02e5
> modprobe: ERROR: could not insert 'test_printf': Invalid argument
> 
> Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>

Great feature!

Reviewed-by: Petr Mladek <pmladek@suse.com>

Best Regards,
Petr
diff mbox series

Patch

diff --git a/lib/test_printf.c b/lib/test_printf.c
index 1ed4a27390cb621715ab..bbea8b807d1eafe67e01 100644
--- a/lib/test_printf.c
+++ b/lib/test_printf.c
@@ -24,6 +24,7 @@ 
 
 #include <linux/property.h>
 
+#define KSTM_RANDOM 1
 #include "../tools/testing/selftests/kselftest_module.h"
 
 #define BUF_SIZE 256
@@ -111,8 +112,14 @@  __test(const char *expect, int elen, const char *fmt, ...)
 	 * be able to print it as expected.
 	 */
 	failed_tests += do_test(BUF_SIZE, expect, elen, fmt, ap);
-	rand = 1 + prandom_u32_max(elen+1);
-	/* Since elen < BUF_SIZE, we have 1 <= rand <= BUF_SIZE. */
+	rand = prandom_u32_range_state(&rnd_state, 1, elen + 1);
+	/*
+	 * Except for elen == 0, we have 1 <= rand <= elen < BUF_SIZE,
+	 * i.e., the output is guaranteed to be truncated somewhere in
+	 * the middle, and we're not pretending the buffer to be
+	 * larger than it really is. For the boring case of elen == 0,
+	 * rand is 1, which is of course also <= BUF_SIZE.
+	 */
 	failed_tests += do_test(rand, expect, elen, fmt, ap);
 	failed_tests += do_test(0, expect, elen, fmt, ap);