Kconfig test scripts

Commit Message

Sam Ravnborg July 20, 2013, 2:19 p.m. UTC

Hi all.

Often when I hack on Kconfig I have missed a set of test cases,
that would allow me to verify that I did not introduce any regressions.

I have not anything fancy in my mind and I spent
a little time on the attached today.
The following is obviously missing a lot a features - but this
would allow me to get started adding simple test cases.

The idea is that each test cases consist of a full
Kconfig file and the resulting output.

Anyone have something better than this,
or maybe some brilliant ideas how to do this much better?

	Sam

--
To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

Yann E. MORIN July 20, 2013, 2:44 p.m. UTC | #1

Sam, All,

On 2013-07-20 16:19 +0200, Sam Ravnborg spake thusly:
> Often when I hack on Kconfig I have missed a set of test cases,
> that would allow me to verify that I did not introduce any regressions.
> 
> I have not anything fancy in my mind and I spent
> a little time on the attached today.
> The following is obviously missing a lot a features - but this
> would allow me to get started adding simple test cases.
> 
> The idea is that each test cases consist of a full
> Kconfig file and the resulting output.
> 
> Anyone have something better than this,
> or maybe some brilliant ideas how to do this much better?

How would one express a test for 'choice'? Test randconfig?

I'm all for a test-suite. But we need to be able to test two things:
  - the parser
  - the behaviour

For the first, your static tests are doing (can do) fine.
For the second, we'd need something a bit more sophisticated.

For example, we need a way to tell that one of such or such symbol is
expected, but not both at the same time. Eg, for a choice between A and
B:
    CONFIG_A=y | CONFIG_B=y     (inclusive OR, for tristate choice)
or:
    CONFIG_A=y ^ CONFIG_B=y     (exclusive OR, for boolean choice)

Also, for randconfig, we'd need to be able to repeatedly run a test and
check symbols probability. Eg:
    CONFIG_A=y [25] ^ CONFIG_B=y [75]

Of course, allow for a slight distortion of a few percent to account for
bias in the RNG).

No idea (yet) on how to do that...

Regards,
Yann E. MORIN.

Patch

diff --git a/scripts/kconfig/tests/run.sh b/scripts/kconfig/tests/run.sh
new file mode 100644
index 0000000..fbd7613
--- /dev/null
+++ b/scripts/kconfig/tests/run.sh
@@ -0,0 +1,20 @@ 
+# set up test environment
+mkdir -p include/config
+mkdir -p include/generated
+
+tests="single_symbol.sh two_symbols.sh"
+
+
+for t in ${tests}; do
+
+	rm -f dot_config.expect
+
+	sh $t
+
+	if [ -f dot_config.expect ]; then
+		grep -v ^# .config > dot_config.actual
+		if ! cmp -s dot_config.actual dot_config.expect ; then
+			diff -u dot_config.expect dot_config.actual
+		fi
+	fi
+done
diff --git a/scripts/kconfig/tests/single_symbol.sh b/scripts/kconfig/tests/single_symbol.sh
new file mode 100644
index 0000000..6459fcf
--- /dev/null
+++ b/scripts/kconfig/tests/single_symbol.sh
@@ -0,0 +1,11 @@ 
+cat << EOF > Kconfig.test
+config SINGLE
+	def_bool y
+
+EOF
+
+cat << EOF > dot_config.expect
+CONFIG_SINGLE=y
+EOF
+
+../conf Kconfig.test > /dev/null
diff --git a/scripts/kconfig/tests/two_symbols.sh b/scripts/kconfig/tests/two_symbols.sh
new file mode 100644
index 0000000..c31902d
--- /dev/null
+++ b/scripts/kconfig/tests/two_symbols.sh
@@ -0,0 +1,14 @@ 
+cat << EOF > Kconfig.test
+config FIRST
+	def_bool y
+
+config SECOND
+	def_bool y
+EOF
+
+cat << EOF > dot_config.expect
+CONFIG_FIRST=y
+CONFIG_SECOND=y
+EOF
+
+../conf Kconfig.test > /dev/null