[v3,4/8] cmake: support for testing git with ctest

Commit Message

Koji Nakamaru via GitGitGadget May 29, 2020, 1:40 p.m. UTC

From: Sibi Siddharthan <[e-mail]>

This patch provides an alternate way to test git using ctest.
CTest ships with CMake, so there is no additional dependency being
introduced.

To perform the tests with ctest do this after building:
ctest -j[number of jobs]

NOTE: -j is optional, the default number of jobs is 1

Each of the jobs does this:
cd t/ && sh t[something].sh

The reason for using CTest is that it logs the output of the tests
in a neat way, which can be helpful during diagnosis of failures.

After the tests have run ctest generates three log files located in
`build-directory`/Testing/Temporary/

These log files are:

CTestCostData.txt:
This file contains the time taken to complete each test.

LastTestsFailed.log:
This log file contains the names of the tests that have failed in the
run.

LastTest.log:
This log file contains the log of all the tests that have run.
A snippet of the file is given below.

10/901 Testing: D:/my/git-master/t/t0009-prio-queue.sh
10/901 Test: D:/my/git-master/t/t0009-prio-queue.sh
Command: "sh.exe" "D:/my/git-master/t/t0009-prio-queue.sh"
Directory: D:/my/git-master/t
"D:/my/git-master/t/t0009-prio-queue.sh"
Output:
----------------------------------------------------------
ok 1 - basic ordering
ok 2 - mixed put and get
ok 3 - notice empty queue
ok 4 - stack order
passed all 4 test(s)
1..4
<end of output>
Test time =   1.11 sec

NOTE: Testing only works when building in source for now.

Changes:
Renamed the variable test_helper_sources to test-tool_SOURCES
to be consistent with the naming of source variables.

Signed-off-by: Sibi Siddharthan <[e-mail]>
---
 contrib/buildsystems/CMakeLists.txt | 124 ++++++++++++++++++++++++++++
 1 file changed, 124 insertions(+)

Comments

Đoàn Trần Công Danh May 30, 2020, 1:49 p.m. UTC | #1

On 2020-05-29 13:40:20+0000, Sibi Siddharthan via GitGitGadget <[e-mail]> wrote:
> From: Sibi Siddharthan <[e-mail]>
> 
> This patch provides an alternate way to test git using ctest.
> CTest ships with CMake, so there is no additional dependency being
> introduced.
> 
> To perform the tests with ctest do this after building:
> ctest -j[number of jobs]

Or we can just run: make test
CMake will run: ctest itself.

Ah, OK, that's not equivalence. make -j9 test doesn't work :/

Anyway, there're test is failing in Linux with this CMake.

	$git_repo/t/../build/bin-wrappers/git is not executable; using GIT_EXEC_PATH

It looks like CMake Generator forgets "chmod +x bin-wrappers/git"

> Test time =   1.11 sec
> 
> NOTE: Testing only works when building in source for now.

OK, so this maybe the pain point, let me build in source again.
Hm, no, I still see the same problems.
Worse, CMake overrides my current Makefile.
Luckily, I don't have any changes in Makefile.

> +endif()#BUILD_TESTING

We can use:

	endif(BUILD_TESTING)

Sibi Siddharthan May 30, 2020, 7:04 p.m. UTC | #2

On Sat, May 30, 2020 at 7:19 PM Đoàn Trần Công Danh
<[e-mail]> wrote:
>
> On 2020-05-29 13:40:20+0000, Sibi Siddharthan via GitGitGadget <[e-mail]> wrote:
> > From: Sibi Siddharthan <[e-mail]>
> >
> > This patch provides an alternate way to test git using ctest.
> > CTest ships with CMake, so there is no additional dependency being
> > introduced.
> >
> > To perform the tests with ctest do this after building:
> > ctest -j[number of jobs]
>
> Or we can just run: make test
> CMake will run: ctest itself.
>
> Ah, OK, that's not equivalence. make -j9 test doesn't work :/
>

No, it does not.
make test -> ctest --force-new-ctest-process.

> Anyway, there're test is failing in Linux with this CMake.
>
>         $git_repo/t/../build/bin-wrappers/git is not executable; using GIT_EXEC_PATH
>
> It looks like CMake Generator forgets "chmod +x bin-wrappers/git"
>

CMake does not support changing permissions of files easily.
You have to juggle them around to a temporary location and
change the permissions when you do `file(COPY ...)`.

Will look into this.

> > Test time =   1.11 sec
> >
> > NOTE: Testing only works when building in source for now.
>
> OK, so this maybe the pain point, let me build in source again.
> Hm, no, I still see the same problems.
> Worse, CMake overrides my current Makefile.
> Luckily, I don't have any changes in Makefile.
>

Please read the next PATCH, it supports out of source building and testing.

> > +endif()#BUILD_TESTING
>
> We can use:
>
>         endif(BUILD_TESTING)
>

The docs suggest that this way of endif() is for backwards compat only.
Not sure if I should change this, as CMake might remove this in the
future(very unlikely though).

Thank You,
Sibi Siddharthan

> --
> Danh

Đoàn Trần Công Danh May 31, 2020, 1:28 a.m. UTC | #3

On 2020-05-31 00:34:52+0530, Sibi Siddharthan <[e-mail]> wrote:
> > Anyway, there're test is failing in Linux with this CMake.
> >
> >         $git_repo/t/../build/bin-wrappers/git is not executable; using GIT_EXEC_PATH
> >
> > It looks like CMake Generator forgets "chmod +x bin-wrappers/git"
> >
> 
> CMake does not support changing permissions of files easily.
> You have to juggle them around to a temporary location and
> change the permissions when you do `file(COPY ...)`.

Does it work better to use `configure_file` and `file(COPY)`

> > OK, so this maybe the pain point, let me build in source again.
> > Hm, no, I still see the same problems.
> > Worse, CMake overrides my current Makefile.
> > Luckily, I don't have any changes in Makefile.
> >
> 
> Please read the next PATCH, it supports out of source building and testing.

It's the remind to NOT mess with Git's Makefile ;)

> > > +endif()#BUILD_TESTING
> >
> > We can use:
> >
> >         endif(BUILD_TESTING)
> >
> 
> The docs suggest that this way of endif() is for backwards compat only.
> Not sure if I should change this, as CMake might remove this in the
> future(very unlikely though).

I don't have strong opinion for this syntax.

Patch

diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 0cde1a02cbc..33a3559eb8c 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -86,6 +86,7 @@  include(CheckIncludeFile)
 include(CheckFunctionExists)
 include(CheckSymbolExists)
 include(CheckStructHasMember)
+include(CTest)
 
 find_package(ZLIB REQUIRED)
 find_package(CURL)
@@ -731,3 +732,126 @@  install(DIRECTORY ${CMAKE_BINARY_DIR}/templates/blt/ DESTINATION share/git-core/
 if(MSGFMT_EXE)
 	install(DIRECTORY ${CMAKE_BINARY_DIR}/po/build/locale DESTINATION share)
 endif()
+
+
+if(BUILD_TESTING)
+
+#tests-helpers
+add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)
+target_link_libraries(test-fake-ssh common-main)
+
+add_executable(test-line-buffer ${CMAKE_SOURCE_DIR}/t/helper/test-line-buffer.c)
+target_link_libraries(test-line-buffer common-main vcs-svn)
+
+add_executable(test-svn-fe ${CMAKE_SOURCE_DIR}/t/helper/test-svn-fe.c)
+target_link_libraries(test-svn-fe common-main vcs-svn)
+
+#test-tool
+parse_makefile_for_sources(test-tool_SOURCES "TEST_BUILTINS_OBJS")
+
+list(TRANSFORM test-tool_SOURCES PREPEND "${CMAKE_SOURCE_DIR}/t/helper/")
+add_executable(test-tool ${CMAKE_SOURCE_DIR}/t/helper/test-tool.c ${test-tool_SOURCES})
+target_link_libraries(test-tool common-main)
+
+set_target_properties(test-fake-ssh test-line-buffer test-svn-fe test-tool
+			PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/t/helper)
+
+#wrapper scripts
+set(wrapper_scripts
+	git git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext)
+
+set(wrapper_test_scripts
+	test-fake-ssh test-line-buffer test-svn-fe test-tool)
+
+
+foreach(script ${wrapper_scripts})
+	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
+	string(REPLACE "@@BUILD_DIR@@" "${CMAKE_BINARY_DIR}" content "${content}")
+	string(REPLACE "@@PROG@@" "${script}" content "${content}")
+	file(WRITE ${CMAKE_BINARY_DIR}/bin-wrappers/${script} ${content})
+endforeach()
+
+foreach(script ${wrapper_test_scripts})
+	file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
+	string(REPLACE "@@BUILD_DIR@@" "${CMAKE_BINARY_DIR}" content "${content}")
+	string(REPLACE "@@PROG@@" "t/helper/${script}" content "${content}")
+	file(WRITE ${CMAKE_BINARY_DIR}/bin-wrappers/${script} ${content})
+endforeach()
+
+file(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)
+string(REPLACE "@@BUILD_DIR@@" "${CMAKE_BINARY_DIR}" content "${content}")
+string(REPLACE "@@PROG@@" "git-cvsserver" content "${content}")
+file(WRITE ${CMAKE_BINARY_DIR}/bin-wrappers/git-cvsserver ${content})
+
+#options for configuring test options
+option(PERL_TESTS "Perform tests that use perl" ON)
+option(PYTHON_TESTS "Perform tests that use python" ON)
+
+#GIT-BUILD-OPTIONS
+set(TEST_SHELL_PATH ${SHELL_PATH})
+set(DIFF diff)
+set(PYTHON_PATH /usr/bin/python)
+set(TAR tar)
+set(NO_CURL )
+set(NO_EXPAT )
+set(USE_LIBPCRE1 )
+set(USE_LIBPCRE2 )
+set(NO_LIBPCRE1_JIT )
+set(NO_PERL )
+set(NO_PTHREADS )
+set(NO_PYTHON )
+set(PAGER_ENV "LESS=FRX LV=-c")
+set(DC_SHA1 YesPlease)
+set(RUNTIME_PREFIX true)
+set(NO_GETTEXT )
+
+if(NOT CURL_FOUND)
+	set(NO_CURL 1)
+endif()
+
+if(NOT EXPAT_FOUND)
+	set(NO_EXPAT 1)
+endif()
+
+if(NOT Intl_FOUND)
+	set(NO_GETTEXT 1)
+endif()
+
+if(NOT PERL_TESTS)
+	set(NO_PERL 1)
+endif()
+
+if(NOT PYTHON_TESTS)
+	set(NO_PYTHON 1)
+endif()
+
+file(WRITE ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "SHELL_PATH='${SHELL_PATH}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "TEST_SHELL_PATH='${TEST_SHELL_PATH}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "PERL_PATH='${PERL_PATH}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "DIFF='${DIFF}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "PYTHON_PATH='${PYTHON_PATH}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "TAR='${TAR}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "NO_CURL='${NO_CURL}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "NO_EXPAT='${NO_EXPAT}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "USE_LIBPCRE1='${USE_LIBPCRE1}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "NO_LIBPCRE1_JIT='${NO_LIBPCRE1_JIT}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "NO_PERL='${NO_PERL}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "NO_PTHREADS='${NO_PTHREADS}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "NO_UNIX_SOCKETS='${NO_UNIX_SOCKETS}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "PAGER_ENV='${PAGER_ENV}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "DC_SHA1='${DC_SHA1}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "X=''\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "NO_GETTEXT='${NO_GETTEXT}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "RUNTIME_PREFIX='${RUNTIME_PREFIX}'\n")
+file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "NO_PYTHON='${NO_PYTHON}'\n")
+
+file(GLOB test_scipts "${CMAKE_SOURCE_DIR}/t/t[0-9]*.sh")
+
+#test
+foreach(tsh ${test_scipts})
+	add_test(NAME ${tsh}
+		COMMAND ${SH_EXE} ${tsh}
+		WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/t)
+endforeach()
+
+endif()#BUILD_TESTING