Repository navigation
[Proposal] Introducing new flags to run tests #542
Description
I and @czgdp1807 had a discussion on adding some new features for running the tests like running all the tests with --show-c or any other such flags. We can also add a few more features to efficiently run a variety of tests. @czgdp1807 Can you please provide some demo CLIs that you would like to have?
Reacted by Thirumalai Shaktivel
Activity
czgdp1807 commentedon Jun 1, 2022
A simple example is,
python script_to_run_tests_with_given_flag.py --flag path/to/test/folderA concrete example,
python script_to_run_tests_with_given_flag.py --show-c integration_tests/Now
script_to_run_tests_with_given_flag.pywill executelpython --show-ccommand for all the test files inintegration_tests/.@certik What do you think?
certik commentedon Jun 1, 2022
We could do that, or we could implement an equivalent of what we do in LFortran: https://gitlab.com/lfortran/lfortran/-/blob/391ff8562350a9942bb02adb83a191219c42704f/integration_tests/CMakeLists.txt#L85 where we mark each test what backends it is supposed to compile with. As not all tests work with all backends, and at the same time, you want to ensure that those tests that work with say LLVM and C backends continue working.
Smit-create commentedon Jun 4, 2022
I have the following suggestion to create a single
run_tests.pythat can do the following:test_me(by defaultllvmbackend):C) for some foldertest_me:What are your views on this? @certik @czgdp1807
certik commentedon Jun 4, 2022
I would like to move the test infrastructure to libasr, and share it with LFortran as much as possible. See also this MR that we need to finish that makes it run in parallel: https://gitlab.com/lfortran/lfortran/-/merge_requests/1749.
For
Cintegration tests, I think we should just use cmake, because we need to link in other C files etc.I would not couple integration tests with tests.
But to be able to run a subset of tests, I think we can do that. Any change must be very carefully tested, to ensure the test runner will fail when it has to fail.
czgdp1807 commentedon Jun 10, 2022
I think we can close this because gh-565 has been merged.
Smit-create commentedon Aug 14, 2023
This can be closed now.