Skip to content

Commit c58e98a

Browse files
authored
Initial version of integration tests (#330)
* Initial commit for integration tests. Experimental. Playing with potential syntax * Some experimental code to setup tests * Piecewise building of CMakeLists * First check * Alternative approach. Using ruby's test/unit * Parse CMakeCache. Separate lib * First integration test * Latest Format.cmake. Passing style * Allow user-provided integration test dir. Allow reuse * Separate class with utils for cache (no longer pure Hash) * Allow running of tests from any dir * Add integration tests to CI * Use an in-source integration test directory * Allow relative integration test dir from env * Custom assertion for a success of CommandResult * Windows-latest-latest * Enrich CMakeCache class with more CPM data * Added test for CPM-specific CMakeCache values * Style * Style * test_update_single_package * WiP for source cache test * Small source_cache test * Style * Moved env clean to cleanup to make setup methods simpler (not require super) * WiP for integration test documentation * WiP for integration test documentation * Project file creation tweaks * Split docs into multiple files. Complete tutorial. Reference. * Tips * Typo * Setup Ruby inistead of requiring windows-2022 * Revert "Setup Ruby inistead of requiring windows-2022" This reverts commit 8aa2732.
1 parent a27c66a commit c58e98a

20 files changed

+849
-3
lines changed

Diff for: .github/workflows/test.yml

+9-1
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,10 @@ jobs:
1313
runs-on: ${{ matrix.os }}
1414
strategy:
1515
matrix:
16-
os: [ubuntu-latest, windows-latest, macos-latest]
16+
# windows-latest is windows-2019 which carries a pretty old version of ruby (2.5)
17+
# we need at least ruby 2.7 for the tests
18+
# instead of dealing with installing a modern version of ruby on 2019, we'll just use windows-2022 here
19+
os: [ubuntu-latest, windows-2022, macos-latest]
1720

1821
steps:
1922
- name: clone
@@ -23,3 +26,8 @@ jobs:
2326
run: |
2427
cmake -Htest -Bbuild/test
2528
cmake --build build/test --target test-verbose
29+
30+
- name: integration tests
31+
run: ruby test/integration/runner.rb
32+
env:
33+
CPM_INTEGRATION_TEST_DIR: ./build/integration

Diff for: test/integration/.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Use this to have a local integration test which is for personal experiments
2+
test_local.rb

Diff for: test/integration/README.md

+44
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# CPM.cmake Integration Tests
2+
3+
The integration tests of CPM.cmake are written in Ruby. They use a custom integration test framework which extends the [Test::Unit](https://www.rubydoc.info/github/test-unit/test-unit/Test/Unit) library.
4+
5+
They require Ruby 2.7.0 or later.
6+
7+
## Running tests
8+
9+
To run all tests from the repo root execute:
10+
11+
```
12+
$ ruby test/integration/runner.rb
13+
```
14+
15+
The runner will run all tests and generate a report of the exeuction.
16+
17+
The current working directory doesn't matter. If you are in `<repo-root>/test/integration`, you can run simply `$ ruby runner.rb`.
18+
19+
You can execute with `--help` (`$ ruby runner.rb --help`) to see various configuration options of the runner like running individual tests or test cases, or ones that match a regex.
20+
21+
The tests themselves are situated in the Ruby scripts prefixed with `test_`. `<repo-root>/test/integration/test_*`. You can also run an individual test script. For example to only run the **basics** test case, you can execute `$ ruby test_basics.rb`
22+
23+
The tests generate CMake scripts and execute CMake and build toolchains. By default they do this in a directory they generate in your temp path (`/tmp/cpm-test/` on Linux). You can configure the working directory of the tests with an environment variable `CPM_INTEGRATION_TEST_DIR`. For example `$ CPM_INTEGRATION_TEST_DIR=~/mycpmtest; ruby runner.rb`
24+
25+
## Writing tests
26+
27+
Writing tests makes use of the custom integration test framework in `lib.rb`. It is a relatively small extension of Ruby's Test::Unit library.
28+
29+
### The Gist
30+
31+
* Tests cases are Ruby scripts in this directory. The file names must be prefixed with `test_`
32+
* The script should `require_relative './lib'` to allow for individual execution (or else if will only be executable from the runner)
33+
* A test case file should contain a single class which inherits from `IntegrationTest`. It *can* contain multiple classes, but that's bad practice as it makes individual execution harder and implies a dependency between the classes.
34+
* There should be no dependency between the test scripts. Each should be executable individually and the order in which multiple ones are executed mustn't matter.
35+
* The class should contain methods, also prefixed with `test_` which will be executed by the framework. In most cases there would be a single test method per class.
36+
* In case there are multiple test methods, they will be executed in the order in which they are defined.
37+
* The test methods should contain assertions which check for the expected state of things at varous points of the test's execution.
38+
39+
### More
40+
41+
* [A basic tutorial on writing integration tests.](tutorial.md)
42+
* [A brief reference of the integration test framework](reference.md)
43+
* Make sure you're familiar with the [idiosyncrasies](idiosyncrasies.md) of writing integration tests
44+
* [Some tips and tricks](tips.md)

Diff for: test/integration/idiosyncrasies.md

+98
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Notable Idiosyncrasies When Writing Integration Tests
2+
3+
As an integration test framework based on a unit test framework the one created for CPM.cmake suffers from several idiosyncrasies. Make sure you familiarize yourself with them before writing integration tests.
4+
5+
## No shared instance variables between methods
6+
7+
The runner will create an instance of the test class for each test method. This means that instance variables defined in a test method, *will not* be visible in another. For example:
8+
9+
```ruby
10+
class MyTest < IntegrationTest
11+
def test_something
12+
@x = 123
13+
assert_equal 123, @x # Pass. @x is 123
14+
end
15+
def test_something_else
16+
assert_equal 123, @x # Fail! @x would be nil here
17+
end
18+
end
19+
```
20+
21+
There are hacks around sharing Ruby state between methods, but we choose not to use them. If you want to initialize something for all test methods, use `setup`.
22+
23+
```ruby
24+
class MyTest < IntegrationTest
25+
def setup
26+
@x = 123
27+
end
28+
def test_something
29+
assert_equal 123, @x # Pass. @x is 123 thanks to setup
30+
end
31+
def test_something_else
32+
assert_equal 123, @x # Pass. @x is 123 thanks to setup
33+
end
34+
end
35+
```
36+
37+
## `IntegrationTest` makes use of `Test::Unit::TestCase#cleanup`
38+
39+
After each test method the `cleanup` method is called thanks to Test::Unit. If you require the use of `cleanup` in your own tests, make sure you call `super` to also run `IntegrationTest#cleanup`.
40+
41+
```ruby
42+
class MyTest < IntegrationTest
43+
def cleanup
44+
super
45+
my_cleanup
46+
end
47+
# ...
48+
end
49+
```
50+
51+
## It's better to have assertions in test methods as opposed to helper methods
52+
53+
Test::Unit will display a helpful message if an assertion has failed. It will also include the line of code in the test method which caused the failure. However if an assertion is not in the test method, it will display the line which calls the method in which it is. So, please try, to have most assertions in test methods (though we acknowledge that in certain cases this is not practical). For example, if you only require scopes, try using lambdas.
54+
55+
Instead of this:
56+
57+
```ruby
58+
class MyTest < IntegrationTest
59+
def test_something
60+
do_a
61+
do_b
62+
do_c
63+
end
64+
def do_a
65+
# ...
66+
end
67+
def do_b
68+
# ...
69+
assert false # will display failed line as "do_b"
70+
end
71+
def do_c
72+
# ...
73+
end
74+
end
75+
```
76+
77+
...write this:
78+
79+
```ruby
80+
class MyTest < IntegrationTest
81+
def test_something
82+
do_a = -> {
83+
# ...
84+
}
85+
do_b = -> {
86+
# ...
87+
assert false # will display failed line as "assert false"
88+
}
89+
do_c = -> {
90+
# ...
91+
}
92+
93+
do_a.()
94+
do_b.()
95+
do_c.()
96+
end
97+
end
98+
```

Diff for: test/integration/lib.rb

+200
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
require 'fileutils'
2+
require 'open3'
3+
require 'tmpdir'
4+
require 'test/unit'
5+
6+
module TestLib
7+
TMP_DIR = File.expand_path(ENV['CPM_INTEGRATION_TEST_DIR'] || File.join(Dir.tmpdir, 'cpm-test', Time.now.strftime('%Y_%m_%d-%H_%M_%S')))
8+
CPM_PATH = File.expand_path('../../cmake/CPM.cmake', __dir__)
9+
10+
TEMPLATES_DIR = File.expand_path('templates', __dir__)
11+
12+
# Environment variables which are read by cpm
13+
CPM_ENV = %w(
14+
CPM_USE_LOCAL_PACKAGES
15+
CPM_LOCAL_PACKAGES_ONLY
16+
CPM_DOWNLOAD_ALL
17+
CPM_DONT_UPDATE_MODULE_PATH
18+
CPM_DONT_CREATE_PACKAGE_LOCK
19+
CPM_INCLUDE_ALL_IN_PACKAGE_LOCK
20+
CPM_USE_NAMED_CACHE_DIRECTORIES
21+
CPM_SOURCE_CACHE
22+
)
23+
def self.clear_env
24+
CPM_ENV.each { ENV[_1] = nil }
25+
end
26+
end
27+
28+
puts "Warning: test directory '#{TestLib::TMP_DIR}' already exists" if File.exist?(TestLib::TMP_DIR)
29+
raise "Cannot find 'CPM.cmake' at '#{TestLib::CPM_PATH}'" if !File.file?(TestLib::CPM_PATH)
30+
31+
puts "Running CPM.cmake integration tests"
32+
puts "Temp directory: '#{TestLib::TMP_DIR}'"
33+
34+
# Clean all CPM-related env vars
35+
TestLib.clear_env
36+
37+
class Project
38+
def initialize(src_dir, bin_dir)
39+
@src_dir = src_dir
40+
@bin_dir = bin_dir
41+
end
42+
43+
attr :src_dir, :bin_dir
44+
45+
def create_file(target_path, text, args = {})
46+
target_path = File.join(@src_dir, target_path)
47+
48+
# tweak args
49+
args[:cpm_path] = TestLib::CPM_PATH if !args[:cpm_path]
50+
args[:packages] = [args[:package]] if args[:package] # if args contain package, create the array
51+
args[:packages] = args[:packages].join("\n") if args[:packages] # join all packages if any
52+
53+
File.write target_path, text % args
54+
end
55+
56+
def create_file_from_template(target_path, source_path, args = {})
57+
source_path = File.join(@src_dir, source_path)
58+
raise "#{source_path} doesn't exist" if !File.file?(source_path)
59+
src_text = File.read source_path
60+
create_file target_path, src_text, args
61+
end
62+
63+
# common function to create ./CMakeLists.txt from ./lists.in.cmake
64+
def create_lists_from_default_template(args = {})
65+
create_file_from_template 'CMakeLists.txt', 'lists.in.cmake', args
66+
end
67+
68+
CommandResult = Struct.new :out, :err, :status
69+
def configure(extra_args = '')
70+
CommandResult.new *Open3.capture3("cmake -S #{@src_dir} -B #{@bin_dir} #{extra_args}")
71+
end
72+
def build(extra_args = '')
73+
CommandResult.new *Open3.capture3("cmake --build #{@bin_dir} #{extra_args}")
74+
end
75+
76+
class CMakeCache
77+
class Entry
78+
def initialize(val, type, advanced, desc)
79+
@val = val
80+
@type = type
81+
@advanced = advanced
82+
@desc = desc
83+
end
84+
attr :val, :type, :advanced, :desc
85+
alias_method :advanced?, :advanced
86+
def inspect
87+
"(#{val.inspect} #{type}" + (advanced? ? ' ADVANCED)' : ')')
88+
end
89+
end
90+
91+
Package = Struct.new(:ver, :src_dir, :bin_dir)
92+
93+
def self.from_dir(dir)
94+
entries = {}
95+
cur_desc = ''
96+
file = File.join(dir, 'CMakeCache.txt')
97+
return nil if !File.file?(file)
98+
File.readlines(file).each { |line|
99+
line.strip!
100+
next if line.empty?
101+
next if line.start_with? '#' # comment
102+
if line.start_with? '//'
103+
cur_desc += line[2..]
104+
else
105+
m = /(.+?)(-ADVANCED)?:([A-Z]+)=(.*)/.match(line)
106+
raise "Error parsing '#{line}' in #{file}" if !m
107+
entries[m[1]] = Entry.new(m[4], m[3], !!m[2], cur_desc)
108+
cur_desc = ''
109+
end
110+
}
111+
CMakeCache.new entries
112+
end
113+
114+
def initialize(entries)
115+
@entries = entries
116+
117+
package_list = self['CPM_PACKAGES']
118+
@packages = if package_list
119+
# collect package data
120+
@packages = package_list.split(';').map { |name|
121+
[name, Package.new(
122+
self["CPM_PACKAGE_#{name}_VERSION"],
123+
self["CPM_PACKAGE_#{name}_SOURCE_DIR"],
124+
self["CPM_PACKAGE_#{name}_BINARY_DIR"]
125+
)]
126+
}.to_h
127+
else
128+
{}
129+
end
130+
end
131+
132+
attr :entries, :packages
133+
134+
def [](key)
135+
e = @entries[key]
136+
return nil if !e
137+
e.val
138+
end
139+
end
140+
def read_cache
141+
CMakeCache.from_dir @bin_dir
142+
end
143+
end
144+
145+
class IntegrationTest < Test::Unit::TestCase
146+
self.test_order = :defined # run tests in order of defintion (as opposed to alphabetical)
147+
148+
def cleanup
149+
# Clear cpm-related env vars which may have been set by the test
150+
TestLib.clear_env
151+
end
152+
153+
# extra assertions
154+
155+
def assert_success(res)
156+
msg = build_message(nil, "command status was expected to be a success, but failed with code <?> and STDERR:\n\n#{res.err}", res.status.to_i)
157+
assert_block(msg) { res.status.success? }
158+
end
159+
160+
def assert_same_path(a, b)
161+
msg = build_message(nil, "<?> expected but was\n<?>", a, b)
162+
assert_block(msg) { File.identical? a, b }
163+
end
164+
165+
# utils
166+
class << self
167+
def startup
168+
@@test_dir = File.join(TestLib::TMP_DIR, self.name.
169+
# to-underscore conversion from Rails
170+
gsub(/::/, '/').
171+
gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
172+
gsub(/([a-z\d])([A-Z])/,'\1_\2').
173+
tr("-", "_").
174+
downcase
175+
)
176+
end
177+
end
178+
179+
def cur_test_dir
180+
@@test_dir
181+
end
182+
183+
def make_project(template_dir = nil)
184+
test_name = local_name
185+
test_name = test_name[5..] if test_name.start_with?('test_')
186+
187+
base = File.join(cur_test_dir, test_name)
188+
src_dir = base + '-src'
189+
190+
FileUtils.mkdir_p src_dir
191+
192+
if template_dir
193+
template_dir = File.join(TestLib::TEMPLATES_DIR, template_dir)
194+
raise "#{template_dir} is not a directory" if !File.directory?(template_dir)
195+
FileUtils.copy_entry template_dir, src_dir
196+
end
197+
198+
Project.new src_dir, base + '-bin'
199+
end
200+
end

0 commit comments

Comments
 (0)