Testing and benchmarking
Prerequisites:
- You should be familiar with Rust
- check out the results and options chapter
One of the often cited features of Rust is built-in support for testing. It allows you to write both integration and unit tests easily, as well as facilitating benchmarking and helping you keep your documentation up to date.
Tests in the code
The testing syntax is pretty simple, you use the #[test] attribute,
#![allow(unused)] fn main() { #[test] fn this_test_will_fail() { panic!("good bye"); } }
Tests that you expect to panic! (ie. testing functionality that should panic) are
marked with the #[should_panic] attribute:
#![allow(unused)] fn main() { #[should_panic] #[test] fn this_test_will_fail_but_thats_okay() { panic!("farewell"); } }
Tests can be temporarily (or conditionally, if combined with the appropriate attribute) disabled
via the #[ignore] attribute:
#![allow(unused)] fn main() { #[ignore] #[test] fn this_test_wont_run() { panic!("i am not talking"); } }
Putting unit tests into a module
It is a common practice to put tests into their own module, named so, and making it conditionally compiled:
#![allow(unused)] fn main() { pub fn add(a: i32, b: i32) -> i32 { a + b } // This is a really bad adding function, its purpose is to fail in this // example. #[allow(dead_code)] fn bad_add(a: i32, b: i32) -> i32 { a - b } #[cfg(test)] mod tests { // Note this useful idiom: importing names from outer (for mod tests) scope. use super::*; #[test] fn test_add() { assert_eq!(add(1, 2), 3); } #[test] fn test_bad_add() { // This assert would fire and test will fail. // Please note, that private functions can be tested too! assert_eq!(bad_add(1, 2), 3); } } }
Check out the assert_eq and assert macros, they are handy for panicking on condition false
or inequality of values, you might be familiar with asserts from C/C++.
Benchmarks
For performance-critical sections of your project, Rust also provides support for benchmark tests. Keep in mind that at the time of this writing, benchmarking is a nightly feature but should be stabilized relatively soon.
#![allow(unused)] #![feature(test)] fn main() { extern crate test; pub fn add_two(a: i32) -> i32 { a + 2 } #[cfg(test)] mod tests { use super::*; use test::Bencher; #[test] fn it_works() { assert_eq!(4, add_two(2)); } #[bench] fn bench_add_two(b: &mut Bencher) { b.iter(|| add_two(2)); } } }
Thanks to integration with tests, as you can see above, you can use a test both as a benchmark and as a means to verify correctness.
If you want to benchmark an entire program, check out a Rust tool called hyperfine.
Integration tests
Rust also has a support for integration tests. While syntactically, they are the same, their location is different.
- Unit tests are either situated right by the code they are testing, or typically under
src/tests/ - Integration tests are typically under
tests/and are freestanding Rust files
Rust files containing integration tests view the library of your crate as an external library and you have to import the items you are testing accordingly.
Additionally, you can use dev-dependencies in integration tests, which have their own section in Cargo manifest of
the same name.i
Doc tests
Rust has a first-class support for in-code documentation, which will be no doubt featured in another future chapter. Part of it is support for runnable code examples.
Rust considers these code examples to be so called doc-tests and whenever you use cargo test,
they will be automatically tested.
#![allow(unused)] fn main() { /// # Examples /// /// ``` /// let x = 5; /// ``` }
Please note that by default, if no language is set for the block code, rustdoc assumes it is Rust code. So the following:
```rust
let x = 5;
```
is strictly equivalent to:
```
let x = 5;
```
Sometimes, you need some setup code, or other things that would distract from your example, but are important to make the tests work. Consider an example block that looks like this:
/// ```
/// /// Some documentation.
/// # fn foo() {} // this function will be hidden
/// println!("Hello, World!");
/// ```
Assortment of handy notes and tips
- This is a good practice in general kinda tip, but if you have initialization code that has to be run all over,
consider placing it into a function within the
testsmodule
Next, if you run tests via the cargo test command, keep in mind that this builds a runnable program with
a wide assortment of command-line arguments:
» cargo test -- --help
Compiling what v0.1.0 (/home/magnusi/braiins-university-projects/what)
Finished test [unoptimized + debuginfo] target(s) in 0.68s
Running unittests (target/debug/deps/what-6e58141455fd2a6d)
Usage: --help [OPTIONS] [FILTERS...]
Options:
--include-ignored
Run ignored and not ignored tests
--ignored Run only ignored tests
--force-run-in-process
Forces tests to run in-process when panic=abort
--exclude-should-panic
Excludes tests marked as should_panic
--test Run tests and not benchmarks
--bench Run benchmarks instead of tests
--list List all tests and benchmarks
-h, --help Display this message
--logfile PATH Write logs to the specified file
--nocapture don't capture stdout/stderr of each task, allow
printing directly
--test-threads n_threads
Number of threads used for running tests in parallel
--skip FILTER Skip tests whose names contain FILTER (this flag can
be used multiple times)
-q, --quiet Display one character per test instead of one line.
Alias to --format=terse
--exact Exactly match filters rather than by substring
--color auto|always|never
Configure coloring of output:
auto = colorize if stdout is a tty and tests are run
on serially (default);
always = always colorize output;
never = never colorize output;
--format pretty|terse|json|junit
Configure formatting of output:
pretty = Print verbose output;
terse = Display one character per test;
json = Output a json document;
junit = Output a JUnit document
--show-output Show captured stdout of successful tests
-Z unstable-options Enable nightly-only flags:
unstable-options = Allow use of experimental features
--report-time Show execution time of each test.
Threshold values for colorized output can be
configured via
`RUST_TEST_TIME_UNIT`, `RUST_TEST_TIME_INTEGRATION`
and
`RUST_TEST_TIME_DOCTEST` environment variables.
Expected format of environment variable is
`VARIABLE=WARN_TIME,CRITICAL_TIME`.
Durations must be specified in milliseconds, e.g.
`500,2000` means that the warn time
is 0.5 seconds, and the critical time is 2 seconds.
Not available for --format=terse
--ensure-time Treat excess of the test execution time limit as
error.
Threshold values for this option can be configured via
`RUST_TEST_TIME_UNIT`, `RUST_TEST_TIME_INTEGRATION`
and
`RUST_TEST_TIME_DOCTEST` environment variables.
Expected format of environment variable is
`VARIABLE=WARN_TIME,CRITICAL_TIME`.
`CRITICAL_TIME` here means the limit that should not
be exceeded by test.
--shuffle Run tests in random order
--shuffle-seed SEED
Run tests in random order; seed the random number
generator with SEED
The FILTER string is tested against the name of all tests, and only those
tests whose names contain the filter are run. Multiple filter strings may
be passed, which will run all tests matching any of the filters.
By default, all tests are run in parallel. This can be altered with the
--test-threads flag or the RUST_TEST_THREADS environment variable when running
tests (set it to 1).
By default, the tests are run in alphabetical order. Use --shuffle or set
RUST_TEST_SHUFFLE to run the tests in random order. Pass the generated
"shuffle seed" to --shuffle-seed (or set RUST_TEST_SHUFFLE_SEED) to run the
tests in the same order again. Note that --shuffle and --shuffle-seed do not
affect whether the tests are run in parallel.
All tests have their standard output and standard error captured by default.
This can be overridden with the --nocapture flag or setting RUST_TEST_NOCAPTURE
environment variable to a value other than "0". Logging is not captured by default.
Test Attributes:
`#[test]` - Indicates a function is a test to be run. This function
takes no arguments.
`#[bench]` - Indicates a function is a benchmark to be run. This
function takes one argument (test::Bencher).
`#[should_panic]` - This function (also labeled with `#[test]`) will only pass if
the code causes a panic (an assertion failure or panic!)
A message may be provided, which the failure string must
contain: #[should_panic(expected = "foo")].
`#[ignore]` - When applied to a function which is already attributed as a
test, then the test runner will ignore these tests during
normal test runs. Running with --ignored or --include-ignored will run
these tests.
Pay particular heed to --nocapture, which comes in handy if you want to see the standard and error
output of your application, and --test-threads n_threads if you want to run the tests single-threaded,
suspecting a race condition type error.
Being particular about should_panic
You can make should_panic expect a specific reason for panicking, for example:
#[test]
#[should_panic(expected = "assertion failed")]
fn it_works() {
assert_eq!("Hello", "world");
}
This can make should_panic tests less fragile, since it will catch instances where the test fails for an
unrelated reason. The test harness will make sure that the failure message contains the provided text.
Architecture/Platform specific tests
Tests, just like any other item in Rust, can be adorned with attributes for conditional compilation, which allows for having platform-specific tests (or conversely, disabling tests on a particular platform).
Here is a couple handy attributes:
#![allow(unused)] fn main() { #[cfg(target_arch = "arm")] // only compile this in on arm #[cfg(not(target_arch = "x86_64"))] // compile this anywhere else but x64 #[cfg(target_family = "unix")] // compile on all unix systems #[cfg(target_env = "musl")] // compile if using musl libc }
The task: Testing the previous and standard library
For this project, you will need at least three of the functions from the final exercise of the chapter on Results and Options.
Part 1: Unit testing
Write tests (as unit tests) for the functions you have selected.
It is up to you how many, and if you will write any #[should_panic] tests also.
Part 2: Integration testing
In this part, we can pretty much test anything. Write tests for the following scenarios:
- Accessing an element outside of slice bounds should panic
- Replacing "_" with "-" in a string should actually do that
- Using the saturating add on usize should not wrap around
Final product
In the end you should be left with a well prepared project, that has the following:
- documented code explaining your reasoning where it isn't self-evident
- optionally tests
- and an example or two where applicable
- clean git history that does not contain fix-ups, merge commits or malformed/misformatted commits
Your Rust code should be formatted by rustfmt / cargo fmt and should produce no
warnings when built. It should also work on stable Rust and follow the Braiins Standard