Unit testing


The why of testing

Why test

  • Validation

  • Reproducibility

  • Better code structure

  • Better ability to refactor

  • Better code quality

  • Better documentation

  • Reduces frustration

Testing as validation

Software validation is hard (really really hard)

  • Software involves a huge number of complex processes interacting with one another

  • Proving

A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. - Douglas Adams

Testing as documentation

It is pretty easy to write down basic feature requirements for software (see HW2), it is really hard to write down all of the requirements without any ambiguity.

There is an impedance mismatch between spoken/written language and code - testing is a chance to explicitly show requirements instead of just describing them.

Testing workflow

Your code will always have bugs, and we should try to minimize them as much as possible - but being careful isn’t enough.

Necessary to plan for what to do if and when we find a bug (most important thing is that once we have made a mistake we want to make sure that we will not make it again).

Idealized workflow:

  • Find a bug

  • Write a test to detect the bug - test(s) should fail initially

  • Fix the bug - test(s) should pass now

TDD - a more extreme approach

Introduction to testthat

testthat’s structure

testthat uses a heirarchical structure for the organization of tests:

  • Individual tests are constructed using an expectation, which runs code and compares it to an expected result.

  • expectations for the same or very similar functions are grouped together within a test along with any scaffolding code (e.g. object initialization).

  • Finally, collections of tests are grouped into a context

expectations (old way)

In testthat expectations are constructed using the expect_that function which is given an object and some condition to test using that object. Conditions are specified using builtin condition functions.

expect_that(obj, cond())


  • is_true: truth

  • is_false: falsehood

  • is_a: inheritance

  • equals: equality with numerical tolerance

  • equals_reference: equality relative to a reference

  • is_equivalent_to: equality ignoring attributes

  • is_identical_to: exact identity

  • matches: string matching

  • prints_text: output matching

  • throws_error: error matching

  • gives_warning: warning matching

  • shows_message: message matching

  • takes_less_than: performance

Examples

expect_that(1, equals(1))
expect_that(1, equals(2))
## Error: 1 not equal to 2
## 1 - 2 == -1
m = matrix(1:4,2,2)
expect_that(m, is_a("matrix"))
expect_that(m, is_equivalent_to(1:4))
## Error: m not equal to expected
## target is matrix, current is numeric
expect_that(m, is_identical_to(1:4))
## Error: m is not identical to 1:4. Differences: 
## Attributes: < target is NULL, current is list >
## target is numeric, current is matrix
v1 = 1:4
v2 = c(a=1L,b=2L,c=3L,d=4L)
expect_that(v1, is_equivalent_to(v1))
expect_that(v1, is_identical_to(v2))
## Error: v1 is not identical to v2. Differences: 
## names for target but not for current
expect_that(1, equals(1.01, tolerance=0.01))
## Error: 1 not equal to 1.01
## 1 - 1.01 == -0.01
expect_that(1, equals(1.01, tolerance=0.1))

Expectation shortcuts (new way)

  • expect_true

  • expect_false

  • expect_is

  • expect_equal

  • expect_equal_to_reference

  • expect_equivalent

  • expect_identical

  • expect_match

  • expect_output

  • expect_error

  • expect_warning

  • expect_message

  • expect_more_than

  • expect_less_than

  • expect_named

  • expect_null

Examples (redux)

expect_equal(1, 1)
expect_equal(1, 2)
## Error: 1 not equal to 2
## 1 - 2 == -1
m = matrix(1:4,2,2)
expect_is(m, "matrix")
expect_equivalent(m, 1:4)
## Error: m not equal to expected
## target is matrix, current is numeric
expect_identical(m, 1:4)
## Error: m is not identical to 1:4. Differences: 
## Attributes: < target is NULL, current is list >
## target is numeric, current is matrix
v1 = 1:4
v2 = c(a=1L,b=2L,c=3L,d=4L)
expect_equivalent(v1, v2)
expect_identical(v1, v2)
## Error: v1 is not identical to v2. Differences: 
## names for target but not for current
expect_equal(1.01,1,tolerance=0.01)
expect_equal(1.01,1,tolerance=0.1)


Grouping expectations (tests)

totitle = function(x) 
{
  s = strsplit(x, " ")[[1]]
  paste(toupper(substring(s, 1, 1)), substring(s, 2), sep = "", collapse = " ")
}

test_that("string tests", {
  str1 = "Hello World!"
  str2 = totitle(str1)
  str3 = totitle(tolower(str2))
  str4 = totitle(toupper(str3))

  expect_equal(str1, str2)
  expect_equal(str1, str3)
  expect_equal(str1, str4)
})
## Error: Test failed: 'string tests'
## Not expected: str1 not equal to str4
## 1 string mismatches:
## x[1]: "Hello World!"
## y[1]: "HELLO WORLD!".

Revising totitle

totitle = function(x) 
{
  s = strsplit(x, " ")[[1]]
  paste(toupper(substring(s, 1, 1)), tolower(substring(s, 2)), sep = "", collapse = " ")
}

test_that("string tests", {
  str1 = "Hello World!"
  str2 = totitle(str1)
  str3 = totitle(tolower(str2))
  str4 = totitle(toupper(str3))

  expect_equal(str1, str2)
  expect_equal(str1, str3)
  expect_equal(str1, str4)
})

contexts, files, and directories

The general approach is to group tests into logical subgroups:

  • a context is a description / label that is applied to all subsequent tests, and (maybe) shown by the reporter when running the tests

  • in general the preferences is for one context per test file (R script) - a single test file can be run using test_file (like source but it doesn’t pollute the global environment)

    • the goal for these test files is to be entirely self contained
  • all test files are contained in single directory - these test files can be run (in alphabetical order) using test_dir

Exercise 1

Write at least a half-dozen tests using testthat to evaluate if the following function is correct. Your tests should include checks for correct output when given valid input as well as the appropriate behavior when given invalid input.

Once you have your tests rewrite fib as fib2 such that the new function successfully passes all of your tests.

fib = function(n)
{
  stopifnot(is.integer(n))

  if (n < 2)
      return(n);

  return( fib(n-1L) + fib(n-2L) );
}

Acknowledgments

Acknowledgments

Above materials are derived in part from the following sources: