In the past weeks, you’ve learned about our overarching testing philosophy:
> Test all of your code with pretty, isolated, fast tests.
Feel free to reply to this email or give me a call to discuss how we can easily add passive metering to your communities for unparalled behavioral data and insights.
You also learned about our investment in Elixir, and our belief that we value *excellent syntax*, in addition to a performant language supporting great concurrency and reliability. Today, we’re going into detail on the first two of our investments in the Elixir ecosystem, the testing frameworks ShouldI and Blacksmith.
Let’s start with the anatomy of a typical Elixir ExUnit test. Here’s a typical ExUnit test:
Believe it or not, this test case represents a huge improvement over test cases in some languages. You can see the test and setup macros on lines 8 and 4 that are shorter and cleaner than typical test cases based on functions.
ONE EXPERIMENT, SEVERAL MEASUREMENTS
We can do better, though. At icanmakeitbetter, we write a bunch of tests. We believe that the setup block should run an experiment, and the test blocks should run measurements. For each setup experiment, we expect users to run several measurements.
For this philosophy, the verb test is not really descriptive enough. We chose to use the verb should, which will help our test users think about what they are actually doing: should-style testing. Each should-style test asserts that some measurement is true. We adopted this strategy from the now-defunct ThoughtBot library, shoulda. We’ve also chosen to emulate their matcher style, but we’ve made some enhancements on the overall testing flow and philosophy that we think address the shortcomings in that library related to performance and flow.
Let’s rewrite our test with our more descriptive macro:
It’s a tiny change, but our intentions are marvelously clear. Not only do we communicate that we’re testing, we also communicate the type of test we’re writing. We’re seeing an important idea in play here: the function names we’ve chosen participate in the overall thought for that line of code:
> should “assign context key”, context do
CONDENSE AND COMBINE THE MEASUREMENTS
We can do still better. In the previous test, we have only one measurement. Typically, we will have multiple measurements. When that happens, it helps to have some mechanisms to group and help document tests, like this:
The grouping is an improvement. It does two things:
- It tells ShouldI that those tests are related.
- It helps the writer group and document the code.
Let’s improve this test. We’d like to run setup once and then run both tests, but we can’t because the tests are do blocks, where anything can happen. For example, we could change database state, which means the tests should run in isolation.
Wait a minute. Our strategy is to run experiments in the setup, and then multiple measurements. And experiments are stateless. Typically, a measurement will just assert something about what’s in the context. We have a short hand for such a measurement, called a matcher. It looks like this:
We’re running not one test, but four. Since all of these measurements are stateless, ShouldI can run them with a single setup.
Also, even if one of these tests fails, the rest can run through completion. You can get all of the failure information from your tests with a single invocation. An error would stop all tests, as you would expect.
You can also do all of the typical things you’d like to do in a context comparison, check for a matching key/value set, do an Elixir pattern match, or check for the existence or nonexistence of a key.
ShouldI comes with a defined set of matchers, for comparing against the context you define in setup, or to compare against a Plug connection:
CONTROLLING SETUP REPETITION WITH NESTED CONTEXTS
When you’re working with a bare-bones testing framework like ExUit, you’ll experience significant repetition when you set up code. Consider this testing script:
This code runs some simple tests, and correctly uses the setup macro to control the duplication of the code to set up the web application.
Still, you can see more duplication. Every test that has to log in replicates code to log in and create a user. The problem is that in real-world testing situations, you’ll always have significant setup code, and this code causes significant duplication.
We can eliminate this code in ShouldI through nesting with blocks, like this:
The with statements serve to organize and document the tests by groups. Though the code is a little longer, it’s much less repetitive and has proven easier to maintain and read for our team at icanmakeitbetter.com.
Of all the features in ShouldI, nested contexts and with is the one that gives us the most value, and is the one that prompted us to build the library.
As you can see, we’ve built on the foundation of ExUnit. We’ve defined a testing language with an overarching philosophy of one test, multiple experiments. We’ve layered onto that motif matchers to simplify quick experiments, nested contexts to help control duplication, and some macros to clarify the vocabulary for these kinds of tests.
We’d love to have your contributions to ShouldI!
Next time, we’ll walk you through Blacksmith, another testing tool. This one is great for building data templates.