goldplate 🏅
$ goldplate -j2 tests/
Found 32 specs
Running 49 executions in 2 jobs
Finished in 0.84s
Ran 32 specs, 49 executions, 146 asserts, all A-OK!
goldplate
is a cute and simple opaque golden test runner for CLI
applications. You place your test cases in a directory, annotate them with
.goldplate
files, and that's it. It is completely language agnostic. And
perhaps its best feature is that it can automatically✨🪄 fix your tests outputs!
Give goldplate
a try if:
- You are testing a CLI application
- The application produces deterministic output (e.g. compilers, prettifiers,
convertors)
- The application is quick to start (a process is created for every test)
- Your output is text-based and not huge in size
At Fugue, we've been using internal versions of this tool
since 2016, so it should be pretty stable.
Table of Contents
Tutorial
You can follow along with the tutorial by cloning the repository and running
this command:
$ goldplate tests
As you can see, goldplate
itself is tested using goldplate
. In this
tutorial, we'll walk through some examples. By the end, you should have a good
idea of how to test your CLI application using goldplate
.
Simple asserts
View example: tests/echo.goldplate
In this very simple example, we just run echo "Hello, world!"
. This is
specified in the command
and arguments
fields.
The actual tests that we're executing live in the asserts
field. This simple
test has two asserts:
- We verify that the exit code is 0 (success).
- We check the
stdout
(output) of the command against the file
${GOLDPLATE_NAME}.stdout
. GOLDPLATE_NAME
is the name of the
specification without the extension; so our expected output lives in
tests/echo.stdout
in this case.
We can check that our asserts are correct:
$ goldplate tests/echo.goldplate
If we want to regenerate the expected output, we can simply do:
$ rm tests/echo.stdout
$ goldplate --fix --pretty-diff tests/echo.goldplate
And goldplate
will show you that it fixed one file.
View example: tests/cat.goldplate
You can pass one or multiple lines of input to the command by using the stdin
field.
Setting environment variables
View example: tests/env.goldplate
The environment
field can be used to set environment variables for the
program.
View example: tests/glob.goldplate
.goldplate
files are fairly small but if you have a whole directory of files
that you just want to run the same command on, they can get very repetitive.
This is why goldplate
provides a simple way to pull in multiple input files.
If the input_files
field is set to a glob, all asserts will be ran for every
matching input file. goldplate
will set the following variables:
${GOLDPLATE_INPUT_FILE}
: the path to the input file
${GOLDPLATE_INPUT_NAME}
: the input file without extension
Post processing: find and replace
View example: tests/replace.goldplate
Sometimes you may want to do a find-and-replace on the actual output, for
example to filter out timestamps or other information that you do not expect to
match up against the expected output.
Post processing: prettify JSON
View example: tests/prettify-json.goldplate
Many modern CLI tools output JSON. You can use the prettify_json
post
processor to make sure the JSON is converted to a normalized form with sorted
keys and consistent indentation.
Created files and directories
View example: tests/create.goldplate
goldplate
is geared towards checking the stdout
and stderr
outputs of a
program, but you can also check that files were created with specific contents.
If you do this, goldplate
will remove these files and directories afterwards
to leave a clean repository behind.
Installation
Installation through source is done using standard Haskell tooling -- Cabal
and stack both work well.
Using stack
- Install stack for your platform.
- Clone this repository and
cd
into it.
- Run
stack install
.
- Make sure
$HOME/.local/bin
is in your $PATH
.
Using Cabal
- Install Cabal for your platform.
- Clone this repository and
cd
into it.
- Run
cabal install
.
- Make sure
$HOME/.cabal/bin
is in your $PATH
.
Reference
Syntax
Environment variables can be spliced into the configuration using ${VAR}
syntax within strings. To escape this syntax, use $${VAR}
to get a literal
${VAR}
, $$${VAR}
to get a literal $${VAR}
, and so on.
Environment variables
The test is always executed in the directory that holds the .goldplate
file.
goldplate
will always set the following environment variables:
GOLDPLATE_FILE
: The filename of the .goldplate
file, e.g.
echo.goldplate
.
GOLDPLATE_NAME
: The filename of the .goldplate
file without the
extension, e.g. echo
.
When dealing with multiple input files, the following
additional variables are set:
GOLDPLATE_INPUT_FILE
: The input file name, relative to the current
directory.
GOLDPLATE_INPUT_NAME
: The same as GOLDPLATE_INPUT_FILE
but without
any extensions.
Similar projects
A similar project is smoke. I think goldplate
has two major advantages
over smoke:
- It can fix the output files automatically using
--fix
! This is very
useful if you make a change to your tool that affects a lot of test
files. You still need to manually review the diff, but at least you don't
need to manually update the specs.
- You can avoid most repetitive configs by using
simple globbing.