Safe Haskell | None |
---|---|
Language | Haskell2010 |
FitSpec: refining property-sets for functional testing
FitSpec provides automated assistance in the task of refining test properties for Haskell functions. FitSpec tests mutant variations of functions under test against a given property set, recording any surviving mutants that pass all tests. FitSpec then reports:
- surviving mutants: indicating incompleteness of properties, prompting the user to amend a property or to add a new one;
- conjectures: indicating redundancy in the property set, prompting the user to remove properties so to reduce the cost of testing.
Example, refining a sort
specification:
import Test.FitSpec import Data.List (sort) properties sort = [ property $ \xs -> ordered (sort xs) , property $ \xs -> length (sort xs) == length xs , property $ \x xs -> elem x (sort xs) == elem x xs , property $ \x xs -> notElem x (sort xs) == notElem x xs , property $ \x xs -> minimum (x:xs) == head (sort (x:xs)) ] where ordered (x:y:xs) = x <= y && ordered (y:xs) ordered _ = True main = mainWith args { names = ["sort xs"] , nMutants = 4000 , nTests = 4000 , timeout = 0 } (sort::[Word2]->[Word2]) properties
The above program reports the following:
Apparent incomplete and non-minimal specification based on 4000 test cases for each of properties 1, 2, 3, 4 and 5 for each of 4000 mutant variations. 3 survivors (99% killed), smallest: \xs -> case xs of [0,0,1] -> [0,1,1] _ -> sort xs apparent minimal property subsets: {1,2,3} {1,2,4} conjectures: {3} = {4} 96% killed (weak) {1,3} ==> {5} 98% killed (weak)
- type Property = [([String], Bool)]
- property :: Testable a => a -> Property
- data Args = Args {}
- data ShowMutantAs
- args :: Args
- fixargs :: Int -> Int -> Args
- report :: (Mutable a, ShowMutable a) => a -> (a -> [Property]) -> IO ()
- reportWith :: (Mutable a, ShowMutable a) => Args -> a -> (a -> [Property]) -> IO ()
- reportWithExtra :: (Mutable a, ShowMutable a) => [a] -> Args -> a -> (a -> [Property]) -> IO ()
- mainWith :: (Mutable a, ShowMutable a) => Args -> a -> (a -> [Property]) -> IO ()
- defaultMain :: (Mutable a, ShowMutable a) => a -> (a -> [Property]) -> IO ()
- getArgs :: IO Args
- getArgsWith :: Args -> IO Args
- class Mutable a where
- mutiersEq :: (Listable a, Eq a) => a -> [[a]]
- class ShowMutable a where
- mutantSEq :: (Eq a, Show a) => a -> a -> MutantS
- showMutantAsTuple :: ShowMutable a => [String] -> a -> a -> String
- showMutantDefinition :: ShowMutable a => [String] -> a -> a -> String
- showMutantNested :: ShowMutable a => [String] -> a -> a -> String
- showMutantBindings :: ShowMutable a => [String] -> a -> a -> String
- deriveMutable :: Name -> DecsQ
- deriveMutableE :: [Name] -> Name -> DecsQ
- module Test.FitSpec.TestTypes
- module Test.LeanCheck
Encoding properties
type Property = [([String], Bool)] Source #
An encoded representation of a property suitable for use by FitSpec.
Each list of strings is a printable representation of one possible choice of argument values for the property. Each boolean indicate whether the property holds for this choice.
Configuring reports
Extra arguments / configuration for reportWith
.
See args
for default values.
Args | |
|
data ShowMutantAs Source #
How to show mutants. Use this to fill showMutantAs
.
Default arguments for reportWith
:
nMutants = 500
, start with 500 mutantsnTests = 1000
, start with 1000 test valuestimeout = 5
, keep incresing the number of mutants until 5 seconds elapsenames = []
, default function call template:
["f x y z", "g x y z", "h x y z", ...]
fixargs :: Int -> Int -> Args Source #
Non timed-out default arguments. Make conjectures based on a fixed number of mutants and tests, e.g.:
reportWith (fixargs 100 200) f pmap
This is just a shorthand, see:
fixargs nm nt = args { nMutants = nm, nTests = nt, timeout = 0 }
(fixargs nm nt) { nMutants = 500, nTests = 1000, timeout = 5 } = args
Reporting results
report :: (Mutable a, ShowMutable a) => a -> (a -> [Property]) -> IO () Source #
Report results generated by FitSpec.
Uses standard configuration (see args
).
Needs a function to be mutated and a property map.
Example (specification of boolean negation):
properties not = [ property $ \p -> not (not p) == p , property $ \p -> not (not (not p)) == not p ] main = report not properties
reportWith :: (Mutable a, ShowMutable a) => Args -> a -> (a -> [Property]) -> IO () Source #
reportWithExtra :: (Mutable a, ShowMutable a) => [a] -> Args -> a -> (a -> [Property]) -> IO () Source #
Same as reportWith
, but accepts a list of manually defined (extra)
mutants to be tested alongside those automatically generated.
Parsing command line arguments
mainWith :: (Mutable a, ShowMutable a) => Args -> a -> (a -> [Property]) -> IO () Source #
Same as reportWith
, but allow overriding of configuration via command
line arguments.
defaultMain :: (Mutable a, ShowMutable a) => a -> (a -> [Property]) -> IO () Source #
Same as report
, but allow configuration via command line arguments.
Mutable types
class Mutable a where Source #
This typeclass is similar to Listable
.
A type is Mutable
when there exists a function that
is able to list mutations of a value.
Ideally: list all possible values without repetitions.
Instances are usually defined by a mutiers
function that
given a value, returns tiers of mutants of that value:
the first tier contains the equivalent mutant, of size 0,
the second tier contains mutants of size 1,
the third tier contains mutants of size 2,
and so on.
The equivalent mutant is the actual function without mutations.
The size of a mutant is given by the sum of: the number of mutated points (relations) and the sizes of mutated arguments and results.
To get only inequivalent mutants,
just take the tail
of either mutants
or mutiers
:
tail mutants
tail mutiers
Given that the underlying Listable
enumeration has no repetitions,
parametric instances defined in this file will have no repeated mutants.
Mutable Bool Source # | mutants True = [True,False] |
Mutable Char Source # | |
Mutable Int Source # | mutants 3 = [3,0,1,2,4,5,6,7,8,9,... |
Mutable () Source # | mutants () = [()] |
(Eq a, Listable a) => Mutable [a] Source # | mutants [0] = [ [0], [], [0,0], [1], ... |
(Eq a, Listable a) => Mutable (Maybe a) Source # | mutants (Just 0) = [Just 0, Nothing, ... |
(Eq a, Listable a, Mutable b) => Mutable (a -> b) Source # | mutants not = [ not , \p -> case p of False -> False; _ -> not p , \p -> case p of True -> True; _ -> not p , \p -> case p of False -> False; True -> True ] |
(Mutable a, Mutable b) => Mutable (a, b) Source # | mutants (0,1) = [(0,1),(0,0),(1,1),(0,-1),...] |
(Mutable a, Mutable b, Mutable c) => Mutable (a, b, c) Source # | |
(Mutable a, Mutable b, Mutable c, Mutable d) => Mutable (a, b, c, d) Source # | |
(Mutable a, Mutable b, Mutable c, Mutable d, Mutable e) => Mutable (a, b, c, d, e) Source # | |
(Mutable a, Mutable b, Mutable c, Mutable d, Mutable e, Mutable f) => Mutable (a, b, c, d, e, f) Source # | For Mutable tuple instances greater than sixtuples, see FitSpec.Mutable.Tuples. Despite being hidden in this Haddock documentation, 7-tuples up to 12-tuples are exported by Test.FitSpec. |
mutiersEq :: (Listable a, Eq a) => a -> [[a]] Source #
Implementation of mutiers
for non-functional data types.
Use this to create instances for user-defined data types, e.g.:
instance MyData where mutiers = mutiersEq
and for parametric datatypes:
instance (Eq a, Eq b) => MyDt a b where mutiers = mutiersEq
Examples:
mutiersEq True = [[True], [False]] mutiersEq 2 = [[2], [0], [1], [], [3], [4], [5], [6], [7], [8], [9], ...] mutiersEq [1] = [[[1]], [[]], [[0]], [[0,0]], [[0,0,0],[0,1],[1,0],[-1]], ...]
class ShowMutable a where Source #
Types that can have their mutation shown.
Has only one function mutantS
that returns a simple AST (MutantS
)
representing the mutant. A standard implementation of mutantS
for Eq
types is given by mutantSEq
.
ShowMutable Bool Source # | |
ShowMutable Char Source # | |
ShowMutable Int Source # | |
ShowMutable () Source # | |
(Eq a, Show a) => ShowMutable [a] Source # | |
(Eq a, Show a) => ShowMutable (Maybe a) Source # | |
(Listable a, Show a, ShowMutable b) => ShowMutable (a -> b) Source # | |
(ShowMutable a, ShowMutable b) => ShowMutable (a, b) Source # | |
(ShowMutable a, ShowMutable b, ShowMutable c) => ShowMutable (a, b, c) Source # | |
(ShowMutable a, ShowMutable b, ShowMutable c, ShowMutable d) => ShowMutable (a, b, c, d) Source # | |
(ShowMutable a, ShowMutable b, ShowMutable c, ShowMutable d, ShowMutable e) => ShowMutable (a, b, c, d, e) Source # | |
(ShowMutable a, ShowMutable b, ShowMutable c, ShowMutable d, ShowMutable e, ShowMutable f) => ShowMutable (a, b, c, d, e, f) Source # | |
mutantSEq :: (Eq a, Show a) => a -> a -> MutantS Source #
For a given type Type
instance of Eq
and Show
,
define the ShowMutable
instance as:
instance ShowMutable Type where mutantS = mutantSEq
showMutantAsTuple :: ShowMutable a => [String] -> a -> a -> String Source #
Show a Mutant as a tuple of lambdas.
> putStrLn $ showMutantAsTuple ["p && q","not p"] ((&&),not) ((||),id) ( \p q -> case (p,q) of (False,False) -> True _ -> p && q , \p -> case p of False -> False True -> True _ -> not p )
Can be easily copy pasted into an interactive session for manipulation.
On GHCi, use :{
and :}
to allow multi-line expressions and definitions.
showMutantDefinition :: ShowMutable a => [String] -> a -> a -> String Source #
Show a Mutant as a new complete top-level definition, with a prime appended to the name of the mutant.
> putStrLn $ showMutantDefinition ["p && q","not p"] ((&&),not) ((==),id) False &&- False = True p &&- q = p && q not' False = False not' True = True not' p = not p
showMutantNested :: ShowMutable a => [String] -> a -> a -> String Source #
Show a Mutant as a tuple of nested lambdas.
Very similar to showMutantAsTuple
, but the underlying data structure is
not flatten: so the output is as close as possible to the underlying
representation.
showMutantBindings :: ShowMutable a => [String] -> a -> a -> String Source #
Show a Mutant as the list of bindings that differ from the original function(s).
> putStrLn $ showMutantBindings ["p && q","not p"] ((&&),not) ((==),id) False && False = True not False = False not True = True
Can possibly be copied into the source of the original function for manipulation.
Automatic derivation
deriveMutable :: Name -> DecsQ Source #
Derives Mutable
, ShowMutable
and (optionally) Listable
instances
for a given type Name
.
Consider the following Stack
datatype:
data Stack a = Stack a (Stack a) | Empty
Writing
deriveMutable ''Stack
will automatically derive the following
Listable
, Mutable
and ShowMutable
instances:
instance Listable a => Listable (Stack a) where tiers = cons2 Stack \/ cons0 Empty instance (Eq a, Listable a) => Mutable a where mutiers = mutiersEq instance (Eq a, Show a) => ShowMutable a where mutantS = mutantSEq
If a Listable
instance already exists, it is not derived.
(cf.: deriveListable
)
Needs the TemplateHaskell
extension.
deriveMutableE :: [Name] -> Name -> DecsQ Source #
Derives a Mutable instance for a given type Name
using a given context for all type variables.
Re-export modules
module Test.FitSpec.TestTypes
module Test.LeanCheck