Stability | experimental |
---|---|
Maintainer | Daniël de Kok <me@danieldk.eu> |
Safe Haskell | Safe-Infered |
This module provides functionality to perform approximate randomization tests (Noreen, 1989).
- data TestResult
- type RandWithError a = ErrorT String Rand a
- approxRandTest :: TestType -> TestStatistic -> Int -> Double -> Sample -> Sample -> Rand TestResult
- approxRandScores :: TestStatistic -> Int -> Sample -> Sample -> Rand [Double]
- approxRandPairTest :: TestType -> TestStatistic -> Int -> Double -> Sample -> Sample -> RandWithError TestResult
- approxRandPairScores :: TestStatistic -> Int -> Sample -> Sample -> RandWithError [Double]
- type TestStatistic = Sample -> Sample -> Double
- differenceMean :: TestStatistic
- meanDifference :: TestStatistic
- varianceRatio :: TestStatistic
Description
Approximate randomization tests rely on a simple premise: given a test statistic, if the null-hypothesis (the samples do not differ) is true, we can randomly swap values between samples without an (extreme) impact on the test statistic. Otherwise, the null-hypothesis must be rejected.
The test works by generating a given number of sample shuffles and computing the test statistic for each shuffle. If r is the number of shuffled samples where the test statistic is at least as high as the test statistic applied on the original samples; and N the number of shuffles, then the null-hypothesis is rejected iff (r + 1):(N + 1) < p-value (for one-sided tests).
Two kinds of test are supported:
- Paired sample (
approxRandPairTest
): values from samples are shuffled pair-wise. This requires the samples to have an equal length. - Unpaired sample (
approxRandTest
): values from samples are shuffled among both samples. Consequently the i-th element of one sample does not bear a relationship with the i-th element of the other sample. The shuffled samples retain the sizes of the original samples.
Both tests can be performed as a one-tailed or two-tailed test.
Examples
Both unpaired and paired sample tests use the Rand
monad to obtain
random numbers. We can obtain a pseudo-random number generator that
is seeded using the system clock using the
newPureMT
function (please refer to
the documentation of Pure64
for more
information):
prng <- newPureMT
Suppose that we have the samples s1
and s2
. We could now perform
a Two-Tailed randomization test with 10,000 shuffles and the mean
difference as the test statistic, by running approxRandTest
in the Rand
monad (at the p = 0.01 level):
evalRandom (approxRandTest TwoSided meanDifference 10000 0.01 s1 s2) prng
It is also possible to obtain the test scores of the shuffled samples directly (e.g. to inspect the distribution of test scores) using the 'approxRandScores'/'approxRandPiarScores' functions:
evalRandom (approxRandScores meanDifference 10000 0.01 s1 s2) prng
Data types
data TestResult Source
The result of hypothesis testing.
Significant Double | The null hypothesis should be rejected |
NotSignificant Double | Data is compatible with the null hypothesis |
type RandWithError a = ErrorT String Rand aSource
Computations with random numbers that can fail.
Approximate randomization tests
:: TestType | |
-> TestStatistic | Test statistic |
-> Int | Number of shuffled sample to create |
-> Double | The p-value at which to test (e.g. 0.05) |
-> Sample | First sample |
-> Sample | Second sample |
-> Rand TestResult | The test result |
Apply an approximate randomization test.
In approximate randomization tests, the values of two samples are shuffled among those samples. A test statistic is calculated for the original samples and the shuffled samples, to detect whether the difference of the samples is extreme or not.
:: TestStatistic | Test statistic |
-> Int | Number of shuffled samples to create |
-> Sample | First sample |
-> Sample | Second sample |
-> Rand [Double] | The scores of each shuffle |
Generate a given number of shuffled samples, and calculate the test score for each shuffle.
This function does not require the samples to have an equal length.
:: TestType | |
-> TestStatistic | Test statistic |
-> Int | Number of shuffled samples to create |
-> Double | The p-value at which to test (e.g. 0.05) |
-> Sample | First sample |
-> Sample | Second sample |
-> RandWithError TestResult | The test result |
Apply a pair-wise approximate randomization test.
In pair-wise approximate randomization tests the scores at a given index are swapped between samples with a probability of 0.5. Since swapping is pairwise, the samples should have the same length.
:: TestStatistic | Test statistic |
-> Int | Number of shuffled samples to create |
-> Sample | First sample |
-> Sample | Second sample |
-> RandWithError [Double] | The scores of each shuffle |
Generate a given number of pairwise shuffled samples, and calculate the test score for each shuffle.
Since the scores at a given index are swapped (with a probability of 0.5), the samples should have the same length.
Test statistics
type TestStatistic = Sample -> Sample -> DoubleSource
A test stastic calculates the difference between two samples. See
meanDifference
and varianceRatio
for examples.
differenceMean :: TestStatisticSource
Calculates the difference mean of two samples (mean(s1 - s2)). When the two samples do not have an equal length, the trailing elements of the longer vector are ignored.
meanDifference :: TestStatisticSource
Calculates the mean difference of two samples (mean(s1) - mean(s2)).
varianceRatio :: TestStatisticSource
Calculate the ratio of sample variances (var(s1) : var(s2)).