hsbencher: Launch and gather data from Haskell and non-Haskell benchmarks.

[ bsd3, development, library ] [ Propose Tags ]

Benchmark frameworks are usually very specific to the host language/environment. Hence they are usually about as reusable as compiler passes (that is, not).

Nevertheless, hsbencher is an attempt at a reusable benchmark framework. It knows fairly little about what the benchmarks do, and is mostly concerned with defining and iterating through configuration spaces (e.g. varying the number of threads), and managing the data that results.

Benchmark data is stored in simple text files, and optionally uploaded via pluggable backend packages such as `hsbencher-fusion`, which uploads to Google Fusion Tables.

hsbencher attempts to stradle the divide between language-specific and language-agnostic by having an extensible set of BuildMethods. As shipped, hsbencher knows a little about cabal, ghc, and less about Make, but it can be taught more.

The general philosophy is to have benchmarks follow a simple protocol, for example printing out a line "SELFTIMED: 3.3" if they wish to report their own timing, in seconds. The focus is on benchmarks that run long enough to run in their own process. This is typical of parallelism benchmarks and different than the fine-grained benchmarks that are well supported by Criterion.

hsbencher is used by creating a script or executable that imports HSBencher and provides a list of benchmarks, each of which is decorated with its parameter space. Below is a minimal example that creates a two-configuration parameter space:

import HSBencher
main = defaultMainModifyConfig $ addBenchmarks
.      [ mkBenchmark "bench1/bench1.cabal" ["1000"] $
.        Or [ Set NoMeaning (RuntimeParam "+RTS -qa -RTS")
.            , Set NoMeaning (RuntimeEnv "HELLO" "yes") ] ]

The output would appear as in this gist: https://gist.github.com/rrnewton/5667800

More examples can be found here: https://github.com/rrnewton/HSBencher/tree/master/hsbencher/example

ChangeLog:

  • (1.3.8) Added --skipto and --runid arguments

  • (1.3.4) Added ability to prune benchmarks with patterns on command line.

  • (1.4.2) Breaking changes, don't use Benchmark constructor directly. Use mkBenchmark.

  • (1.5) New columns in schema.

  • (1.8) Backend plugins, hsbencher-fusion package factored out.

  • (1.15) Add systemCleaner field to Config

  • (1.17) Add cpu affinity control; lspci off by default

  • (1.18) Fix COMPILE_FLAGS upload; upload "unwords" of COMPILE_FLAGS to avoid quotes in output.

  • (1.19.1) Change the semantics of naked runtime args to conjunction, not disjunction. Also allow filtering of benchmarks by BenchSpace as well.

  • (1.20) Add "--bindir" command line argument.

Flags

Manual Flags

NameDescriptionDefault
hydra

Add support for (and dependency on) the hydra-print library.

Disabled

Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info

Downloads

Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees

Candidates

Versions [RSS] 1.0, 1.1, 1.1.0.1, 1.1.0.2, 1.2, 1.3.1, 1.3.4, 1.3.6, 1.3.8, 1.3.9, 1.5.1, 1.5.3, 1.5.3.1, 1.8.0.4, 1.12, 1.14, 1.14.1, 1.20, 1.20.0.1, 1.20.0.2, 1.20.0.3, 1.20.0.5
Dependencies async (>=2.0), base (>=4.5 && <4.9), bytestring, containers, data-default (>=0.5.3), directory, filepath, GenericPretty (>=1.2), io-streams (>=1.1), mtl, process (>=1.2), random, time, unix [details]
License BSD-3-Clause
Copyright (c) Ryan Newton 2013
Author Ryan Newton, Joel Svensson
Maintainer bo.joel.svensson@gmail.com
Category Development
Source repo head: git clone https://github.com/rrnewton/HSBencher
Uploaded by RyanNewton at 2015-04-26T12:18:51Z
Distributions
Reverse Dependencies 3 direct, 0 indirect [details]
Downloads 15312 total (63 in the last 30 days)
Rating (no votes yet) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs available [build log]
Last success reported on 2015-04-26 [all 1 reports]