hw-dsv: Unbelievably fast streaming DSV file parser
Flags
Automatic Flags
| Name | Description | Default |
|---|---|---|
| avx2 | Enable avx2 instruction set | Disabled |
| bmi2 | Enable bmi2 instruction set | Disabled |
| sse42 | Enable SSE 4.2 optimisations. | Enabled |
Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info
Downloads
- hw-dsv-0.3.5.tar.gz [browse] (Cabal source package)
- Package description (revised from the package)
Note: This package has metadata revisions in the cabal description newer than included in the tarball. To unpack the package including the revisions, use 'cabal get'.
Maintainer's Corner
For package maintainers and hackage trustees
Candidates
| Versions [RSS] | 0.1.0.0, 0.2, 0.2.1, 0.3.0, 0.3.1, 0.3.2, 0.3.3, 0.3.4, 0.3.5, 0.3.6, 0.3.7, 0.3.8, 0.4.0, 0.4.1.0, 0.4.1.1, 0.4.1.2 |
|---|---|
| Change log | ChangeLog.md |
| Dependencies | base (>=4.7 && <5), bits-extra (>=0.0.1.2 && <0.1), bytestring (>=0.10 && <0.11), deepseq (>=1.4 && <1.5), generic-lens (>=1.0.0.0 && <1.3), ghc-prim, hedgehog (>=0.5 && <1.1), hw-bits (>=0.7.0.2 && <0.8), hw-dsv, hw-prim (>=0.6.2.14 && <0.7), hw-rankselect (>=0.12.0.2 && <0.14), hw-rankselect-base (>=0.3.2.0 && <0.4), hw-simd (>=0.1.1.3 && <0.2), lens (>=4.15 && <5), optparse-applicative (>=0.13 && <0.16), resourcet (>=1.1 && <1.3), semigroups (>=0.8.4 && <0.19), transformers (>=0.4 && <0.6), vector (>=0.12.0.1 && <0.13) [details] |
| Tested with | ghc ==8.6.5, ghc ==8.4.4, ghc ==8.2.2, ghc ==8.0.2 |
| License | BSD-3-Clause |
| Copyright | 2018-2019 John Ky |
| Author | John Ky |
| Maintainer | newhoggy@gmail.com |
| Revised | Revision 2 made by haskellworks at 2019-09-09T01:40:44Z |
| Category | Text, CSV, SIMD, Succinct Data Structures, Data Structures |
| Home page | https://github.com/haskell-works/hw-dsv#readme |
| Bug tracker | https://github.com/haskell-works/hw-dsv/issues |
| Source repo | head: git clone https://github.com/haskell-works/hw-dsv |
| Uploaded | by haskellworks at 2019-04-03T11:19:13Z |
| Distributions | |
| Reverse Dependencies | 3 direct, 3 indirect [details] |
| Executables | hw-dsv |
| Downloads | 8445 total (67 in the last 30 days) |
| Rating | (no votes yet) [estimated by Bayesian average] |
| Your Rating | |
| Status | Docs available [build log] Last success reported on 2019-04-03 [all 1 reports] |
Readme for hw-dsv-0.3.5
[back to package description]hw-dsv
Unbelievably fast streaming DSV file parser that reads based on succinct data structures.
This library will use support for some BMI2 or AVX2 CPU instructions on some x86 based
CPUs if compiled with the appropriate flags on ghc-8.4.1 or later.
Compilation
Pre-requisites:
- Install Haskell Stack
It is sufficient to build, test and benchmark the library as follows for basic performance. The library will be compiled to use broadword implementation of rank & select, which has reasonable performance.
stack build
stack test
stack bench
For best performance, add the bmi2 and avx2 flag to target the BMI2 and AVS2 instruction sets:
stack build --flag bits-extra:bmi2 --flag hw-rankselect-base:bmi2 --flag hw-rankselect:bmi2 --flag hw-simd:bmi2 --flag hw-simd:avx2 --flag hw-dsv:bmi2 --flag hw-dsv:avx2
stack test --flag bits-extra:bmi2 --flag hw-rankselect-base:bmi2 --flag hw-rankselect:bmi2 --flag hw-simd:bmi2 --flag hw-simd:avx2 --flag hw-dsv:bmi2 --flag hw-dsv:avx2
stack bench --flag bits-extra:bmi2 --flag hw-rankselect-base:bmi2 --flag hw-rankselect:bmi2 --flag hw-simd:bmi2 --flag hw-simd:avx2 --flag hw-dsv:bmi2 --flag hw-dsv:avx2
stack install --flag bits-extra:bmi2 --flag hw-rankselect-base:bmi2 --flag hw-rankselect:bmi2 --flag hw-simd:bmi2 --flag hw-simd:avx2 --flag hw-dsv:bmi2 --flag hw-dsv:avx2
For slightly older CPUs, add only the bmi2 flag to target the BMI2 instruction set:
stack build --flag bits-extra:bmi2 --flag hw-rankselect-base:bmi2 --flag hw-rankselect:bmi2 --flag hw-simd:bmi2 --flag hw-dsv:bmi2
stack test --flag bits-extra:bmi2 --flag hw-rankselect-base:bmi2 --flag hw-rankselect:bmi2 --flag hw-simd:bmi2 --flag hw-dsv:bmi2
stack bench --flag bits-extra:bmi2 --flag hw-rankselect-base:bmi2 --flag hw-rankselect:bmi2 --flag hw-simd:bmi2 --flag hw-dsv:bmi2
stack install --flag bits-extra:bmi2 --flag hw-rankselect-base:bmi2 --flag hw-rankselect:bmi2 --flag hw-simd:bmi2 --flag hw-dsv:bmi2
Benchmark results
The following benchmark shows the kinds of performance gain that can be expected from enabling the BMI2 instruction set for CPU targets that support them. Benchmarks were run on 2.9 GHz Intel Core i7, macOS High Sierra.
With BMI2 disabled:
$ stack install
$ cat 7g.csv | pv -t -e -b -a | hw-dsv query-lazy -k 1 -k 2 -d , -e '|' > /dev/null
7.08GiB 0:07:25 [16.3MiB/s]
With BMI2 and AVX2 enabled:
$ stack install --flag bits-extra:bmi2 --flag hw-bits:bmi2 --flag hw-rankselect-base:bmi2 --flag hw-rankselect:bmi2 --flag hw-dsv:bmi2 --flag hw-dsv:avx2
$ cat 7gb.csv | pv -t -e -b -a | hw-dsv query-lazy -k 1 -k 2 -d , -e '|' > /dev/null
7.08GiB 0:00:39 [ 181MiB/s]
With only BMI2 enabled:
$ stack install --flag bits-extra:bmi2 --flag hw-bits:bmi2 --flag hw-rankselect-base:bmi2 --flag hw-rankselect:bmi2 --flag hw-dsv:bmi2
$ cat 7gb.csv | pv -t -e -b -a | hw-dsv query-lazy -k 1 -k 2 -d , -e '|' > /dev/null
7.08GiB 0:00:43 [ 165MiB/s]
hw-dsv command line options
The hw-dsv application accepts 1-based column indexes rather than 0-based. The library is 0-based.
Using hw-dsv as a library
{-# LANGUAGE ScopedTypeVariables #-}
module Example where
import qualified Data.ByteString.Lazy as LBS
import qualified Data.Vector as DV
import qualified HaskellWorks.Data.Dsv.Lazy.Cursor as SVL
example :: IO ()
example = do
bs <- LBS.readFile "sample.csv"
let c = SVL.makeCursor ',' bs
let rows :: [DV.Vector LBS.ByteString] = SVL.toListVector c
return ()