Haskell Stream Processor
Haskell Stream Processor is a command line utility to process streams
using Haskell code.
There are many reasons why Haskell is suitable for stream processing from the
command line. Code written in Haskell is concise thanks to a clean syntax and
the type inference which allows code without type decoration. Also it is very
easy to define one-line transformations by combining functions.
For example:
hsp "L.map (L.head . words) . lines"
prints the first word of each line of the input stream.
Installation
From the project directory
cabal install
This will compile and install the executable hsp and the library
HSProcess.Representable.
Usage
hsp supports different modes:
Evaluate an expression
It is possible to use hsp to evaluate a user expression without
input using the option -e:
hsp -e "1"
Work on the stream
The standard mode of hsp process the whole stream. It accepts a
string representing a transformation from the stream, that has type
Data.ByteString.Lazy.ByteString, to some value with type that is an
instance of Rows:
ByteString -> Rows a
Rows is a special case of Show for representing data on the
command line . For example, to print on stdout what it gets from stdin:
hsp "id"
Split stream in chunks and process them
Many times, stream processing is about splitting the stream on some delimiter,
like '\n', and process each chunk of data. With the standard mode of
hsp this can be achieved using the split function of ByteString:
hsp "L.filter (not . null) . split '\n'"
This happens so often that hsp has a mode to split automatically the
stream on a delimiter using -d [<delimiter>]. If
<delimiter> is omitted, then it is set to \n. With -d, the
function provided must have type:
[ByteString] -> Rows a
The command before can be rewritten as:
hsp -d "L.filter (not . null)"
Map a function on each chunk of data
A specific case of hsp -d <delimiter> is hsp -d <delimiter> -m that
is equivalent of mapping the supplied function to the input list. In this case
the function must have type:
ByteString -> Row a
For example, to take the first word of each line:
hsp -m "L.head . words"
When -m is specified, -d can be omitted and the delimiter is
automatically set to \n.
Configuration
Haskell Stream Processor is a command line utility and for this reason it needs
informations, like which modules should be loaded, that cannot be easily passed
as arguments. There are two configuration files located under
$HOME/.hsp, one to import modules and one to import user defined
functions.
Modules
Haskell Stream Processor reads a list of modules to load from the file
$HOME/.hsp/modules. Each line of this file is composed by the name of a
module eventually followed by a space and it's qualified name. An example could
be:
Control.Monad
Data.List L
which means that all the functions from Control.Monad and Data.List
will be available to the user, but for Data.List functions you must
qualify them with L..
There are some modules that are loaded automatically without qualification. In particular,
the module Data.ByteString.Lazy.Char8 is automatically loaded
because hsp works on lazy bytestrings. This means functions like that
in Prelude work on list, like map, in hsp work on
ByteStrings. Same for function that work on String.
Note that Prelude is loaded with the qualified name P, so its
functions are not directly visible.
An example of module file can be found in the example directory.
User defined functions
It is possible to define new function to be used in Haskell Stream Processor
inside the file $HOME/.hsp/toolkit.hs.
An example of toolkit can be found in the example directory.
Differences with the Glasgow Haskell Compiler
It is already possible to evaluate an function using the
Glasgow Haskell Compiler using the option
-e and by passing the custom function to interact:
ghc -e "interact id"
The main differences are that Haskell Stream Processor works on (lazy)
ByteString instead of the slower String, it can load modules
automatically from the module file and can load user defined functions
from the toolkit.hs file. Also, Haskell Stream Processor supports
different modes from working on the entire stream, like working on each line.
Examples
In all the examples, Data.ByteString is loaded without qualification
whereas Data.List is qualified as L. The function match is an
alias for Text.Regex.Posix.=~.
Evaluate 2^100:
hsp -e "2^100"
Print numbers from 1 to 100:
hsp -e "[1 .. 100]"
Take the first line of a stream:
... | hsp -d "L.take 1"
Take the last two lines of a stream:
... | hsp -d "L.reverse . L.take 2 . L.reverse"
Print the 10th element of each line:
... | hsp -m "(L.!! 10) . words"
Print the elements from the 2nd to the 20th of each line:
... | hsp -m "L.take 20 . L.drop 1 . words"
Get the number of words:
... | hsp -d "L.length . L.concatMap words"
Get the number of lines:
... | hsp -d "L.length"
Sort integers and remove duplicates:
... | hsp -d "L.nub . L.sort . L.map asInt"
Sum the 2nd elements of every line:
... | hsp -d "P.sum . L.map (asFloat . (L.!! 1) . words)"
Split each line on a delimiter ':' and print the second element:
... | hsp -m "(L.!! 1) . split ':'"
Remove empty lines:
... | hsp -d "L.filter (not . null)"
Filter lines that match a pattern:
... | hsp -d "L.filter (`match` "t\\w\\wt")"
