A monad for writing pure tokenizers in an imperative-looking way.

Main idea: You walk through the input string like a turtle, and everytime you find a token boundary, you call emit. If some specific kinds of tokens should be suppressed, you can discard them instead (or filter afterwards).

This module supports is specialized for lazy ByteStrings. The module Control.Monad.Tokenizer provides more general types, but does not export a Tokenizable instance for ByteStrings, as its implementation depends on the encoding. This module assumes ASCII encoding (you have been warned!).

Example for a simple tokenizer, that splits words by whitespace and discards stop symbols:

tokenizeWords :: ByteString -> [ByteString]
tokenizeWords = runTokenizer $ untilEOT $ do
  c <- pop
  if isStopSym c
    then discard
    else if c `elem` (" \t\r\n" :: [Char])
         then discard
         else do
           walkWhile (\c -> (c=='_') || not (isSpace c || isPunctuation' c))
           emit