tokenizer-monad-0.2.2.0: An efficient and easy-to-use tokenizer monad.

Safe HaskellSafe
LanguageHaskell2010

Control.Monad.Tokenizer.Class

Contents

Description

A monad class for writing pure tokenizers in an imperative-looking way.

Main idea: You walk through the input string like a turtle, and everytime you find a token boundary, you call emit. If some specific kinds of tokens should be suppressed, you can discard them instead (or filter afterwards).

An concrete instance of this class is provided in Control.Monad.Tokenizer.

Synopsis

Monad class

class Monad m => MonadTokenizer m where Source #

A monad for turtle tokenization.

Minimal complete definition

walk, walkBack, peek, restore, emit, discard, isEOT, lookAhead

Methods

walk :: m () Source #

Proceed to the next character

walkBack :: m () Source #

Walk back to the previous character, unless it was discarded/emitted.

peek :: m Char Source #

Peek the current character

restore :: m () Source #

Restore the state after the last emit/discard.

emit :: m () Source #

Break at the current position and emit the scanned token

discard :: m () Source #

Break at the current position and discard the scanned token

isEOT :: m Bool Source #

Have I reached the end of the input text?

lookAhead :: [Char] -> m Bool Source #

Check if the next input chars agree with the given string

pop :: m Char Source #

Peek the current character and proceed

untilEOT :: MonadTokenizer m => m () -> m () Source #

Repeat a given tokenizer as long as the end of text is not reached

Utilities

pop :: MonadTokenizer m => m Char Source #

Peek the current character and proceed

walkWhile :: MonadTokenizer m => (Char -> Bool) -> m () Source #

Proceed as long as a given function succeeds

walkFold :: MonadTokenizer m => a -> (Char -> a -> Maybe a) -> m () Source #

Proceed as long as a given fold returns Just (generalization of walkWhile)

Text types

class Tokenizable t where Source #

Text types that can be split by the Tokenizer monad. In this module, instances are provided for String, strict Text, and lazy Text. If you are dealing with ASCII ByteStrings, you can find instances in the modules Control.Monad.Tokenizer.Char8.Strict and Control.Monad.Tokenizer.Char8.Lazy

Methods

tnull :: t -> Bool Source #

thead :: t -> Char Source #

ttail :: t -> t Source #

ttake :: Int -> t -> t Source #

tdrop :: Int -> t -> t Source #

tlower :: t -> t Source #

Instances
Tokenizable ByteString Source #

Assuming ASCII encoding

Instance details

Defined in Control.Monad.Tokenizer.Char8.Lazy

Tokenizable ByteString Source #

Assuming ASCII encoding

Instance details

Defined in Control.Monad.Tokenizer.Char8.Strict

Tokenizable Text Source # 
Instance details

Defined in Control.Monad.Tokenizer.Class

Tokenizable Text Source # 
Instance details

Defined in Control.Monad.Tokenizer.Class

Tokenizable [Char] Source # 
Instance details

Defined in Control.Monad.Tokenizer.Class

Methods

tnull :: [Char] -> Bool Source #

thead :: [Char] -> Char Source #

ttail :: [Char] -> [Char] Source #

ttake :: Int -> [Char] -> [Char] Source #

tdrop :: Int -> [Char] -> [Char] Source #

tlower :: [Char] -> [Char] Source #