Portability	non-portable
Stability	experimental
Maintainer	ekmett@gmail.com
Safe Haskell	Trustworthy

Text.Parser.Token

Contents

Token Parsing
Identifiers
- Lenses and Traversals

Description

Parsers that comprehend whitespace and identifier styles

 idStyle    = haskellIdents { styleReserved = ... }
 identifier = ident idStyle
 reserved   = reserve idStyle

Synopsis

Token Parsing

whiteSpace :: TokenParsing m => m ()Source

Skip zero or more bytes worth of white space. More complex parsers are free to consider comments as white space.

charLiteral :: TokenParsing m => m Char Source

This token parser parses a single literal character. Returns the literal character value. This parsers deals correctly with escape sequences. The literal character is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

stringLiteral :: (TokenParsing m, IsString s) => m sSource

This token parser parses a literal string. Returns the literal string value. This parsers deals correctly with escape sequences and gaps. The literal string is parsed according to the grammar rules defined in the Haskell report (which matches most programming languages quite closely).

stringLiteral' :: (TokenParsing m, IsString s) => m sSource

This token parser behaves as stringLiteral, but for single-quoted strings.

natural :: TokenParsing m => m Integer Source

This token parser parses a natural number (a positive whole number). Returns the value of the number. The number can be specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the Haskell report.

integer :: TokenParsing m => m Integer Source

This token parser parses an integer (a whole number). This parser is like natural except that it can be prefixed with sign (i.e. '-' or '+'). Returns the value of the number. The number can be specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the Haskell report.

double :: TokenParsing m => m Double Source

This token parser parses a floating point value. Returns the value of the number. The number is parsed according to the grammar rules defined in the Haskell report.

naturalOrDouble :: TokenParsing m => m (Either Integer Double)Source

This token parser parses either natural or a float. Returns the value of the number. This parsers deals with any overlap in the grammar rules for naturals and floats. The number is parsed according to the grammar rules defined in the Haskell report.

integerOrDouble :: TokenParsing m => m (Either Integer Double)Source

This token parser is like naturalOrDouble, but handles leading - or +.

symbol :: TokenParsing m => String -> m String Source

Token parser symbol s parses string s and skips trailing white space.

textSymbol :: TokenParsing m => Text -> m Text Source

Token parser textSymbol t parses text s and skips trailing white space.

symbolic :: TokenParsing m => Char -> m Char Source

Token parser symbolic s parses char s and skips trailing white space.

parens :: TokenParsing m => m a -> m aSource

Token parser parens p parses p enclosed in parenthesis, returning the value of p.

braces :: TokenParsing m => m a -> m aSource

Token parser braces p parses p enclosed in braces ('{' and '}'), returning the value of p.

angles :: TokenParsing m => m a -> m aSource

Token parser angles p parses p enclosed in angle brackets ('<' and '>'), returning the value of p.

brackets :: TokenParsing m => m a -> m aSource

Token parser brackets p parses p enclosed in brackets ('[' and ']'), returning the value of p.

comma :: TokenParsing m => m Char Source

Token parser comma parses the character ',' and skips any trailing white space. Returns the string ",".

colon :: TokenParsing m => m Char Source

Token parser colon parses the character ':' and skips any trailing white space. Returns the string ":".

dot :: TokenParsing m => m Char Source

Token parser dot parses the character '.' and skips any trailing white space. Returns the string ".".

semiSep :: TokenParsing m => m a -> m [a]Source

Token parser semiSep p parses zero or more occurrences of p separated by semi. Returns a list of values returned by p.

semiSep1 :: TokenParsing m => m a -> m [a]Source

Token parser semiSep1 p parses one or more occurrences of p separated by semi. Returns a list of values returned by p.

commaSep :: TokenParsing m => m a -> m [a]Source

Token parser commaSep p parses zero or more occurrences of p separated by comma. Returns a list of values returned by p.

commaSep1 :: TokenParsing m => m a -> m [a]Source

Token parser commaSep1 p parses one or more occurrences of p separated by comma. Returns a list of values returned by p.

Token Parsing Class

class CharParsing m => TokenParsing m whereSource

Additional functionality that is needed to tokenize input while ignoring whitespace.

Methods

someSpace :: m ()Source

Usually, someSpace consists of one or more occurrences of a space. Some parsers may choose to recognize line comments or block (multi line) comments as white space as well.

nesting :: m a -> m aSource

Called when we enter a nested pair of symbols. Overloadable to enable disabling layout

semi :: m Char Source

The token parser |semi| parses the character ';' and skips any trailing white space. Returns the character ';'. Overloadable to permit automatic semicolon insertion or Haskell-style layout.

highlight :: Highlight -> m a -> m aSource

Tag a region of parsed text with a bit of semantic information. Most parsers won't use this, but it is indispensible for highlighters.

token :: m a -> m aSource

token p first applies parser p and then the whiteSpace parser, returning the value of p. Every lexical token (token) is defined using token, this way every parse starts at a point without white space. Parsers that use token are called token parsers in this document.

The only point where the whiteSpace parser should be called explicitly is the start of the main parser in order to skip any leading white space.

Alternatively, one might define token as first parsing whiteSpace and then parser p. By parsing whiteSpace first, the parser is able to return before parsing additional whiteSpace, improving laziness.

 mainParser  = sum <$ whiteSpace <*> many (token digit) <* eof

Instances

TokenParsing ReadP
Chunk t => TokenParsing (Parser t)
(TokenParsing m, MonadPlus m) => TokenParsing (IdentityT m)
TokenParsing m => TokenParsing (Unlined m)
TokenParsing m => TokenParsing (Unspaced m)
TokenParsing m => TokenParsing (Unhighlighted m)
(TokenParsing m, MonadPlus m) => TokenParsing (ReaderT e m)
(TokenParsing m, MonadPlus m) => TokenParsing (StateT s m)
(TokenParsing m, MonadPlus m) => TokenParsing (StateT s m)
(TokenParsing m, MonadPlus m, Monoid w) => TokenParsing (WriterT w m)
(TokenParsing m, MonadPlus m, Monoid w) => TokenParsing (WriterT w m)
Stream s m Char => TokenParsing (ParsecT s u m)
(TokenParsing m, MonadPlus m, Monoid w) => TokenParsing (RWST r w s m)
(TokenParsing m, MonadPlus m, Monoid w) => TokenParsing (RWST r w s m)

Token Parsing Transformers

newtype Unspaced m a Source

This is a parser transformer you can use to disable the automatic trailing space consumption of a Token parser.

Constructors

Unspaced
Fields runUnspaced :: m a

Instances

MonadTrans Unspaced
Monad m => Monad (Unspaced m)
Functor m => Functor (Unspaced m)
MonadPlus m => MonadPlus (Unspaced m)
Applicative m => Applicative (Unspaced m)
Alternative m => Alternative (Unspaced m)
Parsing m => Parsing (Unspaced m)
CharParsing m => CharParsing (Unspaced m)
TokenParsing m => TokenParsing (Unspaced m)

newtype Unlined m a Source

This is a parser transformer you can use to disable the automatic trailing newline (but not whitespace-in-general) consumption of a Token parser.

Constructors

Unlined
Fields runUnlined :: m a

Instances

MonadTrans Unlined
Monad m => Monad (Unlined m)
Functor m => Functor (Unlined m)
MonadPlus m => MonadPlus (Unlined m)
Applicative m => Applicative (Unlined m)
Alternative m => Alternative (Unlined m)
Parsing m => Parsing (Unlined m)
CharParsing m => CharParsing (Unlined m)
TokenParsing m => TokenParsing (Unlined m)

newtype Unhighlighted m a Source

This is a parser transformer you can use to disable syntax highlighting over a range of text you are parsing.

Constructors

Unhighlighted
Fields runUnhighlighted :: m a

Instances

MonadTrans Unhighlighted
Monad m => Monad (Unhighlighted m)
Functor m => Functor (Unhighlighted m)
MonadPlus m => MonadPlus (Unhighlighted m)
Applicative m => Applicative (Unhighlighted m)
Alternative m => Alternative (Unhighlighted m)
Parsing m => Parsing (Unhighlighted m)
CharParsing m => CharParsing (Unhighlighted m)
TokenParsing m => TokenParsing (Unhighlighted m)

Non-Token Parsers

decimal :: TokenParsing m => m Integer Source

Parses a positive whole number in the decimal system. Returns the value of the number.

This parser does NOT swallow trailing whitespace

hexadecimal :: TokenParsing m => m Integer Source

Parses a positive whole number in the hexadecimal system. The number should be prefixed with "x" or "X". Returns the value of the number.

This parser does NOT swallow trailing whitespace

octal :: TokenParsing m => m Integer Source

Parses a positive whole number in the octal system. The number should be prefixed with "o" or "O". Returns the value of the number.

This parser does NOT swallow trailing whitespace

characterChar :: TokenParsing m => m Char Source

This parser parses a character literal without the surrounding quotation marks.

This parser does NOT swallow trailing whitespace

integer' :: TokenParsing m => m Integer Source

This parser parses an integer (a whole number). This parser is like natural except that it can be prefixed with sign (i.e. '-' or '+'). Returns the value of the number. The number can be specified in decimal, hexadecimal or octal. The number is parsed according to the grammar rules in the Haskell report.

This parser does NOT swallow trailing whitespace.

Also, unlike the integer parser, this parser does not admit spaces between the sign and the number.

Identifiers

data IdentifierStyle m Source

Used to describe an input style for constructors, values, operators, etc.

Constructors

IdentifierStyle
Fields _styleName :: String _styleStart :: m Char _styleLetter :: m Char _styleReserved :: HashSet String _styleHighlight :: Highlight _styleReservedHighlight :: Highlight

liftIdentifierStyle :: (MonadTrans t, Monad m) => IdentifierStyle m -> IdentifierStyle (t m)Source

Lift an identifier style into a monad transformer

Using over from the lens package:

liftIdentifierStyle = over styleChars lift

ident :: (TokenParsing m, Monad m, IsString s) => IdentifierStyle m -> m sSource

Parse a non-reserved identifier or symbol

reserve :: (TokenParsing m, Monad m) => IdentifierStyle m -> String -> m ()Source

parse a reserved operator or identifier using a given style

reserveText :: (TokenParsing m, Monad m) => IdentifierStyle m -> Text -> m ()Source

parse a reserved operator or identifier using a given style given Text.

Lenses and Traversals

styleName :: Functor f => (String -> f String) -> IdentifierStyle m -> f (IdentifierStyle m)Source

This lens can be used to update the name for this style of identifier.

styleName :: Lens' (IdentifierStyle m) String

styleStart :: Functor f => (m Char -> f (m Char)) -> IdentifierStyle m -> f (IdentifierStyle m)Source

This lens can be used to update the action used to recognize the first letter in an identifier.

styleStart :: Lens' (IdentifierStyle m) (m Char)

styleLetter :: Functor f => (m Char -> f (m Char)) -> IdentifierStyle m -> f (IdentifierStyle m)Source

This lens can be used to update the action used to recognize subsequent letters in an identifier.

styleLetter :: Lens' (IdentifierStyle m) (m Char)

styleChars :: Applicative f => (m Char -> f (n Char)) -> IdentifierStyle m -> f (IdentifierStyle n)Source

This is a traversal of both actions in contained in an IdentifierStyle.

styleChars :: Traversal (IdentifierStyle m) (IdentifierStyle n) (m Char) (n Char)

styleReserved :: Functor f => (HashSet String -> f (HashSet String)) -> IdentifierStyle m -> f (IdentifierStyle m)Source

This is a lens that can be used to modify the reserved identifier set.

styleReserved :: Lens' (IdentifierStyle m) (HashSet String)

styleHighlight :: Functor f => (Highlight -> f Highlight) -> IdentifierStyle m -> f (IdentifierStyle m)Source

This is a lens that can be used to modify the highlight used for this identifier set.

styleHighlight :: Lens' (IdentifierStyle m) Highlight

styleReservedHighlight :: Functor f => (Highlight -> f Highlight) -> IdentifierStyle m -> f (IdentifierStyle m)Source

This is a lens that can be used to modify the highlight used for reserved identifiers in this identifier set.

styleReservedHighlight :: Lens' (IdentifierStyle m) Highlight

styleHighlights :: Applicative f => (Highlight -> f Highlight) -> IdentifierStyle m -> f (IdentifierStyle m)Source

This is a traversal that can be used to modify the highlights used for both non-reserved and reserved identifiers in this identifier set.

styleHighlights :: Traversal' (IdentifierStyle m) Highlight