Documentation
tokenize :: Maybe Lexer -> String -> Either String TokensSource
Tokenize a string, returning either an error or a sequence
of tokens. If the first argument is Just
a lexer, use
the lexer to tokenize. If Nothing
, return a single Plain
token with the whole source. tokenize
is designed to be
used with lexerByName
or lexerByFilename
: for example,
tokenize (lexerByName "Haskell") input
lexerByName :: String -> Maybe LexerSource
Matches a lexer by name or alias (case-insensitive).
lexerByFilename :: String -> Maybe LexerSource
Matches a lexer by the filename of the source file.