| Safe Haskell | None |
|---|---|
| Language | Haskell2010 |
JL.Tokenizer
Description
Duet syntax tokenizer.
Synopsis
- tokenize :: FilePath -> Text -> Either ParseError [(Token, Location)]
- tokensTokenizer :: Parser [(Token, Location)]
- tokenTokenizer :: [Char] -> Parser (Token, Location)
- spaces1 :: Parser ()
- ellipsis :: Int -> [Char] -> [Char]
- specialParsing :: (t1 -> t) -> Parser t1 -> String -> Parser (t, Location)
- atom :: t -> String -> Parser (t, Location)
- atomThenSpace :: t -> String -> Parser (t, Location)
- parsing :: (Text -> t) -> Parser Text -> String -> Parser (t, Location)
- parseNumbers :: [a] -> Parser (Token, Location)
- smartQuotes :: [Char] -> [Char]
- equalToken :: Token -> TokenParser Location
- satisfyToken :: (Token -> Bool) -> TokenParser (Token, Location)
- anyToken :: TokenParser (Token, Location)
- consumeToken :: (Token -> Maybe a) -> TokenParser (a, Location)
- tokenString :: (Token, Location) -> [Char]
- tokenStr :: Token -> [Char]
- tokenPosition :: SourcePos -> (Token, Location) -> t -> SourcePos
- type TokenParser e = forall s m. Stream s m (Token, Location) => ParsecT s Int m e
- notFollowedBy' :: TokenParser (Token, Location) -> TokenParser ()
- endOfTokens :: TokenParser ()
- curlyQuotes :: [Char] -> [Char]
Documentation
smartQuotes :: [Char] -> [Char] Source #
equalToken :: Token -> TokenParser Location Source #
satisfyToken :: (Token -> Bool) -> TokenParser (Token, Location) Source #
Consume the given predicate from the token stream.
anyToken :: TokenParser (Token, Location) Source #
The parser anyToken accepts any kind of token. It is for example
used to implement eof. Returns the accepted token.
consumeToken :: (Token -> Maybe a) -> TokenParser (a, Location) Source #
Consume the given predicate from the token stream.
tokenString :: (Token, Location) -> [Char] Source #
Make a string out of the token, for error message purposes.
tokenPosition :: SourcePos -> (Token, Location) -> t -> SourcePos Source #
Update the position by the token.
notFollowedBy' :: TokenParser (Token, Location) -> TokenParser () Source #
notFollowedBy p only succeeds when parser p fails. This parser
does not consume any input. This parser can be used to implement the
'longest match' rule. For example, when recognizing keywords (for
example let), we want to make sure that a keyword is not followed
by a legal identifier character, in which case the keyword is
actually an identifier (for example lets). We can program this
behaviour as follows:
keywordLet = try (do{ string "let"
; notFollowedBy alphaNum
})endOfTokens :: TokenParser () Source #
This parser only succeeds at the end of the input. This is not a
primitive parser but it is defined using notFollowedBy.
eof = notFollowedBy anyToken <?> "end of input"
curlyQuotes :: [Char] -> [Char] Source #