tlex: A lexer generator

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.

[maintain] [Publish]

Tlex is haskell libraries and toolchains for generating lexical analyzer.

[Skip to Readme]

Properties

Versions	0.1.0.0, 0.1.0.0, 0.2.0.0, 0.3.0.0, 0.4.0.0, 0.4.0.1, 0.5.0.0
Change log	CHANGELOG.md
Dependencies	base (>=4.12.0 && <4.15), containers (>=0.6.0 && <0.7), tlex-core (>=0.1.0 && <0.2) [details]
License	(Apache-2.0 OR MPL-2.0)
Copyright	(c) 2020 Mizunashi Mana
Author	Mizunashi Mana
Maintainer	mizunashi-mana@noreply.git
Category	Parsing
Home page	https://github.com/mizunashi-mana/tlex
Bug tracker	https://github.com/mizunashi-mana/tlex/issues
Source repo	head: git clone https://github.com/mizunashi-mana/tlex.git
Uploaded	by mizunashi_mana at 2021-01-07T12:18:06Z

Modules

[Index] [Quick Jump]

Language
- Lexer
  - Language.Lexer.Tlex

Flags

Manual Flags

Name	Description	Default
develop	Turn on some options for development	Disabled

Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info

Downloads

tlex-0.1.0.0.tar.gz [browse] (Cabal source package)
Package description (as included in the package)

Maintainer's Corner

Package maintainers

mizunashi_mana

For package maintainers and hackage trustees

edit package information

Readme for tlex-0.1.0.0

[back to package description]

Tlex: A Generator for Lexical Analysers

Installation

Add dependencies on package.cabal:

build-depends:
    base,
    bytestring,
    tlex,          -- main
    tlex-encoding, -- for utf8 parsing
    tlex-th,       -- for outputing lexer with Template Haskell
    charset,
    template-haskell,

Usage

Setup:

import qualified Data.CharSet                        as CharSet
import qualified Data.Word                           as Word
import qualified Language.Haskell.TH                 as TH
import qualified Language.Lexer.Tlex                 as Tlex
import qualified Language.Lexer.Tlex.Plugin.Encoding as TlexEnc
import qualified Language.Lexer.Tlex.Plugin.TH       as TlexTH


type LexerState = ()
type LexerAction = [LexerCodeUnit] -> Token
type LexerCodeUnit = Word.Word8

type ScannerBuilder = TlexTH.THScannerBuilder LexerState LexerCodeUnit LexerAction
type Pattern = Tlex.Pattern LexerCodeUnit

rule :: Pattern -> TH.Q (TH.TExp LexerAction) -> ScannerBuilder ()
rule = TlexTH.thLexRule [()]

Setup charSetP:

charSetP :: CharSet.CharSet -> Pattern
charSetP cs = TlexEnc.charSetP TlexEnc.charSetPUtf8 cs

chP :: Char -> Pattern
chP c = TlexEnc.chP TlexEnc.charSetPUtf8 c

Write lexer rules:

buildLexer :: TH.Q [TH.Dec]
buildLexer = do
    lexer <- TlexTH.buildTHScannerWithReify lexerRules
    TlexTH.outputScanner lexer

data Token
    = TokWhiteSpace [LexerCodeUnit]
    | TokSmallAlpha [LexerCodeUnit]
    | TokLargeAlpha [LexerCodeUnit]
    | TokDigit [LexerCodeUnit]

lexerRules :: ScannerBuilder ()
lexerRules = do
    rule (Tlex.someP whitecharP) [||TokWhiteSpace||]
    rule (charSetP $ CharSet.range 'a' 'z') [||TokSmallAlpha||]
    rule (charSetP $ CharSet.range 'A' 'Z') [||TokLargeAlpha||]
    rule (charSetP $ CharSet.range '0' '9') [||TokDigit||]

whitecharP = Tlex.orP
    [ chP ' '
    , '\t'
    , '\n'
    , '\r'
    ]

Build lexer:

$(Lexer.Rules.buildLexer)

newtype InputByteString a = InputByteString
    { unInputByteString :: ByteString -> Int -> (a, Int)
    }
    deriving (Functor, Applicative, Monad)
        via (ReaderT ByteString (State Int))

runInputByteString :: InputByteString a -> ByteString -> (a, Int)
runInputByteString (InputByteString runner) input = runner input 0

instance TlexContext Int Word8 InputByteString where
    tlexGetInputPart = InputString $ \bs i -> (bs `indexMaybe` i, i)
    tlexGetMark = InputByteString $ \bs i -> (i, i)

lexByteString :: ByteString.ByteString -> Maybe [ByteString.ByteString]
lexByteString s0 = go s0 id where
    go s acc = case runInputByteString (tlexScan ()) s of
        (TlexEndOfInput, _)            -> Just $ acc []
        (TlexError, _)                 -> Nothing
        (TlexAccepted n act, _) ->
            let (consumed, rest) = splitAt n s
                token = act consumed
            in go rest $ \n -> acc act:n

Examples

Small language: https://github.com/mizunashi-mana/tlex/tree/master/example/small-lang
Haskell2010: https://github.com/mizunashi-mana/tlex/tree/master/example/haskell2010