Portability	ghc
Stability	experimental
Maintainer	bjpop@csse.unimelb.edu.au

Language.Python.Version3.Lexer

Contents

Lexical analysis
Tokens
Parse errors

Description

A lexer and set of tokens for Python version 3 programs. See: http://docs.python.org/dev/3.0/reference/lexical_analysis.html.

Synopsis

Lexical analysis

lex Source

Arguments

:: String	The input stream (python source code).
-> String	The name of the python source (filename or input device).
-> Either ParseError [Token]	An error or a list of tokens.

Parse a string into a list of Python Tokens, or return an error.

lexOneToken Source

Arguments

:: String	The input stream (python source code).
-> String	The name of the python source (filename or input device).
-> Either ParseError (Token, String)	An error or the next token and the rest of the input after the token.

Try to lex the first token in an input string. Return either a parse error or a pair containing the next token and the rest of the input after the token.

Tokens

data Token Source

Lexical tokens.

Constructors

Indent SrcLocation	Indentation: increase.
Dedent SrcLocation	Indentation: decrease.
Newline SrcLocation	Newline.
Identifier SrcLocation !String	Identifier.
String SrcLocation !String	Literal: string.
ByteString SrcLocation !ByteString	Literal: byte string.
Integer SrcLocation !Integer	Literal: integer.
Float SrcLocation !Double	Literal: floating point.
Imaginary SrcLocation !Double	Literal: imaginary number.
Def SrcLocation	Keyword: 'def'.
While SrcLocation	Keyword: 'while'.
If SrcLocation	Keyword: 'if'.
True SrcLocation	Keyword: 'True'.
False SrcLocation	Keyword: 'False'.
Return SrcLocation	Keyword: 'Return'.
Try SrcLocation	Keyword: 'try'.
Except SrcLocation	Keyword: 'except'.
Raise SrcLocation	Keyword: 'raise'.
In SrcLocation	Keyword: 'in'.
Is SrcLocation	Keyword: 'is'.
Lambda SrcLocation	Keyword: 'lambda'.
Class SrcLocation	Keyword: 'class'.
Finally SrcLocation	Keyword: 'finally'.
None SrcLocation	Keyword: 'None'
For SrcLocation	Keyword: 'for'.
From SrcLocation	Keyword: 'from'.
NonLocal SrcLocation	Keyword: 'nonlocal'.
Global SrcLocation	Keyword: 'global'.
With SrcLocation	Keyword: 'with'.
As SrcLocation	Keyword: 'as'.
Elif SrcLocation	Keyword: 'elif'.
Yield SrcLocation	Keyword: 'yield'.
Assert SrcLocation	Keyword: 'assert'.
Import SrcLocation	Keyword: 'import'.
Pass SrcLocation	Keyword: 'pass'.
Break SrcLocation	Keyword: 'break'.
Continue SrcLocation	Keyword: 'continue'.
Delete SrcLocation	Keyword: 'del'.
Else SrcLocation	Keyword: 'else'.
Not SrcLocation	Keyword: 'not'.
And SrcLocation	Keyword: boolean conjunction 'and'.
Or SrcLocation	Keyword: boolean disjunction 'or'.
At SrcLocation	Delimiter: at sign '@'.
LeftRoundBracket SrcLocation	Delimiter: left round bracket '('.
RightRoundBracket SrcLocation	Delimiter: right round bracket ')'.
LeftSquareBracket SrcLocation	Delimiter: left square bracket '['.
RightSquareBracket SrcLocation	Delimiter: right square bracket ']'.
LeftBrace SrcLocation	Delimiter: left curly bracket '{'.
RightBrace SrcLocation	Delimiter: right curly bracket '}'.
Dot SrcLocation	Delimiter: dot (full stop) '.'.
Comma SrcLocation	Delimiter: comma ','.
SemiColon SrcLocation	Delimiter: semicolon ';'.
Colon SrcLocation	Delimiter: colon ':'.
Ellipsis SrcLocation	Delimiter: ellipses (three dots) '...'.
RightArrow SrcLocation	Delimiter: right facing arrow '->'.
Assign SrcLocation	Delimiter: assignment '='.
PlusAssign SrcLocation	Delimiter: plus assignment '+='.
MinusAssign SrcLocation	Delimiter: minus assignment '-='.
MultAssign SrcLocation	Delimiter: multiply assignment '*='
DivAssign SrcLocation	Delimiter: divide assignment '/='.
ModAssign SrcLocation	Delimiter: modulus assignment '%='.
PowAssign SrcLocation	Delimiter: power assignment '**='.
BinAndAssign SrcLocation	Delimiter: binary-and assignment '&='.
BinOrAssign SrcLocation	Delimiter: binary-or assignment '\|='.
BinXorAssign SrcLocation	Delimiter: binary-xor assignment '^='.
LeftShiftAssign SrcLocation	Delimiter: binary-left-shift assignment '<<='.
RightShiftAssign SrcLocation	Delimiter: binary-right-shift assignment '>>='.
FloorDivAssign SrcLocation	Delimiter: floor-divide assignment '='.
Plus SrcLocation	Operator: plus '+'.
Minus SrcLocation	Operator: minus: '-'.
Mult SrcLocation	Operator: multiply '*'.
Div SrcLocation	Operator: divide '/'.
GreaterThan SrcLocation	Operator: greater-than '>'.
LessThan SrcLocation	Operator: less-than '<'.
Equality SrcLocation	Operator: equals '=='.
GreaterThanEquals SrcLocation	Operator: greater-than-or-equals '>='.
LessThanEquals SrcLocation	Operator: less-than-or-equals '<='.
Exponent SrcLocation	Operator: exponential '**'.
BinaryOr SrcLocation	Operator: binary-or '\|'.
Xor SrcLocation	Operator: binary-xor '^'.
BinaryAnd SrcLocation	Operator: binary-and '&'.
ShiftLeft SrcLocation	Operator: binary-shift-left '<<'.
ShiftRight SrcLocation	Operator: binary-shift-right '>>'.
Modulo SrcLocation	Operator: modulus '%'.
FloorDiv SrcLocation	Operator: floor-divide ''.
Tilde SrcLocation	Operator: tilde '~'.
NotEquals SrcLocation	Operator: not-equals '!='.
EOF	End of file (no source location).

Instances

Eq Token
Ord Token
Show Token
Location Token

Parse errors

newtype ParseError Source

Parse error. A list of error messages and a source location.

Constructors

ParseError ([String], SrcLocation)

Instances

Show ParseError