superdoc-0.1.2.6: Additional documentation markup and Unicode support

Safe HaskellSafe
LanguageHaskell2010

Distribution.Superdoc.UTF8

Description

This module provides a simple parser for UTF8. It converts a string of bytes into a list of unicode tokens. Invalid input bytes are converted to special Invalid tokens, leaving it up to the consuming application to decide what to do with them.

Synopsis

Documentation

data Token Source #

A token in a parsed UTF8 stream is either a valid Unicode character or an invalid input character.

Constructors

Unicode Char 
Invalid Char 

parse_utf8 :: String -> [Token] Source #

Parse a UTF8 stream into tokens. Rejects overlong forms and code points above 0x10ffff. Does not check validity of individual unicode code points.