The Flat class

One bit is plenty for a Bool.

>>> test False
(True,1,"0")

>>> test True
(True,1,"1")

Char's are mapped to Word32 and then encoded.

For ascii characters, the encoding is standard ascii.

>>> test 'a'
(True,8,"01100001")

For unicode characters, the encoding is non standard.

>>> test 'È'
(True,16,"11001000 00000001")

>>> test '不'
(True,24,"10001101 10011100 00000001")

>>> test "\x1F600"
(True,26,"11000000 01110110 00000011 10")

Doubles are encoded as standard IEEE binary64 values:

IEEE_754_binary64 ≡ IEEE_754_binary64 {sign :: Sign,
                                        exponent :: MostSignificantFirst Bits11,
                                        fraction :: MostSignificantFirst Bits52}

Floats are encoded as standard IEEE binary32 values:

IEEE_754_binary32 ≡ IEEE_754_binary32 {sign :: Sign,
                                        exponent :: MostSignificantFirst Bits8,
                                        fraction :: MostSignificantFirst Bits23}

>>> test (0::Float)
(True,32,"00000000 00000000 00000000 00000000")

>>> test (1.4012984643E-45::Float)
(True,32,"00000000 00000000 00000000 00000001")

>>> test (1.1754942107E-38::Float)
(True,32,"00000000 01111111 11111111 11111111")

Integer, Int, Int16, Int32 and Int64 are defined as the ZigZag encoded version of the equivalent unsigned Word:

Int   ≡  Int   (ZigZag Word)

Int64 ≡  Int64 (ZigZag Word64)

Int32 ≡  Int32 (ZigZag Word32)

Int16 ≡  Int16 (ZigZag Word16)

Int8  ≡  Int8  (ZigZag Word8)

ZigZag a ≡ ZigZag a

ZigZag encoding alternates between positive and negative numbers, so that numbers whose absolute value is small can be encoded efficiently:

>>> test (0::Int)
(True,8,"00000000")

>>> test (-1::Int)
(True,8,"00000001")

>>> test (1::Int)
(True,8,"00000010")

>>> test (-2::Int)
(True,8,"00000011")

>>> test (2::Int)
(True,8,"00000100")

>>> test (0::Int8)
(True,8,"00000000")

>>> test (127::Int8)
(True,8,"11111110")

>>> test (-128::Int8)
(True,8,"11111111")

>>> test (0::Int16)
(True,8,"00000000")

>>> test (1::Int16)
(True,8,"00000010")

>>> test (-1::Int16)
(True,8,"00000001")

>>> test (minBound::Int16)
(True,24,"11111111 11111111 00000011")

equivalent to 0b1111111111111111

>>> test (maxBound::Int16)
(True,24,"11111110 11111111 00000011")

equivalent to 0b1111111111111110

>>> test (0::Int32)
(True,8,"00000000")

>>> test (minBound::Int32)
(True,40,"11111111 11111111 11111111 11111111 00001111")

>>> test (maxBound::Int32)
(True,40,"11111110 11111111 11111111 11111111 00001111")

>>> test (0::Int64)
(True,8,"00000000")

>>> test (minBound::Int64)
(True,80,"11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 00000001")

>>> test (maxBound::Int64)
(True,80,"11111110 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 00000001")

Integers are encoded just as the fixed size Ints.

>>> test (0::Integer)
(True,8,"00000000")

>>> test (-1::Integer)
(True,8,"00000001")

>>> test (1::Integer)
(True,8,"00000010")

>>> test (-(2^4)::Integer)
(True,8,"00011111")

>>> test (2^4::Integer)
(True,8,"00100000")

>>> test (-(2^120)::Integer)
(True,144,"11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 00000011")

>>> test (2^120::Integer)
(True,144,"10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 00000100")

Naturals are encoded just as the fixed size Words.

>>> test (0::Natural)
(True,8,"00000000")

>>> test (2^120::Natural)
(True,144,"10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 10000000 00000010")

Natural, Word, Word16, Word32 and Word64 are encoded as a non empty list of 7 bits chunks (least significant chunk first and most significant bit first in every chunk).

Words are always encoded in a whole number of bytes, as every chunk is 8 bits long (1 bit for the List constructor, plus 7 bits for the value).

The actual definition is:

Word64 ≡   Word64 Word

Word32 ≡   Word32 Word

Word16 ≡   Word16 Word

Word ≡   Word (LeastSignificantFirst (NonEmptyList (MostSignificantFirst Word7)))

LeastSignificantFirst a ≡   LeastSignificantFirst a

NonEmptyList a ≡   Elem a
                 | Cons a (NonEmptyList a)

MostSignificantFirst a ≡   MostSignificantFirst a

Word7 ≡   V0
        | V1
        | V2
        ...
        | V127

Values between as 0 and 127 fit in a single byte.

127 (0b1111111) is represented as Elem V127 and encoded as: Elem=0 127=1111111

>>> test (127::Word)
(True,8,"01111111")

254 (0b11111110) is represented as Cons V126 (Elem V1) (254=128+126) and encoded as: Cons=1 V126=1111110 (Elem=0 V1=0000001):

>>> test (254::Word)
(True,16,"11111110 00000001")

Another example, 32768 (Ob1000000000000000 = 0000010 0000000 0000000):

>>> test (32768::Word32)
(True,24,"10000000 10000000 00000010")

As this is a variable length encoding, values are encoded in the same way, whatever their type:

>>> all (test (3::Word) ==) [test (3::Word16),test (3::Word32),test (3::Word64)]
True

Word8 always take 8 bits.

>>> test (0::Word8)
(True,8,"00000000")

>>> test (255::Word8)
(True,8,"11111111")

`()`, as all data types with a single constructor, has a zero-length encoding.

>>> test ()
(True,0,"")

Since: 0.4.4

Since: 0.4.4

>>> tst ((False,True,False,SBS.pack [11,22,33]))
(True,51,[65,3,11,22,33,0])

>>> tst ((False,True,False,L.pack [11,22,33]))
(True,51,[65,3,11,22,33,0])

ByteString, ByteString.Lazy and ByteString.Short are all encoded as Prealigned Arrays:

PreAligned a ≡ PreAligned {preFiller :: Filler, preValue :: a}

Filler ≡   FillerBit Filler
          | FillerEnd

Array v = A0
        | A1 v (Array v)
        | A2 v v (Array v)
        ...
        | A255 ... (Array v)

That's to say as a byte-aligned sequence of blocks of up to 255 elements, with every block preceded by the count of the elements in the block and a final 0-length block.

>>> tst (B.pack [11,22,33])
(True,48,[1,3,11,22,33,0])

where:

1= PreAlignment (takes a byte if we are already on a byte boundary)

3= Number of bytes in ByteString

11,22,33= Bytes

0= End of Array

>>> tst (B.pack [])
(True,16,[1,0])

Pre-alignment ensures that a ByteString always starts at a byte boundary:

>>> tst ((False,True,False,B.pack [11,22,33]))
(True,51,[65,3,11,22,33,0])

All ByteStrings are encoded in the same way:

>>> all (tst (B.pack [55]) ==) [tst (L.pack [55]),tst (SBS.pack [55])]
True

Text (and Data.Text.Lazy) is encoded as a byte aligned array of bytes corresponding to its UTF8 encoding.

>>> tst $ T.pack ""
(True,16,[1,0])

>>> tst $ T.pack "aaa"
(True,120,[1,3,97,97,97,0])

>>> tst $ T.pack "¢¢¢"
(True,120,[1,6,194,162,194,162,194,162,0])

>>> tst $ T.pack "日日日"
(True,120,[1,9,230,151,165,230,151,165,230,151,165,0])

>>> tst $ T.pack "𐍈𐍈𐍈"
(True,120,[1,12,240,144,141,136,240,144,141,136,240,144,141,136,0])

Strict and Lazy Text has the same encoding:

>>> tst (T.pack "abc") == tst (TL.pack "abc")
True

Use a special encoding for the filler

For better encoding/decoding performance, it is useful to declare instances of concrete list types, such as [Char].

>>> test ""
(True,1,"0")

>>> test "aaa"
(True,28,"10110000 11011000 01101100 0010")

>>> test ([]::[Bool])
(True,1,"0")

>>> test [False,False]
(True,5,"10100")

>>> test (Nothing::Maybe Bool)
(True,1,"0")

>>> test (Just False::Maybe Bool)
(True,2,"10")

Ratios are encoded as tuples of (numerator,denominator)

>>> test (3%4::Ratio Word8)
(True,16,"00000011 00000100")

>>> test (4 :+ 2 :: Complex Word8)
(True,16,"00000100 00000010")

>>> test (MkFixed 123 :: Fixed E0)
(True,16,"11110110 00000001")

>>> test (MkFixed 123 :: Fixed E0) == test (MkFixed 123 :: Fixed E2)
True

Since: 0.4.4

Since: 0.4.4

Since: 0.4.4

Since: 0.4.4

Since: 0.4.4

Since: 0.4.4

Since: 0.4.4

Since: 0.4.4

Since: 0.4.4

>>> test (B.fromList [True])
(True,2,"10")

>>> test (B.fromList [False,False])
(True,4,"0100")

Maps are defined as a list of (Key,Value) tuples:

Map = List (Key,Value)

List a = Nil | Cons a (List a)

>>> tst (Data.IntMap.empty :: IntMap ())
(True,1,[0])

>>> asList Data.IntMap.fromList [(1,"a"),(2,"b")]
True

>>> tst (Node (1::Word8) [Node 2 [Node 3 []], Node 4 []])
(True,39,[1,129,64,200,32])

Data.Sequence.Seq is encoded as a list.

>>> asList Data.Sequence.fromList [3::Word8,4,7]
True

In flat <0.4, it was encoded as an Array.

If you want to restore the previous behaviour, use AsArray:

>>> tst $ AsArray (Data.Sequence.fromList [11::Word8,22,33])
(True,40,[3,11,22,33,0])

>>> tst $ Data.Sequence.fromList [11::Word8,22,33]
(True,28,[133,197,164,32])

Data.Set is encoded as a list

>>> asList Data.Set.fromList [3::Word8,4,7]
True

>>> test (Data.DList.fromList [7::Word,7])
(True,19,"10000011 11000001 110")

>>> let l = [7::Word,7] in flat (Data.DList.fromList l) == flat l
True

>>> test (Data.HashSet.fromList [1..3::Word])
(True,28,"10000000 11000000 10100000 0110")

Vectors are encoded as arrays.

>>> tst (V.fromList [11::Word8,22,33])
(True,40,[3,11,22,33,0])

All Vectors are encoded in the same way:

>>> let l = [11::Word8,22,33] in all (tst (V.fromList l) ==) [tst (U.fromList l),tst (S.fromList l)]
True

>>> test (Left False::Either Bool ())
(True,2,"00")

>>> test (Right ()::Either Bool ())
(True,1,"1")

Tuples are supported up to 7 elements.

>>> test (False,())
(True,1,"0")

>>> test ((),())
(True,0,"")

"7 elements tuples ought to be enough for anybody" (Bill Gates - apocryphal)

>>> test (False,True,True,True,False,True,True)
(True,7,"0111011")

tst (1::Int,"2","3","4","5","6","7","8") ...error

Array is encoded as (lowBound,highBound,AsArray (elems array)):

>>> let arr = A.array ((1::Word,4::Word),(2,5)) [((1,4),11::Word),((1,5),22),((2,4),33),((2,5),44)] in tst (bounds arr,AsArray(elems arr)) == tst arr
True

As it's easy to see:

>>> tst $ A.array ((1::Word,4::Word),(2,5)) [((1,4),11::Word),((1,5),22),((2,4),33),((2,5),44)]
(True,80,[1,4,2,5,4,11,22,33,44,0])

>>> tst $ A.array ((1,4),(2,5)) [((1,4),"1.4"),((1,5),"1.5"),((2,4),"2.4"),((2,5),"2.5")]
(True,160,[2,8,4,10,4,152,203,166,137,140,186,106,153,75,166,137,148,186,106,0])

Arrays and Unboxed Arrays are encoded in the same way:

>>> let bounds = ((1::Word,4::Word),(2,5));elems=[11::Word,22,33,44] in tst (U.listArray bounds elems :: U.UArray (Word,Word) Word) == tst (A.listArray bounds elems)
True

Maps are encoded as lists:

>>> tst (Data.Map.empty :: Map () ())
(True,1,[0])

>>> asList Data.Map.fromList [("a","aa"),("b","bb")]
True

Key/Values are encoded in order:

>>> let l = [("a","aa"),("b","bb")] in tst (Data.Map.fromList l) == tst (Data.Map.fromList $ Prelude.reverse l)
True

IntMap and Map are encoded in the same way:

>>> let l = [(2::Int,"b"),(1,"a")] in tst (Data.IntMap.fromList l) == tst (Data.Map.fromList l)
True

>>> test (Data.HashMap.Strict.fromList [(1,11),(2,22)])
(True,35,"10000001 00001011 01000001 00001011 000")

>>> test (Data.HashMap.Lazy.fromList [(1,11),(2,22)])
(True,35,"10000001 00001011 01000001 00001011 000")

>>> let w = Just (11::Word8); a = Alt w <> Alt (Just 24) in tst a == tst w
True

>>> let w = Just (11::Word8); a = Alt Nothing <> Alt w in tst a == tst w
True

Since: 0.4.4