Documentation

Lift a single token to chunk of the stream. The default implementation is:

tokenToChunk pxy = tokensToChunk pxy . pure

However for some types of stream there may be a more efficient way to lift.

The first method that establishes isomorphism between list of tokens and chunk of the stream. Valid implementation should satisfy:

chunkToTokens pxy (tokensToChunk pxy ts) == ts

The second method that establishes isomorphism between list of tokens and chunk of the stream. Valid implementation should satisfy:

tokensToChunk pxy (chunkToTokens pxy chunk) == chunk

Return length of a chunk of the stream.

Check if a chunk of the stream is empty. The default implementation is in terms of the more general chunkLength:

chunkEmpty pxy ts = chunkLength pxy ts <= 0

However for many streams there may be a more efficient implementation.

Set source position at given token. By default, the given SourcePos (second argument) is just returned without looking at the token. This method is important when your stream is a collection of tokens where every token knows where it begins in the original input.

The same as positionAt1, but for chunks of the stream. The function should return the position where the entire chunk begins. Again, by default the second argument is returned without modifications and the chunk is not looked at.

Advance position given a single token. The returned position is the position right after the token, or the position where the token ends.

Advance position given a chunk of stream. The returned position is the position right after the chunk, or the position where the chunk ends.

Extract a single token form the stream. Return Nothing if the stream is empty.

takeN_ n s should try to extract a chunk of length n, or if the stream is too short, the rest of the stream. Valid implementation should follow the rules:

• If the requested length n is 0 (or less), Nothing should never be returned, instead Just ("", s) should be returned, where "" stands for the empty chunk, and s is the original stream (second argument).
• If the requested length is greater than 0 and the stream is empty, Nothing should be returned indicating end of input.
• In other cases, take chunk of length n (or shorter if the stream is not long enough) from the input stream and return the chunk along with the rest of the stream.

Extract chunk of the stream taking tokens while the supplied predicate returns True. Return the chunk and the rest of the stream.

For many types of streams, the method allows for significant performance improvements, although it is not strictly necessary from conceptual point of view.