Represents a component of a tokenised input string. Words in the input are represented as the type parameter a — which for this reason will usually, though not always, be PWord.

Given a tokenised input string, return only the Words within it.

Given a Component containing multiple values in a Word, split it apart into a list of Components in which the given String is used as a Separator between multiple results.

For instance:

>>> splitMultipleResults " " (Word ["abc", "def", "ghi"])
[Word "abc", Separator " ", Word "def", Separator " ", Word "ghi"]

>>> splitMultipleResults " " (Word ["abc"])
[Word "abc"]

Tokenise a String input word into a PWord by splitting it up into Graphemes. A list of available multigraphs is supplied as the first argument.

Note that this tokeniser is greedy: if one of the given multigraphs is a prefix of another, the tokeniser will prefer the longest if possible. If there are no matching multigraphs starting at a particular character in the String, tokeniseWord will treat that character as its own Grapheme. For instance:

>>> tokeniseWord [] "cherish"
Right [GMulti "c",GMulti "h",GMulti "e",GMulti "r",GMulti "i",GMulti "s",GMulti "h"]

>>> tokeniseWord ["e","h","i","r","s","sh"] "cherish"
Right [GMulti "c",GMulti "h",GMulti "e",GMulti "r",GMulti "i",GMulti "sh"]

>>> tokeniseWord ["c","ch","e","h","i","r","s","sh"] "cherish"
Right [GMulti "ch",GMulti "e",GMulti "r",GMulti "i",GMulti "sh"]

Given a list of available multigraphs, tokenise an input string into a list of words and other Components. This uses the same tokenisation strategy as tokeniseWords, but also recognises Glosses (in square brackets) and Separators (in the form of whitespace).

Given a function to convert Components to strings, converts a list of Components to strings.

Specialisation of detokeniseWords' for PWords, converting words to strings using concatWithBoundary.

Render a PWord as a String. Very much like concat, but treating GBoundarys specially. Word-external boundaries are deleted, while word-internal boundaries are converted to "#".

Given a list of sound changes, extract the list of multigraphs defined in the first categories declaration of the SoundChanges.

CPS'd form of findFirstCategoriesDecl. Nice for doing things like withFirstCategoriesDecl tokeniseWords changes words (to tokenise using the graphemes from the first categories declaration) and so on.

Megaparsec parser for PWords — see tokeniseWord documentation for details on the parsing strategy and the meaning of the second parameter. For most usecases tokeniseWord should suffice; wordParser itself is only really useful in unusual situations (e.g. as part of a larger parser). The first parameter gives a list of characters (aside from whitespace) which should be excluded from words, i.e. the parser will stop if any of them are found. The second gives a list of multigraphs which might be expected.

Note: the second parameter must be sortByDescendingLength-ed; otherwise multigraphs will not be parsed correctly.

Megaparsec parser for Components. Similarly to wordParser, usually it’s easier to use tokeniseWords instead.