tiktoken: Haskell implementation of tiktoken

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.

[maintain] [Publish]

Warnings:

No 'category' field.
These packages miss upper bounds: - bytestring - containers - deepseq - filepath - pcre-light - raw-strings-qq - text - unordered-containers Please add them, using `cabal gen-bounds` for suggestions. For more information see: https://pvp.haskell.org/

This packages only implements tokenization. In other words, given an existing encoding (cl100k_base) you can tokenize an input.

[Skip to Readme]

Properties

Versions	1.0.0, 1.0.1, 1.0.2, 1.0.2, 1.0.3
Change log	CHANGELOG.md
Dependencies	base (>=4.18.0.0 && <5), base64 (>=1.0 && <1.1), bytestring (>=0.11.3.0), containers (>=0.5.0.0), deepseq (>=1.4.0.0), filepath, megaparsec (<9.7), pcre-light (>=0.2), raw-strings-qq, text, unordered-containers [details]
License	BSD-3-Clause
Author	Gabriella Gonzalez
Maintainer	GenuineGabriella@gmail.com
Uploaded	by GabrielGonzalez at 2024-09-02T05:08:52Z

Modules

[Index] [Quick Jump]

Tiktoken

Downloads

tiktoken-1.0.2.tar.gz [browse] (Cabal source package)
Package description (as included in the package)

Maintainer's Corner

Package maintainers

GabrielGonzalez

For package maintainers and hackage trustees

edit package information

Readme for tiktoken-1.0.2

[back to package description]

`tiktoken`

This is a Haskell implementation of tiktoken, but just the tokenization logic. In other words, given an existing encoding (like cl100k_base) you can tokenize a string (into smaller strings or token ranks).

This means that you can't (yet) use this package to create your own new encodings, but you can use it to consume encodings. In particular, this comes in handy for prompt engineering where you want to use as much of the available prompt tokens as possible (which requires accurately counting tokens).

Encoding speed is ≈2.6-3.1 MB/s on an M1 MacBook Pro (using only one core since this package does not yet support parallel tokenization):

All
  Encode 10 MB of Wikipedia
    r50k_base:   OK (23.88s)
      3.356 s ± 151 ms
    p50k_base:   OK (10.39s)
      3.445 s ±  31 ms
    p50k_edit:   OK (11.13s)
      3.693 s ± 240 ms
    cl100k_base: OK (11.16s)
      3.685 s ± 143 ms
    o200k_base:  OK (11.01s)
      3.648 s ± 134 ms