[ library, mit, program, unclassified ] [ Propose Tags ]


Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees


  • No Candidates
Versions [RSS]
Change log
Dependencies aeson (>=1.4.5 && <2), base (>=4.7 && <5), bytestring (>0.10.8 && <1), conduit, hashable, process (>=1.6 && <2), streaming-commons, text (>=1.2.3), token-search, unordered-containers [details]
License MIT
Copyright 2019 Josh Clayton
Author Josh Clayton
Home page
Bug tracker
Source repo head: git clone
Uploaded by joshuaclayton at 2019-12-22T02:20:03Z
Distributions NixOS:
Executables token-search
Downloads 354 total (3 in the last 30 days)
Rating (no votes yet) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs available [build log]
Last success reported on 2019-12-22 [all 1 reports]

Readme for token-search-

[back to package description]


This is a library for efficient substring detection across a codebase.


Unused leverages ctags' token generation in conjunction with a tool to search the file system (either ripgrep or The Silver Searcher).

During execution, Unused shells out from Haskell for each unique token and searches the appropriate files for each token. This means each file searched is searched thousands of times each run, which takes a significant amount of time.


Instead of searching each file git tracks for each of potentially thousands of tokens, token-search processes each file once.

With the tokens:


and the text:

lorem ipsum
dolor sit amet

token-search then:

  1. Builds a trie with the three tokens
  2. Iterates over each character in the text, while also:
    • creating a new copy of the trie
    • adding the created trie to a list of all non-terminated tries
    • walks each trie by the character
    • maintains a list of terminal nodes as tries are walked
    • increments a terminal node count for tokens as they're encountered


stack install


stack test


Copyright 2019 Josh Clayton. See the LICENSE.