ngram: Ngram models for compressing and classifying text.

[ bsd3, library, machine-learning, natural-language-processing, program ] [ Propose Tags ]

A library and collection of commands for training, evaluating, and applying n-gram-based sequence models.


[Skip to Readme]

Downloads

Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees

Candidates

Versions [RSS] 0.1.0.0, 0.1.0.1
Dependencies base (>=4.7 && <5), bytestring (>=0.10.8.1), cereal (>=0.5.4.0), cereal-text (>=0.1.0.2), containers (>=0.5.10.2), ngram, optparse-generic (>=1.2.2), text (>=1.2.2), zlib (>=0.6.1) [details]
License BSD-3-Clause
Copyright 2018 Tom Lippincott
Author Tom Lippincott
Maintainer tom@cs.jhu.edu
Category natural-language-processing, machine-learning
Home page https://github.com/TomLippincott/ngram#readme
Bug tracker https://github.com/TomLippincott/ngram/issues
Source repo head: git clone https://github.com/TomLippincott/ngram
Uploaded by TomLippincott at 2018-11-13T18:13:29Z
Distributions NixOS:0.1.0.1
Executables ngramClassifier
Downloads 1125 total (13 in the last 30 days)
Rating (no votes yet) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs available [build log]
Last success reported on 2018-11-13 [all 1 reports]

Readme for ngram-0.1.0.1

[back to package description]

NGram

This is a code base for experimenting with various approaches to n-gram-based text modeling.

Compiling

First install Stack somewhere on your PATH. For example, for ~/.local/bin:

wget https://get.haskellstack.org/stable/linux-x86_64.tar.gz -O -|tar xpfz - -C /tmp
cp /tmp/stack-*/stack ~/.local/bin
rm -rf /tmp/stack-*

Then, while in the directory of this README file, run:

stack build

The first time this runs will take a while, 10 or 15 minutes, as it builds an entire Haskell environment from scratch. Subsequent compilations are very fast.

Running

Generally, the commands expect data to be text files where each line has the format:

${id}<TAB>${label}<TAB>${text}

When a model is applied to data, the output will generally have a header with the format:

ID<TAB>GOLD<TAB>${label_1_name}<TAB>${label_2_name}<TAB>...

and lines with the corresponding format:

${doc_id}<TAB>${gold_label_name}<TAB>${label_1_prob}<TAB>${label_2_prob}<TAB>...

where probabilities are represented as natural logarithms.

The remainder of this document describes the implemented models, most of which have a corresponding command that stack will have installed. The library aims to be parametric over the sequence types, and most commands allow users to specify whether to consider bytes, unicode characters, or whitespace-delimited tokens.

Prediction by Partial Matching

PPM is essentially an n-gram model with a particular backoff logic that can't quite be reduced to more widespread approaches to smoothing, but empirically tends to outperform them on short documents. To create a PPM model, run:

sh> stack exec -- ngramClassifier train --train train.txt --dev dev.txt --n 4 --modelFile model.gz
Dev accuracy: 0.8566666666666667

The model can then be applied to new data:

sh> stack exec -- ngramClassifier apply --test test.txt --modelFile model.gz --n 4 --scoresFile scores.txt

The value of --n can also be less than the model size, which will run a bit faster, and (perhaps) less tuned to the original training data.