nerf-0.5.4.1: Nerf, a named entity recognition tool based on linear-chain CRFs

Safe HaskellSafe
LanguageHaskell2010

NLP.Nerf.Tokenize

Contents

Description

The module implements the tokenization used within Nerf and some other tokenization-related stuff.

Synopsis

Tokenization

tokenize :: String -> [String] Source #

Tokenize sentence using the default tokenizer.

Synchronization

class Word a where Source #

A class of objects which can be converted to String.

Methods

word :: a -> String Source #

Instances
Word Text Source # 
Instance details

Defined in NLP.Nerf.Tokenize

Methods

word :: Text -> String Source #

Word Text Source # 
Instance details

Defined in NLP.Nerf.Tokenize

Methods

word :: Text -> String Source #

Word String Source # 
Instance details

Defined in NLP.Nerf.Tokenize

Methods

word :: String -> String Source #

sync Source #

Arguments

:: (Word b, Word c) 
=> NeForest a b

NE forest

-> [c]

New tokenization

-> NeForest a c

Resulting NE forest

Synchronize the list of NE trees with the new tokenization.