com.aliasi.lm

Interface LanguageModel.Tokenized

All Superinterfaces:

LanguageModel

All Known Implementing Classes:

CompiledTokenizedLM, TokenizedLM

Enclosing interface:

LanguageModel

public static interface LanguageModel.Tokenized
extends LanguageModel

A LanguageModel.Tokenized provides a means of estimating the probability of a sequence of tokens. These may be returned in either linear or log form.

Since:

LingPipe2.2

Version:

2.2

Author:

Bob Carpenter

Nested Class Summary

Nested classes/interfaces inherited from interface com.aliasi.lm.LanguageModel

LanguageModel.Conditional, LanguageModel.Dynamic, LanguageModel.Process, LanguageModel.Sequence, LanguageModel.Tokenized

Method Summary

Methods

Modifier and Type	Method and Description
double	tokenLog2Probability(String[] tokens, int start, int end) Returns the log (base 2) probability of the specified token slice in the underlying token n-gram distribution.
double	tokenProbability(String[] tokens, int start, int end) Returns the probability of the specified token slice in the token n-gram distribution.

Methods inherited from interface com.aliasi.lm.LanguageModel

log2Estimate, log2Estimate

Method Detail

tokenLog2Probability

double tokenLog2Probability(String[] tokens,
                          int start,
                          int end)

Returns the log (base 2) probability of the specified token slice in the underlying token n-gram distribution. This includes the estimation of the actual token for unknown tokens.

Parameters:

tokens - Underlying array of tokens.

start - Index of first token in slice.

end - Index of one past the last token in the slice.

Returns:

The log (base 2) probability of the token slice.

tokenProbability

double tokenProbability(String[] tokens,
                      int start,
                      int end)

Returns the probability of the specified token slice in the token n-gram distribution. This estimate includes the estimates of the actual token for unknown tokens.

Parameters:

tokens - Underlying array of tokens.

start - Index of first token in slice.

end - Index of one past the last token in the slice.

Returns:

The probability of the token slice.