com.aliasi.chunk

Class BioTagChunkCodec

java.lang.Object

com.aliasi.chunk.BioTagChunkCodec

All Implemented Interfaces:

TagChunkCodec, Serializable

public class BioTagChunkCodec
extends Object
implements Serializable

The BioTagChunkCodec implements a chunk to tag coder/decoder based on the BIO encoding scheme and a specified tokenizer factory.

BIO Encoding

The basis of the BIO encoding of a chunking is to break the chunking down into tokens. The first token in a chunk of type X is labeled B_X, and subsequent tokens in a chunk of type X are labeled I_X. All other chunks are labeled with the tag O.

For instance, consider the following input string:

 John Jones Mary and Mr. J. J. Jones ran to Washington.
 012345678901234567890123456789012345678901234567890123
 0         1         2         3         4         5

and chunking consisting of the string and chunks:

 (0,10):PER, (11,15):PER, (24,35):PER, (43,53):LOC

Recall that indexing is of the first character and one past the last character. Note that the two person names "John Jones" and "Mary", are separate chunks of type PER (for persons), and the location chunk for "Washington" ends before the period.

If we have a tokenizer that breaks on whitespace and punctuation, we have tokens starting at + and continuing through the - signs.

 John Jones Mary and Mr. J. J. Jones ran to Washington.
 +--- +---- +--- +-- +-+ ++ ++ +---- +-- +- +---------+

In particular, note that the the four periods form their own tokens, even though they are adjacent to characters in other tokens. Writing the tokens out in a column, we show the tags used by the default BIO encoding to the right:

Token	Tag
John	B_PER
Jones	I_PER
Mary	B_PER
and	O
Mr	O
.	O
J	B_PER
.	I_PER
J	I_PER
.	I_PER
Jones	I_PER
ran	O
to	O
Washington	B_LOC
.	O

Note that chunks may be any number of tokens long.

Non-Default Prefixes and Suffixes

The default prefix for begin tags is B_ and the default prefix for in tags is I_ and the default out tag is O. Different begin and in prefixes and out tags may be specified using the constructor BioTagChunkCodec(TokenizerFactory,boolean,String,String,String).

Set of Tags

There is a single tag O, as well as tags B_X and I_X for all chunk types X.

Legal Tag Sequences

The legal tag sequences are described in the following table, where the first column lists tags schematically and the second column shows the tags that may follow them, with the variables ranging over all chunk types:

Tag	Legal Following Tags
O	O, B_X
B_X	O, I_X, B_Y
I_X	O, I_X, B_Y

Note that the begin and in tags have the same legal followers.

Attempts to encode taggings with illegal tag sequences will result in exceptions.

Enforcing Tokenization Consistency

If the consistency flag is set on the constructor, attempts to encode chunkings or decode taggings that are inconsistent with the tokenizer will throw illegal argument exceptions.

In order for a tokenizer to be consistent with a chunking, the tokenization of the characterer sequence for the chunking must be such that every chunk start and end occurs at a token start or end. The same rule applies for tagging, in that the chunking produced has to obey the same rules.

For example, if a regular-expression based tokenizer that breaks on whitespace were used for the above example, the character sequence "Washington." is a token, including the final period. This conflicts with the location-type entity, which ends with the last character before the period.

Serialization

Instances of this class are serializable if their underlying tokenizer factories are serializable. Reading them back in produces an instance of the same class with the same behavior.

Thread Safety

After constructino, a BIO tag-chunk codec is thread safe.

Since:

LingPipe3.9.1

Version:

3.9.2

Author:

Bob Carpenter

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	BEGIN_TAG_PREFIX Default prefix for begin tags, "B_".
static String	IN_TAG_PREFIX Default prefix for continuation tags, "I_".
static String	OUT_TAG Default name of out tag, "O".

Constructor Summary

Constructors

Constructor and Description
BioTagChunkCodec() Construct a BIO-encoded tag/chunk decoder with a null tokenizer that does not enforce consistency and uses the default begin, in, and out tags.
BioTagChunkCodec(TokenizerFactory tokenizerFactory, boolean enforceConsistency) Construct a BIO-encoding based tag-chunk coder/decoder based on the specified tokenizer factory, enforcing consistency of chunkings and tagging coded if the specified flag is set, and using the default being, in, and out tags.
BioTagChunkCodec(TokenizerFactory tokenizerFactory, boolean enforceConsistency, String beginTagPrefix, String inTagPrefix, String outTag) Construct a BIO-encoding based tag-chunk coder/decoder based on the specified tokenizer factory, enforcing consistency of chunkings and tagging coded if the specified flag is set.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	enforceConsistency() Returns true if this codec enforces consistency of the chunkings relative to the tokenizer factory.
boolean	isDecodable(StringTagging tagging) Returns true if the specified tagging may be consistently decoded into a chunking.
boolean	isEncodable(Chunking chunking) Returns true if the specified chunking may be consistently encoded as a tagging.
boolean	legalTags(String... tags) Returns true if the specified sequence of tags is a complete legal tag sequence.
boolean	legalTagSubSequence(String... tags) Returns true if the specified sequence of tags is a legal subsequence of tags.
Iterator<Chunk>	nBestChunks(TagLattice<String> lattice, int[] tokenStarts, int[] tokenEnds, int maxResults) Returns an iterator over chunks extracted in order of highest probability up to the specified maximum number of results.
Set<String>	tagSet(Set<String> chunkTypes) Returns the complete set of tags used by this codec for the specified set of chunk types.
Chunking	toChunking(StringTagging tagging) Return the result of decoding the specified tagging into a chunking.
TokenizerFactory	tokenizerFactory() Return the tokenizer factory for this codec.
String	toString() Return a string-based representation of this codec.
StringTagging	toStringTagging(Chunking chunking) Return the string tagging that fully encodes the specified chunking.
Tagging<String>	toTagging(Chunking chunking) Return the tagging that partially encodes the specified chunking.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

OUT_TAG

public static final String OUT_TAG

Default name of out tag, "O".

See Also:

Constant Field Values

BEGIN_TAG_PREFIX

public static final String BEGIN_TAG_PREFIX

Default prefix for begin tags, "B_".

See Also:

Constant Field Values

IN_TAG_PREFIX

public static final String IN_TAG_PREFIX

Default prefix for continuation tags, "I_".

See Also:

Constant Field Values

Constructor Detail

BioTagChunkCodec

public BioTagChunkCodec()

Construct a BIO-encoded tag/chunk decoder with a null tokenizer that does not enforce consistency and uses the default begin, in, and out tags. A codec constructed with this method only supports the conversion of a string tagging to a chunking, not vice-versa.

BioTagChunkCodec

public BioTagChunkCodec(TokenizerFactory tokenizerFactory,
                boolean enforceConsistency)

Construct a BIO-encoding based tag-chunk coder/decoder based on the specified tokenizer factory, enforcing consistency of chunkings and tagging coded if the specified flag is set, and using the default being, in, and out tags.

Parameters:

tokenizerFactory - Tokenizer factory for generating tokens.

enforceConsistency - Set to true to ensure all coded chunkings and decoded taggings are consistent for round trips.

BioTagChunkCodec

public BioTagChunkCodec(TokenizerFactory tokenizerFactory,
                boolean enforceConsistency,
                String beginTagPrefix,
                String inTagPrefix,
                String outTag)

Construct a BIO-encoding based tag-chunk coder/decoder based on the specified tokenizer factory, enforcing consistency of chunkings and tagging coded if the specified flag is set.

Parameters:

tokenizerFactory - Tokenizer factory for generating tokens.

enforceConsistency - Set to true to ensure all coded chunkings and decoded taggings are consistent for round trips.

Method Detail

enforceConsistency

public boolean enforceConsistency()

Returns true if this codec enforces consistency of the chunkings relative to the tokenizer factory. Consistency requires each chunk to start on the first character of a token and requires each chunk to end on the last character of a token (as usual, ends are one past the last character).

Returns:

true if this codec enforces consistency of chunkings relative to tokenization.

tagSet

public Set<String> tagSet(Set<String> chunkTypes)

Description copied from interface: TagChunkCodec

Returns the complete set of tags used by this codec for the specified set of chunk types.

Modifying the returned set will not affect the codec.

Specified by:

tagSet in interface TagChunkCodec

Parameters:

chunkTypes - Set of types for chunks.

Returns:

Set of all tags used to encode chunks of types in the specified set.

legalTagSubSequence

public boolean legalTagSubSequence(String... tags)

Description copied from interface: TagChunkCodec

Returns true if the specified sequence of tags is a legal subsequence of tags. See the companion method TagChunkCodec.legalTags(String[]) to test if a complete sequence is legal.

A sequence of tags is a legal subsequence if a legal sequence may be created by adding more tags to the front and/or end of the specified sequence.

Providing an empty sequence of tags always returns true. The result for a single input tag determines if the tag itself is legal. For longer sequences, the tags must all be legal and their order must be legal.

Specified by:

legalTagSubSequence in interface TagChunkCodec

Parameters:

tags - Sequence of tags to test.

Returns:

true if the sequence of tags is legal as a subsequence of some larger sequence.

legalTags

public boolean legalTags(String... tags)

Description copied from interface: TagChunkCodec

Returns true if the specified sequence of tags is a complete legal tag sequence. The companion method TagChunkCodec.legalTagSubSequence(String[]) tests if a substring of tags is legal.

Specified by:

legalTags in interface TagChunkCodec

Parameters:

tags - Variable length array of tags.

Returns:

true if the specified sequence of tags is a complete legal tag sequence.

toChunking

public Chunking toChunking(StringTagging tagging)

Description copied from interface: TagChunkCodec

Return the result of decoding the specified tagging into a chunking.

Specified by:

toChunking in interface TagChunkCodec

Parameters:

tagging - Tagging to decode.

Returns:

Chunking resulting from tagging.

toStringTagging

public StringTagging toStringTagging(Chunking chunking)

Description copied from interface: TagChunkCodec

Return the string tagging that fully encodes the specified chunking.

Specified by:

toStringTagging in interface TagChunkCodec

Parameters:

chunking - Chunking to encode.

Returns:

Tagging that encodes the chunking.

Throws:

UnsupportedOperationException - If the tokenizer factory is null.

toTagging

public Tagging<String> toTagging(Chunking chunking)

Description copied from interface: TagChunkCodec

Return the tagging that partially encodes the specified chunking. This method does not return the underlying character sequence or token positions -- that functionality is available from the method TagChunkCodec.toStringTagging(Chunking).

This method will typically be more efficient than toStringTagging(), but implementations may just return the same value, because StringTagging extends Tagging<String>.

This method may be implemented by delegating to call to TagChunkCodec.toStringTagging(Chunking), but a direct implementation is often more efficient.

Specified by:

toTagging in interface TagChunkCodec

Parameters:

chunking - Chunking to encode.

Returns:

Tagging that encodes the chunking.

Throws:

UnsupportedOperationException - If the tokenizer factory is null.

nBestChunks

public Iterator<Chunk> nBestChunks(TagLattice<String> lattice,
                          int[] tokenStarts,
                          int[] tokenEnds,
                          int maxResults)

Description copied from interface: TagChunkCodec

Returns an iterator over chunks extracted in order of highest probability up to the specified maximum number of results.

Specified by:

nBestChunks in interface TagChunkCodec

Parameters:

lattice - Lattice from which chunks are extracted.

maxResults - Maximum number of chunks to return.

Returns:

Iterator over the chunks in the lattice in order from highest to lowest probability.

toString

public String toString()

Return a string-based representation of this codec.

Overrides:

toString in class Object

Returns:

A string-based representation of this codec.

tokenizerFactory

public TokenizerFactory tokenizerFactory()

Return the tokenizer factory for this codec. The tokenizer factory may be null if this was only constructed as a decoder.

Returns:

The underlying tokenizer factory.

isEncodable

public boolean isEncodable(Chunking chunking)

Returns true if the specified chunking may be consistently encoded as a tagging. A chunking is encodable if none of the chunks overlap, and if all chunks begin on the first character of a token and end on the character one past the end of the last character in a token.

Subclasses may enforce further conditions as defined in their class documentation.

Specified by:

isEncodable in interface TagChunkCodec

Parameters:

chunking - Chunking to test.

Returns:

true if the chunking is consistently encodable.

Throws:

UnsupportedOperationException - If the tokenizer is null so that this is only a decoder.

isDecodable

public boolean isDecodable(StringTagging tagging)

Returns true if the specified tagging may be consistently decoded into a chunking. A tagging is decodable if its tokens are the tokens produced by the tokenizer for this coded and if the tags form a legal sequence.

Specified by:

isDecodable in interface TagChunkCodec

Parameters:

tagging - Tagging to test for decodability.

Returns:

true if decoding then encoding produces the specified tagging.

Throws:

UnsupportedOperationException - If the tokenizer is null so that this is only a decoder.