StandardTokenizer (thomasjungblut-common 2.0.0 API)

java.lang.Object
- de.jungblut.nlp.StandardTokenizer

All Implemented Interfaces:

Tokenizer
```
public final class StandardTokenizer
extends java.lang.Object
implements Tokenizer
```
Just a basic tokenizer by certain attributes with normalization.

Author:

thomas.jungblut

Constructor Summary

Constructors
Constructor Description

StandardTokenizer()

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`java.lang.String[]`	`tokenize(java.lang.String toTokenize)`	Tokenizes the given String to a array of Strings.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - StandardTokenizer
```
public StandardTokenizer()
```
- Method Detail
  - tokenize
```
public java.lang.String[] tokenize(java.lang.String toTokenize)
```
    Description copied from interface: Tokenizer
    
    Tokenizes the given String to a array of Strings.
    
    Specified by:
    
    tokenize in interface Tokenizer
    
    Parameters:
    
    toTokenize - the string to tokenize.
    
    Returns:
    
    the array of tokenized tokens.