AbnerTokenizer (JCoRe Lingscope AE 2.6.1 API)

java.lang.Object
- lingscope.algorithms.AbnerTokenizer

public class AbnerTokenizer
extends Object

Author:: shashank

Constructor Summary

Constructors
Constructor Description

AbnerTokenizer()

Method Summary

All Methods Static Methods Concrete Methods
Modifier and Type	Method	Description
`static String`	`splitTermsByPunctuation(String input)`	Takes an input and splits the sentence by punctuations and spaces, then stitches it back together with a space and returns
`static String`	`tokenize(String s)`	Take raw text apply ABNER's built-in tokenization on it.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
  - AbnerTokenizer
```
public AbnerTokenizer()
```
- Method Detail
  - tokenize
```
public static String tokenize(String s)
```
    Take raw text apply ABNER's built-in tokenization on it.
  - splitTermsByPunctuation
```
public static String splitTermsByPunctuation(String input)
```
    Takes an input and splits the sentence by punctuations and spaces, then stitches it back together with a space and returns
    
    Parameters:
    
    input - the input string to process
    
    Returns:
    
    processed input string, where all words and punctuations are seperated by space