Package de.jungblut.nlp

Interface Tokenizer

  • All Known Implementing Classes:
    BigramTokenizer, StandardTokenizer
    public interface Tokenizer
    Standard tokenizer interface. Makes multiple strings out of one.
    Author:
    thomas.jungblut

    • Method Summary

      All Methods Instance Methods Abstract Methods 
      Modifier and Type Method Description
      java.lang.String[] tokenize​(java.lang.String toTokenize)
      Tokenizes the given String to a array of Strings.

    • Method Detail

      • tokenize

        java.lang.String[] tokenize​(java.lang.String toTokenize)
        Tokenizes the given String to a array of Strings.
        Parameters:
        toTokenize - the string to tokenize.
        Returns:
        the array of tokenized tokens.