Module com.github.szgabsz91.morpher.languagehandlers.hunmorph
Package com.github.szgabsz91.morpher.languagehandlers.hunmorph.impl

Class HunmorphAnnotationTokenizer

java.lang.Object
com.github.szgabsz91.morpher.languagehandlers.hunmorph.impl.HunmorphAnnotationTokenizer
public class HunmorphAnnotationTokenizer extends Object
Tokenizes Hunmorph-Ocamorph token strings.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final List<String>
    KNOWN_TOKENS
    The list of known tokens.

  • Constructor Summary

    Constructors
    Constructor
    Description
    HunmorphAnnotationTokenizer()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    List<String>
    getSupportedAffixTypes()
     
    com.github.szgabsz91.morpher.languagehandlers.api.model.AnnotationTokenizerResult
    preprocess(com.github.szgabsz91.morpher.languagehandlers.api.model.AnnotationTokenizerResult originalResult)
    Preprocesses the given result and returns the processed version.
    com.github.szgabsz91.morpher.languagehandlers.api.model.AnnotationTokenizerResult
    tokenize(String expression, String grammaticalForm, int frequency)
    Tokenizes the given expression and grammatical form and returns the result.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • KNOWN_TOKENS

      public static final List<String> KNOWN_TOKENS
      The list of known tokens.

  • Constructor Details

    • HunmorphAnnotationTokenizer

      public HunmorphAnnotationTokenizer()

  • Method Details

    • tokenize

      public com.github.szgabsz91.morpher.languagehandlers.api.model.AnnotationTokenizerResult tokenize(String expression, String grammaticalForm, int frequency)
      Tokenizes the given expression and grammatical form and returns the result.
      Parameters:
      expression - the expression containing the tokens
      grammaticalForm - the grammatical form
      frequency - the frequency
      Returns:
      the AnnotationTokenizerResult instance

    • getSupportedAffixTypes

      public List<String> getSupportedAffixTypes()

    • preprocess

      public com.github.szgabsz91.morpher.languagehandlers.api.model.AnnotationTokenizerResult preprocess(com.github.szgabsz91.morpher.languagehandlers.api.model.AnnotationTokenizerResult originalResult)
      Preprocesses the given result and returns the processed version.
      Parameters:
      originalResult - the original result
      Returns:
      the processed result