Interface Tokenizer

All Known Subinterfaces:
CompoundWordTokenizer, SentenceTokenizer
All Known Implementing Classes:
SimpleSentenceTokenizer, SRXSentenceTokenizer, WordTokenizer
public interface Tokenizer
Interface for classes that tokenize text into smaller units.

  • Method Summary

    Modifier and Type
    Method
    Description
    List<String>
    tokenize(String text)