public class TokenTestUtils extends java.lang.Object
Modifier and Type | Field and Description |
---|---|
static java.util.List<java.lang.Integer> |
NO_INDICES
An empty list of integers, used to represent an empty sequence of indices.
|
static java.util.List<ETokenType> |
NO_TYPES
An empty list of token types.
|
static java.util.Comparator<IToken> |
TYPE_TEXT_OFFSET_COMPARATOR
Comparator for ITokens that compares IToken.getOffset() ,
IToken.getType() , and IToken.getText() (in that order). |
Constructor and Description |
---|
TokenTestUtils() |
Modifier and Type | Method and Description |
---|---|
static java.util.EnumSet<ETokenType> |
asEnumSet(ETokenType... sequence)
Creates an enum set from the given token types.
|
static java.util.List<IToken> |
tokenize(java.lang.String input,
ELanguage language)
Tokenizes the input string to a token stream of the given language.
|
public static final java.util.List<java.lang.Integer> NO_INDICES
public static final java.util.List<ETokenType> NO_TYPES
public static final java.util.Comparator<IToken> TYPE_TEXT_OFFSET_COMPARATOR
Comparator
for ITokens that compares IToken.getOffset()
,
IToken.getType()
, and IToken.getText()
(in that order).public TokenTestUtils()
public static java.util.List<IToken> tokenize(java.lang.String input, ELanguage language)
public static java.util.EnumSet<ETokenType> asEnumSet(ETokenType... sequence)