TokenStreamUtils (Teamscale Check Framework API)

java.lang.Object
- eu.cqse.check.framework.shallowparser.TokenStreamUtils

public class TokenStreamUtils
extends java.lang.Object

Utility methods for IToken lists.

Field Summary

Fields
Modifier and Type Field and Description

static int NOT_FOUND
The value returned if nothing was found in the various find methods.

Fields
Modifier and Type	Field and Description
`static int`	`NOT_FOUND` The value returned if nothing was found in the various find methods.

Constructor Summary

Constructors
Constructor and Description

TokenStreamUtils()

Constructors
Constructor and Description
`TokenStreamUtils()`

Method Summary

All Methods Static Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`static void`	`addToken(java.util.List<IToken> tokens, int position, ETokenType type, java.lang.String text)` Adds a new token to the given token list at the given position with the given type and text.
`static java.util.List<java.lang.Integer>`	`allStartingIndicesOfTypeSequence(java.util.List<IToken> tokens, int startOffset, int endIndex, ETokenType... sequence)` Returns the indices of all tokens that starts a sequence of the given token types (and are between startOffset and endIndex).
`static boolean`	`contains(java.util.List<IToken> tokens, ETokenType.ETokenClass tokenClass)` Returns `true` if the list of tokens contains a token of `tokenClass, false otherwise.`
`static boolean`	`contains(java.util.List<IToken> tokens, ETokenType tokenType)` Returns `true` if the list of tokens contains a token of `tokenType, false otherwise.`
`static boolean`	`contains(java.util.List<IToken> tokens, int startIndex, int endIndex, ETokenType tokenType)` Returns `true` if the list of tokens contains a token of `tokenType in the given range, false otherwise.`
`static boolean`	`containsAll(java.util.List<IToken> tokens, ETokenType... tokenTypes)` Returns `true` if the list of tokens contains at least one token of each of the given `tokenTypes, false otherwise.`
`static boolean`	`containsAll(java.util.List<IToken> tokens, int startIndex, int endIndex, ETokenType... tokenTypes)` Returns `true` if the list of tokens contains at least one token of each of the given `tokenTypes in the given range, false otherwise.`
`static boolean`	`containsAny(java.util.List<IToken> tokens, java.util.EnumSet<ETokenType> tokenTypeSet)` Returns `true` if the given token stream contains one of the tokens in the given set.
`static boolean`	`containsAny(java.util.List<IToken> tokens, ETokenType... tokenTypes)` Returns `true` if the list of tokens contains a any token of given `tokenTypes, false otherwise.`
`static boolean`	`containsAny(java.util.List<IToken> tokens, int startIndex, int endIndex, java.util.EnumSet<ETokenType> tokenTypes)` Returns `true` if the list of tokens contains a any token of given `tokenTypes in the given range, false otherwise.`
`static boolean`	`containsAny(java.util.List<IToken> tokens, int startIndex, int endIndex, ETokenType... tokenTypes)` Returns `true` if the list of tokens contains a any token of given `tokenTypes in the given range, false otherwise.`
`static boolean`	`containsSequence(java.util.List<IToken> tokens, ETokenType... tokenTypes)` Returns `true` if the list of tokens contains a consecutive subsequence of tokens that match the sequence of token types., false otherwise.
`static boolean`	`containsSequence(java.util.List<IToken> tokens, int startIndex, int endIndex, ETokenType... tokenTypes)` Returns `true` if the list of tokens contains a consecutive subsequence of tokens that match the sequence of token types., false otherwise.
`static boolean`	`containsTokenIndexPredicate(java.util.List<IToken> tokens, java.util.function.BiPredicate<java.lang.Integer,java.util.List<IToken>> predicate)` Returns whether an index of the given token list matches the given `BiPredicate` (which gets the token index and the given token list).
`static java.util.List<IToken>`	`copyTokens(IToken referenceToken, java.util.List<IToken> tokens)` Creates a new token list copying the given list and using the given reference token.
`static int`	`count(java.util.List<IToken> tokens, ETokenType tokenType)` Counts the number of tokens of a given type in the list of tokens
`static int`	`countWithoutNesting(java.util.List<IToken> tokens, ETokenType tokenType, ETokenType openingType, ETokenType closingType)` Counts the number of tokens of a given type in the list of tokens that are not nested between the given nesting tokens.
`static int`	`countWithoutNesting(java.util.List<IToken> tokens, ETokenType tokenType, java.util.List<ETokenType> openingType, java.util.List<ETokenType> closingType)` Counts the number of tokens of a given type in the list of tokens that are not nested between the given nesting tokens.
`static IToken`	`createToken(IToken referenceToken, java.lang.String text, ETokenType type)` Creates a new token from the given referenceToken.
`static boolean`	`endsWith(java.util.List<IToken> tokens, ETokenType... sequence)` Returns whether the given token list ends with given token type sequence.
`static java.util.List<java.lang.Integer>`	`findAll(java.util.List<IToken> tokens, java.util.EnumSet<ETokenType> types)` Returns all indices of the given token types in the given token list.
`static java.util.List<java.lang.Integer>`	`findAll(java.util.List<IToken> tokens, int startOffset, int endOffset, java.util.EnumSet<ETokenType> types)` Returns all indices of the given token types in the given token list, beginning from startOffset (inclusive) to endOffset (exclusive).
`static java.util.List<IToken>`	`findAllTokens(java.util.List<IToken> tokens, java.util.EnumSet<ETokenType> types)` Returns all tokens of the given token types in the given token list.
`static int`	`findFirstTopLevel(java.util.List<IToken> tokens, java.util.EnumSet<ETokenType> searchTypes, java.util.List<ETokenType> openingTypes, java.util.List<ETokenType> closingTypes)` Returns the index of the first top-level token that has one of the given search types regarding the nesting introduced by the opening and closing types.
`static int`	`findFirstTopLevel(java.util.List<IToken> tokens, ETokenType searchType, java.util.List<ETokenType> openingTypes, java.util.List<ETokenType> closingTypes)` Returns the index of the first top-level token that has the given search type regarding the nesting introduced by the opening and closing types.
`static int`	`findFirstTopLevel(java.util.List<IToken> tokens, int startIndex, java.util.EnumSet<ETokenType> searchTypes, java.util.List<ETokenType> openingTypes, java.util.List<ETokenType> closingTypes)` Returns the index of the first top-level token starting at the given index that has one of the given search types regarding the nesting introduced by the opening and closing types.
`static int`	`findFirstTopLevel(java.util.List<IToken> tokens, int startIndex, java.util.function.Predicate<IToken> searchPredicate, java.util.List<ETokenType> openingTypes, java.util.List<ETokenType> closingTypes)` Returns the index of the first top-level token starting at the given index that matches the given token predicate regarding the nesting introduced by the opening and closing types.
`static int`	`findFirstTopLevelWithIndexPredicate(java.util.List<IToken> tokens, int startIndex, java.util.function.Predicate<java.lang.Integer> searchPredicate, java.util.List<ETokenType> openingTypes, java.util.List<ETokenType> closingTypes)` Returns the index of the first top-level token starting at the given index that matches the given token predicate regarding the nesting introduced by the opening and closing types.
`static int`	`findMatchingClosingToken(java.util.List<IToken> tokens, int currentOffset, java.util.EnumSet<ETokenType> openingTypes, java.util.EnumSet<ETokenType> closingTypes)` Returns the offset of the first token of closingType, thereby counting nesting occurrences of openingType and closingType.
`static int`	`findMatchingClosingToken(java.util.List<IToken> tokens, int currentOffset, ETokenType openingType, ETokenType closingType)` Returns the offset of the first token of closingType, thereby counting nesting occurrences of openingType and closingType.
`static int`	`findMatchingOpeningToken(java.util.List<IToken> tokens, int currentOffset, ETokenType openingType, ETokenType closingType)` Returns the offset of the first token of openingType, thereby counting nesting occurrences of openingType and closingType.
`static int`	`firstTokenMatchingIndexPredicate(java.util.List<IToken> tokens, int startIndex, java.util.function.BiPredicate<java.lang.Integer,java.util.List<IToken>> predicate)` Returns the first index of the given token list that matches the given `BiPredicate` (which gets the token index and the given token list).
`static int`	`firstTokenOfAlternatingTypes(java.util.List<IToken> tokens, int endOffset, ETokenType element, ETokenType separator)` Returns the starting index of a element - separator - element type token sequence ending at `endOffset` with type `element`.
`static int`	`firstTokenOfType(java.util.List<IToken> tokens, ETokenType... tokenTypes)` Finds the index of the first token in the token list that has any of the of given token types.
`static int`	`firstTokenOfType(java.util.List<IToken> tokens, int startIndex, ETokenType... tokenTypes)` Finds the index of the first token in the token list after startIndex that has any of the of given token types.
`static int`	`firstTokenOfType(java.util.List<IToken> tokens, int startIndex, int endIndex, java.util.EnumSet<ETokenType> set)` Finds the index of the first token in the token list (not before startIndex and before endIndex) that has any of the of given token types.
`static int`	`firstTokenOfType(java.util.List<IToken> tokens, int startIndex, int endIndex, ETokenType... tokenTypes)` Finds the index of the first token in the token list (not before startIndex and before endIndex) that has any of the of given token types.
`static int`	`firstTokenOfTypeSequence(java.util.List<IToken> tokens, int startOffset, ETokenType... sequence)` Returns the index of the first occurrence of the given sequence of token types in the given token list, beginning from the given start offset.
`static int`	`firstTokenOfTypeSequence(java.util.List<IToken> tokens, int startOffset, int endIndex, ETokenType... sequence)` Returns the index (between the given start and end indices) of the first token that starts a sequence of the given token types.
`static java.util.List<java.lang.Integer>`	`firstTokenOfTypeSequences(java.util.List<IToken> tokens, int offset, ETokenType... sequence)` Returns all indices of occurrences of the given sequence of token types in the given token list, beginning from the given offset.
`static ELanguage`	`getLanguage(java.util.Collection<IToken> tokens)` Returns the `ELanguage` of the given tokens (assuming all tokens have the same language).
`static IToken`	`getTokenByTypeAndText(java.util.List<IToken> tokens, java.lang.String tokenText, java.util.Set<ETokenType> tokenTypes)` Searches for the first token with the given type and text (ignoring case).
`static java.util.List<IToken>`	`getTokensInSameLine(java.util.List<IToken> tokens, int index)` Returns a list of tokens existing in the same line of a given token, which specified by its index.
`static java.util.List<java.lang.String>`	`getTokenTextsByType(java.util.List<IToken> tokens, ETokenType tokenType)` Filters the tokens by type and returns the text of each filtered token.
`static boolean`	`hasTokenTypeSequence(java.util.List<IToken> tokens, int startOffset, ETokenType... sequence)` Returns whether the given token list contains tokens with the given types in that order starting at the given start offset.
`static boolean`	`hasTypes(java.util.List<IToken> tokens, ETokenType... types)` Returns whether the given list's token types equal the given token types.
`static int`	`indexOfByOffset(java.util.List<IToken> tokens, int offset)` Binary search for an `IToken` in the given list that has exactly the given offset (`IToken.getOffset()`).
`static java.util.List<java.lang.Integer>`	`indicesOfByPredicate(java.util.List<IToken> tokens, java.util.function.Predicate<IToken> includeToken)` Returns the indices of the tokens in the given list that satisfy the given predicate.
`static int`	`lastTokenOfType(java.util.List<IToken> tokens, java.util.EnumSet<ETokenType> tokenTypes)` Returns the index of the last token that has any of the given token types (or `NOT_FOUND`).
`static int`	`lastTokenOfType(java.util.List<IToken> tokens, ETokenType... tokenTypes)` Returns the index of the last token that has any of the given token types (or `NOT_FOUND`).
`static int`	`lastTokenOfType(java.util.List<IToken> tokens, int startIndex, java.util.EnumSet<ETokenType> tokenTypes)` Returns the index of the last token that has any of the given token types not before the start index (or `NOT_FOUND`).
`static int`	`lastTokenOfType(java.util.List<IToken> tokens, int startIndex, ETokenType... tokenTypes)` Returns the index of the last token that has any of the given token types not before the start index (or `NOT_FOUND`).
`static int`	`lastTokenOfType(java.util.List<IToken> tokens, int startIndex, int endIndex, java.util.EnumSet<ETokenType> tokenTypes)` Returns the index of the last token that has any of the given token types not before the start index and before the end index (or `NOT_FOUND`).
`static int`	`lastTokenOfType(java.util.List<IToken> tokens, int startIndex, int endIndex, ETokenType... tokenTypes)` Returns the index of the last token that has any of the given token types not before the start index and before the end index (or `NOT_FOUND`).
`static java.util.List<IToken>`	`removeAtEnd(java.util.List<IToken> tokens, ETokenType type)` Removes the last token of the given token list if it matches the specified token type.
`static java.util.List<IToken>`	`removeAtFront(java.util.List<IToken> tokens, ETokenType type)` Removes the first token of the given token list if it matches the specified token type.
`static java.util.List<java.util.List<IToken>>`	`split(java.util.List<IToken> tokens, java.util.EnumSet<ETokenType> splitTypes)` Splits the given token stream at the tokens of the given types.
`static java.util.List<java.util.List<IToken>>`	`split(java.util.List<IToken> tokens, java.util.EnumSet<ETokenType> splitTypes, int limit)` Splits the given token stream at the tokens of the given types, but at most as often as given in the limit parameter.
`static java.util.List<java.util.List<IToken>>`	`split(java.util.List<IToken> tokens, ETokenType... splitTypes)` Splits the given token stream at the tokens of the given types.
`static java.util.List<java.util.List<IToken>>`	`split(java.util.List<IToken> tokens, java.util.function.Predicate<IToken> splitTokenMatcher, int limit)` Splits the given token stream at the tokens matching the given predicate, but at most as often as given in the limit parameter.
`static java.util.List<java.util.List<IToken>>`	`splitWithNesting(java.util.List<IToken> tokens, java.util.EnumSet<ETokenType> splitTypes, java.util.List<ETokenType> openingTypes, java.util.List<ETokenType> closingTypes, int limit)` Splits the given token list at the tokens of the given split types.
`static java.util.List<java.util.List<IToken>>`	`splitWithNesting(java.util.List<IToken> tokens, ETokenType splitType, ETokenType openingType, ETokenType closingType)` Splits the given token list at the tokens of the given split type.
`static java.util.List<java.util.List<IToken>>`	`splitWithNesting(java.util.List<IToken> tokens, ETokenType splitType, java.util.List<ETokenType> openingTypes, java.util.List<ETokenType> closingTypes)` Splits the given token list at the tokens of the given split type.
`static boolean`	`startsNewStatement(IToken token, IToken lastToken, java.util.EnumSet<ETokenType> continueTokens, java.util.EnumSet<ETokenType> nonContinuationTokens, ETokenType newStatementToken)` Checks if the token is the start of a new statement.
`static boolean`	`startsWith(java.util.List<IToken> tokens, ETokenType... sequence)` Returns whether the given token list starts with given token type sequence.
`static java.util.List<IToken>`	`tokensBetween(java.util.List<IToken> tokens, java.util.EnumSet<ETokenType> startTypes, java.util.EnumSet<ETokenType> endTypes)` Returns the sublist of tokens between the first occurrence of some given start token types and the first occurrence of any of the given end token types (after start).
`static java.util.List<IToken>`	`tokensBetween(java.util.List<IToken> tokens, ETokenType startType, ETokenType endType)` Returns the sublist of tokens between the first occurrence of given start token type and the first occurrence of end token type (after start).
`static java.util.List<IToken>`	`tokensBetweenWithNesting(java.util.List<IToken> tokens, ETokenType openingType, ETokenType closingType)` Returns all tokens between the tokens of the opening and closing types regarding nesting.
`static java.util.List<IToken>`	`tokensBetweenWithNesting(java.util.List<IToken> tokens, int startOffset, ETokenType openingType, ETokenType closingType)` Returns all tokens between the tokens of the opening and closing types regarding nesting beginning from the given start offset.
`static boolean`	`tokenTypesAt(java.util.List<IToken> tokens, int startOffset, ETokenType... tokenTypes)` Deprecated. use `hasTokenTypeSequence(List, int, ETokenType...)` instead. I leave the method here to avoid hidden merge conflicts. Can be removed after some releases.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail
- NOT_FOUND
```
public static final int NOT_FOUND
```
  The value returned if nothing was found in the various find methods.
  
  See Also:
  
  Constant Field Values

Constructor Detail
- TokenStreamUtils
```
public TokenStreamUtils()
```

Method Detail

firstTokenOfType
```
public static int firstTokenOfType(java.util.List<IToken> tokens,
                                   ETokenType... tokenTypes)
```
Finds the index of the first token in the token list that has any of the of given token types.
For finding sequences, see this method: firstTokenOfTypeSequence(List, int, int, ETokenType...)

Returns:

integer index (or NOT_FOUND).

firstTokenOfType
```
public static int firstTokenOfType(java.util.List<IToken> tokens,
                                   int startIndex,
                                   ETokenType... tokenTypes)
```
Finds the index of the first token in the token list after startIndex that has any of the of given token types.
For finding sequences, see this method: firstTokenOfTypeSequence(List, int, int, ETokenType...)

Returns:

integer index (or NOT_FOUND).

firstTokenOfType
```
public static int firstTokenOfType(java.util.List<IToken> tokens,
                                   int startIndex,
                                   int endIndex,
                                   ETokenType... tokenTypes)
```
Finds the index of the first token in the token list (not before startIndex and before endIndex) that has any of the of given token types.
For finding sequences, see this method: firstTokenOfTypeSequence(List, int, int, ETokenType...)

Returns:

integer index (or NOT_FOUND).

firstTokenOfType
```
public static int firstTokenOfType(java.util.List<IToken> tokens,
                                   int startIndex,
                                   int endIndex,
                                   java.util.EnumSet<ETokenType> set)
```
Finds the index of the first token in the token list (not before startIndex and before endIndex) that has any of the of given token types.
For finding sequences, see this method: firstTokenOfTypeSequence(List, int, int, ETokenType...)

Returns:

integer index (or NOT_FOUND).

lastTokenOfType

public static int lastTokenOfType(java.util.List<IToken> tokens,
                                  ETokenType... tokenTypes)

Returns the index of the last token that has any of the given token types (or NOT_FOUND).

lastTokenOfType

public static int lastTokenOfType(java.util.List<IToken> tokens,
                                  java.util.EnumSet<ETokenType> tokenTypes)

Returns the index of the last token that has any of the given token types (or NOT_FOUND).

lastTokenOfType

public static int lastTokenOfType(java.util.List<IToken> tokens,
                                  int startIndex,
                                  ETokenType... tokenTypes)

Returns the index of the last token that has any of the given token types not before the start index (or NOT_FOUND).

lastTokenOfType

public static int lastTokenOfType(java.util.List<IToken> tokens,
                                  int startIndex,
                                  java.util.EnumSet<ETokenType> tokenTypes)

Returns the index of the last token that has any of the given token types not before the start index (or NOT_FOUND).

lastTokenOfType

public static int lastTokenOfType(java.util.List<IToken> tokens,
                                  int startIndex,
                                  int endIndex,
                                  ETokenType... tokenTypes)

Returns the index of the last token that has any of the given token types not before the start index and before the end index (or NOT_FOUND).

lastTokenOfType

public static int lastTokenOfType(java.util.List<IToken> tokens,
                                  int startIndex,
                                  int endIndex,
                                  java.util.EnumSet<ETokenType> tokenTypes)

Returns the index of the last token that has any of the given token types not before the start index and before the end index (or NOT_FOUND).

firstTokenOfAlternatingTypes

public static int firstTokenOfAlternatingTypes(java.util.List<IToken> tokens,
                                               int endOffset,
                                               ETokenType element,
                                               ETokenType separator)

Returns the starting index of a element - separator - element type token sequence ending at endOffset with type element. Returns NOT_FOUND if no such sequence is found. Useful for example for matching qualified names (element=IDENTIFIER, separator=DOT)

contains

public static boolean contains(java.util.List<IToken> tokens,
                               ETokenType tokenType)

Returns true if the list of tokens contains a token of tokenType, false otherwise.







contains
public static boolean contains(java.util.List<IToken> tokens,
                               int startIndex,
                               int endIndex,
                               ETokenType tokenType)
Returns true if the list of tokens contains a token of
 tokenType in the given range, false otherwise.








contains
public static boolean contains(java.util.List<IToken> tokens,
                               ETokenType.ETokenClass tokenClass)
Returns true if the list of tokens contains a token of
 tokenClass, false otherwise.








containsAny
public static boolean containsAny(java.util.List<IToken> tokens,
                                  ETokenType... tokenTypes)
Returns true if the list of tokens contains a any token of given
 tokenTypes, false otherwise.








containsAny
public static boolean containsAny(java.util.List<IToken> tokens,
                                  java.util.EnumSet<ETokenType> tokenTypeSet)
Returns true if the given token stream contains one of the
 tokens in the given set.







containsAny
public static boolean containsAny(java.util.List<IToken> tokens,
                                  int startIndex,
                                  int endIndex,
                                  ETokenType... tokenTypes)
Returns true if the list of tokens contains a any token of given
 tokenTypes in the given range, false otherwise.








containsAny
public static boolean containsAny(java.util.List<IToken> tokens,
                                  int startIndex,
                                  int endIndex,
                                  java.util.EnumSet<ETokenType> tokenTypes)
Returns true if the list of tokens contains a any token of given
 tokenTypes in the given range, false otherwise.








containsAll
public static boolean containsAll(java.util.List<IToken> tokens,
                                  ETokenType... tokenTypes)
Returns true if the list of tokens contains at least one token
 of each of the given tokenTypes, false otherwise.








containsAll
public static boolean containsAll(java.util.List<IToken> tokens,
                                  int startIndex,
                                  int endIndex,
                                  ETokenType... tokenTypes)
Returns true if the list of tokens contains at least one token
 of each of the given tokenTypes in the given range, false
 otherwise.








containsSequence
public static boolean containsSequence(java.util.List<IToken> tokens,
                                       int startIndex,
                                       int endIndex,
                                       ETokenType... tokenTypes)
Returns true if the list of tokens contains a consecutive
 subsequence of tokens that match the sequence of token types., false
 otherwise.







containsSequence
public static boolean containsSequence(java.util.List<IToken> tokens,
                                       ETokenType... tokenTypes)
Returns true if the list of tokens contains a consecutive
 subsequence of tokens that match the sequence of token types., false
 otherwise.







tokensBetween
public static java.util.List<IToken> tokensBetween(java.util.List<IToken> tokens,
                                                   ETokenType startType,
                                                   ETokenType endType)
Returns the sublist of tokens between the first occurrence of given start
 token type and the first occurrence of end token type (after start). If one
 of them is not found, an empty list is returned. The tokens for the start and
 end are not included in the returned sub list.







tokensBetween
public static java.util.List<IToken> tokensBetween(java.util.List<IToken> tokens,
                                                   java.util.EnumSet<ETokenType> startTypes,
                                                   java.util.EnumSet<ETokenType> endTypes)
Returns the sublist of tokens between the first occurrence of some given
 start token types and the first occurrence of any of the given end token
 types (after start). If no matches are found, an empty list is returned. The
 tokens for the start and end are not included in the returned sub list.







findMatchingClosingToken
public static int findMatchingClosingToken(java.util.List<IToken> tokens,
                                           int currentOffset,
                                           ETokenType openingType,
                                           ETokenType closingType)
Returns the offset of the first token of closingType, thereby counting
 nesting occurrences of openingType and closingType. Returns
 NOT_FOUND if the token before currentOffset isn't equal to the
 openingType or if no token with closingType was found.

Parameters:
currentOffset - this is the offset to start the search and must be one token
            after the opening token for which the closing token shall
            be found.








findMatchingClosingToken
public static int findMatchingClosingToken(java.util.List<IToken> tokens,
                                           int currentOffset,
                                           java.util.EnumSet<ETokenType> openingTypes,
                                           java.util.EnumSet<ETokenType> closingTypes)
Returns the offset of the first token of closingType, thereby counting
 nesting occurrences of openingType and closingType. Returns
 NOT_FOUND if none was found.

Parameters:
currentOffset - this is the offset to start the search and must be one token
            after the opening token for which the closing token shall
            be found.








findMatchingOpeningToken
public static int findMatchingOpeningToken(java.util.List<IToken> tokens,
                                           int currentOffset,
                                           ETokenType openingType,
                                           ETokenType closingType)
Returns the offset of the first token of openingType, thereby counting
 nesting occurrences of openingType and closingType. Returns
 NOT_FOUND if none was found.

Parameters:
currentOffset - this is the offset to start the search and must be one token
            before the closing token for which the opening token shall
            be found.








findFirstTopLevel
public static int findFirstTopLevel(java.util.List<IToken> tokens,
                                    ETokenType searchType,
                                    java.util.List<ETokenType> openingTypes,
                                    java.util.List<ETokenType> closingTypes)
Returns the index of the first top-level token that has the given search type
 regarding the nesting introduced by the opening and closing types. If no such
 token is found NOT_FOUND is returned.







findFirstTopLevel
public static int findFirstTopLevel(java.util.List<IToken> tokens,
                                    java.util.EnumSet<ETokenType> searchTypes,
                                    java.util.List<ETokenType> openingTypes,
                                    java.util.List<ETokenType> closingTypes)
Returns the index of the first top-level token that has one of the given
 search types regarding the nesting introduced by the opening and closing
 types. If no such token is found NOT_FOUND is returned.







findFirstTopLevel
public static int findFirstTopLevel(java.util.List<IToken> tokens,
                                    int startIndex,
                                    java.util.EnumSet<ETokenType> searchTypes,
                                    java.util.List<ETokenType> openingTypes,
                                    java.util.List<ETokenType> closingTypes)
Returns the index of the first top-level token starting at the given index
 that has one of the given search types regarding the nesting introduced by
 the opening and closing types. If no such token is found NOT_FOUND
 is returned.







findFirstTopLevel
public static int findFirstTopLevel(java.util.List<IToken> tokens,
                                    int startIndex,
                                    java.util.function.Predicate<IToken> searchPredicate,
                                    java.util.List<ETokenType> openingTypes,
                                    java.util.List<ETokenType> closingTypes)
Returns the index of the first top-level token starting at the given index
 that matches the given token predicate regarding the nesting introduced by
 the opening and closing types. If no such token is found NOT_FOUND
 is returned.







findFirstTopLevelWithIndexPredicate
public static int findFirstTopLevelWithIndexPredicate(java.util.List<IToken> tokens,
                                                      int startIndex,
                                                      java.util.function.Predicate<java.lang.Integer> searchPredicate,
                                                      java.util.List<ETokenType> openingTypes,
                                                      java.util.List<ETokenType> closingTypes)
Returns the index of the first top-level token starting at the given index
 that matches the given token predicate regarding the nesting introduced by
 the opening and closing types. If no such token is found NOT_FOUND
 is returned.







tokenTypesAt
@Deprecated
public static boolean tokenTypesAt(java.util.List<IToken> tokens,
                                               int startOffset,
                                               ETokenType... tokenTypes)
Deprecated. use hasTokenTypeSequence(List, int, ETokenType...)
             instead. I leave the method here to avoid hidden merge conflicts.
             Can be removed after some releases.
Returns whether the next tokens starting at the given offset are of given
 type.







startsWith
public static boolean startsWith(java.util.List<IToken> tokens,
                                 ETokenType... sequence)
Returns whether the given token list starts with given token type sequence.







endsWith
public static boolean endsWith(java.util.List<IToken> tokens,
                               ETokenType... sequence)
Returns whether the given token list ends with given token type sequence.







count
public static int count(java.util.List<IToken> tokens,
                        ETokenType tokenType)
Counts the number of tokens of a given type in the list of tokens







countWithoutNesting
public static int countWithoutNesting(java.util.List<IToken> tokens,
                                      ETokenType tokenType,
                                      ETokenType openingType,
                                      ETokenType closingType)
Counts the number of tokens of a given type in the list of tokens that are
 not nested between the given nesting tokens.







countWithoutNesting
public static int countWithoutNesting(java.util.List<IToken> tokens,
                                      ETokenType tokenType,
                                      java.util.List<ETokenType> openingType,
                                      java.util.List<ETokenType> closingType)
Counts the number of tokens of a given type in the list of tokens that are
 not nested between the given nesting tokens.







findAllTokens
public static java.util.List<IToken> findAllTokens(java.util.List<IToken> tokens,
                                                   java.util.EnumSet<ETokenType> types)
Returns all tokens of the given token types in the given token list.







findAll
public static java.util.List<java.lang.Integer> findAll(java.util.List<IToken> tokens,
                                                        java.util.EnumSet<ETokenType> types)
Returns all indices of the given token types in the given token list.







findAll
public static java.util.List<java.lang.Integer> findAll(java.util.List<IToken> tokens,
                                                        int startOffset,
                                                        int endOffset,
                                                        java.util.EnumSet<ETokenType> types)
Returns all indices of the given token types in the given token list,
 beginning from startOffset (inclusive) to endOffset (exclusive).







split
public static java.util.List<java.util.List<IToken>> split(java.util.List<IToken> tokens,
                                                           java.util.EnumSet<ETokenType> splitTypes)
Splits the given token stream at the tokens of the given types.







split
public static java.util.List<java.util.List<IToken>> split(java.util.List<IToken> tokens,
                                                           ETokenType... splitTypes)
Splits the given token stream at the tokens of the given types.







split
public static java.util.List<java.util.List<IToken>> split(java.util.List<IToken> tokens,
                                                           java.util.EnumSet<ETokenType> splitTypes,
                                                           int limit)
Splits the given token stream at the tokens of the given types, but at most
 as often as given in the limit parameter. The resulting list will have at
 most limit entries). The limit must be bigger than 0.







split
public static java.util.List<java.util.List<IToken>> split(java.util.List<IToken> tokens,
                                                           java.util.function.Predicate<IToken> splitTokenMatcher,
                                                           int limit)
                                                    throws java.lang.AssertionError
Splits the given token stream at the tokens matching the given predicate, but
 at most as often as given in the limit parameter. The resulting list will
 have at most limit entries). The limit must be bigger than 0.

Throws:
java.lang.AssertionError








hasTokenTypeSequence
public static boolean hasTokenTypeSequence(java.util.List<IToken> tokens,
                                           int startOffset,
                                           ETokenType... sequence)
Returns whether the given token list contains tokens with the given types in
 that order starting at the given start offset.
 
 This method returns false when called with invalid offsets.







containsTokenIndexPredicate
public static boolean containsTokenIndexPredicate(java.util.List<IToken> tokens,
                                                  java.util.function.BiPredicate<java.lang.Integer,java.util.List<IToken>> predicate)
Returns whether an index of the given token list matches the given
 BiPredicate (which gets the token index and the given token list).







firstTokenMatchingIndexPredicate
public static int firstTokenMatchingIndexPredicate(java.util.List<IToken> tokens,
                                                   int startIndex,
                                                   java.util.function.BiPredicate<java.lang.Integer,java.util.List<IToken>> predicate)
Returns the first index of the given token list that matches the given
 BiPredicate (which gets the token index and the given token list).
 This method iterates over the indices starting with the given startIndex.







firstTokenOfTypeSequence
public static int firstTokenOfTypeSequence(java.util.List<IToken> tokens,
                                           int startOffset,
                                           int endIndex,
                                           ETokenType... sequence)
Returns the index (between the given start and end indices) of the first
 token that starts a sequence of the given token types. If no such token is
 found NOT_FOUND is returned.







allStartingIndicesOfTypeSequence
public static java.util.List<java.lang.Integer> allStartingIndicesOfTypeSequence(java.util.List<IToken> tokens,
                                                                                 int startOffset,
                                                                                 int endIndex,
                                                                                 ETokenType... sequence)
Returns the indices of all tokens that starts a sequence of the given token
 types (and are between startOffset and endIndex). If no such token is found,
 an empty List is returned.







firstTokenOfTypeSequence
public static int firstTokenOfTypeSequence(java.util.List<IToken> tokens,
                                           int startOffset,
                                           ETokenType... sequence)
Returns the index of the first occurrence of the given sequence of token
 types in the given token list, beginning from the given start offset. If the
 sequence if not found, NOT_FOUND is returned.







firstTokenOfTypeSequences
public static java.util.List<java.lang.Integer> firstTokenOfTypeSequences(java.util.List<IToken> tokens,
                                                                          int offset,
                                                                          ETokenType... sequence)
Returns all indices of occurrences of the given sequence of token types in
 the given token list, beginning from the given offset. If the sequence is not
 found, an empty list is returned.







hasTypes
public static boolean hasTypes(java.util.List<IToken> tokens,
                               ETokenType... types)
Returns whether the given list's token types equal the given token types.







tokensBetweenWithNesting
public static java.util.List<IToken> tokensBetweenWithNesting(java.util.List<IToken> tokens,
                                                              ETokenType openingType,
                                                              ETokenType closingType)
Returns all tokens between the tokens of the opening and closing types
 regarding nesting.







tokensBetweenWithNesting
public static java.util.List<IToken> tokensBetweenWithNesting(java.util.List<IToken> tokens,
                                                              int startOffset,
                                                              ETokenType openingType,
                                                              ETokenType closingType)
Returns all tokens between the tokens of the opening and closing types
 regarding nesting beginning from the given start offset.







splitWithNesting
public static java.util.List<java.util.List<IToken>> splitWithNesting(java.util.List<IToken> tokens,
                                                                      ETokenType splitType,
                                                                      ETokenType openingType,
                                                                      ETokenType closingType)
Splits the given token list at the tokens of the given split type. If the
 split type is nested between the open and close tokens, it is ignored.







splitWithNesting
public static java.util.List<java.util.List<IToken>> splitWithNesting(java.util.List<IToken> tokens,
                                                                      ETokenType splitType,
                                                                      java.util.List<ETokenType> openingTypes,
                                                                      java.util.List<ETokenType> closingTypes)
Splits the given token list at the tokens of the given split type. If the
 split type is nested between one of the open and close tokens, it is ignored.
 The open and close token type lists must have the same size. None of both
 must contain duplicates.







splitWithNesting
public static java.util.List<java.util.List<IToken>> splitWithNesting(java.util.List<IToken> tokens,
                                                                      java.util.EnumSet<ETokenType> splitTypes,
                                                                      java.util.List<ETokenType> openingTypes,
                                                                      java.util.List<ETokenType> closingTypes,
                                                                      int limit)
Splits the given token list at the tokens of the given split types. If one of
 the split types is nested between one of the open and close tokens, it is
 ignored. The resulting list will have at most limit entries. Limit must be
 greater zero. The open and close token type lists must have the same size.
 None of both must contain duplicates.







createToken
public static IToken createToken(IToken referenceToken,
                                 java.lang.String text,
                                 ETokenType type)
Creates a new token from the given referenceToken.

Parameters:
referenceToken - Reference token, whose offset, line number and origin id will be
            copied to the new token.
text - Text of the newly created token.
type - Type of the newly created token.
Returns:
New token with the given text and type.








copyTokens
public static java.util.List<IToken> copyTokens(IToken referenceToken,
                                                java.util.List<IToken> tokens)
Creates a new token list copying the given list and using the given reference
 token.

Parameters:
referenceToken - Reference token, whose offset, line number and origin id will be
            copied to the new tokens.
tokens - List of tokens to be duplicated.








addToken
public static void addToken(java.util.List<IToken> tokens,
                            int position,
                            ETokenType type,
                            java.lang.String text)
Adds a new token to the given token list at the given position with the given
 type and text. The token that is at that position currently is taken as a
 reference. The new token will have the same offset, origin ID and line number
 as the reference token.

Parameters:
tokens - The token list to modify. Must not be unmodifiable or empty.
position - The position at which to insert the new token. Also the position
            of the reference token. If this is equal to the size of the token
            list, the new token will be appended at the end and the last token
            will be taken as the reference token.
type - The token type of the new token.
text - The text of the new token.








removeAtEnd
public static java.util.List<IToken> removeAtEnd(java.util.List<IToken> tokens,
                                                 ETokenType type)
Removes the last token of the given token list if it matches the specified
 token type.

Parameters:
tokens - The token list to modify.
type - Token type, which should be removed.
Returns:
The token list without the given token type at the end.








removeAtFront
public static java.util.List<IToken> removeAtFront(java.util.List<IToken> tokens,
                                                   ETokenType type)
Removes the first token of the given token list if it matches the specified
 token type.

Parameters:
tokens - The token list to modify.
type - Token type, which should be removed.
Returns:
The token list without the given token type at the beginning.








getTokenByTypeAndText
public static IToken getTokenByTypeAndText(java.util.List<IToken> tokens,
                                           java.lang.String tokenText,
                                           java.util.Set<ETokenType> tokenTypes)
Searches for the first token with the given type and text (ignoring case).

Parameters:
tokens - The list of tokens to be searched.
tokenText - The searched token should have this text (ignoring case).
tokenTypes - The searched token should have one of these types.
Returns:
The first token that matches, null if none is found.








getTokenTextsByType
public static java.util.List<java.lang.String> getTokenTextsByType(java.util.List<IToken> tokens,
                                                                   ETokenType tokenType)
Filters the tokens by type and returns the text of each filtered token.

Parameters:
tokens - The list of tokens to be searched.
tokenType - The type for which filtering is performed.
Returns:
A list containing the texts of all tokens of the passed type. Empty
         list if no token was found.








indexOfByOffset
public static int indexOfByOffset(java.util.List<IToken> tokens,
                                  int offset)
Binary search for an IToken in the given list that has exactly the
 given offset (IToken.getOffset()). Assumes that the tokens in the
 given list are ordered by offset. Returns the index of the found token in the
 given list or -1 if no such token is found.
 
 This also works on token lists where offsets are not *strictly* increasing
 (e.g., due to preprocessor expansions). In this case, returns the index of
 the first token that has the given offset.

Parameters:
tokens - The list of tokens in which to search
offset - The offset for which the token is to find
Returns:
index of the token in the list or -1








indicesOfByPredicate
public static java.util.List<java.lang.Integer> indicesOfByPredicate(java.util.List<IToken> tokens,
                                                                     java.util.function.Predicate<IToken> includeToken)
Returns the indices of the tokens in the given list that satisfy the given
 predicate.







getLanguage
public static ELanguage getLanguage(java.util.Collection<IToken> tokens)
Returns the ELanguage of the given tokens (assuming all tokens have
 the same language). Returns null if the given collection is
 empty.







getTokensInSameLine
public static java.util.List<IToken> getTokensInSameLine(java.util.List<IToken> tokens,
                                                         int index)
Returns a list of tokens existing in the same line of a given token, which
 specified by its index.







startsNewStatement
public static boolean startsNewStatement(IToken token,
                                         IToken lastToken,
                                         java.util.EnumSet<ETokenType> continueTokens,
                                         java.util.EnumSet<ETokenType> nonContinuationTokens,
                                         ETokenType newStatementToken)
Checks if the token is the start of a new statement.

Parameters:
token - The current token.
lastToken - The previous token.
continueTokens - Token types that force a statement to continue even in a new line.
nonContinuationTokens - Token types of class OPERATOR that do not force a statement
            continuation.
newStatementToken - Token that indicates the start of a new statement.
Returns:
True if it is the start of a new statement otherwise false.

Class TokenStreamUtils

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

NOT_FOUND

Constructor Detail

TokenStreamUtils

Method Detail

firstTokenOfType

firstTokenOfType

firstTokenOfType

firstTokenOfType

lastTokenOfType

lastTokenOfType

lastTokenOfType

lastTokenOfType

lastTokenOfType

lastTokenOfType

firstTokenOfAlternatingTypes

contains

contains

contains

containsAny

containsAny

containsAny

containsAny

containsAll

containsAll

containsSequence

containsSequence

tokensBetween

tokensBetween

findMatchingClosingToken

findMatchingClosingToken

findMatchingOpeningToken

findFirstTopLevel

findFirstTopLevel

findFirstTopLevel

findFirstTopLevel

findFirstTopLevelWithIndexPredicate

tokenTypesAt

startsWith

endsWith

count

countWithoutNesting

countWithoutNesting

findAllTokens

findAll

findAll

split

split

split

split

hasTokenTypeSequence

containsTokenIndexPredicate

firstTokenMatchingIndexPredicate

firstTokenOfTypeSequence

allStartingIndicesOfTypeSequence

firstTokenOfTypeSequence

firstTokenOfTypeSequences

hasTypes

tokensBetweenWithNesting

tokensBetweenWithNesting

splitWithNesting

splitWithNesting

splitWithNesting

createToken

copyTokens

addToken

removeAtEnd

removeAtFront

getTokenByTypeAndText

getTokenTextsByType

indexOfByOffset

indicesOfByPredicate

getLanguage

getTokensInSameLine

startsNewStatement