The NFA implementation has landed in trunk (for version 1.5). Further quick gains in tokenizer processing speed seems improbable.

The worst performance problems are associated with the
greedy and reluctant quantifiers, as they need to support
backtracking. When the tokenizer buffer doesn't contain a
complete match, a quantifier regular expression will
backtrack until all possibilities have been evaluated. The
backtracking cannot be removed, as regular expressions such
as "a*a" must be allowed.

In order to avoid the backtracking, and thus improve
performance, the possessive quantifiers can be used
instead. These are the '*+' and '++' quantifiers that
only match the maximum number of characters.