Grammatica - Bugs: bug #8202, Tokenizer: Create a single DFA or...
You are not allowed to post comments on this tracker with your current authentication level.
bug #8202: Tokenizer: Create a single DFA or NFA for all tokens
Submitter: | Per Cederberg <cederberg> | ||
Submitted: | Fri 19 Mar 2004 10:08:13 AM UTC | ||
Severity: | 5 - Major | Item Group: | Future Improvement |
Status: | Fixed | Assigned to: | cederberg |
Open/Closed: | Closed |
Discussion locked!
Sat 14 Mar 2009 09:35:45 AM UTC, comment #3: |
Per Cederberg <cederberg>![]() ![]() |
Tue 24 Feb 2009 09:56:08 PM UTC, comment #2: Fixed in trunk. A mixed approach finally provided the optimal performance; using a DFA for strings, an NFA for all simple regular expressions and native Java regular expressions for the rest. The previous built-in regular expression library is now unused and might be removed in a future version. |
Per Cederberg <cederberg>![]() ![]() |
Sun 22 May 2005 09:32:46 AM UTC, comment #1: Hmm... This issue seems more convoluted that I first anticipated. It turns out that modern RE:s aren't regular expressions in the linguistic sense anymore. Actually, they are quite a bit more expressive and impossible to handle correctly with a simple DFA model or with the linguistically equivalent NFA.
|
Per Cederberg <cederberg>![]() ![]() |
Fri 19 Mar 2004 10:08:13 AM UTC, original submission:
Currently (version 1.4) the Tokenizer class creates a DFA (Deterministic Finite Automaton) with the string tokens, while the regular expression tokens are processed one at a time. It would be preferable to process all the tokens at once in a DFA as the tokenizer speed would thereby be much improved (see bug #3603).
|
Per Cederberg <cederberg>![]() ![]() |
No files currently attached
Depends on the following items: None found
There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
Follow 6 latest changes.
Date | Changed by | Updated Field | Previous Value | => | Replaced by |
---|---|---|---|---|---|
2009-03-14 | cederberg | Open/Closed | Open | ![]() |
Closed |
Discussion Lock | Unlocked | ![]() |
Locked | ||
2009-02-24 | cederberg | Status | None | ![]() |
Fixed |
Assigned to | None | ![]() |
cederberg | ||
2005-05-21 | cederberg | Dependencies | - | ![]() |
bugs #11737 is dependent |
2004-03-19 | cederberg | Dependencies | - | ![]() |
bugs # is dependent |
Version 1.5 was released on 2009-03-07. Closing this issue.