Grammatica - Bugs: bug #5060, Tokenizer should be able to run in...
You are not allowed to post comments on this tracker with your current authentication level.
bug #5060: Tokenizer should be able to run in case-insensitive mode
Submitter: | Per Cederberg <cederberg> | ||
Submitted: | Thu 04 Sep 2003 12:24:33 AM UTC | ||
Severity: | 5 - Major | Item Group: | Future Improvement |
Status: | Fixed | Assigned to: | cederberg |
Open/Closed: | Closed |
Discussion locked!
Sat 14 Mar 2009 09:36:11 AM UTC, comment #5: |
Per Cederberg <cederberg>![]() ![]() |
Thu 22 Apr 2004 10:26:31 PM UTC, comment #4: The new case-insensitive Tokenizer mode has been merged some time ago. The fix is available in CVS and will be released in version 1.5. |
Per Cederberg <cederberg>![]() ![]() |
Fri 19 Mar 2004 10:10:40 AM UTC, comment #3: I'll be merging your patch for case-insensitivity but with some modifications. First as we don't have a complete solution, I think it is better to make the tokenizer constructor take a "ignoreCase" flag. That way we won't be promising case-insensitivity per token, which we can't deliver anyway. Next I believe the right way to set this flag in the grammar file would be with an optional header declaration CASE_IGNORE = "yes" (or "true").
|
Per Cederberg <cederberg>![]() ![]() |
Sun 14 Mar 2004 11:33:04 PM UTC, comment #2: %ignorecase% is currently interpreted as %ignore with a message set to "case"
|
Adrian Moore <AgentSmith> |
Sun 14 Mar 2004 11:37:23 AM UTC, comment #1: Maybe I should clarify that this bug really consists of two parts. First the string tokens (with "blahblah" syntax) must support case-insensitive processing. This requires changes to the DFA creation methods in the Tokenizer class in order to add two arcs (one for each case) between the states. In order to make this memory efficient, though, the two arcs should go to the SAME state UNLESS there are other tokens sharing the destination state. If such are detected, two states must be used and that can be a bit tricky to get right.
|
Per Cederberg <cederberg>![]() ![]() |
Thu 04 Sep 2003 12:24:33 AM UTC, original submission:
The Tokenizer class currently only parses tokens in case-sensitive mode, i.e. exactly as they are stated in the grammar file. For some languages, however, this is problematic . In SQL, for instance, all keywords are by default case-insensitive, which makes a correct SQL grammar unneedingly complex to write.
|
Per Cederberg <cederberg>![]() ![]() |
No files currently attached
Depends on the following items: None found
Items that depend on this one: None found
There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.
Version 1.5 was released on 2009-03-07. Closing this issue.