bugGrammatica - Bugs: bug #5060, Tokenizer should be able to run in...


You are not allowed to post comments on this tracker with your current authentication level.

bug #5060: Tokenizer should be able to run in case-insensitive mode

Submitter:  Per Cederberg <cederberg>
Submitted:  Thu 04 Sep 2003 12:24:33 AM UTC
Severity:  5 - Major Item Group:  Future Improvement
Status:  Fixed Assigned to:  cederberg
Open/Closed:  Closed

Discussion locked!

Sat 14 Mar 2009 09:36:11 AM UTC, comment #5: 

Version 1.5 was released on 2009-03-07. Closing this issue.

Per Cederberg <cederberg>
Group administrator
Thu 22 Apr 2004 10:26:31 PM UTC, comment #4: 

The new case-insensitive Tokenizer mode has been merged some time ago. The fix is available in CVS and will be released in version 1.5.

Per Cederberg <cederberg>
Group administrator
Fri 19 Mar 2004 10:10:40 AM UTC, comment #3: 

I'll be merging your patch for case-insensitivity but with some modifications. First as we don't have a complete solution, I think it is better to make the tokenizer constructor take a "ignoreCase" flag. That way we won't be promising case-insensitivity per token, which we can't deliver anyway. Next I believe the right way to set this flag in the grammar file would be with an optional header declaration CASE_IGNORE = "yes" (or "true").

I think the solution outlined previously here should be passed to bug #8202. Also, it seems like %caseignore% is the best name so far.

Per Cederberg <cederberg>
Group administrator
Sun 14 Mar 2004 11:33:04 PM UTC, comment #2: 

%ignorecase% is currently interpreted as %ignore with a message set to "case"

I vote for %caseignore% since this is what I've been using to add this functionality. I've only added case-insensitive support for string token matches since, as you said, this can already be done with regexp.

Adrian Moore <AgentSmith>
Sun 14 Mar 2004 11:37:23 AM UTC, comment #1: 

Maybe I should clarify that this bug really consists of two parts. First the string tokens (with "blahblah" syntax) must support case-insensitive processing. This requires changes to the DFA creation methods in the Tokenizer class in order to add two arcs (one for each case) between the states. In order to make this memory efficient, though, the two arcs should go to the SAME state UNLESS there are other tokens sharing the destination state. If such are detected, two states must be used and that can be a bit tricky to get right.

The second part is to make the regexp tokens case-insensitive. This is not as important as the first part as it can easily be worked around by writing things as <<[a-zAZ]>> but would nonetheless be a good thing to have. For the SQL case it wouldn't be needed, though.

A design decision must also be made regarding the keyword to use after the token pattern. Some possible alternatives would be:


Per Cederberg <cederberg>
Group administrator
Thu 04 Sep 2003 12:24:33 AM UTC, original submission:  

The Tokenizer class currently only parses tokens in case-sensitive mode, i.e. exactly as they are stated in the grammar file. For some languages, however, this is problematic . In SQL, for instance, all keywords are by default case-insensitive, which makes a correct SQL grammar unneedingly complex to write.

An intermediate solution to this, would be to only support case-insensitive string token patterns. This could be accomplished by adding a %caseignore% flag after the token definition. Having a global case-ignore flag is not as flexible, although it is the way that other tools (such as flex) operates.

Per Cederberg <cederberg>
Group administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by cederberg (Posted a comment)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.


    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-03-14 cederberg Open/ClosedOpen Closed
        Discussion LockNone Locked
    2004-04-22 cederberg StatusNone Fixed
    2004-03-19 cederberg Assigned toNone cederberg
    2004-03-14 cederberg Assigned tocederberg None

    Back to the top

    Powered by Savane 3.13-12ab.
    Corresponding source code