bugGrammatica - Bugs: bug #5060, Tokenizer should be able to run in...


You are not allowed to post comments on this tracker with your current authentication level.

bug #5060: Tokenizer should be able to run in case-insensitive mode

Submitted by:  Per Cederberg <cederberg>
Submitted on:  Thu 04 Sep 2003 12:24:33 AM UTC  
Severity:  5 - Major Item Group:  Future Improvement
Status:  Fixed Assigned to:  Per Cederberg <cederberg>
Open/Closed:  Closed

Sat 14 Mar 2009 09:36:11 AM UTC, comment #5: 

Version 1.5 was released on 2009-03-07. Closing this issue.

Per Cederberg <cederberg>
Project AdministratorIn charge of this item.
Thu 22 Apr 2004 10:26:31 PM UTC, comment #4: 

The new case-insensitive Tokenizer mode has been merged some time ago. The fix is available in CVS and will be released in version 1.5.

Per Cederberg <cederberg>
Project AdministratorIn charge of this item.
Fri 19 Mar 2004 10:10:40 AM UTC, comment #3: 

I'll be merging your patch for case-insensitivity but with some modifications. First as we don't have a complete solution, I think it is better to make the tokenizer constructor take a "ignoreCase" flag. That way we won't be promising case-insensitivity per token, which we can't deliver anyway. Next I believe the right way to set this flag in the grammar file would be with an optional header declaration CASE_IGNORE = "yes" (or "true").

I think the solution outlined previously here should be passed to bug #8202. Also, it seems like %caseignore% is the best name so far.

Per Cederberg <cederberg>
Project AdministratorIn charge of this item.
Sun 14 Mar 2004 11:33:04 PM UTC, comment #2: 

%ignorecase% is currently interpreted as %ignore with a message set to "case"

I vote for %caseignore% since this is what I've been using to add this functionality. I've only added case-insensitive support for string token matches since, as you said, this can already be done with regexp.

Adrian Moore <AgentSmith>
Sun 14 Mar 2004 11:37:23 AM UTC, comment #1: 

Maybe I should clarify that this bug really consists of two parts. First the string tokens (with "blahblah" syntax) must support case-insensitive processing. This requires changes to the DFA creation methods in the Tokenizer class in order to add two arcs (one for each case) between the states. In order to make this memory efficient, though, the two arcs should go to the SAME state UNLESS there are other tokens sharing the destination state. If such are detected, two states must be used and that can be a bit tricky to get right.

The second part is to make the regexp tokens case-insensitive. This is not as important as the first part as it can easily be worked around by writing things as <<[a-zAZ]>> but would nonetheless be a good thing to have. For the SQL case it wouldn't be needed, though.

A design decision must also be made regarding the keyword to use after the token pattern. Some possible alternatives would be:


Per Cederberg <cederberg>
Project AdministratorIn charge of this item.
Thu 04 Sep 2003 12:24:33 AM UTC, original submission:  

The Tokenizer class currently only parses tokens in case-sensitive mode, i.e. exactly as they are stated in the grammar file. For some languages, however, this is problematic . In SQL, for instance, all keywords are by default case-insensitive, which makes a correct SQL grammar unneedingly complex to write.

An intermediate solution to this, would be to only support case-insensitive string token patterns. This could be accomplished by adding a %caseignore% flag after the token definition. Having a global case-ignore flag is not as flexible, although it is the way that other tools (such as flex) operates.

Per Cederberg <cederberg>
Project AdministratorIn charge of this item.


No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by cederberg (Posted a comment)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-03-14 cederberg Open/ClosedOpen => Closed
        Discussion LockUnlocked => Locked
    2004-04-22 cederberg StatusNone => Fixed
    2004-03-19 cederberg Assigned toNone => cederberg
    2004-03-14 cederberg Assigned tocederberg => None

    Back to the top

    Powered by Savane 3.6