bugGrammatica - Bugs: bug #23818, Tokenizer can prematurely detect...


You are not allowed to post comments on this tracker with your current authentication level.

bug #23818: Tokenizer can prematurely detect end of file

Submitter:  Jeremy M Stone <jeremystone>
Submitted:  Fri 11 Jul 2008 02:17:14 PM UTC
Severity:  5 - Major Item Group:  Software
Status:  Fixed Assigned to:  cederberg
Open/Closed:  Closed

Discussion locked!

Sat 14 Mar 2009 09:33:06 AM UTC, comment #2: 

Version 1.5 was released on 2009-03-07. Closing this issue.

Per Cederberg <cederberg>
Group administrator
Sat 07 Feb 2009 09:21:35 PM UTC, comment #1: 

Thanks for the detailed bug report and sorry about the slow response.

Fixed in r120 -- http://code.google.com/p/grammatica/source/detail?r=120

Per Cederberg <cederberg>
Group administrator
Fri 11 Jul 2008 02:17:14 PM UTC, original submission:  

Tokenizer.readInput() method stops reading (assuming end of file) when it fails to read a full buffer. This can lead to unexpected end of file errors when parsing data loaded via an url (typically when using http over a slow link it is worse).

Should check length == -1 rather than length < chars.length.

Looks like the problem still exists in v1.5 in the LookAheadReader class.

Jeremy M Stone <jeremystone>


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by cederberg (Posted a comment)
  • -email is unavailable- added by jeremystone (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.


    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-03-14 cederberg Open/ClosedOpen Closed
        Discussion LockNone Locked
    2009-02-07 cederberg StatusNone Fixed
        Assigned toNone cederberg

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code