bugVarnamproject - Bugs: bug #40412, [libvarnam] Duplicate tokens in...

 
 

bug #40412: [libvarnam] Duplicate tokens in the tokens list

Submitter:  Navaneeth <navaneethkn>
Submitted:  Tue 29 Oct 2013 02:11:33 PM UTC
   
 
Category:  libvarnam Severity:  4 - Important
Item Group:  * Bug Status:  Fixed
Privacy:  Public Assigned to:  navaneethkn
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 09 Jan 2014 09:26:00 AM UTC, comment #1: 
Navaneeth <navaneethkn>
Group administrator
Tue 29 Oct 2013 02:11:33 PM UTC, original submission:  

It looks like `libvarnam` is adding duplicate items into the tokens list. For eg, tokenization for the word "അതുറങ്ങിയിട്ടുണ്ടായിരുന്നു".

[1 - 'a']
[4 - 'thu', 4 - 'tu']
[2 - 'ra', 2 - 'ra']
[4 - 'ngi', 4 - 'ngi', 4 - 'ngy', 4 - 'ngy']
[4 - 'yi', 4 - 'yy']
[4 - 'ttu']
[4 - 'nda', 4 - 'ndaa', 4 - 'nda', 4 - 'nda', 4 - 'ndaa', 4 - 'nda', 4 - 'nta', 4 - 'ntaa', 4 - 'nta']
[4 - 'yi', 4 - 'yy']
[4 - 'ru']
[4 - 'nnu']

Navaneeth <navaneethkn>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one

Digest:
   task dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by navaneethkn (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2014-01-09 navaneethkn StatusNone Fixed
        Open/ClosedOpen Closed
    2013-11-18 navaneethkn Dependencies- task #12936 is dependent
    2013-11-18 navaneethkn SummaryRepeated tokens in the tokens list [libvarnam] Duplicate tokens in the tokens list

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code