bugVarnamproject - Bugs: bug #40412, [libvarnam] Duplicate tokens in...

 
 

bug #40412: [libvarnam] Duplicate tokens in the tokens list

Submitted by:  Navaneeth <navaneethkn>
Submitted on:  Tue 29 Oct 2013 02:11:33 PM UTC  
 
Category: libvarnamSeverity: 4 - Important
Item Group: BugStatus: Fixed
Privacy: PublicAssigned to: Navaneeth <navaneethkn>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 09 Jan 2014 09:26:00 AM UTC, comment #1:

Fixed in - https://gitorious.org/varnamproject/libvarnam/commit/34bd1a5138d62ffbd322751f289b3b0f48ff02e4

Navaneeth <navaneethkn>
Project AdministratorIn charge of this item.
Tue 29 Oct 2013 02:11:33 PM UTC, original submission:

It looks like `libvarnam` is adding duplicate items into the tokens list. For eg, tokenization for the word "അതുറങ്ങിയിട്ടുണ്ടായിരുന്നു".

[1 - 'a']
[4 - 'thu', 4 - 'tu']
[2 - 'ra', 2 - 'ra']
[4 - 'ngi', 4 - 'ngi', 4 - 'ngy', 4 - 'ngy']
[4 - 'yi', 4 - 'yy']
[4 - 'ttu']
[4 - 'nda', 4 - 'ndaa', 4 - 'nda', 4 - 'nda', 4 - 'ndaa', 4 - 'nda', 4 - 'nta', 4 - 'ntaa', 4 - 'nta']
[4 - 'yi', 4 - 'yy']
[4 - 'ru']
[4 - 'nnu']

Navaneeth <navaneethkn>
Project AdministratorIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one

Digest:
   task dependencies.

 

Carbon-Copy List
  • -unavailable- added by navaneethkn (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 3 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 09 Jan 2014 09:26:00 AM UTCnavaneethknStatusNone=>Fixed
      Open/ClosedOpen=>Closed
    Mon 18 Nov 2013 09:26:07 AM UTCnavaneethknSummaryRepeated tokens in the tokens list =>[libvarnam] Duplicate tokens in the tokens list

    Back to the top


    Powered by Savane 3.1-cleanup1