bugSilpa Indian Language Processing Project - Bugs: bug #29678, Word loss during reading of...

 
 

bug #29678: Word loss during reading of english dictionary

Submitted by:  Vasudev Kamath <vasudev>
Submitted on:  Mon 26 Apr 2010 04:46:04 PM UTC  
 
Category: NoneSeverity: 2 - Minor
Item Group: NoneStatus: Confirmed
Privacy: PublicAssigned to: Vasudev Kamath <vasudev>
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Thu 29 Apr 2010 04:40:23 PM UTC, comment #1:

Conversion of en_US dictionary encoding to UTF-8 encoding solves this issue. For converting dictionary to UTF-8 we can use the following command
iconv -f ISO-8859-1 -t UTF-8 en_US.dic > en_US_utf-8.dic then remove the old en_US dictionary and rename en_US_utf-8 dictionary.
After conversion the reading with utf-8 doesn't cause any problem

I think we can close this bug

Vasudev Kamath <vasudev>
Project MemberIn charge of this item.
Mon 26 Apr 2010 04:46:04 PM UTC, original submission:

In the spell checker module the reading of dictionaries using codecs.open with utf-8 encoding causes word loss for en_US.dic file. This essentially because the en_US.dic is not a utf-8 text file instead its ISO-8859 text file. I'm attaching a patch file which is difference between the original dictionary and file saved by reading the dictionary in utf-8 encoding and saving via a python script.

Vasudev Kamath <vasudev>
Project MemberIn charge of this item.

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #20337:  difference_dic.patch added by vasudev (7KiB - text/x-diff - Difference between original dictionary and dictionary saved via python script)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by vasudev (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 2 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Thu 29 Apr 2010 04:40:59 PM UTCvasudevOpen/ClosedOpen=>Closed
    Mon 26 Apr 2010 04:46:04 PM UTCvasudevAttached File-=>Added difference_dic.patch, #20337

    Back to the top


    Powered by Savane 3.1-cleanup1