bugSilpa Indian Language Processing Project - Bugs: bug #29678, Word loss during reading of...

 
 

bug #29678: Word loss during reading of english dictionary

Submitter:  Vasudev Kamath <vasudev>
Submitted:  Mon 26 Apr 2010 04:46:04 PM UTC
   
 
Category:  None Severity:  2 - Minor
Item Group:  None Status:  Confirmed
Privacy:  Public Assigned to:  vasudev
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 29 Apr 2010 04:40:23 PM UTC, comment #1: 

Conversion of en_US dictionary encoding to UTF-8 encoding solves this issue. For converting dictionary to UTF-8 we can use the following command
iconv -f ISO-8859-1 -t UTF-8 en_US.dic > en_US_utf-8.dic then remove the old en_US dictionary and rename en_US_utf-8 dictionary.
After conversion the reading with utf-8 doesn't cause any problem

I think we can close this bug

Vasudev Kamath <vasudev>
Group Member
Mon 26 Apr 2010 04:46:04 PM UTC, original submission:  

In the spell checker module the reading of dictionaries using codecs.open with utf-8 encoding causes word loss for en_US.dic file. This essentially because the en_US.dic is not a utf-8 text file instead its ISO-8859 text file. I'm attaching a patch file which is difference between the original dictionary and file saved by reading the dictionary in utf-8 encoding and saving via a python script.

Vasudev Kamath <vasudev>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #20337:  difference_dic.patch added by vasudev (7KiB - text/x-diff - Difference between original dictionary and dictionary saved via python script)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by vasudev (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 2 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2010-04-29 vasudev Open/ClosedOpen Closed
    2010-04-26 vasudev Attached File- Added difference_dic.patch, #20337

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code