patchSamizdat - Patches: patch #6554, guess_lang patch 0.4.1 [vs...


You are not allowed to post comments on this tracker with your current authentication level.

patch #6554: guess_lang patch 0.4.1 [vs 080703-2]

Submitter:  Boud Roukema <boud>
Submitted:  Thu 26 Jun 2008 12:59:48 AM UTC
Category:  None Priority:  5 - Normal
Status:  Works For Me Privacy:  Public
Assigned to:  None Open/Closed:  Open

Jump to the original submission

Mon 20 Apr 2009 03:43:03 PM UTC, comment #7: 

I've just got a response back from libtextcat authors: they do plan to release a new version that would address the encoding issue later this year.

Dmitry Borodaenko <angdraug>
Group administrator
Mon 20 Apr 2009 02:31:32 PM UTC, comment #6: 

As long as it is possible to re-use an existing library for this functionality, it doesn't make sense to implement our own incomplete solution.

libTextCat so far appears the way to go, with one small problem: its signatures are made in a whole range of character encodings, while Samizdat only understands UTF-8. "Downsampling" messages to single-byte encodings such as windows1251 or iso8859-1 would break for any message mixing in non-ASCII characters from another language, so we have to convert libTextCat's language fingerprints to UTF-8.

I have contacted libTextCat authors about that, there is also a Debian bug #452341 about a similar issue. Considering that the last release of libTextCat happened over 5 years ago, it may be a while before upstream authors address this issue, but if we can at least get a Debian package with UTF-8 language fingerprints, this will be enough to move forward with this patch.

Dmitry Borodaenko <angdraug>
Group administrator
Fri 22 Aug 2008 11:44:27 PM UTC, comment #5: 

Version 0.4.1 has a trivial change in the path for the new file for the first file to modify. This is just being uploaded to help a little with systematic treatment of the patches.

(file #16336)

Boud Roukema <boud>
Group Member
Mon 18 Aug 2008 08:49:40 PM UTC, comment #4: 

Version 0.4 moves the check on the length of a string to the guessable_lang method in guess_lang.rb. The motivation is to avoid giving readers comments about a suspected language error in cases where the text is too short to expect a reasonably high accuracy.

Also, an entry for 'es' has been added, but with a minimum length in guessable_lang set to 200, since i don't know the language that well and since it seems to be less trivial for a regexp robot to detect than some of the other languages.

(file #16312)

Boud Roukema <boud>
Group Member
Sat 16 Aug 2008 10:19:39 PM UTC, comment #3: 

0.3.1 has been cleaned of debugging comments which had been left in place in 0.3.

(file #16302)

Boud Roukema <boud>
Group Member
Sat 16 Aug 2008 01:10:41 AM UTC, comment #2: 

Version 0.3 has rough detection algorithms for four languages in utf-8: en, fr, ja, pl. Messages in fr and pl written in ascii (due to people having problems with non-ascii character input) should have a fair chance of being detected as well.

Version 0.3 should also work against 080811-1 since there seems to be only one trivial change in message_controller.rb between these two samizdat snapshots.

For Russian, Belarusian and Ukrainian, someone who knows these languages well and, especially, knows which particular spellings or words distinguish them from one another, could add detectors for one or more of these in utf-8.

This version separates out most of the work as a single method, so is a bit more modular than the previous version.

Reminder: see below for a better long-term solution.

(file #16297)

Boud Roukema <boud>
Group Member
Wed 02 Jul 2008 12:24:46 AM UTC, comment #1: 

Patch version 0.2 is the same as 0.1, except that .to_s is used on body before using the guess_lang method. This avoids having an error in case the body is nil. In situations where a nil body is a problem, this should be flagged earlier - it should not be a problem for guess_lang.

COMMENT: if used together with the multi_publish patch, then this patch should be applied (in the obvious way) to both the preview and multi_preview methods.

(file #15979)

Boud Roukema <boud>
Group Member
Thu 26 Jun 2008 12:59:48 AM UTC, original submission:  

PROBLEM: People publish messages without remembering to check what language was selected by samizdat or by them. The result is a message which is really in, e.g. english but is labelled as being in japanese, or vice versa.

SHORT TERM SOLUTION: Check a small amount of text for some of the most frequent words in that language. My subjective experience using google shows that for non-English languages, this can be extremely effective if just one or two of the most frequent words of the language are used.  This is implemented as helpers/guess_lang.rb as a new method on objects of class String, since we haven't created a class Lang AFAIK. This is called from message_controller.rb. So far it handles en and ja. It uses a few more than just one or two words. It could probably be extended to ru and be by someone who knows the differences between these languages very well.

A parameter in the config file must be turned on to activate this function:

  # If you want a rough language detecting algorithm to warn
  # about people who forget to choose the correct language,
  # then uncomment this option, setting it to 'warn'
  language_guess: 'warn'

LONG TERM SOLUTION: A long term solution would be the language detector "textlangcat" available in lenny/sid as a C package or from source here: Apparently this gets an extremely high-rate of accuracy.

Boud Roukema <boud>
Group Member


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #16336:  080823_guess_lang-0.4.1.patch added by boud (4KiB - text/x-patch)
file #16312:  080818_guess_lang-0.4.patch added by boud (4KiB - text/x-patch)
file #16302:  080815_guess_lang-0.3.1.patch added by boud (4KiB - text/x-patch - 0.3.1 has been cleaned of debugging comments)
file #16297:  080815_guess_lang-0.3.patch added by boud (4KiB - text/x-patch)
file #15979:  080702_guess_lang-0.2.patch added by boud (3KiB - text/x-patch)
file #15945:  080626_guess_lang-0.1.patch added by boud (3KiB - text/x-patch)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by angdraug (Posted a comment)
  • -email is unavailable- added by boud (Submitted the item)

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2008-08-22 boud Attached File- Added 080823_guess_lang-0.4.1.patch, #16336
        Summaryguess_lang patch 0.4 [vs 080703-2] guess_lang patch 0.4.1 [vs 080703-2]
    2008-08-18 boud Attached File- Added 080818_guess_lang-0.4.patch, #16312
        Summaryguess_lang patch 0.3.1 [vs 080703-2] guess_lang patch 0.4 [vs 080703-2]
    2008-08-16 boud Summaryguess_lang patch 0.3 [vs 080703-2] guess_lang patch 0.3.1 [vs 080703-2]
    2008-08-16 boud Attached File- Added 080815_guess_lang-0.3.1.patch, #16302
    2008-08-16 boud Attached File- Added 080815_guess_lang-0.3.patch, #16297
        Summaryguess_lang patch 0.2 guess_lang patch 0.3 [vs 080703-2]
    2008-07-02 boud Attached File- Added 080702_guess_lang-0.2.patch, #15979
        Summaryguess_lang patch 0.1 guess_lang patch 0.2
    2008-06-26 boud Attached File- Added 080626_guess_lang-0.1.patch, #15945

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code