As long as it is possible to re-use an existing library for this functionality, it doesn't make sense to implement our own incomplete solution.

libTextCat so far appears the way to go, with one small problem: its signatures are made in a whole range of character encodings, while Samizdat only understands UTF-8. "Downsampling" messages to single-byte encodings such as windows1251 or iso8859-1 would break for any message mixing in non-ASCII characters from another language, so we have to convert libTextCat's language fingerprints to UTF-8.

I have contacted libTextCat authors about that, there is also a Debian bug #452341 about a similar issue. Considering that the last release of libTextCat happened over 5 years ago, it may be a while before upstream authors address this issue, but if we can at least get a Debian package with UTF-8 language fingerprints, this will be enough to move forward with this patch.

Version 0.4 moves the check on the length of a string to the guessable_lang method in guess_lang.rb. The motivation is to avoid giving readers comments about a suspected language error in cases where the text is too short to expect a reasonably high accuracy.

Also, an entry for 'es' has been added, but with a minimum length in guessable_lang set to 200, since i don't know the language that well and since it seems to be less trivial for a regexp robot to detect than some of the other languages.


(file #16312)

Version 0.3 has rough detection algorithms for four languages in utf-8: en, fr, ja, pl. Messages in fr and pl written in ascii (due to people having problems with non-ascii character input) should have a fair chance of being detected as well.

Version 0.3 should also work against 080811-1 since there seems to be only one trivial change in message_controller.rb between these two samizdat snapshots.

For Russian, Belarusian and Ukrainian, someone who knows these languages well and, especially, knows which particular spellings or words distinguish them from one another, could add detectors for one or more of these in utf-8.

This version separates out most of the work as a single method, so is a bit more modular than the previous version.

Reminder: see below for a better long-term solution.

(file #16297)

PROBLEM: People publish messages without remembering to check what language was selected by samizdat or by them. The result is a message which is really in, e.g. english but is labelled as being in japanese, or vice versa.

SHORT TERM SOLUTION: Check a small amount of text for some of the most frequent words in that language. My subjective experience using google shows that for non-English languages, this can be extremely effective if just one or two of the most frequent words of the language are used.  This is implemented as helpers/guess_lang.rb as a new method on objects of class String, since we haven't created a class Lang AFAIK. This is called from message_controller.rb. So far it handles en and ja. It uses a few more than just one or two words. It could probably be extended to ru and be by someone who knows the differences between these languages very well.

CONFIG:
A parameter in the config file must be turned on to activate this function:

  # If you want a rough language detecting algorithm to warn
  # about people who forget to choose the correct language,
  # then uncomment this option, setting it to 'warn'
  language_guess: 'warn'

LONG TERM SOLUTION: A long term solution would be the language detector "textlangcat" available in lenny/sid as a C package or from source here: http://software.wise-guys.nl/libtextcat/ Apparently this gets an extremely high-rate of accuracy.