bugSamizdat - Bugs: bug #20932, locale2lang-0.1 - BUG + fix:...

 
 

bug #20932: locale2lang-0.1 - BUG + fix: fallback to language_only extracted from Accept-Language http header is needed

Submitter:  Boud Roukema <boud>
Submitted:  Wed 29 Aug 2007 10:04:16 PM UTC
   
 
Category:  None Severity:  3 - Normal
Status:  Fixed Privacy:  Public
Assigned to:  None Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 21 Feb 2008 05:37:30 PM UTC, comment #1: 

This fix is in CVS since 2007-11-04, and is included in Debian/experimental uploads since then.

Dmitry Borodaenko <angdraug>
Group administrator
Wed 29 Aug 2007 10:04:16 PM UTC, original submission:  

PROBLEM:
Even though RFC 2616 recommends that user clients (e.g. firefox)
should recommend to their users to have a backup generic language
without a country code (e.g. "en" in addition to "en-US"),
in practice most users do not do this.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.4

In particular, for non-english language samizdat sites, this
means that people who have only "en-US" sent by their browser
end up getting the default local language of the site. Their
article then gets published with message.language = the local
language, not "en", since formally speaking, they state that
they prefer "en-US" to the local language, but they are not
interested in "en".

This implies that if people want to add a local translation
rather than hiding an article, then moderator intervention
is required to change the language (unless the user chose
open editing).

Moreover, the monolanguage patch https://savannah.nongnu.org/patch/?6167  (still under
development) will fail to exclude these type of articles under
the mono option, since their language is wrongly tagged (except
for a pedantic interpretation of their request).

For these reasons, i'm putting this as a bug (with a proposed
fix) rather than a patch.

PROPOSED SOLUTION:
This requires a reasonably modern version of ruby gettext, e.g.
debian 1.7.0-1 or later.  Copying gettext/locale_object.rb
into an older installation and using an appropriate require
statement is a hack to avoid a full installation of a recent
gettext.

The idea is that if a requested accept-language in the list
is not found, then parse off the language part of it and try
that instead. This could potentially create multiple entries
of the same language, but i suspect that shouldn't be a
problem.


--- s070818/samizdat/lib/samizdat/engine/request.rb     2007-08-14 01:16:53.000000000 +0200
+++ /usr/lib/ruby/1.8/samizdat/engine/request.rb        2007-08-29 23:02:06.869866760 +0200
@@ -165,8 +173,17 @@
       accept.scan(/([^ ,;]+)(?:;q=([^ ,;]+))?/).collect {|l, q|
         [l, (q ? q.to_f : 1.0)]
       }.sort_by {|l, q| -q }.each {|l, q|
-        @accept_language.push l if config_lang.include? l
+#        @accept_language.push l if config_lang.include? l
+        if config_lang.include? l
+          @accept_language.push l
+        else
+          # try converting full locale (language tag) to ISO-639 language only
+          # http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.4
+          lang_only = Locale::Object.new(l).language
+          @accept_language.push lang_only if config_lang.include? lang_only
+        end
       }
+
     # lang cookie overrides Accept-Language
     lang = cookie('lang') and config_lang.include? lang and
       @accept_language.unshift lang


FUTURE EXTENSIONS:
The relations between human languages and how close or distant
they are are well studied. A measure of the distance between
different languages could potentially be used as a backup to
find the likely closest language that a user would prefer
rather than just taking what is considered the "language"
part of the locale/Accept-Language string.

Since the "narratives" which claim different national identities
often try to claim sharp distinctions between closely related
languages, this could potentially be a quite politically
sensitive issue. This is not surprising, and is not IMHO an
argument against doing this: an RDF engine specifically aimed
for grassroots, non-authoritarian media is necessarily going
to challenge artificial linguistic barriers if it's to get
somewhere near doing its task.

In any case, users with their own notions of language preferences
would still be able to state this by all the presently available
methods; adding a language metric would only be used as a
fallback.

COMMENT:
The Locale:: module could probably also be used to check the
config files for valid languages and warn about invalid
languages/locales.

Boud Roukema <boud>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #13832:  070829_locale2lang-0.1 added by boud (997B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by angdraug (Posted a comment)
  • -email is unavailable- added by boud (Submitted the item)
  •  

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2008-02-21 angdraug StatusWorks For Me Fixed
        Open/ClosedOpen Closed
    2007-08-29 boud Attached File- Added 070829_locale2lang-0.1, #13832

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code