patchSamizdat - Patches: patch #5881, import_feeds version 0.7.1 [vs...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

patch #5881: import_feeds version 0.7.1 [vs 080811-1] new i18n version

Submitter:  Dmitry Borodaenko <angdraug>
Submitted:  Sun 22 Apr 2007 05:57:56 PM UTC
   
 
Category:  None Priority:  5 - Normal
Status:  Wont Do Privacy:  Public
Assigned to:  None Open/Closed:  Open

Jump to the original submission

Thu 09 Apr 2009 11:47:44 AM UTC, comment #12: 

Accept-Language removed from RSS import as of 0.6.2.20090409-1.

Dmitry Borodaenko <angdraug>
Group administrator
Mon 06 Apr 2009 05:39:43 PM UTC, comment #11: 

After some consideration, I'm leaning towards a Solomon's solution to the feed import i18n issue: don't send Accept-Language header at all.

RSS feeds that support HTTP language negotiation are far and wide between, and they all have to deal with the reality that most RSS readers don't do language negotiation, so in absense of Accept-Language header all such feeds are expected to provide sensible default (in case of Samizdat, that would be default_language of the site serving the feed).

Language-conscious admins will still be able to specify language in the feed's URL. To support that approach in Samizdat feeds, I am going to merge patch #6171.

Dmitry Borodaenko <angdraug>
Group administrator
Fri 29 Aug 2008 11:42:52 PM UTC, comment #10: 

version 0.7.1 avoids an error which occurred in 0.7 if 'import_feeds' was not configured at all for a samizdat site. The difference is:


s/config['import_feeds'][/(config['import_feeds'] or {})[/g




(file #16392)

Boud Roukema <boud>
Group Member
Sat 19 Jul 2008 05:06:03 PM UTC, comment #9: 

import_feeds-0.7 vs 080703-2

This is an update of the i18n issue first commented below:
 https://savannah.nongnu.org/patch/?5881#comment4
which should work against samizdat snapshot 080703-2. It has several
small differences from the last few versions of this patch.

WARNING: The patch requires updating of the 'import_feeds' section of
the config.yaml file, even if the default language is used.
An alternative would be to use config['import_feeds_languages']
instead of config['import_feeds']['languages'], but this would seem
to be less modular IMHO.


FEEDS_LANGUAGES: The list of languages in which to try to import
feeds, 'feeds_languages' (previously: 'languages_used'), is no longer
generated automatically from those languages which users actually
use. Instead, a config parameter ['import_feeds']['languages'] is
used, or if that is undefined, the default_language is used. The
feeds_languages from the different sites are collected together.

There are two reasons for this:

  • a site may have feeds in a non-default language and expect them

to be imported in that language even though readers rarely choose
that language as their preferred language

  • if random internet users (or spammers) select "rare" languages,

then all these languages will be used during feed imports - e.g.
Accept-Language could be 7 different languages, and someone managing
a server who is ignorant about language negotiation will think that
the importing client has rapidly made 7 identical requests, and
might consider this abusive.


LANGUAGE IN URL:
For non-(language-negotiating) servers, the simplest solution
seems to be that the URL must be written with the language tag
replaced by 'LANG'. Since URLs are not normally written in uppercase,
there is only a moderate chance of this inducing errors. E.g. a
website discussing the French politician Jack Lang and using the
French convention (for adminstrative purposes) of writing family
names in upper case might be http://some.site.fr/Jack_LANG_music_festival
A way to reduce possible errors would be to introduce special
characters e.g. ( % [ ] % ), at the risk of causing escaping/unescaping
problems. i didn't try this.


EXCEPTIONS ON FAILED FEEDS:
We don't want a single failed feed, e.g. of a url in one given
language when LANG is used, to prevent importing of the same url
in other languages, or of other urls. So i've shifted the error
reporting to the end of the update_feed method.



(file #16159)

Boud Roukema <boud>
Group Member
Sat 03 May 2008 01:03:59 AM UTC, comment #8: 

What i just uploaded as the patch  import_feeds version 0.6
has a bug: if the language list is defined individually in
<site>.yaml files in a multi-site installation, then
default_language for the full multi-site set is either
undefined or is that of the last site evaluated (i haven't
tested this). So this will not work properly on a multisite
installation (apart from hardwiring a hack into it).

Boud Roukema <boud>
Group Member
Sat 03 May 2008 12:17:21 AM UTC, comment #7: 

So now i don't know if i should post further corrections here
or in the bugs list as a new bug... :(  It seems to me that
keeping things together makes the most sense.

Anyway, here's an updated patch against 070818.

The new thing is in samizdat-import-feeds to take into
account of the fact that many existing web sites do not yet
do language negotiation, but they do have a fairly obvious
(to a geek) indicator of the language in the URL using
a standard language code like 'ja' or 'en'. So the idea
is that for feeds of this type, let's say
http://some.site.taz/ja/feed/, the sysadmin should just put
one URL in the config file, choosing the site default language.

When the samizdat-import-feeds script is going through the
list of recently used languages, it will guess the URL by substitution of language codes, and fall back to the unsubstituted version if this fails.

Possible TODO: Some sites might give multiple language mixed up in a single feed and label the individual items with language tags. i haven't seen this in practice so far. What i have seen would require more fuzzy intelligence: a mixed language feed which would require a heuristic for guessing the language by content analysis - but this should probably not be a high priority.


(file #15593)

Boud Roukema <boud>
Group Member
Sat 23 Feb 2008 01:45:32 AM UTC, comment #6: 

Same problem as with patch #5882: you should've raised a bug instead of re-opening the patch.

I will include the nil bug fix in the next snapshot, the i18n issue is not so trivial so it will not be fixed until after 0.6.1.

Dmitry Borodaenko <angdraug>
Group administrator
Sun 26 Aug 2007 11:23:41 PM UTC, comment #5: 

i18n is missing from the 070818-1 version, so i'm switching from Done to Works For Me status.

Boud Roukema <boud>
Group Member
Mon 20 Aug 2007 09:56:33 PM UTC, comment #4: 

This patch corrects a minor bug and adds i18n functionality to
the import feeds feature of 0.6.0.070818:

http://lists.gnu.org/archive/html/samizdat-devel/2007-08/msg00005.html

MINOR BUG:
syndication_helper.rb

feed = persistent_cache[ '/import_feeds/' + url ]   # first slash to avoid clashes with site_name


If url == nil due to an error in the config, then this crashes
the frontpage, since + cannot operate on nil. Solution: url.to_s.

I18N FUNCTIONALITY:

People clicking on the frontpage having different preferred
languages - the array @request.accept_language - should be
able to see the imported feeds in at least their most preferred
language if that's available. Version 070818 hardwired the
language to be the default_language - which could either be
hardwired for the server for all samizdat sites, or else if
languages are configured individually for different sites,
then i'm not quite sure what would happen, but i guess that
samizdat-import-helper must choose one or the other
default_language and at least one of the two sites will have
the wrong default_language (assuming they have different
preferred languages).

So should samizdat-import-helper download RDF feeds for all
available languages? In the interests of not overstressing
the servers exporting the feeds, it would be better only to
import those preferred languages which users actually want.
In the long term (let's allow for scalability), the language
list could go up to 100 or more, and there's no point every
samizdat site trying to pull in a single feed 100 times
with different Accept-Language values when chances are the
feed doesn't exist in most of those languages.

So the solution proposed here is to keep a persistent_cache
object of languages which have been preferred by the same
people using render_feeds, and pull in feeds for each of
those languages.

The effect to a user with a non-default preferred language
is that after a recent reboot of the samizdat/apache caches,
s/he will see the imported feeds only in the default language,
but his/her interest will be remembered by the system. When
samizdat-import-feeds gets called again (e.g. by a cron job),
it will include this additional language. So if the user
clicks again after an hour or so (assuming an hourly cron)
or a day or so later, then s/he will see the imported feeds
in his/her preferred language. It's something like "the system
is responsive to your interests but in some cases takes some
time to react".

TODO: i couldn't find out how to make varying cache time
lengths for a reasonably long term cache. IMHO something like
a week would be reasonable - e.g. if there's just one person
interested in language X and s/he reads just once a week, that's
getting to the limit of whether or not s/he needs a feed list
in his/her preferred language. If two people read at random
times each week, then a week ttl would keep their language
constantly in the '/languages_used/' cache.

The patch is attached below.


(file #13743)

Boud Roukema <boud>
Group Member
Sat 18 Aug 2007 06:08:46 PM UTC, comment #3: 

Refactored and integrated in snapshot 2007-08-18.

Differences from Boud's version:

RSS import is configured pretty much the same as in Boud's patch, but it works quite differently: feeds are fetched and parsed by an
external script (bin/samizdat-import-feeds) that can be placed in crontab on an arbitrary schedule (remember though that fetching feeds more often than hourly is not polite). The script puts the results into the DRb cache as persistent entries available across all sites, so that two sites importing the same feed will only fetch it once. The script fetches maximum number of entries, it is up to each site to cut these entries down to the limit configured for given site.

Dmitry Borodaenko <angdraug>
Group administrator
Sun 08 Jul 2007 02:24:35 PM UTC, comment #2: 

Version 0.4 fails to catch a timeout error on at least one
URI RDF feed tested.

This patch 0.4.1 is a patch relative to 0.4. (It's simple enough
that you can do it by hand if you have problems patching the
patch.)

The patch is here and below as a file: it uses the
timeout module and adds Timeout::Error to the list of
exceptions to catch.

TODO: The timeout of 10 seconds is hardwired and should
be made configurable.


--- old/samizdat/helpers/import_feeds_helper.rb~ 2007-05-18 16:23:44.000000000 -0500
+++ samizdat/helpers/import_feeds_helper.rb 2007-07-08 09:17:58.157086648 -0500
@@ -19,7 +19,7 @@
 require 'rss/1.0'
 require 'rss/dublincore'
 require 'rss/2.0'
-
+require 'timeout'
 
 # TODO: The format_date method is from template.rb. In principle,
 # imported feeds should (could) be treated as resources - somewhat
@@ -78,12 +79,13 @@
       response= ""
       valid_URI=0
       begin
+       status = Timeout::timeout(10) {
         open(anURI.untaint,  
-#             "Accept-Language" => config['locale']['languages'][0]) do  |file| 
              "Accept-Language" => lang) do  |file| 
           response += file.read
           valid_URI=1
         end
+       }
       rescue SocketError
         valid_URI=0
         import_feeds_body += _('<li><em>Error opening ') + %{<a href="} +
@@ -92,6 +94,10 @@
         valid_URI=0
         import_feeds_body += _('<li><em>Error opening ') + %{<a href="} +
    anURI + %{">} + _('this feed') + "</a></em></li>\n"
+      rescue Timeout::Error
+        valid_URI=0
+        import_feeds_body += _('<li><em>Error opening ') + %{<a href="} +
+   anURI + %{">} + _('this feed') + "</a></em></li>\n"
       rescue
         valid_URI=0
         import_feeds_body += _('<li><em>Error opening ') + %{<a href="} +


(file #13269)

Boud Roukema <boud>
Group Member
Tue 08 May 2007 10:23:22 PM UTC, comment #1: 

version 0.4 works for samizdat-0.6.0.20070506 - attached as a file




(file #12721)

Boud Roukema <boud>
Group Member
Sun 22 Apr 2007 05:57:56 PM UTC, original submission:  

http://lists.nongnu.org/archive/html/samizdat-devel/2006-10/msg00004.html

(1) index.rb patch


--- /tmp/samizdat_snapshot060924/cgi-bin/index.rb       2006-09-22
09:38:03.000000000 +0200
+++ /usr/share/samizdat/cgi-bin/index.rb        2006-10-08 05:39:33.609135024
+0200
@@ -12,6 +12,8 @@


 require 'samizdat/engine'


+require 'import_feeds.rb'  # TODO - should this be load or require?
+
 # messages that are related to any focus (and are not comments or old
 # versions), ordered chronologically by date of relation to a focus (so that
 # when message is edited, it doesn't flow up)
@@ -172,6 +174,13 @@
       t.nav_rss(rss_updates) + t.nav(updates.size, skip + 1))
   end


+ imported_feeds = "" # default is zero-length string
+ if( config['import_feeds'] )
+ imported_feeds = %{<tr><td class="links-head">}+ _('RDF Feeds')+ + '</td></tr>
+ <tr><td class="links">' + import_feeds_method + '</td></tr>'
+ end +
page =
if full_front_page
%{<table>
@@ -182,8 +191,8 @@
<td class="focuses">#{focuses}</td>
<td class="features" rowspan="3">#{features}</td>
<td class="updates" rowspan="3">#{updates}</td>
- </tr>
- <tr><td class="links-head">}+_('Links')+'</td></tr>
+ </tr>} + imported_feeds + + %{<tr><td class="links-head">}+_('Links')+'</td></tr>
<tr><td class="links">
<div class="focus"><a href="query.rb?run&amp;query='+CGI.escape('SELECT ?resource WHERE (dc::date ?resource ?date) (s::inReplyTo ?resource ?parent) LITERAL ?parent IS NOT NULL ORDER BY ?date DESC')+'">'+_('All Replies')+'</a></div>
<div class="focus"><a href="foci.rb">'+_('All Focuses (verbose)')+'</a></div>


(2) /usr/share/samizdat/cgi-bin/import_feeds.rb


#!/usr/bin/env ruby
#
# Samizdat logout
#
#   Copyright (c) 2002-2006  Dmitry Borodaenko <address@hidden>,
#   Boud (Indymedia) <address@hidden>
#
#   This program is free software.
#   You can distribute/modify this program under the terms of
#   the GNU General Public License version 2 or later.
#
# vim: et sw=2 sts=2 ts=8 tw=0


require 'samizdat/engine'


require 'open-uri'
require 'rss/1.0'
require 'rss/dublincore'
require 'rss/2.0'


# TODO: The format_date method is from template.rb. In principle,
# imported feeds should (could) be treated as resources - somewhat
# similar to messages, but with some properties distinct from ordinary
# messages. In that case, there would be no need to have redundancy
# for the format_date method.
def format_date(date)
  date = date.to_time if date.methods.include? 'to_time'   # duck
  date = date.strftime '%Y-%m-%d %H:%M' if date.kind_of? Time
  date
end



def import_feeds_method()


  import_feeds_body = "<ul>"


  interval = config['timeout']['import_feeds'] # time interval for importing
  interval = 3600 if (interval == nil)  # failsafe default
  timenow = Time.now  # object of Time class


  # The expected caching time is the last "round number" time interval,
  # based on total time in seconds defined in the Time class.
  expected_caching_time = timenow.to_i.divmod(interval)[0] * interval
  import_feeds_cache_key = 'imported_feeds/' + expected_caching_time.to_s


  import_feeds_list_array  = cache[import_feeds_cache_key]


  if(import_feeds_list_array == nil)


    import_feeds_list = Hash.new


    config['import_feeds'].each do | feed_key, feed_value |
      rss_source = feed_key


      # At some point in the future, people might want to have e.g. https
      # feeds, but there is no need to force people to write http:// when
      # this is a very widely used default value. So protocol is optional
      # here.


      protocol = feed_value['protocol']
      protocol = "http://";  if( protocol == nil)


      host = feed_value['host']
      host = _(' Hostname missing.') if (host == nil)
      filename = feed_value['filename']
      filename = _(' Filename missing.') if (filename == nil)
      anURI = protocol + host + filename
      #    anURI = protocol + feed_value['host'] + feed_value['filename']


      # TODO: security - check before untainting?
      # TODO: store and prepare rdf feeds in all available languages
      #       and give the user the one s/he wants?
      response= ""
      valid_URI=0
      begin
        open(anURI.untaint,
             "Accept-Language" => config['locale']['languages'][0]) do  |file|
          response += file.read
          valid_URI=1
        end
      rescue SocketError
        valid_URI=0
        import_feeds_body += _('<li><it>Error opening feed:</it> ') +
          anURI + "</li>\n"
      rescue URI::InvalidURIError
        valid_URI=0
        import_feeds_body += _('<li><it>Error opening feed:</it> ') +
          anURI + "</li>\n"
      end


      if(valid_URI==1)


        # The parsing of the feed initially allows non-RSS-1.0 compliant
        # feeds, but the  do_validate  method is used on individual items
        # later on to check their validity.
        begin
          rss = RSS::Parser.parse(response)  # for RSS 1.0 compliant feeds
        rescue RSS::InvalidRSSError
          rss = RSS::Parser.parse(response, false) # allow non RSS 1.0 compliant
        end


        if(rss)
          # rss.channel in RSS 2.0 seems to contain info in "rss" for RSS 1.0
          # So rss_channel is used here as a commmon name for either.
          rss_channel = rss
          if rss.rss_version == "2.0"
            rss_channel = rss.channel
          end


          # if there is a 'max_entries' parameter, then use at most that
          # number of items for that feed
          n_items=rss_channel.items.length
          if(feed_value['max_entries'])
            if(n_items > feed_value['max_entries'])
              n_items = feed_value['max_entries']
            end
          end


          for item_number in 0...n_items
            if rss_channel.item(item_number).do_validate
              rss_link = rss_channel.item(item_number).link.strip
              title = rss_channel.item(item_number).title.strip
              date = format_date(rss_channel.item(item_number).date)


              # add this feed to the list of valid feeds
              import_feeds_list[rss_link] = { "rss_source" => rss_source,
                "title" => title, "date" => date }


            end
          end  #     import_feeds_list.each { | feed_key, feed_value |
        end  #    if(rss)
      end #  if(valid_URI==1)
    end # for feed_number in ...





    # Sort the import feeds list by date.  The result is an array of
    # pairs.  The first element of each pair is the link (in principle,
    # this should be unique).  The second element of each pair is
    # a hash, containing the other useful pieces of feed
    # information (such as source, title, date)
    import_feeds_list_array = import_feeds_list.sort {
      |a,b| b[1]['date'] <=> a[1]['date'] }


    # update the cache
    cache[import_feeds_cache_key] = import_feeds_list_array


  end #    if(import_feeds_list_array == nil)


  import_feeds_list_array.each do | feed |
    import_feeds_body +=
      "<li> <it>" + feed[1]['rss_source'] +
      '</it> <a href="' + feed[0] + '">' +
      feed[1]['title'] + "</a> " +
      feed[1]['date'] + "</li><br />\n"
  end


  import_feeds_body +=  "</ul>"


  import_feeds_body


end # def import_feeds_method




(3a) Add to config.yaml


# RDF/RSS Feeds
# # This section defines the RDF/RSS feeds which you wish to # automatically import to your website from other websites.
# The maximum number of entries you define for each feed will
# be put together in a list and sorted by reverse chronological
# order, from most recent to oldest. #
# Uncomment the following line to enable this section.
# import_feeds:


# The default installation will include this sorted list in # the lefthand column of the front page of your samizdat installation.
# # The entry for each feed includes:
# 'sourcename': your own arbitrary name for the feed (short is good)
# protocol: (OPTIONAL) the protocol prefix, including e.g. "://"
# host: the hostname of the server
# filename: the filename including the path including the initial "/"
# max_entries: (OPTIONAL) the maximum number of entries from this feed

# Example entry (simple) to import: http://myfriend.site.org/friend/feed
#
# 'mon ami':
#   host: myfriend.site.org
#   filename: /friend/feed
#
# Example entry (with options):
#
# 'mon ami':
#   protocol: http://
#   host: myfriend.site.org
#   filename: /friend/feed
#   max_entries: 5




(3b) Add caching timeout to timeout section of defaults.yaml


timeout: # example value only, 3600 seconds
import_feeds: 3600

Dmitry Borodaenko <angdraug>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #16159:  080719_import_feeds-0.7.patch added by boud (6KiB - text/x-patch)
file #15593:  080502_import_feeds-0.6-patch added by boud (4KiB - application/octet-stream)
file #13743:  070820_import_feeds-0.5-patch added by boud (3KiB - application/octet-stream)
file #13269:  070707_import_feeds-0.4.1-patch added by boud (1KiB - application/octet-stream)
file #12721:  imp added by boud (8KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by boud (Updated the item)
  • -email is unavailable- added by angdraug (Submitted the item)
  •  

    Follow 17 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2009-04-06 angdraug StatusPostponed Wont Do
    2008-08-29 boud Attached File- Added 080830_import_feeds-0.7.1.patch, #16392
        Summaryimport_feeds version 0.7 [vs 080703-2] new i18n version import_feeds version 0.7.1 [vs 080811-1] new i18n version
    2008-07-19 boud Attached File- Added 080719_import_feeds-0.7.patch, #16159
        Summaryimport_feeds version 0.6 (for importing RDF etc. feeds) vs 070818 import_feeds version 0.7 [vs 080703-2] new i18n version
    2008-05-03 boud Attached File- Added 080502_import_feeds-0.6-patch, #15593
        Summaryimport_feeds version 0.5 (for importing RDF etc. feeds) vs 070818 import_feeds version 0.6 (for importing RDF etc. feeds) vs 070818
    2008-02-23 angdraug Priority7 - High 5 - Normal
        StatusWorks For Me Postponed
    2007-08-26 boud StatusDone Works For Me
    2007-08-20 boud Attached File- Added 070820_import_feeds-0.5-patch, #13743
        Summaryimport_feeds.rb version 0.4.1 (for importing RDF etc. feeds) import_feeds version 0.5 (for importing RDF etc. feeds) vs 070818
    2007-08-18 angdraug StatusIn Progress Done
    2007-07-08 boud Attached File- Added 070707_import_feeds-0.4.1-patch, #13269
        Summaryimport_feeds.rb version 0.4 (for importing RDF etc. feeds) import_feeds.rb version 0.4.1 (for importing RDF etc. feeds)
    2007-05-08 boud Attached File- Added imp, #12721
        Summaryimport_feeds.rb version 0.2 (for importing RDF etc. feeds) import_feeds.rb version 0.4 (for importing RDF etc. feeds)

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code