Accept-Language removed from RSS import as of 0.6.2.20090409-1.

version 0.7.1 avoids an error which occurred in 0.7 if 'import_feeds' was not configured at all for a samizdat site. The difference is:

s/config['import_feeds'][/(config['import_feeds'] or {})[/g

(file #16392)

What i just uploaded as the patch  import_feeds version 0.6
has a bug: if the language list is defined individually in
<site>.yaml files in a multi-site installation, then
default_language for the full multi-site set is either
undefined or is that of the last site evaluated (i haven't
tested this). So this will not work properly on a multisite
installation (apart from hardwiring a hack into it).

Same problem as with patch #5882: you should've raised a bug instead of re-opening the patch.

I will include the nil bug fix in the next snapshot, the i18n issue is not so trivial so it will not be fixed until after 0.6.1.

This patch corrects a minor bug and adds i18n functionality to
the import feeds feature of 0.6.0.070818:

http://lists.gnu.org/archive/html/samizdat-devel/2007-08/msg00005.html

MINOR BUG:
syndication_helper.rb

feed = persistent_cache[ '/import_feeds/' + url ]   # first slash to avoid clashes with site_name

If url == nil due to an error in the config, then this crashes
the frontpage, since + cannot operate on nil. Solution: url.to_s.

I18N FUNCTIONALITY:

People clicking on the frontpage having different preferred
languages - the array @request.accept_language - should be
able to see the imported feeds in at least their most preferred
language if that's available. Version 070818 hardwired the
language to be the default_language - which could either be
hardwired for the server for all samizdat sites, or else if
languages are configured individually for different sites,
then i'm not quite sure what would happen, but i guess that
samizdat-import-helper must choose one or the other
default_language and at least one of the two sites will have
the wrong default_language (assuming they have different
preferred languages).

So should samizdat-import-helper download RDF feeds for all
available languages? In the interests of not overstressing
the servers exporting the feeds, it would be better only to
import those preferred languages which users actually want.
In the long term (let's allow for scalability), the language
list could go up to 100 or more, and there's no point every
samizdat site trying to pull in a single feed 100 times
with different Accept-Language values when chances are the
feed doesn't exist in most of those languages.

So the solution proposed here is to keep a persistent_cache
object of languages which have been preferred by the same
people using render_feeds, and pull in feeds for each of
those languages.

The effect to a user with a non-default preferred language
is that after a recent reboot of the samizdat/apache caches,
s/he will see the imported feeds only in the default language,
but his/her interest will be remembered by the system. When
samizdat-import-feeds gets called again (e.g. by a cron job),
it will include this additional language. So if the user
clicks again after an hour or so (assuming an hourly cron)
or a day or so later, then s/he will see the imported feeds
in his/her preferred language. It's something like "the system
is responsive to your interests but in some cases takes some
time to react".

TODO: i couldn't find out how to make varying cache time
lengths for a reasonably long term cache. IMHO something like
a week would be reasonable - e.g. if there's just one person
interested in language X and s/he reads just once a week, that's
getting to the limit of whether or not s/he needs a feed list
in his/her preferred language. If two people read at random
times each week, then a week ttl would keep their language
constantly in the '/languages_used/' cache.

The patch is attached below.


(file #13743)

Version 0.4 fails to catch a timeout error on at least one
URI RDF feed tested.

This patch 0.4.1 is a patch relative to 0.4. (It's simple enough
that you can do it by hand if you have problems patching the
patch.)

The patch is here and below as a file: it uses the
timeout module and adds Timeout::Error to the list of
exceptions to catch.

TODO: The timeout of 10 seconds is hardwired and should
be made configurable.


--- old/samizdat/helpers/import_feeds_helper.rb~ 2007-05-18 16:23:44.000000000 -0500
+++ samizdat/helpers/import_feeds_helper.rb 2007-07-08 09:17:58.157086648 -0500
@@ -19,7 +19,7 @@
 require 'rss/1.0'
 require 'rss/dublincore'
 require 'rss/2.0'
-
+require 'timeout'
 
 # TODO: The format_date method is from template.rb. In principle,
 # imported feeds should (could) be treated as resources - somewhat
@@ -78,12 +79,13 @@
       response= ""
       valid_URI=0
       begin
+       status = Timeout::timeout(10) {
         open(anURI.untaint,  
-#             "Accept-Language" => config['locale']['languages'][0]) do  |file| 
              "Accept-Language" => lang) do  |file| 
           response += file.read
           valid_URI=1
         end
+       }
       rescue SocketError
         valid_URI=0
         import_feeds_body += _('<li><em>Error opening ') + %{<a href="} +
@@ -92,6 +94,10 @@
         valid_URI=0
         import_feeds_body += _('<li><em>Error opening ') + %{<a href="} +
    anURI + %{">} + _('this feed') + "</a></em></li>\n"
+      rescue Timeout::Error
+        valid_URI=0
+        import_feeds_body += _('<li><em>Error opening ') + %{<a href="} +
+   anURI + %{">} + _('this feed') + "</a></em></li>\n"
       rescue
         valid_URI=0
         import_feeds_body += _('<li><em>Error opening ') + %{<a href="} +


(file #13269)

http://lists.nongnu.org/archive/html/samizdat-devel/2006-10/msg00004.html

(1) index.rb patch


--- /tmp/samizdat_snapshot060924/cgi-bin/index.rb       2006-09-22
09:38:03.000000000 +0200
+++ /usr/share/samizdat/cgi-bin/index.rb        2006-10-08 05:39:33.609135024
+0200
@@ -12,6 +12,8 @@


 require 'samizdat/engine'


+require 'import_feeds.rb'  # TODO - should this be load or require?
+
 # messages that are related to any focus (and are not comments or old
 # versions), ordered chronologically by date of relation to a focus (so that
 # when message is edited, it doesn't flow up)
@@ -172,6 +174,13 @@
       t.nav_rss(rss_updates) + t.nav(updates.size, skip + 1))
   end


+ imported_feeds = "" # default is zero-length string
+ if( config['import_feeds'] )
+ imported_feeds = %{<tr><td class="links-head">}+ _('RDF Feeds')+ + '</td></tr>
+ <tr><td class="links">' + import_feeds_method + '</td></tr>'
+ end +
page =
if full_front_page
%{<table>
@@ -182,8 +191,8 @@
<td class="focuses">#{focuses}</td>
<td class="features" rowspan="3">#{features}</td>
<td class="updates" rowspan="3">#{updates}</td>
- </tr>
- <tr><td class="links-head">}+_('Links')+'</td></tr>
+ </tr>} + imported_feeds + + %{<tr><td class="links-head">}+_('Links')+'</td></tr>
<tr><td class="links">
<div class="focus"><a href="query.rb?run&amp;query='+CGI.escape('SELECT ?resource WHERE (dc::date ?resource ?date) (s::inReplyTo ?resource ?parent) LITERAL ?parent IS NOT NULL ORDER BY ?date DESC')+'">'+_('All Replies')+'</a></div>
<div class="focus"><a href="foci.rb">'+_('All Focuses (verbose)')+'</a></div>


(2) /usr/share/samizdat/cgi-bin/import_feeds.rb


#!/usr/bin/env ruby
#
# Samizdat logout
#
#   Copyright (c) 2002-2006  Dmitry Borodaenko <address@hidden>,
#   Boud (Indymedia) <address@hidden>
#
#   This program is free software.
#   You can distribute/modify this program under the terms of
#   the GNU General Public License version 2 or later.
#
# vim: et sw=2 sts=2 ts=8 tw=0


require 'samizdat/engine'


require 'open-uri'
require 'rss/1.0'
require 'rss/dublincore'
require 'rss/2.0'


# TODO: The format_date method is from template.rb. In principle,
# imported feeds should (could) be treated as resources - somewhat
# similar to messages, but with some properties distinct from ordinary
# messages. In that case, there would be no need to have redundancy
# for the format_date method.
def format_date(date)
  date = date.to_time if date.methods.include? 'to_time'   # duck
  date = date.strftime '%Y-%m-%d %H:%M' if date.kind_of? Time
  date
end



def import_feeds_method()


  import_feeds_body = "<ul>"


  interval = config['timeout']['import_feeds'] # time interval for importing
  interval = 3600 if (interval == nil)  # failsafe default
  timenow = Time.now  # object of Time class


  # The expected caching time is the last "round number" time interval,
  # based on total time in seconds defined in the Time class.
  expected_caching_time = timenow.to_i.divmod(interval)[0] * interval
  import_feeds_cache_key = 'imported_feeds/' + expected_caching_time.to_s


  import_feeds_list_array  = cache[import_feeds_cache_key]


  if(import_feeds_list_array == nil)


    import_feeds_list = Hash.new


    config['import_feeds'].each do | feed_key, feed_value |
      rss_source = feed_key


      # At some point in the future, people might want to have e.g. https
      # feeds, but there is no need to force people to write http:// when
      # this is a very widely used default value. So protocol is optional
      # here.


      protocol = feed_value['protocol']
      protocol = "http://";  if( protocol == nil)


      host = feed_value['host']
      host = _(' Hostname missing.') if (host == nil)
      filename = feed_value['filename']
      filename = _(' Filename missing.') if (filename == nil)
      anURI = protocol + host + filename
      #    anURI = protocol + feed_value['host'] + feed_value['filename']


      # TODO: security - check before untainting?
      # TODO: store and prepare rdf feeds in all available languages
      #       and give the user the one s/he wants?
      response= ""
      valid_URI=0
      begin
        open(anURI.untaint,
             "Accept-Language" => config['locale']['languages'][0]) do  |file|
          response += file.read
          valid_URI=1
        end
      rescue SocketError
        valid_URI=0
        import_feeds_body += _('<li><it>Error opening feed:</it> ') +
          anURI + "</li>\n"
      rescue URI::InvalidURIError
        valid_URI=0
        import_feeds_body += _('<li><it>Error opening feed:</it> ') +
          anURI + "</li>\n"
      end


      if(valid_URI==1)


        # The parsing of the feed initially allows non-RSS-1.0 compliant
        # feeds, but the  do_validate  method is used on individual items
        # later on to check their validity.
        begin
          rss = RSS::Parser.parse(response)  # for RSS 1.0 compliant feeds
        rescue RSS::InvalidRSSError
          rss = RSS::Parser.parse(response, false) # allow non RSS 1.0 compliant
        end


        if(rss)
          # rss.channel in RSS 2.0 seems to contain info in "rss" for RSS 1.0
          # So rss_channel is used here as a commmon name for either.
          rss_channel = rss
          if rss.rss_version == "2.0"
            rss_channel = rss.channel
          end


          # if there is a 'max_entries' parameter, then use at most that
          # number of items for that feed
          n_items=rss_channel.items.length
          if(feed_value['max_entries'])
            if(n_items > feed_value['max_entries'])
              n_items = feed_value['max_entries']
            end
          end


          for item_number in 0...n_items
            if rss_channel.item(item_number).do_validate
              rss_link = rss_channel.item(item_number).link.strip
              title = rss_channel.item(item_number).title.strip
              date = format_date(rss_channel.item(item_number).date)


              # add this feed to the list of valid feeds
              import_feeds_list[rss_link] = { "rss_source" => rss_source,
                "title" => title, "date" => date }


            end
          end  #     import_feeds_list.each { | feed_key, feed_value |
        end  #    if(rss)
      end #  if(valid_URI==1)
    end # for feed_number in ...





    # Sort the import feeds list by date.  The result is an array of
    # pairs.  The first element of each pair is the link (in principle,
    # this should be unique).  The second element of each pair is
    # a hash, containing the other useful pieces of feed
    # information (such as source, title, date)
    import_feeds_list_array = import_feeds_list.sort {
      |a,b| b[1]['date'] <=> a[1]['date'] }


    # update the cache
    cache[import_feeds_cache_key] = import_feeds_list_array


  end #    if(import_feeds_list_array == nil)


  import_feeds_list_array.each do | feed |
    import_feeds_body +=
      "<li> <it>" + feed[1]['rss_source'] +
      '</it> <a href="' + feed[0] + '">' +
      feed[1]['title'] + "</a> " +
      feed[1]['date'] + "</li><br />\n"
  end


  import_feeds_body +=  "</ul>"


  import_feeds_body


end # def import_feeds_method




(3a) Add to config.yaml


# RDF/RSS Feeds
# # This section defines the RDF/RSS feeds which you wish to # automatically import to your website from other websites.
# The maximum number of entries you define for each feed will
# be put together in a list and sorted by reverse chronological
# order, from most recent to oldest. #
# Uncomment the following line to enable this section.
# import_feeds:


# The default installation will include this sorted list in # the lefthand column of the front page of your samizdat installation.
# # The entry for each feed includes:
# 'sourcename': your own arbitrary name for the feed (short is good)
# protocol: (OPTIONAL) the protocol prefix, including e.g. "://"
# host: the hostname of the server
# filename: the filename including the path including the initial "/"
# max_entries: (OPTIONAL) the maximum number of entries from this feed

# Example entry (simple) to import: http://myfriend.site.org/friend/feed
#
# 'mon ami':
#   host: myfriend.site.org
#   filename: /friend/feed
#
# Example entry (with options):
#
# 'mon ami':
#   protocol: http://
#   host: myfriend.site.org
#   filename: /friend/feed
#   max_entries: 5




(3b) Add caching timeout to timeout section of defaults.yaml


timeout: # example value only, 3600 seconds
import_feeds: 3600