I can't replicate this bug with snapshot 20080928, please confirm that you still see this bug. If it's still there, please list expressions that are not excluded.

Fix included in snapshot 20080223.

i agree that it would be easier for sysadmins to be able
to type entries in the config file in the same style in which
they are in the big wikispam file.

Thanks for your note about the difference between each and
collect - it looks like i was close to a solution :).

However, before i test these, i think there's some error
in versioning of antispam.rb somewhere. It seems to me that
it's not yet in the pipeline.

In both the 080214 snapshot and debian file it looks to me like
the change has not been made:

http://download.savannah.gnu.org/releases/samizdat/samizdat-snapshot.tar.gz
14-Feb-2008 18:47         215K

md5sum samizdat/lib/samizdat/antispam.rb
a614ca8565fbd9c4dd2c0ebfa67f3902  samizdat/lib/samizdat/antispam.rb

ls -l samizdat/lib/samizdat/antispam.rb
-rw-r--r--  1 boud boud 2361 2007-11-04 19:23 samizdat/lib/samizdat/antispam.rb

and the debian file:

http://ftp.debian.org/debian/pool/main/s/samizdat/samizdat_0.6.0.20080214-2_all.deb

md5sum usr/lib/ruby/1.8/samizdat/antispam.rb
a614ca8565fbd9c4dd2c0ebfa67f3902  usr/lib/ruby/1.8/samizdat/antispam.rb

i don't have a copy of the 071104 snapshot to check, but the internal
date suggests that the antispam.rb file in 080214 (snapshot)
does not have the correction. (i guess the cvs should have
a track of this.)

My explanation below in my previous comment:

"Hence, in the samizdat defaults.yaml or <site>.yaml
exclusion list, it had to be excluded with example\.com  ...
starting at character 0)."

is a bad explanation, sorry.  Here is a better explanation:

irb(main):090:0> 'example\.com' =~ /example\\.com/
=> 0
irb(main):091:0> 'example\.com' =~ /example\.com/
=> nil
irb(main):092:0> 'example.com' =~ /example\\.com/
=> nil
irb(main):093:0> 'example.com' =~ /example\.com/
=> 0

i tried with 'example\.com' in defaults.yaml as in the recent cvs update,
but this does not work for me.

Some further experiments are consistent with my intuition that if the
slash should be escaped, then the . should be escaped too:


irb(main):112:0> 'example\.com' =~ /example\\\.com/
=> 0
irb(main):113:0> 'example.com' =~ /example\.com/
=> 0
irb(main):114:0> 'example\xcom' =~ /example\\.com/
=> 0


If we want sysadmin to not have to escape regexp's, then IMHO we
probably should use Regexp.escape somehow in this line of antispam.rb:

      exclude = Regexp.new(options['exclude'].join('|')).freeze

i haven't yet found a correct way to do this... - i'm still
a ruby newbie...

i tried:

      long_string = options['exclude'].each { |string| Regexp.escape(string) }.join('|')
      exclude = Regexp.new(long_string).freeze

but that didn't seem to work.

My initial report was based on insufficient testing, and i think
i now have consistent, repeatable behaviour.


SIDE COMMENT: To test this,
remember to try a spam expression with a minimum of 7 characters,
since that is what we put in order to exclude expressions which
are too generic. i say this because the arch.thinkmo.de/cgi-bin/spam-merge
spam list includes at least one expression with only 6 characters.
This was ignored by antispam.rb and (today) got me confused.
Given all the sorts of various things people might want to write
in some long text, i think that a spam expression of 6 characters
could turn up by accident in genuine text. Anyway, back to the
main issue of this bug report...

WHAT WORKS CORRECTLY NOW:
i'll write this as my recommended patch, which
i tried to make as
short as possible while also being reasonably clear, even if
it's not a technically full explanation:

--- old/s070818/samizdat/data/samizdat/defaults.yaml        2007-08-13 23:39:32.000000000 +0200
+++ defaults.yaml   2007-11-09 22:37:18.829718392 +0100
@@ -147,6 +147,9 @@
   url: 'http://arch.thinkmo.de/cgi-bin/spam-merge'

   # Exclude certain URIs from the big wiki-spam URI list
+  # Examples:
+  #  example\\.com to exclude a line with   example\.com
+  #  example\.com  to exclude a line with   example.com
   #
   exclude:
     - google-watch\\.org

EXPLANATION:

Some people put entries in the big wiki-spam list such
as

example.com

even though most entries have their dots escaped.

The effect is that text such as "examplexcom" or "examplescom"
is detected as spam - since the dot is the regex dot with a
special meaning, not a literal dot.

Most probably the non-escaping of these dots
are unintended errors. They probably won't
have much chance of excluding non-spam stuff in practice, but
the expressions catch more "spam" than is intended.

One genuine article that got treated as spam on a live
samizdat site contained the URL of a website which was on the
wiki-spam list with this unescaped dot. 

Hence, in the samizdat defaults.yaml or <site>.yaml
exclusion list, it had to be excluded with example\.com
rather than example\\.com, because

"example\.com" =~ /example\\.com/

gives nil

while

"example\.com" =~ /example\.com/

gives 0 (the expression is found starting at character 0).

For main spam list entries with escaped dots, the samizdat
config exclusion list needs to have a doubly escaped dot,
as in 070818-1, which i now think is correct.

RECOMMENDATION:

So now i just recommend the syadmin hint patch above since
i don't see any short term realistic way of getting the
main wikispam list "cleaned" upstream (i.e. dots escaped unless
they are genuinely intended to be unescaped). 

i've switched the status to "works for me".


PS:
As another
side issue (long term issue), if someone knows which discussion
list or wiki page is used to discuss/update the arch.thinkmo.de
list (or knows a "better" list), then please speak up. i looked
around a bit but didn't find it.

In my defaults.yaml i need a single slash in front of dots for
excluding expressions from the spam list to work:

  exclude:
    - google-watch\.org
    - yahoo-watch\.org
    - wikipedia-watch\.org
    - ln-s\.net
    - namebase\.org

Double slash as in 070818 fails:

  exclude:
    - google-watch\\.org
    - yahoo-watch\\.org
    - wikipedia-watch\\.org
    - ln-s\\.net
    - namebase\\.org