bugSamizdat - Bugs: bug #21490, REOPENED: antispam exclusions [vs...

 
 

bug #21490: REOPENED: antispam exclusions [vs 080703-2]: correct escaping of dots if the main spam list contains some unescaped dots

Submitter:  Boud Roukema <boud>
Submitted:  Thu 01 Nov 2007 03:17:36 PM UTC
   
 
Category:  None Severity:  2 - Minor
Status:  Need Info Privacy:  Public
Assigned to:  boud Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 25 Oct 2008 07:25:28 PM UTC, comment #10: 

I can't replicate this bug with snapshot 20080928, please confirm that you still see this bug. If it's still there, please list expressions that are not excluded.

Dmitry Borodaenko <angdraug>
Group administrator
Thu 17 Jul 2008 06:33:45 PM UTC, comment #9: 

VERSION 080703-2. Sorry, but for me the version in 080703-2 fails. Please test it and/or try my patch here.

PROBLEM: See discussion below. The expressions which should be excluded do not get excluded.

PATCH: i've included a patch file. This reverts to double slashes in the config file and removes the Regexp.escape(  ) in antispam.rb. As well, it includes an example to help guide the sysadmin who edits the config file. IMHO this is necessary if we want to avoid sysadmins wasting too much time having to guess the correct escaping conventions.

DISCUSSION:
An additional complication for any alternative solution, e.g. trying to use Regexp.escape on what is read from the wiki spam regexp list, is that not all special characters are escaped, e.g. '-' is not escaped, and some special characters might be used with the intention of retaining their special meanings.


(file #16130)

Boud Roukema <boud>
Group Member
Sat 23 Feb 2008 12:44:08 AM UTC, comment #8: 

Fix included in snapshot 20080223.

Dmitry Borodaenko <angdraug>
Group administrator
Fri 22 Feb 2008 11:34:08 AM UTC, comment #7: 

The 'single-slash in defaults.yaml' correction is there in CVS and in several recent snapshots. The 'Regexp.escape' correction isn't in CVS yet, I've only tested it locally so far. It will be included in the next snapshot, that's when I will close this bug.

Dmitry Borodaenko <angdraug>
Group administrator
Thu 21 Feb 2008 07:35:33 PM UTC, comment #6: 

i agree that it would be easier for sysadmins to be able
to type entries in the config file in the same style in which
they are in the big wikispam file.

Thanks for your note about the difference between each and
collect - it looks like i was close to a solution :).

However, before i test these, i think there's some error
in versioning of antispam.rb somewhere. It seems to me that
it's not yet in the pipeline.

In both the 080214 snapshot and debian file it looks to me like
the change has not been made:


http://download.savannah.gnu.org/releases/samizdat/samizdat-snapshot.tar.gz
14-Feb-2008 18:47         215K

md5sum samizdat/lib/samizdat/antispam.rb
a614ca8565fbd9c4dd2c0ebfa67f3902  samizdat/lib/samizdat/antispam.rb

ls -l samizdat/lib/samizdat/antispam.rb
-rw-r--r--  1 boud boud 2361 2007-11-04 19:23 samizdat/lib/samizdat/antispam.rb


and the debian file:


http://ftp.debian.org/debian/pool/main/s/samizdat/samizdat_0.6.0.20080214-2_all.deb

md5sum usr/lib/ruby/1.8/samizdat/antispam.rb
a614ca8565fbd9c4dd2c0ebfa67f3902  usr/lib/ruby/1.8/samizdat/antispam.rb



i don't have a copy of the 071104 snapshot to check, but the internal
date suggests that the antispam.rb file in 080214 (snapshot)
does not have the correction. (i guess the cvs should have
a track of this.)



Boud Roukema <boud>
Group Member
Thu 21 Feb 2008 06:13:31 PM UTC, comment #5: 

I've already changed to single-slash in the Bonfire Night (2007-11-04) snapshot, I agree with the following analysis, the reason your last code snippet didn't work is that you need to use collect(), each() merely iterates through the list, while collect() returns array of the results which you can then join.

This fix works for me:

--- lib/samizdat/antispam.rb    cb5156e442c083173f072be5894e7e1ad5264b95
+++ lib/samizdat/antispam.rb    cc9b39541caafa9b15912f53be26aaf19776c320
@@ -60,7 +60,9 @@ class Antispam
   #
   def load_regexp_list(options)
     if options['exclude'].kind_of? Array
-      exclude = Regexp.new(options['exclude'].join('|')).freeze
+      exclude = Regexp.new(
+        options['exclude'].collect {|s| Regexp.escape(s) }.join('|')
+      ).freeze
     end
 
     begin

Dmitry Borodaenko <angdraug>
Group administrator
Sat 10 Nov 2007 01:36:32 PM UTC, comment #4: 

My explanation below in my previous comment:

"Hence, in the samizdat defaults.yaml or <site>.yaml
exclusion list, it had to be excluded with example\.com  ...
starting at character 0)."

is a bad explanation, sorry.  Here is a better explanation:


irb(main):090:0> 'example\.com' =~ /example\\.com/
=> 0
irb(main):091:0> 'example\.com' =~ /example\.com/
=> nil
irb(main):092:0> 'example.com' =~ /example\\.com/
=> nil
irb(main):093:0> 'example.com' =~ /example\.com/
=> 0


i tried with 'example\.com' in defaults.yaml as in the recent cvs update,
but this does not work for me.

Some further experiments are consistent with my intuition that if the
slash should be escaped, then the . should be escaped too:


irb(main):112:0> 'example\.com' =~ /example\\\.com/
=> 0
irb(main):113:0> 'example.com' =~ /example\.com/
=> 0
irb(main):114:0> 'example\xcom' =~ /example\\.com/
=> 0


If we want sysadmin to not have to escape regexp's, then IMHO we
probably should use Regexp.escape somehow in this line of antispam.rb:

      exclude = Regexp.new(options['exclude'].join('|')).freeze

i haven't yet found a correct way to do this... - i'm still
a ruby newbie...

i tried:

      long_string = options['exclude'].each { |string| Regexp.escape(string) }.join('|')
      exclude = Regexp.new(long_string).freeze


but that didn't seem to work.


Boud Roukema <boud>
Group Member
Fri 09 Nov 2007 10:11:48 PM UTC, comment #3: 

Maybe better is something like this, because this we're
dealing with a double negative (excluding an expression
for excluding spam text). Here it's clearer that we are
talking about exclusion from the URI list and we are
not talking about exclusion from some text submitted
for publication.


--- old/s070818/samizdat/data/samizdat/defaults.yaml        2007-08-13 23:39:32.000000000 +0200
+++ defaults.yaml   2007-11-09 22:37:18.829718392 +0100
@@ -147,6 +147,9 @@
   url: 'http://arch.thinkmo.de/cgi-bin/spam-merge'

   # Exclude certain URIs from the big wiki-spam URI list
+  # Examples:
+  #  example\\.com to exclude a line in the URI list such as   example\.com
+  #  example\.com  to exclude a line in the URI list such as example.com
   #
   exclude:
     - google-watch\\.org



Boud Roukema <boud>
Group Member
Fri 09 Nov 2007 10:05:45 PM UTC, comment #2: 

My initial report was based on insufficient testing, and i think
i now have consistent, repeatable behaviour.


SIDE COMMENT: To test this,
remember to try a spam expression with a minimum of 7 characters,
since that is what we put in order to exclude expressions which
are too generic. i say this because the arch.thinkmo.de/cgi-bin/spam-merge
spam list includes at least one expression with only 6 characters.
This was ignored by antispam.rb and (today) got me confused.
Given all the sorts of various things people might want to write
in some long text, i think that a spam expression of 6 characters
could turn up by accident in genuine text. Anyway, back to the
main issue of this bug report...

WHAT WORKS CORRECTLY NOW:
i'll write this as my recommended patch, which
i tried to make as
short as possible while also being reasonably clear, even if
it's not a technically full explanation:


--- old/s070818/samizdat/data/samizdat/defaults.yaml        2007-08-13 23:39:32.000000000 +0200
+++ defaults.yaml   2007-11-09 22:37:18.829718392 +0100
@@ -147,6 +147,9 @@
   url: 'http://arch.thinkmo.de/cgi-bin/spam-merge'

   # Exclude certain URIs from the big wiki-spam URI list
+  # Examples:
+  #  example\\.com to exclude a line with   example\.com
+  #  example\.com  to exclude a line with   example.com
   #
   exclude:
     - google-watch\\.org



EXPLANATION:

Some people put entries in the big wiki-spam list such
as


example.com


even though most entries have their dots escaped.

The effect is that text such as "examplexcom" or "examplescom"
is detected as spam - since the dot is the regex dot with a
special meaning, not a literal dot.

Most probably the non-escaping of these dots
are unintended errors. They probably won't
have much chance of excluding non-spam stuff in practice, but
the expressions catch more "spam" than is intended.

One genuine article that got treated as spam on a live
samizdat site contained the URL of a website which was on the
wiki-spam list with this unescaped dot. 

Hence, in the samizdat defaults.yaml or <site>.yaml
exclusion list, it had to be excluded with example\.com
rather than example\\.com, because


"example\.com" =~ /example\\.com/


gives nil

while


"example\.com" =~ /example\.com/


gives 0 (the expression is found starting at character 0).

For main spam list entries with escaped dots, the samizdat
config exclusion list needs to have a doubly escaped dot,
as in 070818-1, which i now think is correct.

RECOMMENDATION:

So now i just recommend the syadmin hint patch above since
i don't see any short term realistic way of getting the
main wikispam list "cleaned" upstream (i.e. dots escaped unless
they are genuinely intended to be unescaped). 

i've switched the status to "works for me".


PS:
As another
side issue (long term issue), if someone knows which discussion
list or wiki page is used to discuss/update the arch.thinkmo.de
list (or knows a "better" list), then please speak up. i looked
around a bit but didn't find it.


Boud Roukema <boud>
Group Member
Sun 04 Nov 2007 06:41:13 PM UTC, comment #1: 

Can't replicate: both versions work for me.

Dmitry Borodaenko <angdraug>
Group administrator
Thu 01 Nov 2007 03:17:36 PM UTC, original submission:  

In my defaults.yaml i need a single slash in front of dots for
excluding expressions from the spam list to work:


  exclude:
    - google-watch\.org
    - yahoo-watch\.org
    - wikipedia-watch\.org
    - ln-s\.net
    - namebase\.org


Double slash as in 070818 fails:


  exclude:
    - google-watch\\.org
    - yahoo-watch\\.org
    - wikipedia-watch\\.org
    - ln-s\\.net
    - namebase\\.org



Boud Roukema <boud>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by angdraug (Updated the item)
  • -email is unavailable- added by boud (Submitted the item)
  •  

    Follow 14 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2008-10-25 angdraug Severity3 - Normal 2 - Minor
    2008-10-25 angdraug StatusIn Progress Need Info
        Assigned toangdraug boud
    2008-07-18 boud StatusFixed In Progress
    2008-07-17 boud Attached File- Added 080717_antispam_21490_vs_080703.patch, #16130
        Open/ClosedClosed Open
        Summaryantispam exclusions: correct escaping of dots if the main spam list contains some unescaped dots REOPENED: antispam exclusions [vs 080703-2]: correct escaping of dots if the main spam list contains some unescaped dots
    2008-02-23 angdraug StatusWorks For Me Fixed
        Open/ClosedOpen Closed
    2008-02-21 angdraug Assigned toboud angdraug
    2007-11-09 boud StatusNeed Info Works For Me
        Summaryantispam exclusions: one slash or two for dots ? antispam exclusions: correct escaping of dots if the main spam list contains some unescaped dots
    2007-11-04 angdraug StatusNone Need Info
        Assigned toNone boud

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code