taskLinterna Mágica - Tasks: task #11216, The way web sites detection and...


You are not allowed to post comments on this tracker with your current authentication level.

task #11216: The way web sites detection and support is handled should change.

Submitter:  Ivaylo Valkov <valkov>
Submitted:  Wed 29 Jun 2011 12:30:14 PM UTC
Should Start On:  Wed 29 Jun 2011 12:00:00 AM UTC Should be Finished on:  Tue 26 Jul 2011 12:00:00 AM UTC
Category:  Code and framework changes Priority:  7 - High
Status:  Done Privacy:  Public
Assigned to:  valkov Open/Closed:  Closed
Planned Release:  0.0.10

Tue 19 Jul 2011 10:32:34 AM UTC, comment #1: 

Changing the end date secund time. Too much changes are still needed in branch task-11216.

Ivaylo Valkov <valkov>
Group administrator
Wed 29 Jun 2011 12:30:14 PM UTC, original submission:  

The main reason for this change is the site specific support. It is
spread in various files. Another reason for the change is the regular
expression that handles most of the sites, the general purpose one. It
is getting big and hard to work with.

The idea is to keep all site specific support in a single file. Some
sites already have most of the specific code in single file, but some
part of the logic is still in other files.

The task requires framework modifications. There is a basic idea in
the TODO file. Every site that requires specific extraction should have
a config template:

    rexgexp: "bla bla | bla ",
    // For xmlhttprequest extraction
    method: "get",
    "extract_url" : "url",
    "extract_elements" : array,
    "make_url" : "http://a.b.c/ 1+ 2 + 3" // use the above array

This was the first draft. After examining the code it is obvious,
that site specific code is executed at 20 to 30 places
(conditions/entry points). These should be used to make templates for

An object should be defined that would hold all site specific code:

LinternaMagica.prototype.sites = new Object();

A key in the object is a web site domain name. Each value is another
object with standard keys, that match the entry points mentioned

LinternaMagica.prototype.site["example.org"] = new Object();
LinternaMagica.prototype.site["example.org"]["ENTRY_POINT_PREDEFINED_NAME"] = ENTRY_POINT_VALUE;

An ENTRY_POINT_VALUE, should be a function with predefined arguments
and return value. These functions will be executed at the places where
the site specific code is executed now.

An ENTRY_POINT_VALUE could reference a function for another
site. Either by directly pointing the object or by a string value.
This could be used to reduce code duplication.

Only the entry points that are needed for a site will be defined.

A general purpose extraction code will be used if a site specific code
is not found.

The site specific code/templates for detecting remotely embedded objects
could be moved in the same file that will have all site specific code.

Ivaylo Valkov <valkov>
Group administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by valkov (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.


    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-07-26 valkov StatusIn Progress Done
        Open/ClosedOpen Closed
    2011-07-19 valkov Should be Finished on2011-07-18 2011-07-26
    2011-07-13 valkov Should be Finished on2011-07-13 2011-07-18

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code