bugVarnamproject - Bugs: bug #40404, Create a words crawler for...


bug #40404: Create a words crawler for obtaining initial training set

Submitter:  Navaneeth <navaneethkn>
Submitted:  Tue 29 Oct 2013 02:46:00 AM UTC
Category:  None Severity:  3 - Normal
Item Group:  * New feature Status:  In Progress
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup

Tue 29 Oct 2013 02:46:00 AM UTC, original submission:  

When a new language is added, it need to get some initial training set of words. This is done by crawling the web and collecting language specific words. Currently this is done with a simple wget script but this requires multiple levels of crawling and organizing data. To address this,

1 - Create a crawler which reads a simple text file specifying the URLs to crawl
2 - Crawl the data and access only words which are in the needed language. Basic language detection can be done using libvarnam or manually checking for the Unicode code point range.
3 - Segregate the words according to the frequency and generate text files which can be used as the initial training set.

There should be an options to specify which words should not be treated for frequency calculation for each URLs. This is required to avoid the site's name or owners name gets more precedence than the actual content which we are crawling.

Navaneeth <navaneethkn>
Group administrator


(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:

No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by navaneethkn (Submitted the item)

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.


    No changes have been made to this item

    Back to the top

    Powered by Savane 3.13-4b48.
    Corresponding source code