bugVarnamproject - Bugs: bug #40404, Create a words crawler for...

 
 

bug #40404: Create a words crawler for obtaining initial training set

Submitted by:  Navaneeth <navaneethkn>
Submitted on:  Tue 29 Oct 2013 02:46:00 AM UTC  
 
Category: NoneSeverity: 3 - Normal
Item Group: New featureStatus: In Progress
Privacy: PublicAssigned to: None
Open/Closed: Open

Add a New Comment (Rich MarkupRich Markup):
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 29 Oct 2013 02:46:00 AM UTC, original submission:

When a new language is added, it need to get some initial training set of words. This is done by crawling the web and collecting language specific words. Currently this is done with a simple wget script but this requires multiple levels of crawling and organizing data. To address this,

1 - Create a crawler which reads a simple text file specifying the URLs to crawl
2 - Crawl the data and access only words which are in the needed language. Basic language detection can be done using libvarnam or manually checking for the Unicode code point range.
3 - Segregate the words according to the frequency and generate text files which can be used as the initial training set.

There should be an options to specify which words should not be treated for frequency calculation for each URLs. This is required to avoid the site's name or owners name gets more precedence than the actual content which we are crawling.

Navaneeth <navaneethkn>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by navaneethkn (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    No Changes Have Been Made to This Item

    Back to the top


    Powered by Savane 3.1-cleanup