/[aramorph]/aramorph/README.TXT
ViewVC logotype

Contents of /aramorph/README.TXT

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.1 - (show annotations) (download)
Wed Oct 15 17:12:20 2003 UTC (20 years, 6 months ago) by brihaye
Branch: MAIN
CVS Tags: HEAD
File MIME type: text/plain
Initial import.

1 Arabic Analyzer for Java.
2
3 This work is a Java port of the amazing Tim Buckwalter's Arabic Morphological Analyzer Version 1.0 (http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2002L49) originaly written in Perl.
4 The morphological analyzer is available in the class gpl.pierrick.brihaye.aramorph.AraMorph.
5
6 It also provides utility classes for analyzing arabic content with Lucene (http://jakarta.apache.org/lucene/docs/index.html).
7 The Lucene classes are available in the package gpl.pierrick.brihaye.aramorph.lucene.
8 2 command-line programs for testing the analyzers are available in the package gpl.pierrick.brihaye.aramorph.test.
9
10 How does the analyzer work ?
11
12 Like every Lucene analyzer, the input stream is read from a Reader, then :
13 a) tokenized : see gpl.pierrick.brihaye.aramorph.lucene.ArabicTokenizer
14 b) stemmed and grammatically analyzed : see gpl.pierrick.brihaye.aramorph.lucene.ArabicStemmer
15 c) grammatically filtered : see gpl.pierrick.brihaye.aramorph.lucene.ArabicGrammaticalFilter
16
17 The analyzer finally outputs tokens which are grammatically "valuable" stems (nouns, adjectives, verbs...).
18
19 The tokenizer tokenizes the incoming stream at arabic letters and filters everything that is not an arabic "word".
20 - TODO : consider output of arabic digits (with the problem of hindic/arabic digits whose accuracy as an arabic content is locale-dependant)
21 - TODO : consider output of arabic punctuation.
22 - TODO : consider output of non-arabic characters by making use of org.apache.lucene.analysis.Token.type()
23
24 The morphological stemmer makes use of the AraMorph class to output one token per grammatical analysis of a word (see gpl.pierrick.brihaye.aramorph.Solution). If there are several ways to analyze a word, and this is quite usual, every token but the first one has its positionIncrement set to 0. See : http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/analysis/Token.html#setPositionIncrement(int).
25
26 IMPORTANT NOTE : the actual implementation of Lucene's QueryParser is unable to handle such tokens. I'm working on a patch which is able to handle multiple tokens at the same position.
27
28 An arabic word can basically be divided in a combination of prefix + stem + suffix. See : http://www.ldc.upenn.edu/doc/LDC2003T06/POS-info.txt. Only the grammatical category (aka POS) of the *stem* is retained as prefixes and suffixes are not to be considered as relevant in a "word analysis".
29
30 The morphological filter filters the analyzed tokens whose grammatical category is not found to be semantically relevant (prepositions, particles, pronouns...). It is a grammatical approach for filtering "empty words".
31 - TODO : not all possible grammatical categories have been encountered in test documents and the list of managed grammatical categories is incomplete. Feedback is welcome.
32 - TODO : the relevant/not-relevant grammatical categories list is still subject to discussion. Feedback is also welcome here.
33
34 The provided AraMorph class is still very close to he original Perl implementation and many TODOes remain :
35 - TODO : use an external library for regular expressions. For now, the code relies on JDK 1.4+ native support of regular expressions.
36 - TODO : use native Java Unicode Strings : for now, every arabic word is previously transformed in "BuckWalter transliteration" : http://www.qamus.org/transliteration.htm.
37 - TODO : all diacritics are removed before the analysis, resulting in a loss of information that could help in giving more accurate analysis results. Not so easy to implement, I'm afraid...
38 - TODO : use Unicode dictionaries, possibly in XML format. Maybe the arabized dictionaries of the Duali project (http://www.arabeyes.org/project.php?proj=Duali) could be a first step in achieving this goal.
39 - TODO : improve the major performance issue, AraMorph initialization (~1 minute on my machine). All dictionaries are loaded in memory. Well... maybe a Lucene index could help ;-) ? For this purpose, it would be interesting to have a more generic interface for the dictionary handlers.
40 - TODO : make better consideration of synchronization in a multi-thread environment.
41 - TODO : improve/refactor the caching mechanism, which is very basic.
42 - TODO : improve Buckwalter's dictionaries. There are no analysis results when the "w(a)" cunjunction is aggregated with a negative particle ("lA"/"mA") and a following verb (e.g. "wlAyktb" returns nothing whereas "wlA yktb" returns a correct analysis).
43 - TODO : output analysis results to an XML Consumer ?
44 - TODO : get rid of Perlicisms ;-)
45
46 Bonus :
47 Also enclosed is an english/arabic glosser. It works quite the same than the ArabicStemmer, but it ouputs the english glosses of the stems rather than the stem itself. This kind of analyzer is a rare case where the search analyzer shall be different from the one used during indexing.
48
49 I am very indebted in Tim Buckwalter's work ; nothing would have been possible without it.
50
51 Pierrick Brihaye, 2003/10/11
52 pierrick.brihaye@wanadoo.fr

savannah-hackers-public@gnu.org
ViewVC Help
Powered by ViewVC 1.1.26