1 |
Arabic Analyzer for Java. |
2 |
|
3 |
This work is a Java port of the amazing Tim Buckwalter's Arabic Morphological Analyzer Version 1.0 (http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2002L49) originaly written in Perl. |
4 |
The morphological analyzer is available in the class gpl.pierrick.brihaye.aramorph.AraMorph. |
5 |
|
6 |
It also provides utility classes for analyzing arabic content with Lucene (http://jakarta.apache.org/lucene/docs/index.html). |
7 |
The Lucene classes are available in the package gpl.pierrick.brihaye.aramorph.lucene. |
8 |
2 command-line programs for testing the analyzers are available in the package gpl.pierrick.brihaye.aramorph.test. |
9 |
|
10 |
How does the analyzer work ? |
11 |
|
12 |
Like every Lucene analyzer, the input stream is read from a Reader, then : |
13 |
a) tokenized : see gpl.pierrick.brihaye.aramorph.lucene.ArabicTokenizer |
14 |
b) stemmed and grammatically analyzed : see gpl.pierrick.brihaye.aramorph.lucene.ArabicStemmer |
15 |
c) grammatically filtered : see gpl.pierrick.brihaye.aramorph.lucene.ArabicGrammaticalFilter |
16 |
|
17 |
The analyzer finally outputs tokens which are grammatically "valuable" stems (nouns, adjectives, verbs...). |
18 |
|
19 |
The tokenizer tokenizes the incoming stream at arabic letters and filters everything that is not an arabic "word". |
20 |
- TODO : consider output of arabic digits (with the problem of hindic/arabic digits whose accuracy as an arabic content is locale-dependant) |
21 |
- TODO : consider output of arabic punctuation. |
22 |
- TODO : consider output of non-arabic characters by making use of org.apache.lucene.analysis.Token.type() |
23 |
|
24 |
The morphological stemmer makes use of the AraMorph class to output one token per grammatical analysis of a word (see gpl.pierrick.brihaye.aramorph.Solution). If there are several ways to analyze a word, and this is quite usual, every token but the first one has its positionIncrement set to 0. See : http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/analysis/Token.html#setPositionIncrement(int). |
25 |
|
26 |
IMPORTANT NOTE : the actual implementation of Lucene's QueryParser is unable to handle such tokens. I'm working on a patch which is able to handle multiple tokens at the same position. |
27 |
|
28 |
An arabic word can basically be divided in a combination of prefix + stem + suffix. See : http://www.ldc.upenn.edu/doc/LDC2003T06/POS-info.txt. Only the grammatical category (aka POS) of the *stem* is retained as prefixes and suffixes are not to be considered as relevant in a "word analysis". |
29 |
|
30 |
The morphological filter filters the analyzed tokens whose grammatical category is not found to be semantically relevant (prepositions, particles, pronouns...). It is a grammatical approach for filtering "empty words". |
31 |
- TODO : not all possible grammatical categories have been encountered in test documents and the list of managed grammatical categories is incomplete. Feedback is welcome. |
32 |
- TODO : the relevant/not-relevant grammatical categories list is still subject to discussion. Feedback is also welcome here. |
33 |
|
34 |
The provided AraMorph class is still very close to he original Perl implementation and many TODOes remain : |
35 |
- TODO : use an external library for regular expressions. For now, the code relies on JDK 1.4+ native support of regular expressions. |
36 |
- TODO : use native Java Unicode Strings : for now, every arabic word is previously transformed in "BuckWalter transliteration" : http://www.qamus.org/transliteration.htm. |
37 |
- TODO : all diacritics are removed before the analysis, resulting in a loss of information that could help in giving more accurate analysis results. Not so easy to implement, I'm afraid... |
38 |
- TODO : use Unicode dictionaries, possibly in XML format. Maybe the arabized dictionaries of the Duali project (http://www.arabeyes.org/project.php?proj=Duali) could be a first step in achieving this goal. |
39 |
- TODO : improve the major performance issue, AraMorph initialization (~1 minute on my machine). All dictionaries are loaded in memory. Well... maybe a Lucene index could help ;-) ? For this purpose, it would be interesting to have a more generic interface for the dictionary handlers. |
40 |
- TODO : make better consideration of synchronization in a multi-thread environment. |
41 |
- TODO : improve/refactor the caching mechanism, which is very basic. |
42 |
- TODO : improve Buckwalter's dictionaries. There are no analysis results when the "w(a)" cunjunction is aggregated with a negative particle ("lA"/"mA") and a following verb (e.g. "wlAyktb" returns nothing whereas "wlA yktb" returns a correct analysis). |
43 |
- TODO : output analysis results to an XML Consumer ? |
44 |
- TODO : get rid of Perlicisms ;-) |
45 |
|
46 |
Bonus : |
47 |
Also enclosed is an english/arabic glosser. It works quite the same than the ArabicStemmer, but it ouputs the english glosses of the stems rather than the stem itself. This kind of analyzer is a rare case where the search analyzer shall be different from the one used during indexing. |
48 |
|
49 |
I am very indebted in Tim Buckwalter's work ; nothing would have been possible without it. |
50 |
|
51 |
Pierrick Brihaye, 2003/10/11 |
52 |
pierrick.brihaye@wanadoo.fr |