Jun Da's WebCentral	Home | Academic | Chinese | CALL | Systems | Personal | Contact
Chinese text computing (This is the 1998 version. An updated 2004 version is now available)

In the above table:

• Total number of characters refers to all the characters found in a corpus;
• Total number of digrams refers to all the digram tokens in a corpus.

In the bigram lists with mutual information statistics.

• Columns are delimited by a single space;
• Column 1: Individual bigram;
• Column 2: Raw frequency;
• Column 3: Mutual information score;
• Column 4: First character;
• Column 5: First character frequency. That is its frequency in the entire corpus irrespective of its position;
• Column 6: Second character;
• Column 7: Second character frequency. That is its frequency in the entire corpus irrespective of its position.

3. Computing mutual information statistics for collocation

How mutual information scores are computed?

Please refer http://www.umiacs.umd.edu/users/resnik/nlstat_tutorial_summer1998/Lab_ngrams.html, where both computing procedure and formula can be found.

How to interpret mutual information scores?

The following guidelines can be used:

• High (MI >= 5)
• Medium ( 4 >= MI >=3)
• Low (MI <= 1)

If the scores are high or medium, the collocation strength is strong. If MI is below 1, it is less likely that the two tokens are related. MI scores between 1 and 3 are in the gray area. My intuitive judegement of  the bigram lists with MI score larger than 2.5 appear to be bisyllabic words in Chinese, though such intuition needs to be verified.

Other statistical measures of collocation

Other statistical measures such as t-score, likelihood ratio, chi-square and Yule's Y are often used to measure collocation strength. For an introduction and comparison of those measures, please refer to, among others:

• Christopher D. Manning and Hinrich Schütze. 1999. Foundation of Statistical Natural Language Processing (Chapter 5). Cambridge, MA: The MIT Press. (http://www.sultry.arts.usyd.edu.au/fsnlp/promo/)
• Kageura, Kyo. 1999. Bigram Statistics Revisited: A Comparative Examination of Some Statistical Measures in Morphological Analysis of Japanese Kanji Sequences. Journal of Quantitative Linguistics. 6,2,149-166 (http://www.swets.nl/sps/journals/jql0602.html#abs4)

In my informal experiments with likelihood ratios and chi-square measures, I found that those two statistics do not provide a reliable measure of collocation as far as the two diagram lists are concerned. The problem with the two measures is that they are much less discriminative as compared with MI. For preliminary result, please take a look at this comparison page.

Viewing instruction: The comparison page is pretty big (818K) and contains GB-encoded Chinese text. It takes time to download. If you are using a non-localized operating system, it also takes time for your web browser to parse the .html source. PATIENCE is recommended.

The comparison page lists a subset of bigrams from the Feng Huan Yuan corpus. Each bigram contains the character "µÄ
(de, POSSESSIVE of)". In Chinese, there are a only few bisyllabic words which can be formed with the character "µÄ (of)". As you will find out from the page, most of the bigrams in the list are non-sense two-character sequences. If we use 2.5 as the cut-off score for mutual information, we can easily identify those bigrams as non-sense sequence. However, the likelihood ratio statistics have a wide range. It is difficult to use the ratio as a reliable measure to separate meaningful sequence (i.e., bisyllablc words) from non-sense sequences.

A more formal study is underway and I will post my result when it becomes available.