Neighbour unpredictability measure in multiword expression extraction

Abstract

In all natural languages, due to the strong cohesive ties between the composing words, some recurrent combinations of words generate multiword expressions (MWEs). The extraction of MWEs in a text has an important role in natural language processing applications and information retrieval. In this study, we introduce a method of MWE extraction that ranks the candidates by the weakness of outer ties between the candidate and the neighbouring words in the text. The method presents a measure for the weakness of outer ties based on the degree of unpredictability of surrounding words in order to distinguish MWEs from other recurrent groups of consecutive words. Simply in the method, if the words following and preceding a MWE candidate are unpredictable due to the relatively excessive number of different neighbouring words, the candidate is accepted to have a strong evidence to be a real MWE. The method generates a single normalized score of unpredictability, which enables not only the comparison of MWE candidates of different occurrence frequency but also the comparison of MWE candidates with different number of composing words (such as the comparison of two-word candidates with three-word candidates). Comparisons with different groups of well-known methods; statistical measures of association and term hood, vector space models of composition and supervised learning methods; illustrate the effectiveness of the proposed method on two-word MWE candidates and in the merged set of two- and three-word candidates.