Morphosyntactic Evaluation for Text Summarization in Morphologically Rich Languages: A Case Study for Turkish
28th International Conference on Applications of Natural Language to Information Systems, NLDB 2023, Derby, İngiltere, 21 - 23 Haziran 2023, cilt.13913 LNCS, ss.201-214, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Cilt numarası: 13913 LNCS
- Doi Numarası: 10.1007/978-3-031-35320-8_14
- Basıldığı Şehir: Derby
- Basıldığı Ülke: İngiltere
- Sayfa Sayıları: ss.201-214
- Anahtar Kelimeler: Morphologically rich languages, Text summarization, Text summarization evaluation
- Boğaziçi Üniversitesi Adresli: Evet
Özet
The evaluation strategy used in text summarization is critical in assessing the relevancy between system summaries and reference summaries. Most of the current evaluation metrics such as ROUGE and METEOR are based on n-gram exact matching strategy. However, this strategy cannot capture the orthographical variations in abstractive summaries and is highly restrictive especially for languages with rich morphology that make use of affixation extensively. In this paper, we propose several variants of the evaluation metrics that take into account morphosyntactic properties of the words. We make a correlation analysis between each of the proposed approaches and the human judgments on a manually annotated dataset that we introduce in this study. The results show that using morphosyntactic tokenization in evaluation metrics outperforms the commonly used evaluation strategy in text summarization.