Deep Learning-Based Preprocessing Tools for Turkish Natural Language Processing
5th International Conference on Deep Learning Theory and Applications, DeLTA 2024, Dijon, Fransa, 10 - 11 Temmuz 2024, cilt.2172 CCIS, ss.218-234, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Cilt numarası: 2172 CCIS
- Doi Numarası: 10.1007/978-3-031-66705-3_15
- Basıldığı Şehir: Dijon
- Basıldığı Ülke: Fransa
- Sayfa Sayıları: ss.218-234
- Anahtar Kelimeler: Natural language processing, Text preprocessing, Turkish
- Boğaziçi Üniversitesi Adresli: Evet
Özet
As the demand for effective natural language processing applications in Turkish continues to rise, the need for text preprocessing tools tailored to the Turkish language increases. These tools form the initial step of any natural language application and improves the efficiency of complex tasks such as text summarization, question-answering, and machine translation. We propose a novel deep learning-based framework focusing on Turkish preprocessing tasks, including tokenization, sentence splitting, deasciification, part-of-speech tagging, vowelization, spell correction, and morphological analysis. The proposed framework is suitable for independent use of each preprocessing tool as well as the use in an all-in-one scheme. We use the CANINE model to train the character-level tools, and BERT and mT5 models for the token-based tools. We evaluate the framework for each task on the BOUN Treebank in the UD project and make both the tools and the codes publicly available.