A comparative study with human data: Do LLMs have superficial language processing? İnsan verisiyle karşılaştırmalı bir çalışma: Büyük dil modelleri dili ne kadar yüzeysel işliyor?
32nd IEEE Conference on Signal Processing and Communications Applications, SIU 2024, Mersin, Türkiye, 15 - 18 Mayıs 2024, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/siu61531.2024.10600807
- Basıldığı Şehir: Mersin
- Basıldığı Ülke: Türkiye
- Anahtar Kelimeler: sentence processing, LLMs, natural language generation, psycholinguistics
- Boğaziçi Üniversitesi Adresli: Evet
Özet
This study aims to test the superficial language processing hypothesis on a natural language generation (NLG) task with large language models (LLMs) and compare it with human performance. We examined whether LLMs sometimes process sentences superficially like humans or always have deep-syntactic processing. The good-enough language processing experiment conducted by Ferreira et al. with English speakers was repeated in this study with OpenAI"s GPT-3.5-turbo and textdavinci-003 models. The grammatical accuracy of the answers given by the LLMs was compared with the average Surprisal (negative log-likelihood) of the English experimental sentences we predicted with RoBERTa, and with human data. We found that the GPT-3.5-turbo model was more accurate than the textdavinci-003 model, but was less successful than humans in processing sentences with semantic anomalies and these models were not able to do abstraction like humans. When compared with Surprisal values, we found that the GPT-3.5-turbo model processed sentences by paying more attention to syntactic cues, while the text-davinci-003 model processed sentences superficially.