LLM-as-a-Judge: automated evaluation of search query parsing using large language models
Frontiers in Big Data, cilt.8, 2025 (ESCI, Scopus)
- Yayın Türü: Makale / Tam Makale
- Cilt numarası: 8
- Basım Tarihi: 2025
- Doi Numarası: 10.3389/fdata.2025.1611389
- Dergi Adı: Frontiers in Big Data
- Derginin Tarandığı İndeksler: Emerging Sources Citation Index (ESCI), Scopus
- Anahtar Kelimeler: LLM-as-a-Judge, structured output evaluation, search query parsing, large language models, evaluation framework, generative search, automatic evaluation, query understanding
- Açık Arşiv Koleksiyonu: AVESİS Açık Erişim Koleksiyonu
- Boğaziçi Üniversitesi Adresli: Evet
Özet
Introduction: The adoption of Large Language Models (LLMs) in search systems necessitates new evaluation methodologies beyond traditional rule-based or manual approaches. Methods: We propose a general framework for evaluating structured outputs using LLMs, focusing on search query parsing within an online classified platform. Our approach leverages LLMs' contextual reasoning capabilities through three evaluation methodologies: Pointwise, Pairwise, and Pass/Fail assessments. Additionally, we introduce a Contextual Evaluation Prompt Routing strategy to improve reliability and reduce hallucinations. Results: Experiments conducted on both small- and large-scale datasets demonstrate that LLM-based evaluation achieves approximately 90% agreement with human judgments. Discussion: These results validate LLM-driven evaluation as a scalable, interpretable, and effective alternative to traditional evaluation methods, providing robust query parsing for real-world search systems.