📄 中文摘要
大型语言模型（LLMs）在处理高资源语言方面取得了显著进展，但在低资源语言和方言方面仍面临挑战。尤其对于阿拉伯语，其拥有众多方言，各方言之间存在显著的词汇、语法和语音差异，这使得LLMs在处理特定阿拉伯语方言时表现不佳。阿联酋方言是海湾阿拉伯语的一种重要变体，具有其独特的语言特征。为了系统性评估LLMs在理解和生成阿联酋方言方面的能力，构建了一个名为“Alyah ⭐️”的综合基准套件。该基准套件包含多种任务类型，涵盖自然语言理解（NLU）和自然语言生成（NLG）两方面。NLU任务包括情感分析、命名实体识别、问答等，旨在测试模型对方言文本的语义理解能力。NLG任务则涉及文本摘要、机器翻译（从标准阿拉伯语到阿联酋方言，以及从阿联酋方言到标准阿拉伯语）和对话生成，以评估模型生成符合方言语境和语法规则文本的能力。Alyah ⭐️的数据集通过众包、专家标注和现有资源改编等方式构建，确保了数据的多样性和高质量。在Alyah ⭐️基准上对一系列主流阿拉伯语LLMs进行了全面测试，包括开源模型和闭源API模型。评估结果揭示了当前LLMs在处理阿联酋方言时普遍存在的挑战，例如在处理方言特有词汇、俚语和复杂语法结构时的不足。同时，也观察到不同模型在不同任务上的表现差异，为未来模型改进提供了方向。该工作不仅提供了一个用于评估阿拉伯语LLMs方言能力的标准化工具，也为促进LLMs在低资源方言领域的进一步发展奠定了基础，有助于提升模型在实际应用中对多方言环境的适应性。
📄 English Summary
Alyah ⭐️: Toward Robust Evaluation of Emirati Dialect Capabilities in Arabic LLMs
Large Language Models (LLMs) have demonstrated remarkable progress in high-resource languages, yet they continue to face significant challenges with low-resource languages and dialects. This is particularly true for Arabic, which encompasses numerous dialects, each exhibiting substantial lexical, grammatical, and phonological variations. These variations often lead to suboptimal performance of LLMs when processing specific Arabic dialects. Emirati dialect, a prominent variant of Gulf Arabic, possesses distinct linguistic characteristics. To systematically evaluate the capabilities of LLMs in understanding and generating Emirati dialect, a comprehensive benchmark suite named “Alyah ⭐️” has been constructed. This benchmark suite comprises diverse task types, covering both Natural Language Understanding (NLU) and Natural Language Generation (NLG). NLU tasks include sentiment analysis, named entity recognition, and question answering, designed to test the models' semantic comprehension of dialectal text. NLG tasks involve text summarization, machine translation (from Modern Standard Arabic to Emirati dialect and vice versa), and dialogue generation, assessing the models' ability to produce text that adheres to dialectal context and grammatical rules. The Alyah ⭐️ dataset was curated through crowdsourcing, expert annotation, and adaptation of existing resources, ensuring data diversity and high quality. A comprehensive evaluation of a range of prominent Arabic LLMs, including both open-source models and closed-source API models, was conducted using the Alyah ⭐️ benchmark. The evaluation results reveal common challenges faced by current LLMs when processing Emirati dialect, such as deficiencies in handling dialect-specific vocabulary, slang, and complex grammatical structures. Furthermore, variations in performance across different models and tasks were observed, providing valuable insights for future model improvements. This work not only provides a standardized tool for evaluating the dialectal capabilities of Arabic LLMs but also lays the groundwork for promoting further advancements of LLMs in low-resource dialect domains, thereby enhancing their adaptability to multi-dialectal environments in real-world applications.