• Arabic LLMs often evaluated only on Modern Standard Arabic, neglecting dialects. • Dialects like Emirati carry unique vocabulary, syntax, and cultural context. • Existing benchmarks miss culturally grounded expressions, idioms, and local anecdotes. • Alyah benchmark focuses on Emirati dialect, testing cultural and pragmatic understanding. • It includes greetings, proverbs, oral poetry, heritage-related content. • Goal: assess LLMs’ ability to interpret Emirati-specific meaning beyond literal translation.
Article Summaries:
- Alyah ⭐️ is a newly released benchmark aimed at measuring Arabic large language models’ proficiency in the Emirati dialect. The dataset contains 1,173 manually curated multiple‑choice items sourced from native speakers, covering greetings, idioms, folklore, poetry, and culturally specific references. Unlike most Arabic benchmarks that focus on Modern Standard Arabic, Alyah probes deeper linguistic and pragmatic aspects unique to Emirati speech. Distractor options were generated by LLMs and then vetted for plausibility, with correct answers randomly positioned to avoid bias. The benchmark seeks to reveal systematic strengths and weaknesses of Arabic LLMs when confronted with authentic, dialect‑rich language use.
Sources: