• Computer Science > Human-Computer Interaction [Submitted on 10 Jan 2026] Title:A Methodology for Identifying Evaluation Items for Practical Dialogue Systems Based on Business-Dialogue System Alignment Models View PDF HTML (experimental)Abstract:This paper proposes a methodology for identifying evaluation items for practical dialogue systems. • Traditionally, user satisfaction and user experiences have been the primary metrics for evaluating dialogue systems. • However, there are various other evaluation items to consider when developing and operating practical dialogue systems, and such evaluation items are expected to lead to new research topics. • So far, there has been no methodology for identifying these evaluation items. • We propose identifying evaluation items based on business-dialogue system alignment models, which are applications of business-IT alignment models used in the development and operation of practical IT systems. • We also present a generic model that facilitates the construction of a business-dialogue system alignment model for each dialogue system.
Article Summaries:
- A recent computer‑science submission proposes a new framework for evaluating practical dialogue systems. The authors argue that user satisfaction and experience have dominated evaluation metrics, overlooking other important factors that could spur further research. They introduce a methodology that derives evaluation items from business‑dialogue system alignment models-adaptations of business‑IT alignment concepts applied to dialogue system design and operation. The paper also presents a generic model that can be customized for any dialogue system, enabling systematic identification of relevant evaluation criteria beyond traditional user‑centric measures.
Sources: