A New Framework for Evaluating Voice Agents (EVA)

Introduction: The Challenges of Evaluating Voice Agents

Voice agents, similar to chatbots, have been playing a critical role in various fields such as customer service, reservations, and information provision in recent years. However, effectively evaluating these voice agents poses a significant challenge. It’s not enough to simply determine if a task is completed; user satisfaction and natural conversational flow must also be considered. Existing evaluation methods have treated accuracy and conversational experience as separate issues, failing to adequately reflect overall quality.

Hugging Face has developed a new framework, EVA (Evaluation of Voice Agents), to evaluate the overall quality of voice agents. EVA considers both task accuracy and conversational experience simultaneously, designed to measure the performance of voice agents in an environment similar to a real-world usage setting. It includes a dataset of 50 airline scenario data and will be expanded to various fields in the future.

Background and Motivation: Limitations of Existing Evaluation Methods

Existing voice agent evaluation methods primarily focus on individual components. AudioBench, SD-Eval, VoxEval, Kimi-Eval, VoiceBench, VoxDialogue evaluate speech recognition capabilities; EmergentTTS, SHEET evaluate audio quality; FD-Bench, Talking Turns, Full-Duplex-Bench evaluate conversational flow; VoiceAgentBench, CAVA evaluate tool usage capabilities and understanding of complex instructions. However, they lacked consideration for the overall conversational flow that a voice agent must handle. The need for a framework that comprehensively evaluates the accuracy and experience of voice agents has become apparent to overcome these limitations.

Components of the EVA Framework

EVA is a framework designed to evaluate the overall performance of a voice agent, consisting of the following key components:

• User Simulator: An AI with goals and personas performs conversations similar to real users. Implements a natural conversational flow using a high-quality TTS model.
• Voice Agent: The voice agent system being evaluated. Built using the Pipecat framework, supporting both a cascade architecture (STT → LLM → TTS) and audio-native models (S2S or S2T→ TTS).
• Tool Executor: Executes the tools needed for the voice agent to perform specific tasks and provides results. Dynamically queries and modifies a predefined scenario database.
• Validators: Verify conversational completeness and confirm that the user’s intended actions and utterances are accurately reproduced. Regenerate conversations if validation fails, using only valid data for evaluation.
• Metrics Suite: Analyzes conversational records, audio recordings, and tool call logs to evaluate the performance of the voice agent. Measures accuracy (EVA-A) and experience (EVA-X), and also provides diagnostic metrics for problem-solving.

Dataset and Evaluation Methodology

EVA is composed of evaluation records, each of which includes a test scenario. Each record contains the user’s goal, persona, scenario database, and ground truth data. A dataset of 50 English airline scenarios is currently available, testing various situations such as rebooking flights, cancellations, voucher provision, same-day boarding, and compensation vouchers. These datasets provide an environment for time reasoning, policy compliance, constraint satisfaction, and named entity processing.

EVA evaluates the accuracy (EVA-A) and experience (EVA-X) of the voice agent. Accuracy measures task completion, truthfulness of answers, and clarity of speech, while experience evaluates conciseness, conversational flow, and turn-taking. It also provides diagnostic metrics for identifying problems in specific areas such as ASR, speech synthesis, and tool usage. Both deterministic measurement methods based on code and evaluation methods utilizing LLMs/LLALMs are used.

Key Findings and Industry Impact

Evaluation of 20 voice agent systems revealed a trade-off between task completion performance and user experience. Focusing solely on task completion led to a decline in user experience, while focusing solely on improving user experience resulted in a lower task completion rate. It was also found that speech recognition errors significantly affect the overall conversational flow, and the system’s weaknesses are clearly revealed in multi-step task flows. The importance of considering both accuracy and user experience of voice agents has been highlighted.

These research findings provide important implications for the direction of voice agent development. Developers should strive to balance task completion rates and user experience, and particularly need improvements in speech recognition errors and multi-step task flows. It is also necessary to increase the reliability and stability of voice agents through testing in real-world environments and acquiring data.

Future Plans and Prospects

In the future, EVA is scheduled to add features such as voice quality evaluation, robustness testing in noisy environments, multi-language support, and user emotion recognition evaluation. It is also planned to expand to various datasets and support more complex scenarios. A code-based application will be developed to increase user understanding and contribute further to improving voice agent performance.

In-Depth Analysis and Implications

Array

Original Source: A New Framework for Evaluation of Voice Agents (EVA)