The PARADISE Evaluation Framework: Issues and Findings

Hajdinjak, Melita

doi:10.1162/coli.2006.32.2.263

Cited by 39 publications

(26 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(See [70] for criticisms.) Equations were developed to predict dialog efficiency (which depends on mean elapsed time and the mean number of user moves) and dialog quality costs (which depends on the number of missing responses, the number of errors, and many other factors, and task success, measured by the Kappa coefficient and defined below):…”

Section: What Should Be Measured?mentioning

confidence: 99%

Development and Evaluation of Automotive Speech Interfaces: Useful Information from the Human Factors and the Related Literature

Green

2013

International Journal of Vehicular Technology

View full text Add to dashboard Cite

Drivers often use infotainment systems in motor vehicles, such as systems for navigation, music, and phones. However, operating visual-manual interfaces for these systems can distract drivers. Speech interfaces may be less distracting. To help designing easyto-use speech interfaces, this paper identifies key speech interfaces (e.g., CHAT, Linguatronic, SYNC, Siri, and Google Voice), their features, and what was learned from evaluating them and other systems. Also included is information on key technical standards (e.g., ISO 9921, ITU P.800) and relevant design guidelines. This paper also describes relevant design and evaluation methods (e.g., Wizard of Oz) and how to make driving studies replicable (e.g., by referencing SAE J2944). Throughout the paper, there is discussion of linguistic terms (e.g., turn-taking) and principles (e.g., Grice's Conversational Maxims) that provide a basis for describing userdevice interactions and errors in evaluations.

show abstract

Section: What Should Be Measured?mentioning

confidence: 99%

Development and Evaluation of Automotive Speech Interfaces: Useful Information from the Human Factors and the Related Literature

Green

2013

International Journal of Vehicular Technology

View full text Add to dashboard Cite

show abstract

“…On the one hand, this could indicate that the selected individual user-satisfaction measures really measure the performance of the dialogue manager and consequently illustrate the obvious difference between both dialogue-management manners. On the other hand, one could argue that this simply means that the individual user-satisfaction measures are not appropriate measures of attitude because people are likely to vary in the way they interpret the item wording [9]. Though, due to the huge difference in significance, this seems an unlikely explanation.…”

Section: Performance Function Resultsmentioning

confidence: 99%

“…Some important PARADISE details and issues were, however, highlighted by Hajdinjak and Mihelič [9]. Applying PARADISE to dialogue data requires the dialogue corpora to be collected via controlled experiments during which users subjectively rate their satisfaction.…”

Section: Introductionmentioning

confidence: 99%

A Dialogue-Management Evaluation Study

Hajdinjak¹,

Mihelic²

2007

CIT

Self Cite

View full text Add to dashboard Cite

We present a highly portable and cooperative dialoguemanager component of a developing, Slovenian and Croatian spoken dialogue system for weather-information retrieval. In order to evaluate the performance of this component, two Wizard-of-Oz experiments were performed. The only difference between the two experiment settings was in the dialogue-management manner, i.e., while in the first experiment dialogue management was performed by a human, the wizard, in the second experiment it was performed by the newlyimplemented dialogue-manager component. The data from both Wizard-of-Oz experiments was evaluated with the PARADISE evaluation framework,which was proposed as a potential general methodology for evaluating and comparing different versions of spoken-language dialogue systems. The study ascertains a remarkable difference in the performance functions when taking different satisfaction-measure sums, or even individual scores as the target to be predicted, it introduces the dialogue costs database parameters, and it confirms the dialogue manager's cooperativity subject to the incorporated knowledge representation.

show abstract

“…Hassel and Hagen [7] investigated feature selection and introduced indirect parameters into AVMs to overcome the drawbacks of PARADISE. Hajdinjak and Miheliĉ [8] discussed regression parameter selection and related normalization.…”

Section: Related Workmentioning

confidence: 99%

Using finite state machines for evaluating spoken dialog systems

Zhu

Yang

Meng

et al. 2010

2010 IEEE Spoken Language Technology Workshop

View full text Add to dashboard Cite

Development of spoken dialog systems (SDSs) can be facilitated by better evaluation methods. Previous methods seldom consider the efficiency of the system, which is important to users. We study the problem of evaluating SDSs and propose a new framework by generalizing states from utterances of dialogs to build finite state machine (FSM). These states can be regarded as efficiency measurement of SDSs. The FSM framework models dialogs as paths in an FSM to combine efficiency measurement with regression models. The proposed FSM framework can be applied in conjunction with regression models to improve evaluation accuracy. We compare our FSM framework combined with three regression models in several experiments. We obtain promising results on a collection of dialogs from the "Let's Go!" system, with our approach outperforming regression models.

show abstract

The PARADISE Evaluation Framework: Issues and Findings

Cited by 39 publications

References 7 publications

Development and Evaluation of Automotive Speech Interfaces: Useful Information from the Human Factors and the Related Literature

Development and Evaluation of Automotive Speech Interfaces: Useful Information from the Human Factors and the Related Literature

A Dialogue-Management Evaluation Study

Using finite state machines for evaluating spoken dialog systems

Contact Info

Product

Resources

About