Applying SCORE to field‐based performance evaluations of soldier worn sensor technologies

Abstract

Soldiers are often asked to perform missions that last many hours and are extremely stressful. After a mission is complete, the soldiers are typically asked to provide a report describing the most important things that happened during the mission. Due to the various stresses associated with military missions, there are undoubtedly many instances in which important information is missed or not reported and, therefore, not available for use when planning future missions. The ASSIST ͑Advanced Soldier Sensor Information System and Sensors Technology͒ program is addressing this challenge by instrumenting soldiers with sensors that they can wear directly on their uniforms. During the mission, the sensors continuously record what is going on around the soldier. With this information, soldiers are able to give more accurate reports without relying solely on their memory. In order for systems like this ͑often termed autonomous or intelligent systems͒ to be successful, they must be comprehensively and quantitatively evaluated to ensure that they will function appropriately and as expected in a wartime environment. The primary contribution of this paper is to introduce and define a framework and approach to performance evaluation called SCORE ͑System, Component, and Operationally Relevant Evaluation͒ and describe the results of applying it to evaluate the ASSIST technology. As the name implies, SCORE is built around the premise that, in order to get a true picture of how a system performs in the field, it must be evaluated at the component level, the system level, and in operationally relevant environments. The SCORE framework provides proven techniques to aid in the performance evaluation of many types of intelligent systems. To date, SCORE has only been applied to technologies under development ͑for-mative evaluation͒, but the authors believe that this approach would lend itself equally well to the evaluation of technologies ready to be fielded ͑summative evaluation͒.

Ontological perspectives for autonomy performance

“…Figure 4 provides an example of the types of terrain which the US&R robots might have to traverse [11,12].…”

“…The TRANSTAC project uses a System, Component, and Operationally-Relevant Evaluation (SCORE) approach to evaluate the candidate systems. We began to apply some of the key concepts of SCORE in formulating the ontology [10,11].…”

Evolution of the SCORE framework to enhance field-based performance evaluations of emerging technologies

“…SCORE takes the framework a step further by identifying specific system goals and areas of interest. It is built around the premise that, in order to get a comprehensive picture of how a system performs in its actual use-case environment, technical performance should be evaluated at the component and system levels [2]. Additionally, system level utility assessments should be performed to gain an understanding of the value the system provides to the end-users.…”

Evaluating the RoboCup 2009 Virtual Robot Rescue Competition

“…We wished to be able to determine which team had the strongest mapping, which had the best communications strategy, which had the best human-computer interface, AND which had the best overall system. To this end, we implemented the SCORE framework of evaluation [3] and evaluated three elemental tests as well as the overall USAR scenario. In addition, an effort was made to automate as much of the scoring procedure as possible.…”