<title>Problem set guidelines to facilitate ATR research, development, and performance assessments</title>

Abstract

In November of 2000, the Deputy Under Secretary of Defense for Science and Technology Sensor Systems (DUSD (S&T/SS)) chartered the Automatic Target Recognizer Working Group (ATRWG) to develop guidelines for sanctioned Problem Sets. Such Problem Sets are intended for development and test of Automatic Target Recognition (ATR) algorithms and contain comprehensive documentation of the data in them. A Problem Set provides a consistent basis for examining ATR performance and growth. Problem Sets will, in general, serve multiple purposes. First, they will enable informed decisions by government agencies sponsoring ATR development and transition. Problem Sets standardize the testing and evaluation process, offering a consistent assessment of ATR performance. Second, they will measure and guide ATR development progress within this standardized framework. Finally, they quantify the state-of-the-art for the community. Problem Sets provide clearly defined operating condition coverage. This encourages ATR developers to consider these critical challenges and allows evaluators to assess over them. The widely distributed development and self-test portions, along with a documented disciplined methodology, permit ATR developers to address critical issues and describe their accomplishments, while the sequestered portion permits government assessment of state-of-the-art and of transition readiness. This paper discusses the elements of an ATR Problem Set as a package of data and information that presents a standardized ATR challenge relevant to one or more scenarios. The package includes training and test data containing targets and clutter, truth information, required experiments, and a standardized analytical methodology to assess performance. Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/16/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx Proc. of SPIE Vol. 4726 313 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 06/16/2016 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

Development of scale model imagery for ATR investigations

“…Image sets which could be used by universities, research laboratories, and international partners would facilitate the exploration of new ATR methods. The goal of this effort was to produce a large data set for ATR development and evaluation, conforming to the established standards for an ATR problem set [1]. Important elements of the process are to maintain high level of control over the imaging conditions and provide precise, accurate image truth along with the image set.…”

Review of current aided/automatic target acquisition technology for military target acquisition tasks

“…This is the ATR Working Group. 25 Sanctioned problem sets permit the establishment of universal metrics to assess algorithm performance collectively and scientifically. The same problem set can be used to test any number of candidate algorithms and permits quantifiable difference measurements across all candidates.…”