tThis report describes the development and evaluation of prototype hands-on equipment, Job sample performance tests for a high skilled technical (fjOsT An electronic maintenance MOS (26C20) was used as the research vehicle. 'Hie results lead to the conclusion chat valid and reliable performance tests could be constructed, but that equipment, facilities, and standardization requirements reduce the feasibility of their use at other than ideal location, such as a U.S. Army school.
DD FORM I JAN 73 1473 EDITION OF '' ROV V SS IS OBSOLETE Unclassified
SECURITY CLASSIFICATION OP THIS PAOE fWiao Data Knlara«;/^zr o//i y SL
SUMMARY AND CONCLUSIONS
Military ProblemThe US Army has placed emphasis recently on performance-based training and evaluation. Performance-based evaluation, using job sample hands-on equipment, has not been developed or used for complex technical MOSs. Therefore, there is a need to determine the feasibility of developing and using Job sample hands-on equipment performance tests for a complex technical MOS.
Research ObjectiveThe research project was to determine whether it was feasible to develop and use hands-on, job sample performance tests for assessing job performance of highly skilled electronic maintenance technicians. Feasibility was defined in psychometric and administrative terms.
MethodAn electronics maintenance MOS was selected as the research vehicle. An analysis of tfhe Job tasks was undertaken, and performance teste Wtere developed to categorize job activities; further, performance tests were experimentally evaluated for selected job activities. The tests were administered to technicians with a broad range of experience. Two scoring approaches were usetf -a GO/NO-GO product measure and a process measure where task procedures were evaluated. In addition, time-to-perform the test was obtained for each administration of the test.
ResultsAn approach to selecting job tasks to be used in the performance tests was taken that emphasized content validity. In addition, empirical validation using a mastery classification approach was used. Master/non-masfer categories were defined using job experience, MOS test scores and job performance rating criteria. Empirical validity was found for two of the three tests using MOS score and for one test using job performance ratings.An interrater reliability of .73 was obtained for the process measures. There was 100% agreement between raters when the GO/NO-GO scoring procedure was used.•• -A time limit of 60 minutes was set for each test. Of those who successfully completed the tests in that time limit, no significant relationships were found for the time scores and MOS scores, Job experience and Job performance ratings.
ConclusionsValid and reliable performances can be developed for assessing electronic maintenance skill proficiency. However, equipment, facilities and administration standardization requirements may reduce the feasibility of using full hands-on. Job sample performance tests at other than an ideal location, euch as an electronic maintenance sc...
