This work describes in detail the definition of a procedure for calibrating and evaluating coordinate measuring arms (AACMMs or CMAs). CMAs are portable coordinate measuring machines that have been widely accepted in industry despite their sensitivity to the skill and experience of the operator in charge of the inspection task. The procedure proposed here is based on the use of a dimensional gauge that incorporates multiple geometric features, specifically designed for evaluating the measuring technique when CMAs are used, at company facilities (workshops or laboratories) and by the usual operators who handle these devices in their daily work. After establishing the procedure and manufacturing the feature-based gauge, the research project was complemented with diverse in situ field tests performed with the collaboration of companies that use these devices in their inspection tasks. Some of the results are presented here, not only comparing different operators but also comparing different companies. The knowledge extracted from these experiments has allowed the procedure to be validated, the defects of the methodologies currently used for in situ inspections to be detected, and substantial improvements for increasing the reliability of these portable instruments to be proposed.