In tolerancing activities, focusing on geometrical tolerances allocation, many critical issues originate from a non-optimal responsibilities assignment among the involved organization units. This paper aims to depict relations between different tolerancing activities, and relevant specifications, assigning them to the proper actor, therefore expanding the ISO 8015:2011 “responsibility principle”. A classification among tolerancing activities, specifications, and media is proposed; a horizontal hierarchical framework, among functional, manufacturing, and verification specifications and a vertical hierarchical framework, along the supply chain, are discussed. Examples of both hierarchical structures are presented.