We examine the current state of the art and the related research on the automated detection and tracking of small objects—or persons—in the context of a person-overboard (POB) scenario and present the associated governing relationship between different technologies, platforms, and approaches as a system of systems. A novel phase model, structuring a POB scenario, comprises three phases: (1) detection, (2) search and track, and (3) rescue. Within these phases, we identify the central areas of responsibility and describe in detail the phases (1) and (2). We emphasize the importance of a high-level representation of different systems and their interactions to comprehensively represent the complexity and dynamics of POB scenarios. Our systematic classification and detailed description of the technologies and methods used provide valuable insights to support future regulatory and research activities. Our primary aim is to advance the development of corresponding technologies and standards.