Testing the Waters: Design of Replicable Experiments for Performance Assessment of Marine Robotic Platforms

An Euclidean norm based criterion to assess robots’ 2D path-following performance

“…The main aim consists to test a new software framework named DeepRuler for the automatic execution of path-following tasks, collecting experimental data and evaluating the performance of the control architecture through different performance indices defined by the authors. For details about the test campaign see [18]. Here we consider an example of execution of a sinusoidal path-following trial carried out by the Charlie USV, when the path is a sine wave with amplitude 10 m and 3 hemi-periods in 100 m (see Figures 1 and 3).…”

Adaptive Experimental Design for Path-following Performance Assessment of Unmanned Vehicles

“…Our group has addressed the definition of GEMs for path-following tasks from an engineering perspective in [13,14,15,16]. During path-following the vehicle has to follow a predefined target path without any time constraints.…”

“…In [9,10,1,2,3,4,5] the need of GEMs in robotics is advocated. In marine robotics the practical possibility of guaranteeing the desirable reproducibility of results is limited by logistic and environmental constraints [5,15]. The experiments are greatly influenced by environmental constraints, such as waves, wind, sea currents, and by other external disturbances, such as other vehicles, that unavoidably limit the degree of reproducibility of experiments at sea.…”

“…The experiments are greatly influenced by environmental constraints, such as waves, wind, sea currents, and by other external disturbances, such as other vehicles, that unavoidably limit the degree of reproducibility of experiments at sea. Various research groups in Italy [10,15], Croatia [23] and Spain [24] are facing this issue, encouraging the development of protocols and procedures for repeatable experiments, and for sharing the collected experimental data with the larger community of involved researchers, engineers and users. The lack of standardized experimental procedures causes difficulty even in achieving tasks apparently simple, like driving a marine robot in a predefined position [15].…”

“…Various research groups in Italy [10,15], Croatia [23] and Spain [24] are facing this issue, encouraging the development of protocols and procedures for repeatable experiments, and for sharing the collected experimental data with the larger community of involved researchers, engineers and users. The lack of standardized experimental procedures causes difficulty even in achieving tasks apparently simple, like driving a marine robot in a predefined position [15]. In marine robotics replicability refers to all controllable parameters of the experiments and not necessarily to their outcomes.…”