Try to Start It! The Challenge of Reusing Code in Robotics Research

Abstract

This paper reviews the source code published with the papers of a flagship robotics research conference, 2017 International Conference on Robotics and Automation (ICRA). The aim is to investigate whether the code is actually useful, i.e. can be reused by an interested reader without much effort. The interest is twofold: for one side, it makes possible to replicate and validate the results of the research; for another side, it facilitates new progress on the field, since researchers can build new systems on top of existing work. Unfortunately, reusing code is not as straightforward as it could seem, and there is a need for tools that alleviate the effort for integrating someone else's code into the own user's system. We propose the use of Docker, a Linux container technology, to turn the source code repositories into executable images, that can be run and tested locally, in an isolated environment, without the need of a costly integration with the host system.

A Method for Evaluating and Selecting Suitable Hardware for Deployment of Embedded System on UAVs

“…The network setup is run over a dedicated router within 1 m in line-of-sight, operating in the 5 to 2.4 GHz range to minimize network effects. The implementation details to enable reproduction [ 12 ] are detailed in Appendix A .…”

“…Solutions to computer vision problems, such as object detection [ 9 ], and robotics problems, such as localization [ 10 ], have been developed and disseminated at a rapid pace, with promising results on benchmarks [ 11 ]. Despite experiments on real robots being the common mode of evaluation for robotics applications [ 11 ], reproducing results requires a considerable effort [ 12 ]. Furthermore, computational performance has traditionally been considered a secondary factor [ 10 ], assuming that hardware progress will eliminate this issue.…”

“…Software development is an intricate process, especially for robotic systems, which are commonly evaluated in experiments and therefore require implementation on realistic hardware [ 11 ]. Standardized frameworks such as the Robot Operating System ( ROS ) are universal and alleviate issues with interoperability, however, reproducing results remains an issue [ 12 ]. Pinpointing bottlenecks when implementing a software on a hardware system is a difficult task and requires detailed profiling [ 34 ].…”

“…However, improvements implied by these models are gained through hardware-near programming techniques requiring profound knowledge of memory and CPU access. This is difficult to achieve for programmers working in a meta-environment such as ROS, which employs a complex variety of different tools in a software stack [ 12 , 37 ]. Models for evaluating the performance of GPUs in the context of contemporary deep learning approaches inspected the timeliness and performance [ 38 ] and the potential to be used with common embedded platforms [ 26 ].…”

“…The remaining criteria of the ISO are Functional Suitability , Usability , Security , Maintainability , and Portability . ROS as a standardized framework that handles Maintainability and Portability [ 12 ]. Functional Suitability , Usability , and Security are equal across all configurations for the evaluated software and are therefore not considered in this work.…”