Socioeconomic indicators are essential to help design and monitor the impact of public policies on society. Such indicators are usually obtained through census data collected at 10-year intervals, which are not only temporally coarse but expensive. Over recent years other ways of collecting data and producing these indicators have been explored, in particular using the new surveillance capabilities that remote observations can provide. The objective of this paper is to evaluate the assessment of socioeconomic indicators using street-view imagery, through a case study conducted in a region of Brazil, the Vale do Ribeira, one of the poorest semi-rural regions in Brazil. In this study we used socioeconomic indicators collected by the Brazilian Institute of Geography and Statistics (IBGE) and used Google Street View (GSV) images as our source of remote observations. A pre-trained convolutional neural network (CNN) was used to predict socio-economic indicators from GSV. To evaluate the performance of the classifier, we performed five-fold cross-validation between the predicted indicator and its true value. The best performance was obtained for the highest income class, with 80% of correct prediction. We conclude that the method has the potential to predict socioeconomic indicators across a large area with social challenges such as Vale do Ribeira, and that the network model is general enough to be used even when the imagery dataset is from semi-rural areas. This demonstrates the applicability of GSV datasets for similar settings and perhaps ensuring their replicability, which is a scientific requirement that requires further experimentation/evaluation.

The challenges of Reproducibility and Replicability (R & R) in computer science experiments have become a focus of attention in the last decade, as efforts to adhere to good research practices have increased. However, experiments using Deep Learning (DL) remain difficult to reproduce due to the complexity of the techniques used. Challenges such as estimating poverty indicators (e.g., wealth index levels) from remote sensing imagery, requiring the use of huge volumes of data across different geographic locations, would be impossible without the use of DL technology. To test the reproducibility of DL experiments, we report a review of the reproducibility of three DL experiments which analyze visual indicators from satellite and street imagery. For each experiment, we identify the challenges found in the data sets, methods and workflows used. As a result of this assessment we propose a checklist incorporating relevant FAIR principles to screen an experiment for its reproducibility. Based on the lessons learned from this study, we recommend a set of actions aimed to improve the reproducibility of such experiments and reduce the likelihood of wasted effort. We believe that the target audience is broad, from researchers seeking to reproduce an experiment, authors reporting an experiment, or reviewers seeking to assess the work of others.

No abstract