The role of machine intelligence in photogrammetric 3D modeling – an overview and perspectives

Abstract

The process of modern photogrammetry converts images and/or LiDAR data into usable 2D/3D/4D products. The photogrammetric industry offers engineering-grade hardware and software components for various applications. While some components of the data processing pipeline work already automatically, there is still substantial manual involvement required in order to obtain reliable and high-quality results. The recent development of machine learning techniques has attracted a great attention in its potential to address complex tasks that traditionally require manual inputs. It is therefore worth revisiting the role and existing efforts of machine learning techniques in the field of photogrammetry, as well as its neighboring field computer vision. This paper provides an overview of the state-of-the-art efforts in machine learning in bringing the automated and 'intelligent' component to photogrammetry, computer vision and (to a lesser degree) to remote sensing. We will primarily cover the relevant efforts following a typical 3D photogrammetric processing pipeline: (1) data acquisition (2) georeferencing/interest point matching (3) Digital Surface Model generation (4) semantic interpretations, followed by conclusions and our insights.

Vertical geomediation: The automation and platformization of photogrammetry

“…The impacts of SfM – combining high-resolution commercial digital cameras, UASs, automating algorithms and streamlined software – have been profound for aerial photogrammetry. While it still lacks some of the accuracy of high-end systems (Fawcett et al, 2019: 302), SfM has dramatically reduced photogrammetry costs, offers increased user-friendliness, and opens it up to non-expert users (Granshaw, 2018; Qin and Gruen, 2021), thus contributing to the ‘democratization of structural data acquisition’ (Fawcett et al, 2019: 301).…”

“…In the drive towards full automation of the photogrammetry workflow, machine learning (especially using convolutional neural networks) has the potential to become ‘a game changer’ (Heipke and Rottensteiner, 2020: 10; on the importance of machine learning within geography more broadly, see Lavallin and Downs, 2021). Machine learning is identified as having clear application in several key areas: in image matching and scene triangulation, employing dense imaging mapping (where rather than search an entire image for features this process compares two overlapping images row by row – Kodde, 2016); in interpreting aerial/satellite images – a process known as semantic interpretation; and in georeferencing (adding geographic coordinates to an image) (see Qin and Gruen, 2021).…”

Oblique View Selection for Efficient and Accurate Building Reconstruction in Rural Areas Using Large-Scale UAV Images

“…Researchers in the photogrammetry community have proposed methodologies for incorporating semantic information in the photogrammetric pipeline [42][43][44][45]. Research works on improving photogrammetric tasks using semantic information have been reported.…”

Convolutional Neural Network for Background Removal in Close Range Photogrammetry: Application on Cultural Heritage Artefacts

“…In aerial and close-range photogrammetry, automation of the process is extremely important due to the large amount of data to be processed. Machine learning and deep learning are two approaches with many investigations devoted to pipeline automation [5,6]. Concerning deep learning, the relevance of the training dataset on the final robustness of the net represents the major drawback.…”

“…Concerning deep learning, the relevance of the training dataset on the final robustness of the net represents the major drawback. Although shared databases and addition of synthetic scenes for the training sets may help to enlarge the generality of these approaches, as highlighted in [7], the computation efforts may rapidly increase, shifting the bottleneck from the lack of automation to the lack of computational resources [8]. In recent years, Convolutional Neural Networks (CNN) have been increasingly used in image analysis and image segmentation, but only nowadays, they started to be used for the improvement of semantic photogrammetry [9], for increasing the level of automation in close-range applications [10], or for background removal with the aim of detecting moving objects [11].…”