The reassembling of severely damaged tangible heritage is a primordial task for archaeologists who not only aim to further study the past but also to preserve ruined ancient monuments. As a consequence, various researchers have proposed methods to automatically solve this problem by computing and matching geometric properties of counterpart fragments. Although their results are quite promising, experts still carry out this task manually by finding relationships between distinctive matching cues, such as type of decoration, remaining traces, inscriptions’ content, etc. The topic itself poses challenges to both automatic and manual approaches due to the high level of damage ancient broken fragments have undergone over the centuries. Therefore, this paper proposes a Puzzling Engine that combines crucial elements of automatic and manual methodologies to empower experts with registration tools for reassembling fragmented heritage. Unlike similar hybrid human-computer puzzling engines, our approach is capable of automatically proposing matches and rough alignments solely based on the geometry of fractured surfaces. Based on these initial solutions and a set of registration tools, experts can accurately solve the puzzle. The virtual environment has been used and verified to find pairwise puzzle-pieces of actual antique wall decorated fragments, resulting in new discoveries that experts could not have come up with by utilizing classic techniques. Concretely, the contributions are twofold, (i) a feature-based registration pipeline that is able to suggest both matches and alignments to the user and (ii) a virtual interface that integrates automatic and user-assisted techniques to accurately puzzle fragmented surfaces.

An increased interest in computer-aided heritage reconstruction has emerged in recent years due to the maturity of sophisticated computer vision techniques. Concretely, feature-based matching methods have been conducted to reassemble heritage assets, yielding plausible results for data that contains enough salient points for matching. However, they fail to register ancient artifacts that have been badly deteriorated over the years. In particular, for monochromatic incomplete data, such as 3D sunk relief eroded decorations, damaged drawings, and ancient inscriptions. The main issue lies in the lack of regions of interest and poor quality of the data, which prevent feature-based algorithms from estimating distinctive descriptors. This paper addresses the reassembly of damaged decorations by deploying a Generative Adversarial Network (GAN) to predict the continuing decoration traces of broken heritage fragments. By extending the texture information of broken counterpart fragments, it is demonstrated that registration methods are now able to find mutual characteristics that allow for accurate optimal rigid transformation estimation for fragments alignment. This work steps away from feature-based approaches, hence employing Mutual Information (MI) as a similarity metric to estimate an alignment transformation. Moreover, high-resolution geometry and imagery are combined to cope with the fragility and severe damage of heritage fragments. Therefore, the testing data is composed of a set of ancient Egyptian decorated broken fragments recorded through 3D remote sensing techniques. More specifically, structured light technology for mesh models creation, as well as orthophotos, upon which digital drawings are created. Even though this study is restricted to Egyptian artifacts, the workflow can be applied to reconstruct different types of decoration patterns in the cultural heritage domain.