Craquelurenet: Matching The Crack Structure In Historical Paintings For Multi-Modal Image Registration

Abstract

Visual light photography, infrared reflectography, ultraviolet fluorescence photography and x-radiography reveal even hidden compositional layers in paintings. To investigate the connections between these images, a multi-modal registration is essential. Due to varying image resolutions, modality dependent image content and depiction styles, registration poses a challenge. Historical paintings usually show crack structures called craquelure in the paint. Since craquelure is visible by all modalities, we extract craquelure features for our multi-modal registration method using a convolutional neural network. We jointly train our keypoint detector and descriptor using multi-task learning. We created a multi-modal dataset of historical paintings with keypoint pair annotations and class labels for craquelure detection and matching. Our method demonstrates the best registration performance on the multi-modal dataset in comparison to competing methods.

Multi-modal Retinal Image Registration Using a Keypoint-Based Vessel Structure Aligning Network

“…We employ and extend the fully-convolutional RetinaCraquelureNet [20,19] for our end-to-end pipeline (see Fig. 1).…”

“…We set the feature dimension of the description head to 256-D to reduce the parameters for end-to-end learning. We pretrain the network from scratch using multi-modal retinal image patches centered at supervised keypoint positions with a binary cross-entropy loss for keypoint detection and a cross-modal bidirectional quadruplet descriptor loss [20,19]. Then, we fit the network into our pipeline.…”

“…With these modifications the predictions of the detector are scaled to the range zero to one. Then, we apply non-maximum suppression (NMS) to the keypoint confidence heatmap and extract the top N max keypoints from the NMS heatmap [20,19]. This step is non-differentiable, therefore we apply a differentiable subpixel keypoint refinement (DKR) that allows the gradients to flow back to the small regions around the keypoints.…”

“…The refined keypoint positions are the sum of the initial coordinates and the soft subpixel coordinates. The corresponding descriptors are linearly interpolated at the refined keypoint coordinates [20].…”

“…Based on the idea of the bidirectional quadruplet descriptor loss (L Desc ) [20], we design a self-supervised keypoint and descriptor loss (L KD ) that is guided by ground truth homographies instead of labeled matching keypoint pairs as in [20]. Within the detected keypoints in both images, positive keypoint pairs are automatically determined based on mutual nearest neighbor matching of the keypoint coordinates whose reprojection error is smaller than a threshold τ .…”