Image-Based TF Colorization With CNN for Direct Volume Rendering

Abstract

In the direct volume rendering (DVR), it often takes a long time for a novice to manipulate the transfer function (TF) and analyze the volume data. To lessen the difficulty in volume rendering, several researchers have developed deep learning techniques. However, the existing techniques are not easy to apply directly to existing DVR pipelines. In this study, we propose an image-based TF colorization with CNN to automatically generate a direct volume rendering image (DVRI) similar to a target image. Our system includes CNN model training, TF labeling, image-based TF generation, and volume rendering by matching the target image. We introduce a technique for training CNN and labeling the TF with images similar to the input volume dataset. Moreover, we extract the primary colors from the target image according to the labels classified with the CNN model. We render the volume data with the TF to produce the DVRI reproducing the prominent colors in the target image.

Colorizing Multi-Modal Medical Data: An Autoencoder-based Approach for Enhanced Anatomical Information in X-ray Images

“…However, a larger experiment is needed since the experiment was only evaluated using a few datasets. [17]. Shi, Q (et al) describe a VGG-16 Convolutional network model that uses a fully autonomous learning-based colorisation technique based on cross-entropy loss in classification generating plausible colour images while mainly focusing on reducing image quality loss [18].…”

Differentiable transfer function optimization and local color transfer for volume rendering

“…Liao et al [12] demonstrated that similar data attributes of images can be transferred through dense correspondence. Kim et al [13] proposed training a convolutional neural network to determine the transfer function grid for labeling and extract the primary color from the target image. He et al [14] proposed learning colorings using an end-to-end neural network, eliminating the need for human-made coloring rules.…”