A Hierarchical Distributed Processing Framework for Big Image Data

Abstract

This paper introduces an effective processing framework nominated ICP (Image Cloud Processing) to powerfully cope with the data explosion in image processing field. While most previous researches focus on optimizing the image processing algorithms to gain higher efficiency, our work dedicates to providing a general framework for those image processing algorithms, which can be implemented in parallel so as to achieve a boost in time efficiency without compromising the results performance along with the increasing image scale. The proposed ICP framework consists of two mechanisms, i.e. SICP (Static ICP) and DICP (Dynamic ICP). Specifically, SICP is aimed at processing the big image data pre-stored in the distributed system, while DICP is proposed for dynamic input. To accomplish SICP, two novel data representations named P-Image and Big-Image are designed to cooperate with MapReduce to achieve more optimized configuration and higher efficiency. DICP is implemented through a parallel processing procedure working with the traditional processing mechanism of the distributed system. Representative results of comprehensive experiments on the challenging ImageNet dataset are selected to validate the capacity of our proposed ICP framework over the traditional state-of-the-art methods, both in time efficiency and quality of results.

A Spark image processing toolkit

“…The last one, known as image cloud processing (ICP) 24 has also been implemented in Hadoop. Authors propose a static distributed data representation called static ICP (SICP) for processing two algorithms used in computer vision, namely the Harris algorithm and the scale-invariant feature transform (SIFT).…”

“…Also, we use the computational cluster 1, listed in Table 3. For each of the above-mentioned algorithms, the execution time curve is measured in both tools (five times for averaging), making an incremental variation of cores (1,3,6,12,24,48, and 96 cores).…”

“…Once SIPT has been compared with Thunder, the second experiment consists of applying the 16 algorithms shown in Table 1 to the MIRFLICKR-25000 image set sequentially by spark-submitting the experiments with a single core. Performance comparisons are carried out by adjusting the number of cores (3,6,12,24,48, and 96 cores of cluster 1 in Table 3) and every iteration has also been performed five times in SIPT to average the resulting computation times. To9 make fair execution comparisons, the stages ImportImages ( Figure 1A) and ExportImages ( Figure 1C) are not included during this experiment due to the large computation times they imply, for which we highly recommend that these stages should be performed off-line.…”

Multigranulation Supertrust Model for Attribute Reduction

“…These 'five V's' are the key features defining the essence of big data [1] [2]. Big data has attracted much attention from a variety of circles of scientific research, marketing, business management, and government decision making, leading to an upsurge of research [3][4][5] [6][7] [8]. Although a large candidate set of attributes is provided in big data problems, most may be redundant or irrelevant, which highly diminishes the learning performance of decision-making algorithms.…”

Large-scale distributed L-BFGS

“…Many applications require solving optimization problems with a large number of parameters. Problems of this scale are very common in the Big Data era [5][6][7]. Therefore, it is important to study the problem of large-scale optimizations on distributed systems.…”