A distributed computing framework for All-to-All comparison problems

Zhang, Yifan; Tian, Yu-Chu; Kelly, Wayne; Fidge, Colin

doi:10.1109/iecon.2014.7048857

Cited by 6 publications

(2 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The effect of distributed computing depends heavily on the strategies of data distribution, task decomposition and task scheduling. The computing performance may not be desirable if the comparison task faces the following problems: the data are distributed irrationally, the data are not highly localized, and the loads are imbalanced in the distributed system [3].…”

Section: Introductionmentioning

confidence: 99%

Construction and Optimization of a File Distribution Model for All-to-All Comparison of Big Dataset

Li¹,

Gao²,

Wang³

et al. 2019

ISI

View full text Add to dashboard Cite

This paper probes into the all-to-all comparison of large dataset, and gives a formal mathematical description of the problem. Then, a multi-objective file distribution model was constructed based on the LP, aiming to localize the data, balance node storage and loads, minimize the storage occupation, and control the occupied storage within the storage limit of each node. To save storage space, the established model was further optimized, and the file distribution algorithm was designed for the distributed environment. Experimental results show that our model and algorithm successfully balanced the storage occupation and loads between computing nodes, and minimized the occupation of node storage.

show abstract

Section: Introductionmentioning

confidence: 99%

Construction and Optimization of a File Distribution Model for All-to-All Comparison of Big Dataset

Li¹,

Gao²,

Wang³

et al. 2019

ISI

View full text Add to dashboard Cite

show abstract

“…Four file allocation experiments were carried out in the environment of matlab 2018a[24,25], with the aim to verify our model and algorithm.…”

mentioning

confidence: 99%

Optimal Data File Allocation for All-to-All Comparison in Distributed System: A Case Study on Genetic Sequence Comparison

Gao

Ren

2019

INT J COMPUT COMMUN

View full text Add to dashboard Cite

In order to solve the problem of unbalanced load of data les in large-scale data all-to-all comparison under distributed system environment, the differences of les themselves arefully considered. This paper aims to fully utilize the advantages of distributed system to enhance the le allocation of all-to-all comparison between the data les in a large dataset. For this purpose, the author formally described the all-to-all comparison problem, and con-structed a data allocation model via mixed integer linear programming (MILP). Meanwhile, a data allocation algorithm was developed on the Matlab using the intlinprog function of branch-and-bound method. Finally, our model and algorithm were veried through several experiments. The results show that the proposed le allocation strategy can achieve the basic load balance of each node in the distributed system without exceeding the storage capacity of any node, and completely localize the data le. The research ndings can be applied to such elds as bioinformatics, biometrics and data mining.

show abstract

Application of Simulated Annealing to Data Distribution for All-to-All Comparison Problems in Homogeneous Systems

Zhang

Tian

Kelly

et al. 2015

Neural Information Processing

View full text Add to dashboard Cite

This series of ICONIP conferences has been held annually since 1994 in Seoul and has become one of the leading international conferences in the areas of artificial intelligence and neural networks.ICONIP 2015 received a total of 432 submissions by scholars coming from 42 countries/regions across six continents. Based on a rigorous peer-review process where each submission was evaluated by an average of two qualified reviewers, a total of 301 high-quality papers were selected for publication in the reputable series of Lecture Notes in Computer Science (LNCS). The selected papers cover major topics of theoretical research, empirical study, and applications of neural information processing research. ICONIP 2015 also featured the Cybersecurity Data Mining Competition and Workshop (CDMC 2015), which was jointly held with ICONIP 2015. Nine papers from CDMC 2015 were selected for the conference proceedings.In addition to the contributed papers, the ICONIP 2015 technical program also featured four invited speakers, Nik Kasabov

show abstract

A distributed computing framework for All-to-All comparison problems

Cited by 6 publications

References 22 publications

Construction and Optimization of a File Distribution Model for All-to-All Comparison of Big Dataset

Construction and Optimization of a File Distribution Model for All-to-All Comparison of Big Dataset

Optimal Data File Allocation for All-to-All Comparison in Distributed System: A Case Study on Genetic Sequence Comparison

Application of Simulated Annealing to Data Distribution for All-to-All Comparison Problems in Homogeneous Systems

Contact Info

Product

Resources

About