Toward a new approach for sorting extremely large data files in the big data era

Shatnawi, Anas; Alzahouri, Yathrip; Shehab, Mohammed A.; Jararweh, Yaser; Al‐Ayyoub, Mahmoud

doi:10.1007/s10586-018-2860-1

Cited by 6 publications

(3 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This algorithm has not been suggested and tested for large data sets and other popular sorting algorithms like Merge and Quick. In [42], it is shown that using multiple passes over the data set, has resulted in a significant improvement in the number of swaps and reducing the sorting time. The results show the superiority of the proposed technique for CPU-only and hybrid CPU-GPU implementations.…”

Section: Related Work and Why A New Framework Is Requiredmentioning

confidence: 99%

A General Framework for Sorting Large Data Sets Using Independent Subarrays of Approximately Equal Length

Moghaddam

2022

IEEE Access

View full text Add to dashboard Cite

Designing an efficient data sorting algorithm that requires less time and space complexity is essential for computer science, different engineering disciplines, data mining systems, wireless networks, and the Internet of things. This paper proposes a general low-complex data sorting framework that distinguishes the sorted or similar data, makes independent subarrays approximately in equal length, and sorts the subarrays' data using one of the popular comparison-based sorting algorithms. Two frameworks, one for serial realization and another for parallel realization, are proposed. The time complexity analyses of the proposed framework demonstrate an improvement compared to the conventional Merge and Quick sorting algorithms. Following complexity analysis, the simulation results indicate slight improvements in the elapsed time and the number of swaps of the proposed serial Merge-based and Quick-based frameworks compared to the conventional ones for low/high variance integer/non-integer data sets, in different data sizes and the number of divisions. It is about (1−1.6%) to (3.5−4%) and (0.3−1.8%) to (2−4%) improvements in the elapsed times for 1, 2, 3, and 4 divisions, respectively for small and very large data sets in Mergebased and Quick-based scenarios. Although these improvements in serial realization are minor, making independent low-variance subarrays allows the sorted components to be extracted sequentially and gradually before the end of the sorting process. Also, it proposes a general framework for parallelizing conventional sorting algorithms using non-connected (independent) or connected (dependent) multi-core structures. As the second experiment, the numerical analyses that compare the results of the parallel realization of the proposed framework to the serial one in 1, 2, 3, and 4 divisions, show a speedup factor of (2 − 4) for small to (2 − 16) for very large data sets. The third experiment shows the effectiveness of the proposed parallel framework to the parallel sorting based on the random-access machine model. Finally, we prove that the mean-based pivot is as efficient as the median-based and much better than the random pivot for making subarrays of approximately equal length.

show abstract

Section: Related Work and Why A New Framework Is Requiredmentioning

confidence: 99%

A General Framework for Sorting Large Data Sets Using Independent Subarrays of Approximately Equal Length

Moghaddam

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Another problem related to data, in particular large data sets, is their processing [40], including sorting [41][42][43], or processing to get random values from the collected data [44]. The proposed method of data storage allows for further processing, even though they are hidden to the human eye.…”

Section: Filementioning

confidence: 99%

A New Way to Store Simple Text Files

Lawnik¹,

Pełka²,

Kapczyński³

2020

Algorithms

View full text Add to dashboard Cite

In the era of ubiquitous digitization, the Internet of Things (IoT), information plays a vital role. All types of data are collected, and some of this data are stored as text files. An important aspect—regardless of the type of data—is related to file storage, especially the amount of disk space that is required. The less space is used on storing data sets, the lower is the cost of this service. Another important aspect of storing data warehouses in the form of files is the cost of data transmission needed for file transfer and its processing. Moreover, the data that are stored should be minimally protected against access and reading by other entities. The aspects mentioned above are particularly important for large data sets like Big Data. Considering the above criteria, i.e., minimizing storage space, data transfer, ensuring minimum security, the main goal of the article was to show the new way of storing text files. This article presents a method that converts data from text files like txt, json, html, py to images (image files) in png format. Taking into account such criteria as the output size of the file, the results obtained for the test files confirm that presented method enables to reduce the need for disk space, as well as to hide data in an image file. The described method can be used for texts saved in extended ASCII and UTF-8 coding.

show abstract

“…At the end, the top‐rated venues are recommended to the given user. Despite its simplicity, the CF‐based recommendation suffers from various issues, such as cold start, 8,9 data sparseness, 10,11 and scalability 12-14 . In our proposed model, we have handled the problem of data sparseness by introducing a preprocessing phase.…”

Section: Introductionmentioning

confidence: 99%

A scalable model for real‐time venue recommendations using MapReduce

Rehman

Khalid

Haq

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

The popular social networks such as Facebook, Twitter, and Foursquare closely monitor user activities to recommend different services and events. Among others, venue recommendation proposes users the most appropriates venues based on the user preferences. It offers facility to the user to mark the check-ins when a venue is visited. The traditional venue recommendation systems have opted collaborative filtering to propose recommendations. However, collaborative filtering overlooked certain critical issues, including real-time recommendations, cold start, and scalability, for venue recommendations. Moreover, real-time physical factors such as distance from the venue are also not considered in traditional venue recommendation systems. Furthermore, parsing and processing of huge volume of unstructured data is the main challenge for conventional recommender systems, particularly when dealing with real-time recommendations. For efficient scaling, significant computational and storage resources for recommendation systems are desired. This article proposes a Real-Time Venue Recommendation (RTVR) model that utilizes cloud-based MapReduce framework to process, compare, mine, and manage large data sets for generating recommendations. The results showed that the proposed model has improved accuracy for real-time recommendations. The proposed RTVR is more scalable as it exploits a cloud-based architecture. Moreover, the proposed techniques are verified using formal verification methods.

show abstract

Toward a new approach for sorting extremely large data files in the big data era

Cited by 6 publications

References 28 publications

A General Framework for Sorting Large Data Sets Using Independent Subarrays of Approximately Equal Length

A General Framework for Sorting Large Data Sets Using Independent Subarrays of Approximately Equal Length

A New Way to Store Simple Text Files

A scalable model for real‐time venue recommendations using MapReduce

Contact Info

Product

Resources

About