A survey of big data and machine learning

Big data is a large set of structured or unstructured data that can collect, store, manage, and analyze data with existing database management tools. And it means the technique of extracting value from these data and interpreting the results. Big data has three characteristics: The size of existing data and other data (volume), the speed of data generation (velocity), and the variety of information forms (variety). The time series data are obtained by collecting and recording the data generated in accordance with the flow of time. If the analysis of these time series data, found the characteristics of the data implies that feature helps to understand and analyze time series data. The concept of distance is the simplest and the most obvious in dealing with the similarities between objects. The commonly used and widely known method for measuring distance is the Euclidean distance. This study is the result of analyzing the similarity of stock price flow using 793,800 closing prices of 1,323 companies in Korea. Visual studio and Excel presented calculate the Euclidean distance using an analysis tool. We selected “000100” as a target domestic company and prepared for big data analysis. As a result of the analysis, the shortest Euclidean distance is the code “143860” company, and the calculated value is “11.147”. Therefore, based on the results of the analysis, the limitations of the study and theoretical implications are suggested.

Section: Resultsmentioning

confidence: 99%

“…First, normalization work on daily prices should be performed. If you do not do the preceding work, you may not find meaningful results [25][26][27]. After the normalization data preprocessing process, we performed full scale data analysis.…”

Section: Resultsmentioning

confidence: 99%

A forecasting of stock trading price using time series information based on big data

Sootai

Jin

Shin

2021

“…Undersampling is selecting parts of a lot of data, such as the image on the left, and aligning them toward the smaller data. Undersampling can reduce execution time by reducing the size of the data set, but it is also possible that the useful data is not extracted or biased to one side [14][15][16][17]. Oversampling involves copying less data and fitting it toward more data.…”

Section: Data Imbalancementioning

confidence: 99%

Fruit tree disease classification system using generative adversarial networks

Kim

Lee

Jung

2021

Smart farm refers to a farm that can remotely and automatically maintain proper growth and management of crops and livestock by integrating technology with agriculture. Currently, smart farms are concentrated in the field of smart horticulture, and although spreading research is being conducted in limited spaces. In addition, it is difficult to obtain a sufficient amount of data to be used for learning, and there is a problem that data imbalance occurs because it is difficult to obtain a similar amount for each class. In this paper, we propose a method to amplify a small amount of data and to solve the problems of imbalance data by using a feature that can learn to mimic the data of a generative adversarial network. The proposed method can create dataset of various crops and also show high hit rate. Dataset generated from crops would be used to solve problems of data imbalance by learning.

“…As indicated by International data corporation (IDC), digital space is projected to increase more than 44 Z.B. in volume by 2020 [1][2][3]. In the era of digital data, big data is something that can't be overlooked.…”

Section: Introductionmentioning

confidence: 99%

A smart method for spark using neural network for big data

Rahman

Hossen

Sultana

et al. 2021

Apache spark, famously known for big data handling ability, is a distributed open-source framework that utilizes the idea of distributed memory to process big data. As the performance of the spark is mostly being affected by the spark predominant configuration parameters, it is challenging to achieve the optimal result from spark. The current practice of tuning the parameters is ineffective, as it is performed manually. Manual tuning is challenging for large space of parameters and complex interactions with and among the parameters. This paper proposes a more effective, self-tuning approach subject to a neural network called Smart method for spark using neural network for big data (SSNNB) to avoid the disadvantages of manual tuning of the parameters. The paper has selected five predominant parameters with five different sizes of data to test the approach. The proposed approach has increased the speed of around 30% compared with the default parameter configuration.