A Scalable Framework to Analyze Data from Heterogeneous Sources at Different Levels of Granularity

Abstract

There is an enormous amount of data present in many different formats, including databases (MsSql, MySQL, etc.), data repositories (.txt, html, pdf, etc.), and MongoDB (NoSQL, etc.). The processing, storing, and management of the data are complicated by the varied locations in which the data is stored. If combined, this data from several sites can yield a lot of important information. Since many researchers have suggested different methods to extract, examine, and integrate the data. To manage heterogeneous data, researchers propose data warehouse and big data as solutions. However, when it comes to handling a variety of data, each of these methods have limitations. It is necessary to comprehend and use this information, as well as to evaluate the massive quantities that are increasing day by day. We propose a solution that facilitates data extraction from a variety of sources. It involves two steps: first, it extracts the pertinent data, and second, then to identify the machine learning algorithm to analyze the data. This paper proposes a system for retrieving data from many sources, such as databases, data sources, and NoSQL. Later, the framework was put to the test on a variety of datasets to extract and integrate data from diverse sources, and it was found that the integrated dataset performed better than the individual datasets in terms of accuracy, management, storage, and other factors. Thus, our prototype scales and functions effectively as the number of heterogeneous data sources increases.

An Adaptive Classification Framework for Handling the Cold Start Problem in Case of News Items

“…In step 1 of the experiment, it was discovered that business news was the most shared of the categories, and this was also true when we looked at news popularity based on location and news shares. So, based on our dataset and on a certain day, we may infer in step 1 that business news was shared the most among the other categories [48][49][50][51]. New users and items are added to every recommender system.…”

Big data-driven risk decision-making and safety management in agricultural supply chains

“…In the swiftly transforming digital age, big data technology has been recognized as a crucial catalyst propelling advancements across various sectors (Awad et al, 2023;Chandrasekaran et al, 2021;Chennouk et al, 2022;Hasan et al, 2022;He et al, 2022;Kusrini et al, 2022;Lazarevska et al, 2022;Li and Gao, 2022;Zhang et al, 2023;Zhao et al, 2022). The agricultural industry, quintessential for sustaining human life, relies heavily on the efficiency and security of its supply chain.…”