A new Apache Spark-based framework for big data streaming forecasting in IoT networks

Fernández-Gómez, Antonio M.; Gutiérrez‐Avilés, David; Troncoso, Alicia; Martínez‐Álvarez, Francisco

doi:10.1007/s11227-023-05100-x

Cited by 12 publications

(4 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Effects and Achievements: Through the implementation of the intelligent traffic management system, the urban traffic situation has been significantly improved, with reductions in traffic congestion, accident rates, and air pollution levels. The smart traffic management system provides urban traffic management departments with more scientific and precise data support, helping them better plan urban traffic networks, optimize traffic flow, and improve the efficiency of urban traffic operations and residents' travel experience [11].…”

Section: Case Study: Intelligent Traffic Management In Smart Citiesmentioning

confidence: 99%

The Characteristics of Cloud Computing in the Internet of Things and the Application of Key Technologies

2024

acss

View full text Add to dashboard Cite

This paper explores the characteristics and key technologies of cloud computing in the Internet of Things (IoT) application. IoT, as an emerging technology, is rapidly evolving and profoundly impacting human life and work. Cloud computing, as one of the fundamental technologies supporting IoT development, provides robust computing and storage support. Firstly, we analyze the characteristics of cloud computing in IoT, including its highly flexible resource scheduling capability, scalability, reliability, and challenges such as security and privacy protection. Next, we introduce key technologies in IoT and cloud computing, including data collection and transmission, storage and computation, edge computing, and virtualization technology. In terms of key technology applications, we delve into encryption algorithms for data security and privacy protection, machine learning and data mining algorithms for big data analysis and mining, and task allocation and scheduling algorithms for edge computing and collaborative processing. Through case studies, we demonstrate the practical application of these key technologies in areas such as smart homes, smart cities, and industrial IoT, and provide insights into the future development trends of cloud computing in IoT, emphasizing the importance of security, intelligence, and sustainable development, to further promote the development of IoT technology.

show abstract

Section: Case Study: Intelligent Traffic Management In Smart Citiesmentioning

confidence: 99%

The Characteristics of Cloud Computing in the Internet of Things and the Application of Key Technologies

2024

acss

View full text Add to dashboard Cite

show abstract

“…This allows Spark to be used in a wide variety of use cases ranging from streaming ones, such as finances, telecommunications, and IoT, to batch data ones like ML model training, data analysis, etc. [27][28][29].…”

Section: Benefits Of Apache Sparkmentioning

confidence: 99%

Building Advanced Web Applications Using Data Ingestion and Data Processing Tools

Šprem,

Tomažin,

Matečić

et al. 2024

Electronics

View full text Add to dashboard Cite

Today, advanced websites serve as robust data repositories that constantly collect various user-centered information and prepare it for subsequent processing. The data collected can include a wide range of important information from email addresses, usernames, and passwords to demographic information such as age, gender, and geographic location. User behavior metrics are also collected, including browsing history, click patterns, and time spent on pages, as well as different preferences like product selection, language preferences, and individual settings. Interactions, device information, transaction history, authentication data, communication logs, and various analytics and metrics contribute to the comprehensive range of user-centric information collected by websites. A method to systematically ingest and transfer such differently structured information to a central message broker is thoroughly described. In this context, a novel tool—Dataphos Publisher—for the creation of ready-to-digest data packages is presented. Data acquired from the message broker are employed for data quality analysis, storage, conversion, and downstream processing. A brief overview of the commonly used and freely available tools for data ingestion and processing is also provided.

show abstract

“…Analyzing data acquired in a continuous flow is a major challenge for various fields; being able to analyze a large volume of data stream from various sources, such as sensors (like IoT), networks, and the internet, is essential for improving the efficiency of processes [13]. In the context of a multi-dimensional data analysis, where data come from various channels while resource constraints limit our observations, the critical challenge is channel selection, particularly in scenarios with high dimensionality.…”

Section: Introductionmentioning

confidence: 99%

Greedy control group selection for multi-explanatory multi-output regression problem

Szűcs,

Németh,

Kiss

2024

Preprint

View full text Add to dashboard Cite

The problem of multi-output learning involves the simultaneous prediction of multiple outputs based on given inputs. This paper focuses on addressing this challenge, assuming that we can only monitor a subset of variables. This resource constraint led to a definition of a new kind of problem, that we call Multi-Explanatory Multi-Output Regression (MEMOR) task. The goal of MEMOR is to select explanatory variables that minimize the prediction error for target variables. The central question pertains to the optimal choice of a given number of variables to maximize the goodness of the regression. We propose two greedy approaches for identifying good explanatory variables, along with a linear approximation as a baseline. To evaluate the performance of the proposed algorithms, we compared the resulting explanatory variables with the optimal set obtained through an exhaustive search. Our greedy algorithms surpass the linear method with better regression results, while they are faster than the exhausted method. Both the MEMOR problem and the methods developed for it are well-suited for multi-dimensional data analysis with resource constraints.

show abstract

A new Apache Spark-based framework for big data streaming forecasting in IoT networks

Cited by 12 publications

References 57 publications

The Characteristics of Cloud Computing in the Internet of Things and the Application of Key Technologies

The Characteristics of Cloud Computing in the Internet of Things and the Application of Key Technologies

Building Advanced Web Applications Using Data Ingestion and Data Processing Tools

Greedy control group selection for multi-explanatory multi-output regression problem

Contact Info

Product

Resources

About