Applying machine learning to big data streams : An overview of challenges

Augenstein, Christoph; Spangenberg, Norman; Franczyk, Bogdan

doi:10.1109/iscmi.2017.8279592

Cited by 12 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even traditional (non-ML) iterative or feedback systems are having challenges in terms of computation and link delays to meet the real-time requirements of dynamic services and highly mobile users [19]. Whereas, the delay in training ML models, for instance in streaming applications, will make it very difficult to match the latency requirements [20].…”

Section: B Challenges For Latency-critical Iot Systemsmentioning

confidence: 99%

The Challenges of Artificial Intelligence in Wireless Networks for the Internet of Things: Exploring Opportunities for Growth

Ahmad

Shahabuddin

Sauter

et al. 2021

EEE Ind. Electron. Mag.

View full text Add to dashboard Cite

The Internet of Things (IoT), hailed as the enabler of the next industrial revolution, will require ubiquitous connectivity, context-aware and dynamic service mobility, and extreme security through the wireless network infrastructure. Artificial Intelligence (AI), thus, will play a major role in the underlying network infrastructure. However, a number of challenges will surface while using the concepts, tools and algorithms of AI in wireless networks used by IoT. In this article, the main challenges in using AI in the wireless network infrastructure that facilitate end-to-end IoT communication are highlighted with potential generalized solution and future research directions.

show abstract

Section: B Challenges For Latency-critical Iot Systemsmentioning

confidence: 99%

The Challenges of Artificial Intelligence in Wireless Networks for the Internet of Things: Exploring Opportunities for Growth

Ahmad

Shahabuddin

Sauter

et al. 2021

EEE Ind. Electron. Mag.

View full text Add to dashboard Cite

show abstract

“…As and when the complexity of data increases, machine learning poses several challenges as explained in [46] pertaining to processing speed, concept drift, variance and bias issues, noisy data, class imbalance, etc. Thus, automating a machine learning task becomes vitally important.…”

Section: Automated Machine Learningmentioning

confidence: 99%

Towards Effective Network Intrusion Detection: From Concept to Creation on Azure Cloud

2021

View full text Add to dashboard Cite

Network Intrusion Detection is one of the most researched topics in the field of computer security. Hacktivists use sophisticated tools to launch numerous attacks that hamper the confidentiality, integrity and availability of computer resources. There is an incessant need to safeguard these resources to avoid further damage. In the proposed study, we have presented a meta-classification approach using decision jungle to perform both binary and multiclass classification. We have established the robustness of our approach by configuring an optimal set of hyper-parameters coupled with relevant feature subsets using a production-ready environment namely Azure machine learning. We have validated the efficiency of the proposed design using three contemporary datasets namely UNSW NB-15, CICIDS 2017, and CICDDOS 2019. We could achieve an accuracy of 99.8% pertaining to UNSW NB-15 whereas the accuracy in the case of CICIDS 2017 and CICDDOS 2019 datasets has been 98% and 97% respectively. A distinctive ability of the proposed model lies in its finesse to detect thirty-three modern attack types considerably well. Unlike conventional stacking ensembles, the proposed solution relies on a train-test ratio of 40:60 to establish the legitimacy of predictions. We also conducted statistical significance tests to compare the performance of classifiers involved in the study. To extend the functionalities further, we have automated the proposed model that can be a reliable candidate for real-time network intrusion detection.

show abstract

“…Hence, f ðlÞ should be a nonlinear monotonically increasing function and it is considered that the classification ability depends on the definition of f ðlÞ. We choose of ol ¼ f ðl; tÞð1 À f ðl; tÞÞ, where t is the time step and f ðl; tÞ is a sigmoid function with 1 1þe Àlt as proposed in [28].…”

Section: Generalized Learning Vector Quantizationmentioning

confidence: 99%

“…Further, not all training data are available at training time, raising the need for constantly updating the model, e.g., online or incremental. Stream classification algorithms [1] tackle these requirements.…”

Section: Introductionmentioning

confidence: 99%

Passive concept drift handling via variations of learning vector quantization

Heusinger

Raab

Schleif

2020

Neural Comput & Applic

View full text Add to dashboard Cite

Concept drift is a change of the underlying data distribution which occurs especially with streaming data. Besides other challenges in the field of streaming data classification, concept drift has to be addressed to obtain reliable predictions. Robust Soft Learning Vector Quantization as well as Generalized Learning Vector Quantization has already shown good performance in traditional settings and is modified in this work to handle streaming data. Further, momentum-based stochastic gradient descent techniques are applied to tackle concept drift passively due to increased learning capabilities. The proposed work is tested against common benchmark algorithms and streaming data in the field and achieved promising results.

show abstract

Applying machine learning to big data streams : An overview of challenges

Cited by 12 publications

References 21 publications

The Challenges of Artificial Intelligence in Wireless Networks for the Internet of Things: Exploring Opportunities for Growth

The Challenges of Artificial Intelligence in Wireless Networks for the Internet of Things: Exploring Opportunities for Growth

Towards Effective Network Intrusion Detection: From Concept to Creation on Azure Cloud

Passive concept drift handling via variations of learning vector quantization

Contact Info

Product

Resources

About