The implementation of data storage and analytics platform for big data lake of electricity usage with spark

Yang, Chao‐Tung; Chen, Tzu-Yang; Kristiani, Endah; Wu, S. Felix

doi:10.1007/s11227-020-03505-6

Cited by 24 publications

(6 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the process of e-commerce transactions, some user privacy data is easily leaked, posing a threat to user privacy security. erefore, it is necessary to use access control and encrypted storage technology and privacy protection of e-commerce data to ensure that users' privacy is not leaked as shown in Figure 1 [3]. By adopting the method of joint network feature analysis, the feature extraction and statistical analysis of e-commerce group user access data in the era of big data is realized, and the level of information management and e-commerce information access is improved.…”

Section: Introductionmentioning

confidence: 99%

E-Commerce Data Access Control and Encrypted Storage Based on Internet of Things

Yue

2022

Mathematical Problems in Engineering

View full text Add to dashboard Cite

In order to discuss the access control and encrypted storage of Internet of Things data for e-commerce data, a method of Internet of Things data access control and encrypted storage based on optical fiber network communication is proposed. Build an IoT application model based on optical fiber network communication, which consists of a perception layer, a network layer, and an application layer. The network layer is through the optical fiber communication network. The information collected by the optical fiber sensor in the perception layer is transmitted to the application layer for processing. And the model-aware layer adopts the dynamic transformation encryption method for attribute decomposition, and the Internet of Things data access control strategy based on key management and implement IoT data access control and encrypted storage based on the IoT application model of optical fiber network communication. The experimental results show that when this method implements access control and encrypted storage of IoT data, it has the advantages of high efficiency, low storage space overhead, and low computational cost of fiber network nodes, and the security of the processed IoT data is as high as 97.77%. Conclusion. This method can effectively deal with the limited computing performance and storage performance of the perception layer nodes of the application model; larger and larger application requirements with a large number of nodes.

show abstract

Section: Introductionmentioning

confidence: 99%

E-Commerce Data Access Control and Encrypted Storage Based on Internet of Things

Yue

2022

Mathematical Problems in Engineering

View full text Add to dashboard Cite

show abstract

“…Whether it is traffic management, energy consumption optimization, or emergency response coordination, the agility afforded by data lakes ensures that decision-makers are equipped with timely and actionable intelligence [38]. Solutions to increase the real-time performance of data lakes for smart cities were recently described in [39][40][41], adopting and integrating big data technologies and frameworks, such as Hadoop, Spark, Solr, HDFS, and Hue, while defining architectures specifically tailored for Smart City requirements and features.…”

Section: Related Workmentioning

confidence: 99%

An Urban Intelligence Architecture for Heterogeneous Data and Application Integration, Deployment and Orchestration

Silvestri,

Tricomi,

Bassolillo

et al. 2024

Sensors

View full text Add to dashboard Cite

This paper describes a novel architecture that aims to create a template for the implementation of an IT platform, supporting the deployment and integration of the different digital twin subsystems that compose a complex urban intelligence system. In more detail, the proposed Smart City IT architecture has the following main purposes: (i) facilitating the deployment of the subsystems in a cloud environment; (ii) effectively storing, integrating, managing, and sharing the huge amount of heterogeneous data acquired and produced by each subsystem, using a data lake; (iii) supporting data exchange and sharing; (iv) managing and executing workflows, to automatically coordinate and run processes; and (v) to provide and visualize the required information. A prototype of the proposed IT solution was implemented leveraging open-source frameworks and technologies, to test its functionalities and performance. The results of the tests performed in real-world settings confirmed that the proposed architecture could efficiently and easily support the deployment and integration of heterogeneous subsystems, allowing them to share and integrate their data and to select, extract, and visualize the information required by a user, as well as promoting the integration with other external systems, and defining and executing workflows to orchestrate the various subsystems involved in complex analyses and processes.

show abstract

“…When it is normal data, the output result identifies the data as the normal data type, and when it is bad data, the category to which the bad data belongs can be identified. The interval time of the load data collected by the power consumption metering collection system consists of 15, 30 and 60 minutes, so that the input variables are of three dimensions: 24, 48 and 96 [1][2]. The number of dimensions of the output variable is equal to the number of dimensions of the output variable in order to be able to distinguish the number of load data categories to which normal data and bad data belong, as the output results are expressed in binary.…”

Section: Collecting Data From Electricity System Usersmentioning

confidence: 99%

A deep learning-based approach for identifying bad data in power systems

Dong

Chen

et al. 2023

Eighth International Conference on Electronic Technology and Information Science (ICETIS 2023)

View full text Add to dashboard Cite

In the actual operation process, some of the power system bad data identification methods have the problem of low accuracy, for this reason, a deep learning-based power system bad data identification method is designed to improve this defect. The data is collected from power system users, the phase deviation caused by non-integer sampling is reduced by high sampling rate, the measurement signal period is obtained, the operational state of the distribution network is evaluated based on deep learning, the state vector is calculated, the maximum standard residual value is found, the location of the bad data is obtained, and the bad data identification method is designed. Experimental results: The mean accuracy of the power system bad data identification method in the paper is: 78.26%, which indicates that the designed power system bad data identification method performs better after fully integrating the deep learning.

show abstract

The implementation of data storage and analytics platform for big data lake of electricity usage with spark

Cited by 24 publications

References 31 publications

E-Commerce Data Access Control and Encrypted Storage Based on Internet of Things

E-Commerce Data Access Control and Encrypted Storage Based on Internet of Things

An Urban Intelligence Architecture for Heterogeneous Data and Application Integration, Deployment and Orchestration

A deep learning-based approach for identifying bad data in power systems

Contact Info

Product

Resources

About