Performance Analysis of Not Only SQL Semi-Stream Join Using MongoDB for Real-Time Data Warehousing

Mehmood, Erum; Anees, Tayyaba

doi:10.1109/access.2019.2941925

Cited by 19 publications

(6 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The traditional methods of data analysis and interpretation for sound decision-making are not sufficient and effective with the enormous amount of organizational data being available in today's competitive environment (Niu et al , 2021), resulting in an increasing number of requests for a new generation of technologies including data analytics and visualization tools to assist companies in voluminous data management, analysis and fed into the decision-making process (Chen and Lin, 2021). Since business data analytics is related to data warehouse systems, data warehouse has been considered as a key platform for the integrated management of decision support data in various industries and countries (Mehmood and Anees, 2019). One of the prominent consequences of the big data revolution is heavy investment in data warehouses to take advantage of the rich data sources for a variety of tasks such as planning, target marketing, decision-making, data analysis and customer services in unpredictable market fluctuations and competitive environments that put much pressure on businesses (Moscoso-Zea et al , 2018).…”

Section: Introductionmentioning

confidence: 99%

An empirical study on data warehouse systems effectiveness: the case of Jordanian banks in the business intelligence era

Al-Okaily

Teoh

et al. 2022

EMJB

View full text Add to dashboard Cite

PurposeDespite the increasing role of the data warehouse as a supportive decision-making tool in today's business world, academic research for measuring its effectiveness has been lacking. This paucity of academic interest stimulated us to evaluate data warehousing effectiveness in the organizational context of Jordanian banks.Design/methodology/approachThis paper develops a theoretical model specific to the data warehouse system domain that builds on the DeLone and McLean model. The model is empirically tested by means of structural equation modelling applying the partial least squares approach and using data collected in a survey questionnaire from 127 respondents at Jordanian banks.FindingsEmpirical data analysis supported that data quality, system quality, user satisfaction, individual benefits and organizational benefits have made strong contributions to data warehousing effectiveness in our organizational data context.Practical implicationsThe results provide a better understanding of the data warehouse effectiveness and its importance in enabling the Jordanian banks to be competitive.Originality/valueThis study is indeed one of the first empirical attempts to measure data warehouse system effectiveness and the first of its kind in an emerging country such as Jordan.

show abstract

Section: Introductionmentioning

confidence: 99%

An empirical study on data warehouse systems effectiveness: the case of Jordanian banks in the business intelligence era

Al-Okaily

Teoh

et al. 2022

EMJB

View full text Add to dashboard Cite

show abstract

“…The method performs stream processing in distributed way, and stream processing engines are designed to perform such joins. Similarly, to handle the unstructured and structured streams, an Mongo DB model is discussed in Mehmood and Anees ( 2019 ), which involves in ETL by joining the semi-stream data coming from various sources with the disk-based master data according to the keys. Similarly, the problem of distributed streaming with ETL is handled with novel Strilim model which supports the development and deployment of streaming application in rapid manner.…”

Section: Introductionmentioning

confidence: 99%

RETRACTED ARTICLE: Dynamic multi-variant relational scheme-based intelligent ETL framework for healthcare management

Manickam

Minu

2022

Soft Comput

View full text Add to dashboard Cite

The growth of information technology has opened the gate for the organizations to maintain their data in various forms and at various volumes. This increases the volume and dimension of data being maintained. However, they store their data in their data servers or in cloud environment. Such data have been used to generate various intelligence to support various problems. To support such analysis process, different data have been used and the big data comes to play in this part. Optimizing techniques to help improve the process of ETL could greatly help in real-time analysis of data. ETL optimization could be achieved through several factors simplest being increasing the frequency of the process. Other ways to achieve optimization are through the use of various architectures, programming models, intelligence in transformation and security. To improve the performance of ETL, an efficient dynamic multi-variant relational intelligent ETL framework has been presented in this article. The distributed approach maintains various ontology’s and data dictionaries which have been dynamically updated by different threads of ETL process. Initially, the process is start by applying the extraction process which extracts the data from different sources and finds set of dimensions and their characteristics. Such data extracted have been verified over the data dictionary. Further, the relational score has been measured for each data source with the existing one. Similarly, the method computes the value of multi-variant relational similarity (MVRS) for the data obtained from a single source. This will be performed by different threads of ETL process. According to the value of MVRS, the method performs map reduce and merging of data. According to the value of MVRS the method selects the data node and merges the data to store in the data warehouse. The threads of ETL are capable of reading the changes in data dictionaries and ontology’s to iterate the process of transformation and loading. The method improves the performance of ETL with least time complexity and higher performance.

show abstract

“…The write speed of MongoDB is better than the write speed of relational databases. This feature provides advantages in scenarios such as storing large-scale IoT data [18] and being used in the ETL layer of Real-Time Data Warehousing [19].…”

Section: Mongodbmentioning

confidence: 99%

A method to improve full-text search performance of MongoDB

Mesut¹,

Öztürk²

2022

Pamukkale J Eng Sci

View full text Add to dashboard Cite

B-Tree based text indexes used in MongoDB are slow compared to different structures such as inverted indexes. In this study, it has been shown that the full-text search speed can be increased significantly by indexing a structure in which each different word in the text is included only once. The Multi-Stream Word-Based Compression Algorithm (MWCA), developed in our previous work, stores word dictionaries and data in different streams. While adding the documents to a MongoDB collection, they were encoded with MWCA and separated into six different streams. Each stream was stored in a different field, and three of them containing unique words were used when creating a text index. In this way, the index could be created in a shorter time and took up less space. It was also seen that Snappy and Zlib block compression methods used by MongoDB reached higher compression ratios on data encoded with MWCA. Search tests on text indexes created on collections using different compression options shows that our method provides 19 to 146times speed increase and 34% to 40% less memory usage. Tests on regex searches that do not use the text index also shows that the MWCA model provides 7 to 13 times speed increase and 29% to 34% less memory usage. MongoDB'de kullanılan B-Tree tabanlı metin dizinleri, ters çevrilmiş dizinler gibi farklı yapılara kıyasla yavaştır. Bu çalışmada, metindeki her farklı kelimenin yalnızca bir kez yer aldığı bir yapı indekslenerek tam metin arama hızının önemli ölçüde artırılabileceği gösterilmiştir. Daha önceki çalışmalarımızda geliştirilen Çok AkışlıKelime Tabanlı Sıkıştırma Algoritması (MWCA), kelime sözlüklerini ve verileri farklı akışlarda saklar. Belgeler bir MongoDB koleksiyonuna eklenirken MWCA ile kodlanmış ve altı farklı akışa ayrılmıştır. Her akış farklı bir alan ismi ile saklanmış ve bunlardan benzersiz kelimeler içeren üçü metin dizini oluşturulurken kullanılmıştır. Bu sayede indeks daha kısa sürede oluşturulabilmiş ve daha az yer kaplamıştır. MongoDB'de kullanılan Snappy ve Zlib blok sıkıştırma yöntemlerinin MWCA ile kodlanan veriler üzerinde daha yüksek sıkıştırma oranlarına ulaştığı da görülmüştür. Farklı yöntemler ile sıkıştırılan koleksiyonlar üzerinde oluşturulan metin dizinlerinde yapılan arama testleri, yöntemimizin 19 ila 146 kat hız artışı ve %34 ila %40 daha az bellek kullanımı sağladığını göstermiştir. Metin dizinini kullanmayan regex aramaları ile ilgili testler de MWCA modelinin 7 ila 13 kat hız artışı ve %29 ila %34 daha az bellek kullanımı sağladığını göstermiştir.

show abstract

Performance Analysis of Not Only SQL Semi-Stream Join Using MongoDB for Real-Time Data Warehousing

Cited by 19 publications

References 19 publications

An empirical study on data warehouse systems effectiveness: the case of Jordanian banks in the business intelligence era

An empirical study on data warehouse systems effectiveness: the case of Jordanian banks in the business intelligence era

RETRACTED ARTICLE: Dynamic multi-variant relational scheme-based intelligent ETL framework for healthcare management

A method to improve full-text search performance of MongoDB

Contact Info

Product

Resources

About