A Distributed Cache for Hadoop Distributed File System in Real-Time Cloud Services

Zhang, Jing; Wu, Gongqing; Hu, Xuegang; Wu, Xindong

doi:10.1109/grid.2012.17

Cited by 63 publications

(34 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Assume that 200,000 users will visit the system at the same time, then there would be more than 1.15 billion messages per day that need to be successfully written into the database. In this case, if a single, no cache traditional database is used, there would be read and write bottlenecks due to the large-scale data [14]. However, in this system, the cache queues and database server cluster are adopted.…”

Section: B Data Processmentioning

confidence: 93%

An Anti-theft Electric Bicycle Tracking System Supporting Large-Scale Users

Zeng

Jia

2014

2014 International Conference on Identification, Information and Knowledge in the Internet of Things

View full text Add to dashboard Cite

Electric bicycle is one of the most common transportation means in China. To improve the safety of the bicycles, an anti-theft electric bicycle tracking system (ATEBTS) for large-scale users was designed to track electric bicycles by obtaining various information of bicycles' states and analyzing running conditions. In this system, a microprocessor with Global Positioning System (GPS) modules embedded in bicycles sends data in real-time through General Packet Radio Service (GPRS) and Global System for Mobile Communication (GSM), and an inner Gsensor can fetch real-time state information, such as acceleration and shaking signals. The system designed a message protocol to encapsulate and send these data to servers and also adopted distributed architecture, including OpenFire server cluster, cache queue cluster, database cluster and web server cluster. This specific architecture provides a better performance when large-scale users visit concurrently. To further improve query efficiency, relevant data are inserted into one single node in database indexed by the terminal serial number. With the increasing number of users, tables can be split into several tables according to different dates. And this system will provide services for more than 200 thousands users in Shandong Province this year.

show abstract

Section: B Data Processmentioning

confidence: 93%

An Anti-theft Electric Bicycle Tracking System Supporting Large-Scale Users

Zeng

Jia

2014

2014 International Conference on Identification, Information and Knowledge in the Internet of Things

View full text Add to dashboard Cite

show abstract

“…A user-specific model usually consumes several megabytes to a hundred megabytes of memory. Since the low performance of small file storage is a widely known weakness of HDFS, we have proposed a distributed cache system HDCache [21] that accelerates the speed of model retrieval and loading. HDCache is a distributed layered cache system built on the top of HDFS.…”

Section: System Architecturementioning

confidence: 99%

Knowledge Transfer Using User-Generated Data within Real-Time Cloud Services

Zhang¹,

Pan²,

Cai³

et al. 2020

KSII TIIS

View full text Add to dashboard Cite

When automatic speech recognition (ASR) is provided as a cloud service, it is easy to collect voice and application domain data from users. Harnessing these data will facilitate the provision of more personalized services. In this paper, we demonstrate our transfer learning-based knowledge service that built with the user-generated data collected through our novel system that deliveries personalized ASR service. First, we discuss the motivation, challenges, and prospects of building up such a knowledge-based service-oriented system. Second, we present a Quadruple Transfer Learning (QTL) method that can learn a classification model from a source domain and transfer it to a target domain. Third, we provide an overview architecture of our novel system that collects voice data from mobile users, labels the data via crowdsourcing, utilises these collected user-generated data to train different machine learning models, and delivers the personalised real-time cloud services. Finally, we use the E-Book data collected from our system to train classification models and apply them in the smart TV domain, and the experimental results show that our QTL method is effective in two classification tasks, which confirms that the knowledge transfer provides a value-added service for the upper-layer mobile applications in different domains.

show abstract

“…processing [25]. Besides, Hadoop has also been adopted to prompt the development of large-scale image processing.…”

Section: Related Workmentioning

confidence: 99%

A Hierarchical Distributed Processing Framework for Big Image Data

Dong

Lin

Liang

et al. 2016

IEEE Trans. Big Data

View full text Add to dashboard Cite

This paper introduces an effective processing framework nominated ICP (Image Cloud Processing) to powerfully cope with the data explosion in image processing field. While most previous researches focus on optimizing the image processing algorithms to gain higher efficiency, our work dedicates to providing a general framework for those image processing algorithms, which can be implemented in parallel so as to achieve a boost in time efficiency without compromising the results performance along with the increasing image scale. The proposed ICP framework consists of two mechanisms, i.e. SICP (Static ICP) and DICP (Dynamic ICP). Specifically, SICP is aimed at processing the big image data pre-stored in the distributed system, while DICP is proposed for dynamic input. To accomplish SICP, two novel data representations named P-Image and Big-Image are designed to cooperate with MapReduce to achieve more optimized configuration and higher efficiency. DICP is implemented through a parallel processing procedure working with the traditional processing mechanism of the distributed system. Representative results of comprehensive experiments on the challenging ImageNet dataset are selected to validate the capacity of our proposed ICP framework over the traditional state-of-the-art methods, both in time efficiency and quality of results.

show abstract

A Distributed Cache for Hadoop Distributed File System in Real-Time Cloud Services

Cited by 63 publications

References 16 publications

An Anti-theft Electric Bicycle Tracking System Supporting Large-Scale Users

An Anti-theft Electric Bicycle Tracking System Supporting Large-Scale Users

Knowledge Transfer Using User-Generated Data within Real-Time Cloud Services

A Hierarchical Distributed Processing Framework for Big Image Data

Contact Info

Product

Resources

About