Hopsworks: Improving User Experience and Development on Hadoop with Scalable, Strongly Consistent Metadata

Ismail, Mahmoud; Gebremeskel, Ermias; Kakantousis, Theofilos; Berthou, Gautier; Dowling, Jim

doi:10.1109/icdcs.2017.41

Cited by 11 publications

(4 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of the experiments were performed on a Hadoop cluster consisting of a single node with 32 cores Intel(R) Xeon(R) CPU E5-4603 v2 @ 2.20GHz 20 and 128GB DDR3 RAM, 1.6 Tb mechanical disk. The experiment presented in Figure 12(left) was performed on a small cluster that runs the Hopsworks data platform [10]. The main module of Hopsworks is Hops 21 , which is a next generation distribution of Apache Hadoop, using a novel implementation of HDFS, called HopsFS [18].…”

Section: Methodsmentioning

confidence: 99%

“…As a concrete example consider the ice monitoring application presented in [15], which facilitates the safety of ship navigation in the Arctic through Geospatial Interlinking. Its input data comprise in situ observations about icebergs (e.g., their location at speciic times and other important properties of sea ice) and satellite images from the EU Copernicus Programme 10 . Its goal is to interlink these large data sources for visualization purposes as well as for performing analytics.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Static and Dynamic Progressive Geospatial Interlinking

Papadakis

Mandilaras

Mamoulis

et al. 2022

ACM Trans. Spatial Algorithms Syst.

View full text Add to dashboard Cite

Geospatial data constitute a considerable part of Semantic Web data, but at the moment, its sources are insufficiently interlinked with topological relations in the Linked Open Data cloud. Geospatial Interlinking aims to cover this gap through space tiling techniques, which significantly restrict the search space. Yet, the state-of-the-art techniques operate exclusively in a batch manner that produces results only after processing all their geometries. In this work, we address this issue by defining the task of Progressive Geospatial Interlinking, which produces results in a pay-as-you-go manner when the available computational or temporal resources are limited. We propose a static progressive algorithm, which employs a fixed processing order, and a dynamic one, whose processing order is updated whenever new topological relations are discovered. We equip both algorithms with a series of weighting schemes and explain how they can be adapted to massive parallelization with Apache Spark. We conduct a thorough experimental study over a six large, real datasets, demonstrating the superiority of our techniques over the current state-of-the-art. Special care is also taken to analyze the performance of the various weighting schemes.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Static and Dynamic Progressive Geospatial Interlinking

Papadakis

Mandilaras

Mamoulis

et al. 2022

ACM Trans. Spatial Algorithms Syst.

View full text Add to dashboard Cite

show abstract

“…TensorFlow Extended (TFX) [21] is Google's solution for the orchestration of components in machine learning pipelines, and it provides a configuration framework to express machine learning pipelines based on the predefined metadata of artifacts, executions, and pipelines. Hopsworks [22] presented another feature store that defines a metadata model for describing the machine learning artifacts and also provides a framework to engineer and store machine learning features at scale.…”

Section: Related Workmentioning

confidence: 99%

Transfer Learning in Smart Environments

Anjomshoaa

Curry

2021

MAKE

View full text Add to dashboard Cite

The knowledge embodied in cognitive models of smart environments, such as machine learning models, is commonly associated with time-consuming and costly processes such as large-scale data collection, data labeling, network training, and fine-tuning of models. Sharing and reuse of these elaborated resources between intelligent systems of different environments, which is known as transfer learning, would facilitate the adoption of cognitive services for the users and accelerate the uptake of intelligent systems in smart building and smart city applications. Currently, machine learning processes are commonly built for intra-organization purposes and tailored towards specific use cases with the assumption of integrated model repositories and feature pools. Transferring such services and models beyond organization boundaries is a challenging task that requires human intervention to find the matching models and evaluate them. This paper investigates the potential of communication and transfer learning between smart environments in order to empower a decentralized and peer-to-peer ecosystem for seamless and automatic transfer of services and machine learning models. To this end, we explore different knowledge types in the context of smart built environments and propose a collaboration framework based on knowledge graph principles for describing the machine learning models and their corresponding dependencies.

show abstract

“…Hopswork [19] is a data science platform for the design and operation of data analytics applications. The system applies HopsFS, a highly scalable distributed file system, to improve its efficiency.…”

Section: Related Workmentioning

confidence: 99%

SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis

Xing

Yeung

Cai

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Deep learning has achieved great success in a wide spectrum of multimedia applications such as image classification, natural language processing and multimodal data analysis. Recent years have seen the development of many deep learning frameworks that provide a high-level programming interface for users to design models, conduct training and deploy inference. However, it remains challenging to build an efficient end-to-end multimedia application with most existing frameworks. Specifically, in terms of usability, it is demanding for non-experts to implement deep learning models, obtain the right settings for the entire machine learning pipeline, manage models and datasets, and exploit external data sources all together. Further, in terms of adaptability, elastic computation solutions are much needed as the actual serving workload fluctuates constantly, and scaling the hardware resources to handle the fluctuating workload is typically infeasible. To address these challenges, we introduce SINGA-Easy, a new deep learning framework that provides distributed hyper-parameter tuning at the training stage, dynamic computational cost control at the inference stage, and intuitive user interactions with multimedia contents facilitated by model explanation. Our experiments on the training and deployment of multi-modality data analysis applications show that the framework is both usable and adaptable to dynamic inference loads. We implement SINGA-Easy on top of Apache SINGA and demonstrate our system with the entire machine learning life cycle. CCS CONCEPTS• Information systems → Multimedia information systems;• Computing methodologies → Machine learning; • Humancentered computing → Systems and tools for interaction design.

show abstract

Hopsworks: Improving User Experience and Development on Hadoop with Scalable, Strongly Consistent Metadata

Cited by 11 publications

References 5 publications

Static and Dynamic Progressive Geospatial Interlinking

Static and Dynamic Progressive Geospatial Interlinking

Transfer Learning in Smart Environments

SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis

Contact Info

Product

Resources

About