Data automation at light sources

Blaiszik, Benjamin J.; Chard, Kyle; Chard, Ryan; Foster, Ian; Ward, Logan

doi:10.1063/1.5084563

Cited by 10 publications

(4 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DLHub also provides a model serving framework for executing inference and other tasks over arbitrary data. DLHub has been exploited by several research projects with success, such as [30], [31].…”

Section: Related Work Regarding Machine Learning Platformsmentioning

confidence: 99%

A Cloud-Based Framework for Machine Learning Workloads and Applications

et al. 2020

View full text Add to dashboard Cite

In this paper we propose a distributed architecture to provide machine learning practitioners with a set of tools and cloud services that cover the whole machine learning development cycle: ranging from the models creation, training, validation and testing to the models serving as a service, sharing and publication. In such respect, the DEEP-Hybrid-DataCloud framework allows transparent access to existing e-Infrastructures, effectively exploiting distributed resources for the most compute-intensive tasks coming from the machine learning development cycle. Moreover, it provides scientists with a set of Cloud-oriented services to make their models publicly available, by adopting a serverless architecture and a DevOps approach, allowing an easy share, publish and deploy of the developed models. INDEX TERMS Cloud computing, computers and information processing, deep learning, distributed computing, machine learning, serverless architectures.

show abstract

Section: Related Work Regarding Machine Learning Platformsmentioning

confidence: 99%

A Cloud-Based Framework for Machine Learning Workloads and Applications

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The autocategorization model perdictions needs a similar automatic data pipeline to uphold the load of each prediction and at the same time allow analysis of the result that will be brought back to the model in a feedback loop. Blaiszik et al made a scalable service that manages machine learning models through a structured API [4]. Blaiszik is using docker containers to wrap the model logic into an environment with controlled variables.…”

Section: Related Workmentioning

confidence: 99%

Productify News Article Classification Model with Sagemaker

Linden¹,

Wang²,

Forsström³

et al. 2020

Adv. sci. technol. eng. syst. j.

View full text Add to dashboard Cite

News companies have a need to automate and make the process of writing about popular and new events more effective. Current technologies involve robotic programs that fill in values in templates and website listeners that notify editors when changes are made so that the editor can read up on the source change on the actual website. Editors can provide news faster and better if directly provided with abstracts of the external sources and categorical meta-data that supports what the text is about. To make categorical meta-data a reality an auto-categorization model was created and optimized for Swedish articles written by local news journalists. The problem was that it was not scale-able enough to use out of the box. Instead of having this local model that could make good predictions of the text documents, the model is to be deployed in the cloud and an API interface is created. The API can be accessed from the tools where the articles is being written and therefore these services can automatically assign categories to the articles once the journalist is done writing it. To allow scale-ability to several thousands of simultaneously categorized articles and at the same time improving the workflow of deploying new models easier the API is uploaded to Sagemaker where several models are trained and once an improved model is found that model will be used in production in such a way that the system organically adapts to new written articles. An evaluation of Sagemaker API was done and it was concluded that the complexity of this solution was polynomial.

show abstract

“…Real-time experimental data analysis [49][50][51] and steering have been active research areas [52,53]. ASTRA is a popular GPU-based toolkit for processing and reconstruction of x-ray data [51].…”

Section: Overhead Analysismentioning

confidence: 99%

Deep Learning Accelerated Light Source Experiments

Liu

Biçer

Kettimuthu

et al. 2019

2019 IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS)

Self Cite

View full text Add to dashboard Cite

Experimental protocols at synchrotron light sources typically process and validate data only after an experiment has completed, which can lead to undetected errors and cannot enable online steering. Real-time data analysis can enable both detection of, and recovery from, errors, and optimization of data acquisition. However, modern scientific instruments, such as detectors at synchrotron light sources, can generate data at GBs/sec rates. Data processing methods such as the widely used computational tomography usually require considerable computational resources, and yield poor quality reconstructions in the early stages of data acquisition when available views are sparse. We describe here how a deep convolutional neural network can be integrated into the real-time streaming tomography pipeline to enable better-quality images in the early stages of data acquisition. Compared with conventional streaming tomography processing, our method can significantly improve tomography image quality, deliver comparable images using only 32% of the data needed for conventional streaming processing, and save 68% experiment time for data acquisition.

show abstract

Data automation at light sources

Cited by 10 publications

References 39 publications

A Cloud-Based Framework for Machine Learning Workloads and Applications

A Cloud-Based Framework for Machine Learning Workloads and Applications

Productify News Article Classification Model with Sagemaker

Deep Learning Accelerated Light Source Experiments

Contact Info

Product

Resources

About