A Reusable Process Control System Framework for the Orbiting Carbon Observatory and NPP Sounder PEATE Missions

Mattmann, Chris A.; Freeborn, Dana; Crichton, D. J.; Foster, B.; Hart, Andrew F.; Woollard, David; Hardman, Sean; Ramírez, Paul; Kelly, Seán; Chang, Albert Y.; Miller, Charles E.

doi:10.1109/smc-it.2009.27

Cited by 22 publications

(16 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…OODT focuses on two canonical use cases: big data processing [21] and information integration [22]. It provides three core services: (1) a file manager is responsible for tracking file locations and transferring files from a staging area to controlled access storage, and for transferring their metadata to Lucene or Solr; (2) a workflow manager captures the control flow and data flow for complex processes, and allows for reproducibility and the construction of scientific pipelines; and (3) a resource manager handles allocation of workflow tasks and other jobs to underlying resources, based on the resource monitoring information from Ganglia or other monitoring software.…”

Section: Oodt: a Data Integration Frameworkmentioning

confidence: 99%

Big Data Integration in Remote Sensing across a Distributed Metadata-Based Spatial Infrastructure

Fan

Yan

et al. 2017

Remote Sensing

View full text Add to dashboard Cite

Since Landsat-1 first started to deliver volumes of pixels in 1972, the volumes of archived data in remote sensing data centers have increased continuously. Due to various satellite orbit parameters and the specifications of different sensors, the storage formats, projections, spatial resolutions, and revisit periods of these archived data are vastly different. In addition, the remote sensing data received continuously by each data center arrives at a faster code rate; it is best to ingest and archive the newly received data to ensure users have access to the latest data retrieval and distribution services. Hence, an excellent data integration, organization, and management program is urgently needed. However, the multi-source, massive, heterogeneous, and distributed storage features of remote sensing data have not only caused difficulties for integration across distributed data center spatial infrastructures, but have also resulted in the current modes of data organization and management being unable meet the rapid retrieval and access requirements of users. Hence, this paper proposes an object-oriented data technology (OODT) and SolrCloud-based remote sensing data integration and management framework across a distributed data center spatial infrastructure. In this framework, all of the remote sensing metadata in the distributed sub-centers are transformed into the International Standardization Organization (ISO) 19115-based unified format, and then ingested and transferred to the main center by OODT components, continuously or at regular intervals. In the main data center, in order to improve the efficiency of massive data retrieval, we proposed a logical segmentation indexing (LSI) model-based data organization approach, and took SolrCloud to realize the distributed index and retrieval of massive metadata. Finally, a series of distributed data integration, retrieval, and comparative experiments showed that our proposed distributed data integration and management program is effective and promises superior results. Specifically, the LSI model-based data organization and the SolrCloud-based distributed indexing schema was able to effectively improve the efficiency of massive data retrieval.

show abstract

Section: Oodt: a Data Integration Frameworkmentioning

confidence: 99%

Big Data Integration in Remote Sensing across a Distributed Metadata-Based Spatial Infrastructure

Fan

Yan

et al. 2017

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…I published the results of this initial refactoring at the IEEE Space Mission Challenges for Information Technology conference with my co-authors that included computer scientists, and experts in chemistry and spectroscopy, and in climate science [11]. This particular experience has helped to influence research agenda item (1) rapid and unobtrusive science algorithm integration described later in the Looking towards the future section of the paper.…”

Section: Flexible Cataloging and Archivingmentioning

confidence: 99%

Cultivating a research agenda for data science

Mattmann

2014

Journal of Big Data

View full text Add to dashboard Cite

I describe a research agenda for data science based on a decade of research and operational work in data-intensive systems at NASA, the University of Southern California, and in the context of open source work at the Apache Software Foundation. My vision is predicated on understanding the architecture for grid computing; on flexible and automated approaches for selecting data movement technologies and on their use in data systems; on the recent emergence of cloud computing for processing and storage, and on the unobtrusive and automated integration of scientific algorithms into data systems. Advancements in each of these areas are a core need, and they will fundamentally improve our understanding of data science, and big data. This paper identifies and highlights my own personal experience and opinion growing into a data scientist.

show abstract

“…In our own work in WINGS, we investigated the retrieval of workflows based on high-level requests, such as finding workflows that generate a desired type of result or process a given type of data [Bergmann and Gil 2012]. Also in our own work on Apache OODT [Mattmann et al 2009], workflows can be retrieved based on a series of dynamic multi-valued metadata that is also used in file cataloging, metadata extraction, and curation and in resource management. This metadata can correspond to workflow status (e.g., FINISHED, EXECUTING; or FAILED); current task wall clock time or workflow wall clock time; workflow id; workflow instance id; and other information.…”

Section: Related Workmentioning

confidence: 99%

“…Apache OODT is a distributed data management and processing framework greatly facilitates learning performance models (item 1 above), since it can extract metadata characteristics upon workflow execution that can be used to do profiling [Mattmann et al 2009], and with distributed execution of workflows with provenance tracking [Mattmann et al 2006]. OODT represents metadata characteristics using OODT's canonical key value pairs that supports a multi-valued metadata representation.…”

Section: Present Users With Optionsmentioning

confidence: 99%

Time-bound analytic tasks on large datasets through dynamic configuration of workflows

Gil

Ratnakar

Verma

et al. 2013

Proceedings of the 8th Workshop on Workflows in Support of Large-Scale Science

Self Cite

View full text Add to dashboard Cite

Domain experts are often untrained in big data technologies and this limits their ability to exploit the data they have available. Workflow systems hide the complexities of high-end computing and software engineering by offering pre-packaged analytic steps combined into multi-step methods commonly used by experts. A current limitation of workflow systems is that they do not take into account user deadlines: they run workflows selected by the user, but take their time to do so. This is impractical when large datasets are at stake, since users often prefer to see an answer faster even if it has lower precision or quality. In this paper, we present an extension to workflow systems that enables them to take into account user deadlines by automatically generating alternative workflow candidates and ranking them according to performance estimates. The system makes these estimates based on workflow performance models created from workflow executions, and uses semantic technologies to reason about workflow options. Possible workflow candidates are presented to the user in a compact manner, and are ranked according to their runtime estimates. We have implemented this approach in the WOOT system, which combines and extends capabilities from the WINGS semantic workflow system and the Apache OODT Object Oriented Data Technology and workflow execution system.

show abstract

A Reusable Process Control System Framework for the Orbiting Carbon Observatory and NPP Sounder PEATE Missions

Cited by 22 publications

References 12 publications

Big Data Integration in Remote Sensing across a Distributed Metadata-Based Spatial Infrastructure

Big Data Integration in Remote Sensing across a Distributed Metadata-Based Spatial Infrastructure

Cultivating a research agenda for data science

Time-bound analytic tasks on large datasets through dynamic configuration of workflows

Contact Info

Product

Resources

About