No abstract

No abstract

E ach year across the US, mesoscale weather events-flash floods, tornadoes, hail, strong winds, lightning, and localized winter storms-cause hundreds of deaths, routinely disrupt transportation and commerce, and lead to economic losses averaging more than US$13 billion.1 Although mitigating the impacts of such events would yield enormous economic and societal benefits, research leading to that goal is hindered by rigid IT frameworks that can't accommodate the real-time, on-demand, dynamically adaptive needs of mesoscale weather research; its disparate, high-volume data sets and streams; or the tremendous computational demands of its numerical models and data-assimilation systems.In response to the increasingly urgent need for a comprehensive national cyberinfrastructure in mesoscale meteorology-particularly one that can interoperate with those being developed in other relevant disciplines-the US National Science Foundation (NSF) funded a large information technology research (ITR) grant in 2003, known as Linked Environments for Atmospheric Discovery (LEAD). A multidisciplinary effort involving nine institutions and more than 100 scientists, students, and technical staff in meteorology, computer science, social science, and education, LEAD addresses the fundamental research challenges needed to create an integrated, scalable framework for adaptively analyzing and predicting the atmosphere.LEAD's foundation is dynamic workflow orchestration and data management in a Web services framework. These capabilities provide for the use of analysis tools, forecast models, and data repositories,

The National Oceanic and Atmospheric Administration’s (NOAA) Big Data Partnership (BDP) was established in April 2015 through cooperative research agreements between NOAA and selected commercial and academic partners. The BDP is investigating how the value inherent in NOAA’s data may be leveraged to broaden their utilization through modern cloud infrastructures and advanced “big data” techniques. NOAA’s Next Generation Weather Radar (NEXRAD) data were identified as an ideal candidate for such collaborative efforts. NEXRAD Level II data are valuable yet challenging to utilize in their entirety, and recent advances in weather radar science can be applied to both the archived and real-time data streams. NOAA’s National Centers for Environmental Information (NCEI) transferred the complete NEXRAD Level II historical archive, originating in 1991, through North Carolina State University’s Cooperative Institute for Climate and Satellites (CICS-NC) to interested BDP collaborators. Amazon Web Services (AWS) has received and made freely available the complete archived Level II data through its AWS platform. AWS then partnered with Unidata/University Corporation for Atmospheric Research (UCAR) to establish a real-time NEXRAD feed, thereby providing on-demand dissemination of both archived and current data seamlessly through the same access mechanism by October 2015. To organize, verify, and utilize the NEXRAD data on its platform, AWS further partnered with the Climate Corporation. This collective effort among federal government, private industry, and academia has already realized a number of new and novel applications that employ NOAA’s NEXRAD data, at no net cost to the U.S. taxpayer. The volume of accessed NEXRAD data, including this new AWS platform service, has increased by 130%, while the amount of data delivered by NOAA/NCEI has decreased by 50%.

With a very modest investment in computer hardware and the open source local data manger (LDM) software from UCAR's Unidata Program Center, an individual researcher can receive a variety of NEXRAD Level III gridded rainfall products, and the unprocessed Level II data in real-time from most NEXRAD radars. Additionally, the National Climatic Data Center has vast archives of these products and Level II data. Still, significant obstacles remain in order to unlock the full potential of the data. One set of obstacles is related to effective management of multi-terrabyte data sets: storing, compressing, and backing up. A second set of obstacles, for hydrologists and hydrometeorologists in particular, is that the NEXRAD Level III products are not well suited for application in hydrology. There is a strong need for the generation of highquality products directly from the Level II data with well-documented steps that include quality control, removal of false echoes, rainfall estimation algorithms with variety of corrections, coordinate conversion and georeferencing, conversion to a convenient data format(s), and integration with GIS. For hydrologists it is imperative that these procedures are basin-centered as opposed to radar-centered. Thirdly, the amount of data present in a multi-year, multi-radar dataset is such that simple cataloging and indexing of the data is not sufficient. Rather, sophisticated metadata extraction and management techniques are required. The authors describe and discuss the Hydro-NEXRAD software system that addresses the above three challenges. With support from the National Science Foundation through its ITR program, the authors are developing a basin-centered framework for addressing all these issues in a comprehensive manner, tailored specifically for use of NEXRAD data in hydrology and hydrometeorology. Through a flexible web interface users can search a large metadata database base, managed by a World Environmental and Water Resources Congress 2007: Restoring Our Natural Habitat