Twenty Years of Data Management in the British Atmospheric Data Centre

Abstract

The British Atmospheric Data Centre (BADC) has existed in its present form for 20 years, having been formally created in 1994. It evolved from the GDF (Geophysical Data Facility), a SERC (Science and Engineering Research Council) facility, as a result of research council reform where NERC (Natural Environment Research Council) extended its remit to cover atmospheric data below 10km altitude. With that change the BADC took on data from many other atmospheric sources and started interacting with NERC research programmes. The BADC has now hit early adulthood. Prompted by this milestone, we examine in this paper whether the data centre is creaking at the seams or is looking forward to the prime of its life, gliding effortlessly into the future. Which parts of it are bullet proof and which parts are held together with double-sided sticky tape? Can we expect to see it in its present form in another twenty years’ time? To answer these questions, we examine the interfaces, technology, processes and organisation used in the provision of data centre services by looking at three snapshots in time, 1994, 2004 and 2014, using metrics and reports from the time to compare and contrasts the services using BADC. The repository landscape has changed massively over this period and has moved the focus for technology and development as the broader community followed emerging trends, standards and ways of working. The incorporation of these new ideas has been both a blessing and a curse, providing the data centre staff with plenty of challenges and opportunities. We also discuss key data centre functions including: data discovery, data access, ingestion, data management planning, preservation plans, agreements/licences and data policy, storage and server technology, organisation and funding, and user management. We conclude that the data centre will probably still exist in some form in 2024 and that it will most likely still be reliant on a file system. However, the technology delivering this service will change and the host organisation and funding routes may vary.

Developing an Open Data Portal for the ESA Climate Change Initiative

“…The Centre for Environmental Data Analysis (CEDA) provides the necessary archival and curation environment -archival provides reliable data availability including multiple copies of data and curation provides active management of the data, including format and metadata migration (if necessary). Systems have been built up over decades (Pepler and Callaghan, 2015) to deliver reliable data services and complex metadata systems are in place (Parton et al, 2015) which exploit a wide range of information categories from archive metadata, to the necessary browse and discovery metadata (Lawrence et al, 2009). These data systems are supported by data scientists who have expertise in the relevant science areas and can actively curate the metadata content to maximise utility as systems and target communities evolve.…”