Dealing with Archaeology's Data Avalanche

Abstract

Newly commissioned state of the art MX beamlines fitted with current advanced hybrid pixel detectors are now in operation. At the NSLS-II, AMX and FMX, two high-brightness micro-focusing beamlines (> 10 11 and > 5x10 12 ph/s/um 2 respectively) are fitted with Dectris Eiger detectors and are equipped with advanced automation that will ultimately allow screening of up to 1000 crystals per day. We have seen throughput greater than 1 GB/s per beamline during demanding experiments and are expecting this to increase in the upcoming months. With this level of throughput, near real time data analysis feedback is a necessity. This requires infrastructure with a high bandwidth network, fast-I/O large storage and significant computational capacity. Optimized data processing software and pipelines are being developed to help in coping with the throughput. We will present the state of current problems that the community is facing and some of the solutions that are currently deployed at various facilities. This will be followed by two talks by Intel and a facilitated discussion. The first talk is on Intel Scalable System Framework (HPC) The second talk is on Intel software and tools Ecosystem (HPC) and will focus on adapting MX code to use Intel processors effectively. Acta Cryst. (2017). A73, a209

The Digital Dilemma

“…The world of open access academia and data publishing is shaping the future of digital data curation and reuse, and data preservation is part of this movement (Baker and Yarmey 2009;Conway et al 2011;Cultural Heritage Partners 2012;Dürr 2008;Hall 2013). While the amount of digital data grows and the solutions for their preservation remain fundamentally misaligned with existing workflows and proprietary claims, both individually and internationally, the digital archaeological record is at risk of loss (Petrovic et al 2011). If scholars wish to ensure the long-term viability of their digital data, they must navigate the potential conflicts that arise from these advances.…”

ArtifactVis2: Managing real-time archaeological data in immersive 3D environments

“…At the core of our system is a geographic information system that is capable of storing and organizing all of the spatial location data, related databases, and generated 3D and 2D media with minimal loss of fidelity (see [18,24,26]). This acquisition pipeline provides real-time access of an excavation's generated digital datasets directly into ArtifactVis2.…”

“…3). It is the attention to detail of their recording in the field and the precision of tools used to extract this information that enables us to properly handle the visualization of these datasets in ArtifactVis2 and manage the 'data avalanche' [24]. Although there are proprietary software tools available for processing the data of many of these devices, they are not necessarily designed for incorporation into a cyber-infrastructure or adapted specifically to the needs of archaeological excavation or cultural heritage.…”

Terrestrial laser scanning (LiDAR) as a means of digital documentation in rescue archaeology: Two examples from the Faynan of Jordan

“…Realizing the need for a dexterously manipulatable format in which to visualize the billions of points necessitated by landscape scanning for archaeological purposes, ELRAP, as part of the Center of Interdisciplinary Science for Art, Architecture, and Archaeology (CISA3) and California Institute for Telecommunications and Information Technology (CalIT2), had previously begun to develop innovative software which could gracefully handle the anticipated 'data avalanche.' [4] With the ability to handle such vast point clouds, ELRAP could therefore attempt to collect as much data as possible in the field.…”