The research philosophy and new capabilities installed at NSLS beamline X26-C to support electronic absorption and Raman spectroscopies coupled with X-ray diffraction are reviewed. This beamline is dedicated full time to multidisciplinary studies with goals that include revealing the relationship between the electronic and atomic structures in macromolecules. The beamline instrumentation has been fully integrated such that optical absorption spectra and X-ray diffraction images are interlaced. Therefore, optical changes induced by X-ray exposure can be correlated with X-ray diffraction data collection. The installation of Raman spectroscopy into the beamline is also briefly reviewed. Data are now routinely generated almost simultaneously from three complementary types of experiments from the same sample. The beamline is available now to the NSLS general user population.

Three-dimensional structures derived from X-ray diffraction of protein crystals provide a wealth of information. Features and interactions important for the function of macromolecules can be deduced and catalytic mechanisms postulated. Still, many questions can remain, for example regarding metal oxidation states and the interpretation of 'mystery density', i.e. ambiguous or unknown features within the electron density maps, especially at $ 2 Å resolutions typical of most macromolecular structures. Beamline X26-C at the National Synchrotron Light Source (NSLS), Brookhaven National Laboratory (BNL), provides researchers with the opportunity to not only determine the atomic structure of their samples but also to explore the electronic and vibrational characteristics of the sample before, during and after X-ray diffraction data collection. When samples are maintained under cryo-conditions, an opportunity to promote and follow photochemical reactions in situ as a function of X-ray exposure is also provided. Plans are in place to further expand the capabilities at beamline X26-C and to develop beamlines at NSLS-II, currently under construction at BNL, which will provide users access to a wide array of complementary spectroscopic methods in addition to high-quality X-ray diffraction data.

This manuscript chronicles the evolution of software used originally to control a diffractometer at a macromolecular crystallography beamline. The system has been augmented and rewritten. A modular and carefully organized suite of programs now handles the whole experimental environment from a single vantage point. It provides automatic logging of the experiment and communication with the user, all the way from an initial proposal to perform the work to the end of data collection. This has included construction of a relational database to organize all details of the experiment and incorporation of a robotic specimen changer to provide automation for high-throughput applications.

Advances in detectors and computational technologies provide new opportunities for applied research and the fundamental sciences. Concurrently, dramatic increases in the three V's (Volume, Velocity, and Variety) of experimental data and the scale of computational tasks produced the demand for new real-time processing systems at experimental facilities.Recently, this demand was addressed by the Spark-MPI approach connecting the Spark data-intensive platform with the MPI high-performance framework. In contrast with existing data management and analytics systems, Spark introduced a new middleware based on resilient distributed datasets (RDDs), which decoupled various data sources from high-level processing algorithms. The RDD middleware significantly advanced the scope of data-intensive applications, spreading from SQL queries to machine learning to graph processing. Spark-MPI further extended the Spark ecosystem with the MPI applications using the Process Management Interface. The paper explores this integrated platform within the context of online ptychographic and tomographic reconstruction pipelines.

No abstract