Logging of application and system information is crucial to running any service. Whether it is looking for trends in use, investigating specific problems or analysing security incidents, logs are an essential source of information for any service manager or system administrator. However, distributed applications produce large volumes of logging information from a number of disparate, but related, daemons running on an increasing number of machines. In this scenario, log analysis with traditional command line tools such as awk and grep is hugely inefficient. In addition, presenting the results of analysis using tools like spreadsheets is a time consuming process for a busy administrator. At STFC, the Scientific Computing Department has adopted the ELK stack (www.elastic.co) for collating of logs from the mass storage system. The ELK stack is built from three open source components -Elasticsearch, Logstash, and Kibana. It is currently gathering information from about 300 machines at the Rutherford Appleton Laboratory covering both the WLCG Tier 1 and local facilities storage systems. In this presentation, we discuss the evolution of Elasticsearch within STFC and present specific use cases where it has clear advantages over traditional methods. We also compare it with the previous logging system developed at CERN (DLF) and show that the ELK stack is a more generically useful toolset. Finally, we demonstrate its usefulness for looking at both trends and specific event analysis.

Much interest has been shown in the Ceph storage platform and this paper presents an introduction to Ceph, object stores, and the initial results of testing Ceph for possible use at large sites. We look at its performance when used as a disk cache for a traditional HSM and compare its performance with competing systems. In addition, we present a model comparing hardware usage for various Ceph configurations with RAL's existing CASTOR instance and discuss future plans for Ceph deployment at RAL.