MEMOPS: Data modelling and automatic code generation

Abstract

SummaryIn recent years the amount of biological data has exploded to the point where much useful information can only be extracted by complex computational analyses. Such analyses are greatly facilitated by metadata standards, both in terms of the ability to compare data originating from different sources, and in terms of exchanging data in standard forms, e.g. when running processes on a distributed computing infrastructure. However, standards thrive on stability whereas science tends to constantly move, with new methods being developed and old ones modified. Therefore maintaining both metadata standards, and all the code that is required to make them useful, is a non-trivial problem. Memops is a framework that uses an abstract definition of the metadata (described in UML) to generate internal data structures and subroutine libraries for data access (application programming interfaces -APIs -currently in Python, C and Java) and data storage (in XML files or databases). For the individual project these libraries obviate the need for writing code for input parsing, validity checking or output. Memops also ensures that the code is always internally consistent, massively reducing the need for code reorganisation. Across a scientific domain a Memops-supported data model makes it easier to support complex standards that can capture all the data produced in a scientific area, share them among all programs in a complex software pipeline, and carry them forward to deposition in an archive. The principles behind the Memops generation code will be presented, along with example applications in Nuclear Magnetic Resonance (NMR) spectroscopy and structural biology. IntroductionIn recent times, the combination of digitization, high-throughput approaches and modern computing techniques has revolutionized the relationship between scientists and data in terms of size and access. These advances present great opportunities but also create considerable problems. Most data now exists in electronic form at some point in its life, and it is therefore extremely important that data can be passed seamlessly between the many different programs that might be used to process and analyse it. If all scientific software was always written to some common data standard then this would not be difficult. In practice, however, this is a non-trivial problem. Science is primarily driven by the need to generate results rather than conform to standards, even if such standards existed and could be agreed upon in constantly evolving fields. The need for standards remains, however. As high throughput methodologies have proliferated, and networks have made it increasingly simple to move data to wherever it is needed, there has been increased interest in defining data standards across a large number of fields where there are immense amounts of data that need to be organised and exploited. Recent reviews by Brazma et al.,[1] on data standards and by Swertz and Jansen [2] on software infrastructure give a good account of both current efforts and the underlying...

CcpNmr AnalysisAssign: a flexible platform for integrated NMR analysis

“…Its development was aimed to alleviate some of the limitations of these two programs whilst combining their desirable aspects. Analysis was built on top of the CCPN data model (Fogh et al 2010) and its data access libraries, which determined the program’s philosophy. This exact modelling of all the data contained within the CCPN project ensures that its persistence is guaranteed and its interpretation is precisely defined.…”

“…Moreover, we have created a so-called “wrapper layer” around the main CCPN data API (Fogh et al 2010) to enable access to the data via a simple, Python-based, command interface. Using this wrapper layer, a user can create their own macros in a language that spectroscopists understand.…”

Straightforward and complete deposition of NMR data to the PDBe

“…For software development, it is more important that data can be transformed directly and unambiguously between files and data structures in memory, and that the data consistency and validity can be assured. This is where the UML-based CCPN data model excels (Fogh et al 2010 ). CCPN projects consist of many XML files, which are less intelligible to humans, but are read and written directly by the subroutine libraries that come with the CCPN implementation.…”

“…All the code is written in Python and uses the CCPN API (Application Programming Interface) libraries to read CCPN XML (eXtensible Markup Language) project files into Python objects (Fogh et al 2010 ). Reading and writing of external data files (for example, coordinates from PDB files, or NMR restraints files from various formats) are performed using FormatConverter libraries (Vranken et al 2005 ).…”

“…To reduce and simplify the extra work required by data depositors, we have developed a complete and easy-to-use set of tools for depositing NMR data for biological macromolecules. These desktop and web-based software tools use the framework provided by the Collaborative Computational Project for NMR (CCPN; Fogh et al 2002 , 2005 , 2010 ) to gather all the mandatory data together in a single CCPN project where all data is stored in an interlinked and consistent way. These CCPN projects can then be deposited using AutoDep at PDBe (Velankar et al 2010 ) or exported as full NMR-STAR files ready to be uploaded to ADIT-NMR at BMRB or PDBj-BMRB (Ulrich et al 2008 ) (Fig.…”