Pilot-Job systems play an important role in supporting distributed scientific computing. They are used to consume more than 700 million CPU hours a year by the Open Science Grid communities, and by processing up to 1 million jobs a day for the ATLAS experiment on the Worldwide LHC Computing Grid. With the increasing importance of task-level parallelism in high-performance computing, Pilot-Job systems are also witnessing an adoption beyond traditional domains. Notwithstanding the growing impact on scientific research, there is no agreement upon a definition of Pilot-Job system and no clear understanding of the underlying abstraction and paradigm. Pilot-Job implementations have proliferated with no shared best practices or open interfaces and little interoperability. Ultimately, this is hindering the realization of the full impact of Pilot-Jobs by limiting their robustness, portability, and maintainability. This paper offers a comprehensive analysis of Pilot-Job systems critically assessing their motivations, evolution, properties, and implementation. The three main contributions of this paper are: (i) an analysis of the motivations and evolution of Pilot-Job systems; (ii) an outline of the Pilot abstraction, its distinguishing logical components and functionalities, its terminology, and its architecture pattern; and (iii) the description of core and auxiliary properties of Pilot-Jobs systems and the analysis of seven exemplar Pilot-Job implementations. Together, these contributions illustrate the Pilot paradigm, its generality, and how it helps to address some challenges in distributed scientific computing.

High performance computing systems have historically been designed to support applications comprised of mostly monolithic, singlejob workloads. Pilot systems decouple workload specification, resource selection, and task execution via job placeholders and late-binding. Pilot systems help to satisfy the resource requirements of workloads comprised of multiple tasks. RADICAL-Pilot (RP) is a modular and extensible Python-based pilot system. In this paper we describe RP's design, architecture and implementation, and characterize its performance. RP is capable of spawning more than 100 tasks/second and supports the steady-state execution of up to 16K concurrent tasks. RP can be used stand-alone, as well as integrated with other application-level tools as a runtime system.

Pilot-Jobs support effective distributed resource utilization, and are arguably one of the most widely-used distributed computing abstractions -as measured by the number and types of applications that use them, as well as the number of production distributed cyberinfrastructures that support them. In spite of broad uptake, there does not exist a well-defined, unifying conceptual model of Pilot-Jobs which can be used to define, compare and contrast different implementations. Often Pilot-Job implementations are strongly coupled to the distributed cyberinfrastructure they were originally designed for. These factors present a barrier to extensibility and interoperability. This paper is an attempt to (i) provide a minimal but complete model (P*) of Pilot-Jobs, (ii) establish the generality of the P* Model by mapping various existing and well known Pilot-Job frameworks such as Condor and DIANE to P*, (iii) derive an interoperable and extensible API for the P* Model (Pilot-API), (iv) validate the implementation of the Pilot-API by concurrently using multiple distinct Pilot-Job frameworks on distinct production distributed cyberinfrastructures, and (v) apply the P* Model to Pilot-Data.

Holoprosencephaly is a pathology of forebrain development characterized by high phenotypic heterogeneity. The disease presents with various clinical manifestations at the cerebral or facial levels. Several genes have been implicated in holoprosencephaly but its genetic basis remains unclear: different transmission patterns have been described including autosomal dominant, recessive and digenic inheritance. Conventional molecular testing approaches result in a very low diagnostic yield and most cases remain unsolved. In our study, we address the possibility that genetically unsolved cases of holoprosencephaly present an oligogenic origin and result from combined inherited mutations in several genes. Twenty-six unrelated families, for whom no genetic cause of holoprosencephaly could be identified in clinical settings [whole exome sequencing and comparative genomic hybridization (CGH)-array analyses], were reanalysed under the hypothesis of oligogenic inheritance. Standard variant analysis was improved with a gene prioritization strategy based on clinical ontologies and gene co-expression networks. Clinical phenotyping and exploration of cross-species similarities were further performed on a family-by-family basis. Statistical validation was performed on 248 ancestrally similar control trios provided by the Genome of the Netherlands project and on 574 ancestrally matched controls provided by the French Exome Project. Variants of clinical interest were identified in 180 genes significantly associated with key pathways of forebrain development including sonic hedgehog (SHH) and primary cilia. Oligogenic events were observed in 10 families and involved both known and novel holoprosencephaly genes including recurrently mutated FAT1, NDST1, COL2A1 and SCUBE2. The incidence of oligogenic combinations was significantly higher in holoprosencephaly patients compared to two control populations (P 5 10 -9 ). We also show that depending on the affected genes, patients present with particular clinical features. This study reports novel disease genes and supports oligogenicity as clinically relevant model in holoprosencephaly. It also highlights key roles of SHH signalling and primary cilia in forebrain development. We hypothesize that distinction between different clinical manifestations of holoprosencephaly lies in the degree of overall functional impact on SHH signalling. Finally, we underline that integrating clinical phenotyping in genetic studies is a powerful tool to specify the clinical relevance of certain mutations.