This paper addresses the problem of scheduling concurrent jobs on clusters where application data is stored on the computing nodes. This setting, in which scheduling computations close to their data is crucial for performance, is increasingly common and arises in systems such as MapReduce, Hadoop, and Dryad as well as many grid-computing environments. We argue that data-intensive computation benefits from a fine-grain resource sharing model that differs from the coarser semi-static resource allocations implemented by most existing cluster computing architectures. The problem of scheduling with locality and fairness constraints has not previously been extensively studied under this resource-sharing model.We introduce a powerful and flexible new framework for scheduling concurrent distributed jobs with fine-grain resource sharing. The scheduling problem is mapped to a graph datastructure, where edge weights and capacities encode the competing demands of data locality, fairness, and starvation-freedom, and a standard solver computes the optimal online schedule according to a global cost model. We evaluate our implementation of this framework, which we call Quincy, on a cluster of a few hundred computers using a varied workload of data-and CPU-intensive jobs. We evaluate Quincy against an existing queue-based algorithm and implement several policies for each scheduler, with and without fairness constraints. Quincy gets better fairness when fairness is requested, while substantially improving data locality. The volume of data transferred across the cluster is reduced by up to a factor of 3.9 in our experiments, leading to a throughput increase of up to 40%.

Computer systems increasingly rely on heterogeneity to achieve greater performance, scalability and energy efficiency. Because heterogeneous systems typically comprise multiple execution contexts with different programming abstractions and runtimes, programming them remains extremely challenging. Dandelion is a system designed to address this programmability challenge for data-parallel applications. Dandelion provides a unified programming model for heterogeneous systems that span diverse execution contexts including CPUs, GPUs, FPGAs, and the cloud. It adopts the .NET LINQ (Language INtegrated Query) approach, integrating data-parallel operators into general purpose programming languages such as C# and F#. It therefore provides an expressive data model and native language integration for user-defined functions, enabling programmers to write applications using standard highlevel languages and development tools.Dandelion automatically and transparently distributes data-parallel portions of a program to available computing resources, including compute clusters for distributed execution and CPU and GPU cores of individual nodes for parallel execution. To enable automatic execution of .NET code on GPUs, Dandelion cross-compiles .NET code to CUDA kernels and uses the PTask runtime [85] to manage GPU execution. This paper discusses the design and implementation of Dandelion, focusing on the distributed CPU and GPU implementation. We evaluate the system using a diverse set of workloads.

No abstract

Clouds and the Earth's Radiant Energy System (CERES) is a National Aeronautics and Space Administration (NASA)investigation to examine the role of clouds in the radiative energy flow through the Earth-atmosphere system. The first CERES scanning radiometer was launched on November 27, 1997 into a 35° inclination, 350 km altitude orbit, on the Tropical Rainfall Measuring Mission (TRMM) spacecraft. The CERES instrument consists of a three channel scanning broadband radiometer. The spectral bands measure shortwave (0.3 -5 .t m), window (8 -12 j.m), and total (0.3 -100 jm) radiation reflected or emitted from the Earth-atmosphere system. Each Earth viewing measurement is geolocated to the Earth fixed coordinate system using satellite ephemeris, Earth rotation and geoid, and instrument pointing data. The interactive CERES coastline detection system is used to assess the accuracy of the CERES geolocation process. By analyzing radiative flux gradients at the boundaries of ocean and land masses, the accuracy of the scanner measurement locations may be derived for the CERES/TRMM instrumentlsatellite system. The resulting CERES measurement location errors are within 10% of the nadir footprint size. Precise pointing knowledge of the Visible and Infrared Scanner (VIRS) is required for convolution of cloud properties onto the CERES footprint; initial VIRS coastline results are included.

No abstract