Abstract. We present a study of density estimation, the conversion of discrete particle positions to a continuous field of particle density defined over a 3D Cartesian grid. The study features a methodology for evaluating the accuracy and performance of various density estimation methods, results of that evaluation for four density estimators, and a large-scale parallel algorithm for a self-adaptive method that computes a Voronoi tessellation as an intermediate step. We demonstrate the performance and scalability of our parallel algorithm on a supercomputer when estimating the density of 100 million particles over 500 billion grid points.Key words. Density estimation, cloud in cell, smoothed particle hydrodynamics, Voronoi tessellation, nearest grid point, triangular shaped clouds AMS subject classifications. 51-04, 68-04, 70-04, 85-04. 1. Introduction. In the context of scientific data analysis, density estimation is a transformation from discrete particle data to a continuous density function defined over a 3D field. This field can be interpolated, differentiated, and integrated: operations not possible in the particles' original discrete form. Moreover, the density field is discretized by a regular grid, which offers several advantages. (1) It is compact, in that x, y, z coordinates of the grid points are defined implicitly and need not be stored. (2) Applications using a regular grid are readily parallelizable because subdomain decomposition and processor assignment are also regular and implicit. (3) A regular grid is the most common data model for scientific data analysis and visualization algorithms. For example, most of the volume rendering literature in the past 25 years targets regular grids [9,24,37]; far less exists for adaptively refined grids [23] and unstructured meshes [30].Density estimation is a fundamental step needed whenever a discrete particle dataset is sampled over a continuous field. Our research is motivated by cosmology and astrophysics, but many other applications exist, for example estimating population density in geospatial applications or electron charge density in computational chemistry. Density estimation is also a key visualization step when an algorithm calls for a regular scalar field but particles are provided instead. For example, atom positions from molecular dynamics simulations may be converted to a grid before rendering an isosurface of atomic density.One of the earliest and arguably most popular density estimators, cloud in cell (CIC), was introduced 45 years ago [7]. Since then, smoothed particle hydrodynamics (SPH) and tessellation (TESS) methods have emerged. Combinations and variations also exist: for example, characteristics such as the window shape of CIC and the adaptivity of SPH can be combined. Nevertheless, two questions remain unanswered: Which density estimators are appropriate for a given problem domain and computational budget, and how can the estimator of choice be scaled to today's problem sizes? In this paper, we examine the first question by designi...

Cloud platforms have emerged as a leading solution for computation. In the meantime, large computations have shifted from big parallel tasks to workflows of smaller tasks with data dependencies between them. Task placement is a major issue on Cloud platforms, especially considering the impact of data exchanges on cost and makespan. In this paper, we investigate the consequences of network contention regarding the use of existing scheduling policies on DaaS-based platforms (DaaS for Data as a Service). We show here that the legacy algorithms use inefficient network models. We then modify those algorithms using a new model inspired by DaaS-based Cloud platforms. Thus, we manage to statically pack tasks so that a batch scheduler could deploy many real-time submitted workflows on a dynamic Cloud platform. Simulations of Fork-Join workflows deployment using SimGrid show that our algorithm reduces computation time as well as deployment costs.

Abstract. GridRPC is an international standard of the Open Grid Forum defining an API designed to allow applications to be submitted in a seamless way on large scale, heterogeneous and geographically distributed computing platforms. First versions of the standard did not take into account any data management feature. Data were parameters of the Remote Procedure calls, without any possibility to prefetch them, to use persistence, replication, external sources, etc. , and making GridRPC codes middleware dependent. The data extension of the standard introduced a short set of functions and data structures to complete the API with simple but powerful data management features. In this paper, we present a modular and extensible implementation of both APIs, which needs only a few developments to be usable with any middleware relying on RPC, and which provides access to numerous and easy to extend protocols and data middleware to access data. Gaining data management functions, it introduces interesting potentiality for optimization that such an approach would provide to large scale applications.

Abstract:With the recent development of commercial Cloud offers, Cloud solutions are today the obvious solution for many computing use-cases. However, high performance scientific computing is still among the few domains where Cloud still raises more issues than it solves. Notably, combining the workflow representation of complex scientific applications with the dynamic allocation of resources in a Cloud environment is still a major challenge. In the meantime, users with monolithic applications are facing challenges when trying to move from classical HPC hardware to elastic platforms. In this paper, we present the structure of an autonomous workflow manager dedicated to IaaS-based Clouds (Infrastructure as a Service) with DaaS storage services (Data as a Service). The solution proposed in this paper fully handles the execution of multiple workflows on a dynamically allocated shared platform. As a proof of concept we validate our solution through a biologic application with the WASABI workflow.