Developers of application codes for massively parallel computer systems face daunting performance tuning and optimization problems that must be solved if massively parallel systems are to fulfill their promise. Recording and analyzing the dynamics of application program, system software, and hardware interac.tions is the key to understanding and the prerequisite to performance tuning, but this instrumentation and analysis must not unduly perturb program execution. Pablo is a performance analysis environment designed t o provide unobtrusive performance data capture, analysis, and presentation across a wide variety of scalable parallel systems. Current efforts include dynamic statistical clustering t o reduce the volume of data that must be captured and complete performance data immersion via head-mounted displays.

Recording and analyzing the dynamics of application program, system software, and hardware interactions are the keys to understanding and tuning the performance of massively parallel systems. Because massively parallel systems contain hundreds or thousands of processors, each potentially with many dynamic performance metrics, the performance data occupy a sparsely populated, high-dimensional space. These dynamic performance metrics for each processor dene a group of evolving, n-dimensional points. Understanding the dynamic \shape" of these metric paths is only possible if one can examine multiple projections. We have implemented an immersive virtual world, called Avatar, that shows all the possible three-dimensional projections of a sparsely populated, n-dimensional metric space. The presentation metaphor is a three-dimensional generalization of a two-dimensional scatterplot matrix. Users can walk inside a single scatterplot cube, y about the cube of scatterplot cubes, control selected characteristics of the scatterplot display, listen to the sounds of statistical data attributes, and interactively modify application behavior and performance in real time.