Java is receiving increasing attention as the most popular platform for distributed and collaborative computing. However, it is still subject to significant performance drawbacks in comparison to other programming languages such as C and Fortran. This paper represents the current status of our ongoing project which intends to conduct a detailed experimental evaluation on the suitability of Java in these environments, with particular focus on its message-passing performance for one-to-one as well as one-to-many and manyto-many data exchange patterns. We also emphasize both methodology and evaluation guidelines in order to ensure reproducibility, sound interpretation, and comparative analysis of performance results. Some of the important parameters which characterize the communication performance of MPI and Java-MPI such as latency, asymptotic bandwidth and N-half are investigated. In addition, we introduce two different types of pipeline effects -intra-message and inter-message -that have significant influence on the message-passing performance. For this purpose we have developed a low-level message-passing benchmark suite, which we have used to evaluate and compare different message-passing environments on the IBM SP-2.