In recent years, reputation management schemes have been proposed as promising solutions to alleviate the blindness during peer selection in distributed P2P environment where malicious peers coexist with honest ones. They indeed provide incentives for peers to contribute more resources to the system and thus promote the whole system performance. But few of them have been implemented practically since they still suffer from various security threats, such as collusion, Sybil attack and so on. Therefore, how to detect malicious peers plays a critical role in the successful work of these mechanisms, and it will also be our focus in this paper. Firstly, we define malicious peers and show their influence on the system performance. Secondly, based on Multiscale Principal Component Analysis (MSPCA) and control chart, a Subspace based MAlicious peeRs deTecting algorithm (SMART) is brought forward. SMART first reconstructs the original reputation matrix based on subspace method, and then finds malicious peers out based on Shewhart control chart. Finally, simulation results indicate that SMART can detect malicious peers efficiently and accurately.