An important study in multi-agent systems is the development of cooperative behavior between agents that have a shared goal. In this paper an example of the predator-prey problem is studied in which four predator agents, using the reinforcement learning method, in an attempt to collectively achieve the task of surrounding one prey agent. First, we describe the structure of the predator agents and the prey agent with their state sensing capability, the action learning method and the action choosing mechanism. Next we study two cooperative behavior mechanisms between agents of multi-agent systems in predator-prey problem. Finally, we demonstrate that cooperative agents outperform agents without cooperative behavior according to simulations of the predator-prey problem and display the corresponding experimental results. 0-7803-8963-8/05/$20.00 ©2005 IEEE.