Ballistic clothing is a crucial safety element that helps to save lives of its wearers. According to the protection level, the ballistic vest is reinforced by one or two armor panels designed to resist to attacks destinated to upper parts of the body. The resistance behavior of ballistic body armor depends on several factors. The most important factors are related to the ballistic fabric properties such as the type of fabric, its thickness, its structure, its strength and strain, and the type of the employed resin. In this review paper, the factors influencing the ballistic performances of soft armor panel and hard armor panel are presented. They are investigated based on an in-depth review of previous research studies and experimental test results. This paper also reports the various methods developed by the researchers to improve the performance of ballistic materials. Thus, ballistic material manufactures and researchers could find some guidelines on promising materials and structures that can be used to build efficient body armors.