(1) Physical activity is one of the most influencing factors in people’ quality of life. Likewise, the costaleros of the Holy Week of Andalusia (Spain) carry out an important effort with high intensity during an extended time without any preparation. This study was the aim of knowing the intensity of the physical activity practiced by the costaleros in relation to their quality of life. (2) A transversal study was carried out with 1057 costaleros in Andalusia (Spain), where 930 were male and 127 female, between the ages of 18–61 years old (31.26 ± 7.60). For this purpose, descriptive, inferential, and correlative analyses were developed. Accelerometers (ActiGraph) were used during the procession to know the intensity of physical activity and the SF-36 test to know the self-perceived state of health and quality of life. (3) The intensity of physical activity practiced by costaleros is moderate, and it is related with their quality of life. In addition, positive associations are found between general health and physical activity. (4) Participants’ quality of life is associated with physical activity and freedom from injury. In addition, the measurement by accelerometry provides real data on the intensity of the effort made.