Background Although gender is an important determinant of health behaviour with males less likely to perform health-protective behaviours, samples in health behaviour research are heavily biased towards females. This study investigated the use of online social network, Facebook, to reach and recruit inactive males to a team-based, social, and gamified physical activity randomised controlled trial. Methods Methodological techniques included a narrative literature review, survey of inactive males ( n = 34) who rated advertisement images and text captions on scales of 1–10, and trial Facebook-delivered recruitment campaigns. Advertisement effectiveness was measured by cost-per-click to the study website, number of expressions of interest, and study enrolments from males. Results Survey results showed that vibrant images of men exercising accompanied by concise captions (< 35 words) were most effective. An advertising campaign incorporating these components achieved a cost-per-click of $0.60, with 80% of n = 50 expressions of interest being from men, a marked improvement from baseline campaigns in which only 11% of expressions of interest were from men. Despite this, men who were recruited through the targeted campaign failed to enrol into the study, primarily due to reluctance to invite friends to join their team. An alternative strategy of encouraging females to invite men boosted male participation from 18% of the sample at baseline to 29% in the targeted recruitment phase. Conclusions Evidence-based approaches can improve Facebook recruitment outcomes, however, there are complex barriers hindering male recruitment to health behaviour studies that may necessitate multi-faceted strategies including involvement of family and friends. Electronic supplementary material The online version of this article (10.1186/s12889-019-7087-4) contains supplementary material, which is available to authorized users.