There is currently a lack of understanding how genetic background and sex differences attribute to the heterogeneity of obsessive-compulsive disorder (OCD). An animal model of compulsive-like behaviors has been developed through bidirectional selection of house mice (Mus musculus) for high (big cotton nests; BIG mice) and low levels (small nests; SMALL mice) of nest-building behavior. The BIG male strains have predictive and face validity as a spontaneous animal model of OCD. Here, we evaluated compulsive-, anxiety-, cognitive-, and depression-like behaviors among male and proestrus female replicate strains each of BIG (BIG1, BIG2) and SMALL (SML1, SML2) nest-builders, and randomly-bred Controls (C1, C2). BIG1 and BIG2 males and females had higher nesting scores when compared to SMALL and Control strains. Male BIG1 and BIG2 strains showed more compulsive-like nesting than BIG1 and BIG2 proestrus females, which was not observed among the other strains. Nesting scores were also different between BIG replicate male strains. A similar pattern was observed in the compulsive-like marble burying behavior with BIG strains burying more marbles than SMALL and Control strains. Significant replicate and sex differences were also observed in marble burying among the BIG strains. The open field test revealed replicate effects while the BIG strains showed less anxiety-like behavior in the elevated plus maze test compared to the SMALL strains. For novel object recognition only the Control strains showed replicate and sex differences. In the depression-like forced swim test proestrus females demonstrated less depression-like behavior than males. BIG and SMALL nest-building strains had a higher corticosterone stress response than the Control strains. Together these results indicate a strong interplay of genetic background and sex in influencing expression of behaviors in our compulsive-like mouse model. These results are in congruence with the clinical heterogeneity of OCD.