Coral reefs are complex habitats that support a diversity of fish species, some of which live in specialised relationships with particular coral taxa. The ecology and social organisation of these habitat specialists can be shaped by characteristics of their host coral. Here, we quantified the distribution, density, size, dispersion and health of the gorgonian sea whip Junceella fragilis and examined their influence on population density and group size of the sea whip goby Bryaninops amplus. Visual surveys of sea whips and resident gobies were conducted in 30 m × 4 m belt transects at 16 fringing reef sites in the Palm Island Group, Queensland, Australia, across 3 depths (5, 10 and 15 m) and 4 replicates. To examine if distributions of J. fragilis and their goby inhabitants were related to current speed, a current meter was deployed at each site and depth for a 6 d period. The spatial distribution and density of B. amplus positively related to that of sea whips, and group size positively related to sea whip surface area. B. amplus density positively related to current speed, and group sizes were highest on solitary sea whips in sandy habitats. Surprisingly, group sizes were not affected by sea whip condition, with large groups found on sea whips with high proportions of damaged tissue. Host distribution, configuration and size are clearly major determinants of sea whip goby density and group size. While goby populations may ultimately be threatened by the loss of their host, they do appear to be resilient to the early stages of degradation.