In social insects, the nests of the same species can show a large difference in size and shape. Despite these large variations, the nests share the same substructures, some appearing during nest growth. In ants, the interplay between nest size and digging activity leads to two successive morphological transitions from circular to branched shapes (budding along the perimeter of the circular cavity and tunnelling of the galleries). Like several other self-organized collective behaviours, this phenomenon, as well as the entire nest-digging process, is thought to be modulated by environmental properties. The present study investigates the effect of excavated substrate on the nest morphogenesis and the morphological transitions by using two materials with different cohesions. Here, we show that the two morphological transitions occur more frequently with a cohesive substrate than with a granular one: 96 per cent of cohesive experiments showed both transitions, whereas only 50 per cent did in granular experiments. We found that transitions and excavation cessation follow area -response thresholds: the shape transitions take place and the digging activity stops when the dug area reaches the corresponding threshold values. The shape transition thresholds are lower with the cohesive substrate and that of stopping digging is independent of nest shape and material. According to simulations, the experimental frequencies of transitions found their origin in the competition between transitions and activity cessation and in the difference between the transition threshold values of each substrate. Our results demonstrate how the substrate properties modulate the collective response and lead to various patterns. Considering the non-specific mechanisms at work, such effects of substrate coarseness have their counterparts in various collective behaviours, generating alternative patterns to colonize and exploit the environment.