It is well established that spontaneous activity in the developing mammalian brain plays a fundamental role in setting up the precise connectivity found in mature sensory circuits. Experiments that produce abnormal activity or that systematically alter neural firing patterns during periods of circuit development strongly suggest that the specific patterns and the degree of correlation in firing may contribute in an instructive manner to circuit refinement. In fish and amphibians, unlike amniotic vertebrates, sensory input directly drives patterned activity during the period of initial projection outgrowth and innervation. Experiments combining sensory stimulation with live imaging, which can be performed non-invasively in these simple vertebrate models, have provided important insights into the mechanisms by which neurons read out and respond to activity patterns. This article reviews the classic and recent literature on spontaneous and evoked activity-dependent circuit refinement in sensory systems and formalizes a set of mechanistic rules for the transformation of patterned activity into accurate neuronal connectivity in the developing brain.