The present study was conducted to assess the effects of a 4% defatted silkworm (SWM-DEF) meal dietary incorporation into chickens’ diet at different growth stages on growth performances, carcass, and meat quality traits. A total of 90 Ross 308 one-day-old male broiler chickens were randomly allocated into 3 dietary groups of 5 replicated pens/diet (6 chickens/pen). One group was fed a standard soybean-based diet (C); group SWM1 consumed a starter diet (1–10 days of age) including 4% SWM-DEF and then the C diet up to slaughter (11–42 days of age); group SWM2 was fed with the C diet in the starter phase and the 4% SWM-DEF diet up to slaughter. Individual live weight and pen feed intake were determined at the end of each feeding phase: starter, grower, and finisher. Weight gain and FCR were then calculated. At 42 days of age, chickens were slaughtered and carcass traits determined. Leg and breast physico-chemical meat quality was also evaluated. Results confirmed that SWM-DEF could be a possible alternative feed source for chickens since growth performance, carcass, and meat physical traits were overall comparable in the three treatment groups. The feeding stage seemed to play a relevant impact on the sole meat protein content (SWM1 < SMW2 and C; p < 0.001). Concluding, the results available to date indicate that 4% SWM-DEF can be provided to chickens in different growth phases, and the choice of the inclusion period is more an industrial cost-benefit evaluation. From the scientific point of view, however, an administration in the grower-finisher phase rather than in the starter one provided the best meat nutritional quality. Further investigations should focus on the impact on meat fatty acids profile and sensory traits, which are of utmost importance for consumers.