A possible way of realizing large optical space switches is by interconnecting multiple 2x2 switching elements based on integrated semiconductor optical amplifiers (SOAs). In this paper, the switch size limitations are examined, giving a special emphasis to the phase noise, originated by SOAs. Using a DPSK direct detection scheme and multiple optical filters within the switching network, it is shown that the phase noise becomes the most restrictive factor, preventing to connect more than 200 switching elements, for a bit rate in the gigabit/s range.