In this work, we demonstrate a compact toolkit of inverse-designed, topologically optimized silicon photonic devices that are arranged in a “plug-and-play” fashion to realize many different photonic integrated circuits, both passive and active, each with a small footprint. The silicon-on-insulator 1550-nm toolkit contains a 2 × 2 3-dB splitter/combiner, a 2 × 2 waveguide crossover, and a 2 × 2 all-forward add–drop resonator. The resonator can become a 2 × 2 electro-optical crossbar switch by means of the thermo-optical effect, phase-change cladding, or free-carrier injection. For each of the ten circuits demonstrated in this work, the toolkit of photonic devices enables the compact circuit to achieve low insertion loss and low crosstalk. By adopting the sophisticated inverse-design approach, the design structure, shape, and sizing of each individual device can be made more flexible to better suit the architecture of the greater circuit. For a compact architecture, we present a unified, parallel waveguide circuit framework into which the devices are designed to fit seamlessly, thus enabling low-complexity circuit design.