Nanostructures that feature nonreciprocal light transmission are highly desirable building blocks for realizing photonic integrated circuits. Here, a simple and ultracompact photonic‐crystal structure, where a waveguide is coupled to a single nanocavity, is proposed and experimentally demonstrated, showing very efficient optical diode functionality. The key novelty of the structure is the use of cavity‐enhanced material nonlinearities in combination with spatial symmetry breaking and a Fano resonance to realize nonreciprocal propagation effects at ultralow power and with good wavelength tunability. The nonlinearity of the device relies on ultrafast carrier dynamics, rather than the thermal effects usually considered, allowing the demonstration of nonreciprocal operation at a bit‐rate of 10 Gbit s−1 with a low energy consumption of 4.5 fJ bit−1.