One of the most intriguing observations of recurrent neural circuits is their flexibility. Seemingly, this flexibility extends far beyond the ability to learn, but includes the ability to use learned procedures to respond to novel situations. Here, we report that this flexibility arises from the synergistic interplay between recurrent mutual excitation and recurrent mutual inhibition. Specifically, we show that mutual inhibition is critical in expanding the functionality of the circuit, far beyond what feedback inhibition alone can accomplish. By taking advantage of dynamical systems theory and bifurcation analysis, we show mutual inhibition doubles the number of cusp bifurcations in the system in small neural circuits. As a concrete example, we build a simulation model of a class of functional motifs we call Coupled Recurrent inhibitory and Recurrent excitatory Loops (CRIRELs). These CRIRELs have the advantage of being multi-functional, performing a plethora of functions, including decisions, switches, toggles, central pattern generators, depending solely on the input type. We then use bifurcation theory to show how mutual inhibition gives rise to this broad repertoire of possible functions. Finally, we demonstrate how this trend also holds for larger networks, and how mutual inhibition greatly expands the amount of information a recurrent network can hold.