We consider operator growth for generic large-N gauge theories at finite temperature. Our analysis is performed in terms of Fourier modes, which do not mix with other operators as time evolves, and whose correlation functions are determined by their two-point functions alone, at leading order in the large-N limit. The algebra of these modes allows for a simple analysis of the operators with whom the initial operator mixes over time, and guarantees the existence of boundary CFT operators closing the bulk Poincaré algebra, describing the experience of infalling observers. We discuss several existing approaches to operator growth, such as number operators, proper energies, the many-body recursion method, quantum circuit complexity, and comment on its relation to classical chaos in black hole dynamics. The analysis evades the bulk vs boundary dichotomy and shows that all such approaches are the same at both sides of the holographic duality, a statement that simply rests on the equality between operator evolution itself. In the way, we show all these approaches have a natural formulation in terms of the Gelfand-Naimark-Segal (GNS) construction, which maps operator evolution to a more conventional quantum state evolution, and provides an extension of the notion of operator growth to QFT.