We construct a colored operad whose category of algebras is the category of algebraic quantum field theories. This is achieved by a construction that depends on the choice of a category, whose objects provide the operad colors, equipped with an additional structure that we call an orthogonality relation. This allows us to describe different types of quantum field theories, including theories on a fixed Lorentzian manifold, locally covariant theories and also chiral conformal and Euclidean theories. Moreover, because the colored operad depends functorially on the orthogonal category, we obtain adjunctions between categories of different types of quantum field theories. These include novel and interesting constructions, such as timeslicification and local-to-global extensions of quantum field theories. We compare the latter to Fredenhagen's universal algebra. Report no.: ZMP-HH/17-26, Hamburger Beiträge zur Mathematik Nr. 682Keywords: algebraic quantum field theory, locally covariant quantum field theory, colored operads, change of color adjunctions, Fredenhagen's universal algebra MSC 2010: 81Txx, 18D50