Boundary operators are gauge invariant operators whose form factors correspond to boundary contributions of BCFW shifts. In gauge theory, the boundary operators contain infinite series, which are constrained by gauge symmetry. We compute the boundary operators of all possible BCFW shifts in Yang-Mills theory and QCD, and show that the infinite series can be elegantly organized into Wilson lines, which are natural building blocks for non-local gauge invariant operators. We comment on their connection to jet functions and gauge invariant off-shell amplitudes. We also verify our results by studying various BCFW shifts of four and five-point amplitudes.