We study operators in four-dimensional gauge theories which are localized on a straight line, create electric and magnetic flux, and in the UV limit break the conformal invariance in the minimal possible way. We call them Wilson-'t Hooft operators, since in the purely electric case they reduce to the wellknown Wilson loops, while in general they may carry 't Hooft magnetic flux. We show that to any such operator one can associate a maximally symmetric boundary condition for gauge fields on AdS 2 E S 2 . We show that Wilson-'t Hooft operators are classified by a pair of weights (electric and magnetic) for the gauge group and its magnetic dual, modulo the action of the Weyl group. If the magnetic weight does not belong to the coroot lattice of the gauge group, the corresponding operator is topologically nontrivial (carries nonvanishing 't Hooft magnetic flux). We explain how the spectrum of Wilson-'t Hooft operators transforms under the shift of the -angle by 2. We show that, depending on the gauge group, either SL2; Z or one of its congruence subgroups acts in a natural way on the set of Wilson-'t Hooft operators. This can be regarded as evidence for the S-duality of N 4 super-Yang-Mills theory. We also compute the one-point function of the stress-energy tensor in the presence of a Wilson-'t Hooft operator at weak coupling.