Using profinite Galois descent, we compute the Brauer group of the $K(1)$-local category relative to Morava E-theory. At odd primes this group is generated by a cyclic algebra formed using any primitive $(p-1)$st root of unity, but at the prime two is a group of order $32$ with nontrivial extensions; we give explicit descriptions of the generators, and consider their images in the Brauer group of $KO$. Along the way, we compute the relative Brauer group of completed $KO$, using the étale locally trivial Brauer group of Antieau, Meier and Stojanoska.