Heating of virialized gas by streaming cosmic rays (CRs) may be energetically important in galaxy halos, groups and clusters. We present a linear thermal stability analysis of plasmas heated by streaming CRs. We separately treat equilibria with and without background gradients, and with and without gravity. We include both CR streaming and diffusion along the magnetic-field direction. Thermal stability depends strongly on the ratio of CR pressure to gas pressure, which determines whether modes are isobaric or isochoric. Modes with k · B 0 are strongly affected by CR diffusion. When the streaming time is shorter than the CR diffusion time, thermally unstable modes (with k · B 0) are waves propagating at a speed ∝ the Alfvén speed. Halo gas in photoionization equilibrium is thermally stable independent of CR pressure, while gas in collisional ionization equilbrium is unstable for physically realistic parameters. In gravitationally stratified plasmas, the oscillation frequency of thermally overstable modes can be higher in the presence of CR streaming than the buoyancy/free-fall frequency. This may modify the critical t cool /t ff at which multiphase gas is present. The criterion for convective instability of a stratified, CR-heated medium can be written in the familiar Schwarzschild form ds eff /dz < 0, where s eff is an effective entropy involving the gas and CR pressures. We discuss the implications of our results for the thermal evolution and multiphase structure of galaxy halos, groups and clusters.