In this study, we explored the manipulation of optical properties in the terahertz (THz) frequency band of radio-frequency (RF) sputtered indium tin oxide (ITO) thin films on highly resistive silicon substrate by rapid thermal annealing (RTA). The optical constants of as-deposited and RTA-processed ITO films annealed at 400 °C, 600 °C and 800 °C are determined in the frequency range of 0.2 to 1.0 THz. The transmittance can be changed from ~27% for as-deposited to ~10% and ~39% for ITO films heat-treated at different annealing temperatures (Ta’s). Such variations of optical properties in the far infrared for the samples under study are correlated with their mobility and carrier concentration, which are extracted from Drude–Smith modeling of THz conductivity with plasma frequency, scattering time and the c-parameters as fitting parameters. Resistivities of the films are in the range of 10−3 to 10−4 Ω-cm, confirming that annealed ITO films can potentially be used as transparent conducting electrodes for photonic devices operating at THz frequencies. The highest mobility, μ = 47 cm2/V∙s, with carrier concentration, Nc = 1.31 × 1021 cm−3, was observed for ITO films annealed at Ta = 600 °C. The scattering times of the samples were in the range of 8–21 fs, with c-values of −0.63 to −0.87, indicating strong backscattering of the carriers, mainly by grain boundaries in the polycrystalline film. To better understand the nature of these films, we have also characterized the surface morphology, microscopic structural properties and chemical composition of as-deposited and RTA-processed ITO thin films. For comparison, we have summarized the optical properties of ITO films sputtered onto fused silica substrates, as-deposited and RTA-annealed, in the visible transparency window of 400–800 nm. The optical bandgaps of the ITO thin films were evaluated with a Tauc plot from the absorption spectra.