We describe the design, construction, and characterization of a medium-finesse Fabry-Pérot cavity for simultaneous frequency stabilization of two lasers operating at 960 and 780 nm wavelengths, respectively. The lasers are applied in experiments with ultracold rubidium Rydberg atoms, for which a combined laser linewidth similar to the natural Rydberg linewidth (≈ 10 kHz) is desired. The cavity, with a finesse of ≈ 1500, is used to reduce the linewidth of the lasers to below this level. By using a spacer made of ultra low expansion (ULE ® ) glass with active temperature stabilization, the residual frequency drift is limited to 1 MHz/day. The design optimizes for ease of construction, robustness, and affordability.