Atom-light interactions in micro-and nanoscale systems hold great promise for alternative technologies based on integrated emitters and optical modes. We present the design architecture, construction method, and characterization of an all-glass alkali-metal vapor cell with nanometer-scale internal structure. Our cell has a glue-free design that allows versatile optical access, in particular with high numerical aperture optics, and incorporates a compact integrated heating system in the form of an external deposited indium tin oxide layer. By performing spectroscopy in different illumination and detection schemes, we investigate atomic densities and velocity distributions in various nanoscopic landscapes. We apply a two-photon excitation scheme to atoms confined in one dimension within our cells, achieving resonance line widths more than an order of magnitude smaller than the Doppler width. We also demonstrate sub-Doppler line widths for atoms confined in two dimensions to micron-sized channels. Furthermore, we illustrate control over vapor density within our cells through nanoscale confinement alone, which could offer a scalable route towards room-temperature devices with single atoms within an interaction volume. Our design offers a robust platform for miniaturized devices that could easily be combined with integrated photonic circuits.