Lithium niobate on insulator (LNOI) has emerged as a promising platform for photonic integrated circuits, with a fast‐growing toolbox of components. This paper proposes, designs, and experimentally demonstrates compact subwavelength grating (SWG) waveguides on an LNOI platform for on‐chip mode and polarization manipulation. To overcome the limitation of waveguide fabrication, the SWGs are designed and formed on a silicon nitride thin film deposited onto the surface of LNOI chip. As proof‐of‐concept devices, the SWG‐based spatial mode filters (including a TE1‐mode‐pass filter and a TE2‐mode‐pass filter) and a TM‐pass polarizer are fabricated successfully on the same chip, with the device lengths of only ≈50 μm. The measured insertion losses for the devices are lower than 3.1 dB, with high extinction ratio larger than 30 dB, at a wavelength of 1550 nm. The proposed and demonstrated SWGs can serve as important building blocks in a series of mode and polarization handling devices for LNOI integrated photonics.