Aluminum, as a metallic material for plasmonics, is of great interest because it extends the applications of surface plasmon resonance into the ultraviolet (UV) region and is superior to noble metals in natural abundance, cost, and compatibility with modern semiconductor fabrication processes. Ultrasmooth single-crystalline metallic films are beneficial for the fabrication of high-definition plasmonic nanostructures, especially complex integrated nanocircuits. The absence of surface corrugation and crystal boundaries also guarantees superior optical properties and applications in nanolasers. Here, we present UV to near-infrared plasmonic resonance of single-crystalline aluminum nanoslits and nanoholes. The high-definition nanostructures are fabricated with focused ion-beam milling into an ultrasmooth single-crystalline aluminum film grown on a semiconducting GaAs substrate with a molecular beam epitaxy method. The single-crystalline aluminum film shows improved reflectivity and reduced two-photon photoluminescence (TPPL) due to the ultrasmooth surface. Both linear scattering and nonlinear TPPL are studied in detail. The nanoslit arrays show clear Fano-like resonance, and the nanoholes are found to support both photonic modes and localized surface plasmon resonance. We also found that TPPL generation is more efficient when the excitation polarization is parallel rather than perpendicular to the edge of the aluminum film. Such a counterintuitive phenomenon is attributed to the high refractive index of the GaAs substrate. We show that the polarization of TPPL from aluminum preserves the excitation polarization and is independent of the crystal orientation of the film or substrate. Our study gains insight into the optical property of aluminum nanostructures on a high-index semiconducting GaAs substrate and illustrates a practical route to implement plasmonic devices onto semiconductors for future hybrid nanodevices.