Inspired by biological water channels (e.g., aquaporin with an hourglass shape), there is a growing interest in the use of conical nanopores for water purification; however, the surface roughness of conical nanopores has not been considered in the literature. In this work, a molecular dynamics simulation study is conducted to investigate water permeation through conical nanopores by considering both surface roughness and chemistry. In hydrophilic alumina nanopores, water permeability is found to increase with increasing surface roughness; however, a reverse trend is observed in hydrophobic carbon nanopores. Comprehensive microscopic analysis reveals that surface roughness in the carbon nanopores induces multiple high-energy barriers and frequent forming/breaking of hydrogen bonding network, which impedes water permeation. In the alumina nanopores, water becomes more bulk-like with increasing surface roughness, thus enhancing water permeation. The molecular insights from this simulation study provide quantitative understanding of water permeation in hydrophilic and hydrophobic conical nanopores and unravel the complex interplay between surface roughness and chemistry, facilitating the design of new materials for water purification.