This paper introduces a novel deep learning approach to the semantic segmentation of shoreline environments with a high frames-per-second (fps) performance, making the approach readily applicable to autonomous navigation for Unmanned Surface Vehicles (USV). The proposed ShorelineNet is an efficient deep neural network of high performance relying only on visual input. ShorelineNet uses monocular visual input to produce accurate shoreline separation and obstacle detection compared to the state-of-the-art, and achieves this with realtime performance. Experimental validation on a challenging multi-modal maritime obstacle detection dataset, the MODD2 dataset, achieves a much faster inference (25fps on an NVIDIA Tesla K80 and 6fps on a CPU) with respect to the recent stateof-the-art methods, while keeping the performance equally high (68.8% F-score). This makes ShorelineNet a robust and effective model to be used for a reliable USV navigation that require realtime and high-performance semantic segmentation of maritime environments.