Single-molecule experiments have changed the way we investigate the physical world but data analysis is typically time-consuming and prone to human bias. Here, we present Deep-LASI (Deep-Learning Assisted Single-molecule Imaging analysis), a software package consisting of an ensemble of deep neural networks to rapidly analyze single-, two- and three-color single-molecule data, in particular from single-molecule Foerster Resonance Energy Transfer (FRET) experiments. Deep-LASI automatically sorts single molecule traces, determines FRET correction factors and classifies the state transitions of dynamic traces, all in ~20-100 ms per trajectory. We thoroughly benchmarked Deep-LASI using ground truth simulations as well as experimental data analyzed manually by an expert user and compared the results with a conventional Hidden Markov Model analysis. We illustrate the capabilities of the technique using a highly tunable L-shaped DNA origami structure and use Deep-LASI to perform titrations, analyze protein conformational dynamics and demonstrate its versatility for analyzing both total internal reflection fluorescence microscopy and confocal smFRET data.