In Brazil, there are around 10 million hard of hearing and deaf people. However, the majority of Brazilians are not fluent in the Brazilian Sign Language (Libras). Many members of the hearing impaired and deaf community have communication issues in everyday life situations. Technological solutions can aid in mitigating this problem. This work proposes a semi-supervised method to identify and classify signals in Libras from Youtube videos. The routine starts by segmenting the videos through a measure of movement intensity. We catalog the video segments according to the corresponding subtitles, which we extract by employing Optical Character Recognition (OCR) if embedded. It composes an ad-hoc dataset that we use to train a Libras recognition system. A Convolutional Neural Network (CNN) performs feature extraction frame-by-frame, and a Recurrent Neural Network (RNN) models the time correlation between the features, thus classifying the signal. The proposed method can achieve accuracy up to 61.6% in the ad-hoc dataset used in this work.