Braille system is widely used worldwide for communication by visually impaired people. However, there are still some visually impaired people who are unable to learn Braille system due to various factors, such as the age (too young or too old), brain damage, etc. A wearable and low-cost Braille recognition system may substantially help these people recognize Braille or assist them in Braille learning. In this work, we fabricated polydimethylsiloxane (PDMS)-based flexible pressure sensors to construct an electronic skin (E-skin) for the application of Braille recognition. The E-skin mimics human touch sensing function for collecting Braille information. Braille recognition is realized with a neural network based on memristors. We utilize a binary neural network algorithm with only two bias layers and three fully connected layers. Such neural network design remarkably reduces the calculation burden and, thus, the system cost. Experiments show that the system can achieve a recognition accuracy of up to 91.25%. This work demonstrates the possibility of realizing a wearable and low-cost Braille recognition system and a Braille learning-assistance system.