In recent years, facial expression recognition (FER) has becoming a growing topic in computer vision with promising applications on virtual reality and human-robot interaction. Due to the influence of illumination, individual differences, attitude variation and etc., facial expression recognition with robust accuracy in complex environment is still an unsolved problem. Meanwhile, with the widely use of social communication, massive data is uploaded to the Internet, the effective utilization of those data is still a challenge due to noisy label phenomenon in the study of FER. To resolve the above-mentioned problems, Firstly, a double active layer (DAL) based CNN is established to recognize the facial expression with high accuracy by learning robust and discriminative features from the data, which could enhance the robustness of network. Secondly, an active incremental learning method was utilized to tackle the problem of using Internet data. During the training phase, a two-stage transfer learning method is explored to transfer the relative information from face recognition to FER task to alleviate the inadequate training data in deep convolution network. Besides, in order to make better use of facial expression data from website and further improve the FER accuracy, UFEDW (Unconstrained Facial Expression Database from Website) database is built in this paper. Extensive experiments performed on two public facial expression recognition database FER 2013 and SFEW 2.0 have demonstrated that the proposed scheme outperforms the state-of-the-art methods, which could achieve 67.08% and 51.90% respectively.