Code summaries are human-readable text that describes the functionality of code blocks. Software developers use code summaries to understand the specification of API while code retrieve system relies on code summaries for effective code search. However, code summaries are often written by software developers. Writing good code summaries usually requires great effort. It could be helpful if developers use automatic code summarization system to generate code summaries. Recently, some works have applied deep learning methods to generate code summaries for code snippets. However, those deep learning methods treat code snippets as streams of text tokens while ignoring the inherent code structure information. In this paper, we propose a novel code summarization method named the CDE-Model (Code summarization by Deep learning and Empirical knowledge) that combines inherent code structure information with deep learning models. The CDE-Model proposes several empirical strategies to transform code snippets to refined code representation and feeds them into an encoder-decoder neural network for text generation. We conduct large-scale experiments on 1500 popular Java projects on GitHub 1 with 396,184 pairs of code snippets and summaries. Experimental results show that the quality of code summaries generated by our CDE-Model is better than other two methods. To the best of our knowledge, this paper is the first to combine code structure information with deep learning.