With the rapid growth of Web services, the demand for discovering the optimal services to satisfy the users' requirements is no longer an easy task. The critical issue in the process of service discovery is to conduct a similarity calculation. To solve such an issue, this study proposes an effective approach that combines the Embeddings from Language Models (ELMo) representation and Convolutional Neural Network (CNN) to obtain a more accurate similarity score for retrieving target Web services. More specifically, first, the study adopts the ELMo model to generate effective word representations for capturing the sufficient information from services and queries. Then, the word representations are used to compose a similarity matrix, which will be taken as the input for the CNN to learn the matching relationships. Finally, the combination of the ELMo representation and CNN is used to address the representation and interaction processes within the matching task to improve the service discovery performance. The results demonstrate the effectiveness of our proposed approach for retrieving better targeted Web services.