Nowadays, a large amount of user privacysensitive data is outsourced to the cloud server in ciphertext, which is provided by the data owners and can be accessed by authorized data users. When accessing data, the user should be assigned with the access permission according to his identities or attributes. In addition, the search capabilities in encrypted outsourced data is expected to be enhanced, i.e., the search results can better present user's intentions. To address the above issues, ESAS, an Efficient Semantic and Authorized Search scheme over encrypted outsourced data, is proposed. In ESAS, by integrating PRSCG (the privacy-preserving ranked search based on conceptual graph) and CP-ABE (ciphertext policy attribute-based encryption), semantic search with file-level fine-grained access authorization can be realized. In addition, search authorization can be done in an offline manner, which can improve search efficiency and reduce the response time. The security analysis indicate that the proposed ESAS meets security requirement.