For interactive TV (ITV) services in converged network, the DVB-based DTV and the IP-based IPTV coexist in a single dual mode set-top box (STB) and the traditional one-way DVB-CA cannot be adapted to the variation. In this paper, a simplified and secure conditional access scheme based on the simplified double layer is proposed to realize authentication and access control between servers and STBs. In this scheme, the advanced encryption standard (AES) and the common scramble algorithm (CSA) are combined to secure the ITV service and its entitled control message. The video fingerprint and legitimate watermark is used for the piracy tracking and the conditional playing. The elliptic curve based authentication algorithm is used for secure authentication. Under the tripartite participation, the service key access protocol is designed to dynamically update the service keys. The algorithm in the front-end, the algorithm in the STB client and the service key access protocol are proposed in detail. Through the test in an actual application environment and the analysis of security, the proposed conditional access system is very secure with high performance for high definition ITV service and large scale customers.