The European Installation Bus(EIB) protocol, also known as KNX/EIB, is widely used in building and home automation. An extension of the KNX/EIB protocol, EIBsec, is primarily designed to meet the requirements for data transmission security in distributed building automation systems. However, this protocol has some security issues in the request, key distribution, and identity authentication processes. This paper employs a formal analysis method that combines Colored Petri Net (CPN) theory with the Dolev-Yao attack model to evaluate and enhance the EIBsec protocol. It utilizes the CPN Tools to conduct CPN modeling analysis on the protocol and introduces a security assessment model to carry out intrusion detection and security assessment. Through this analysis, vulnerabilities in the protocol, such as tampering and replay attacks, are identified. To address these security concerns, we introduce hash verification and timestamp judgment methods into the original protocol to enhance its security. Subsequently, based on the improved protocol, we conduct CPN modeling and verify the security of the new scheme. Finally, through a comparison and analysis of the performance and security between the original protocol and the improved scheme, it is found that the improved scheme has higher security.