Security algorithms and protocols are typical essential upgrades that must be involved within systems and their structures to provide the best performance. The protocols and systems should go through verification and testing processes in order to be more efficient and accurate. In the testing of software, traditional methods are used for accuracy check-up. However, this could not fulfill the measurement of all the testing requirements. The usage of formal verification approaches in checking security properties considers their best environment to be applied. The available literature discussed several approaches on developing the most robust formal verification methods for addressing and analyzing errors that face systems. This could be during the implantation process, unknown attacks, and nondeterministic adversary on the security protocols and algorithm. In this paper, a comprehensive review of the main formal verification approaches such as model checking and theorem approving has been conducted. Moreover, the use of verification tools was briefly presented and explained thoroughly. Those formal verification methods could be involved in the design, redesign of security protocols, and algorithms based on standards and determined sizes that is decided by these techniques' analysis. The critical analysis of the methods used in verifying the security of systems showed that model checking approaches and its tools were the most used approaches among all the reviewed methods.