Intelligent systems are capable of performing several tasks with high reliability and efficiency. Hence, these systems were used to perform tasks which are usually done by humans. In the event of facility breach or in times when primary security systems were compromised, a call for secondary line of security is needed. In this study, it is intended to design a convolutional neural network -based computer vision system that can possibly determine whether a person entering a vicinity is authorized or not using face, height, and built recognition with gender sensitivity. The designed system was able to obtain balanced precision and recall as well as achieving more than 0.9 F1 scores. This is a complementary technology that can work with automated locks or security systems.