Automatic number plate recognition makes use of Image processing and Machine learning technology to detect vehicle numbers irrespective of climatic conditions. As the number of vehicles is increasing day by day, it's necessary to consider safety measures. It is used in various fields like traffic control, automatic road tax collection at toll areas and vehicle parking systems in crowded areas. The goal of the proposed system is to develop a capable automatic authorized vehicle recognition system that makes use of the vehicle number plate. For detecting number plate, an infrared (IR) sensor is used, which helps in taking clear images from a camera. Taking images of moving vehicles is the most important and difficult task. Character segmentation is used to extract the vehicle number plate region from an image by using R-CNN method. The Optical Character Recognition (OCR) method is used for identifying the accurate character. The collected data is then compared with the respective authority databases to investigate specific information like vehicle owner, registration location, address and so on. If the vehicle details match those in the database, then only gate barricade is opened. Through this system, criminal activities can be minimized and road safety measures will be considered. The system tries to promote the efficiency and accuracy of number plate detection over climatic conditions.