In this paper, we propose an image recognition based logistic information system in the construction sites. In the construction sites, it is very important to keep the record of the construction materials as well as the logistics information on site as a part of the construction management. Till now, the logistics information is collected manually in the construction sites, which is time consuming and hassle. Focusing on the productivity improvement of the construction industry, several construction sites in Japan have started installing the web cameras in the sites with the aim of remote monitoring the situation on site. By using the video images, we thought of developing a logistic information gathering system based on the image recognition. The main purpose of our system is to automate the logistics information gathering system and replace the manual logistics information gathering process. We developed this system for confirming the possibility to automatically trace the movement of the materials on sites using the images from the web camera. The first place to collect the data of the materials starts from the vehicles that transports the materials into the sites. In this system, we will monitor and gather the information of the vehicles by using the video taken in the gate of the sites. The system extracts the number plate of the vehicle and output the entry and exit time of the vehicles automatically in a CSV format. We developed our system by using the image processing tools, Deep Learning techniques and the OCR. We performed various experiments using the data taken in the construction sites for calculating the effectiveness of our proposed system.