From the early 21st century to the present date, Traffic surveillance has been one of the most challenging problems and over the years many solutions were proposed for tackling this problem. However, flaws are can still be noticed in the currently existing systems for performing this task and some of them will be further addressed in this work. In our project Traffic Surveillance using Computer Vision and Deep Learning, we propose a smart vision-based system which accurately provides live traffic statistics like the speed of the traffic and the count of Vehicles passed on a road in between a time interval. The system takes a video feed as an input, which could be live footage provided by a CCTV camera at a traffic junction, and on querying starting and ending timestamps, it outputs the count of the vehicles and speed of the traffic in between the queried timestamps. We divide this problem into two major sub-problems, firstly detecting a vehicle, followed by tracking the detected vehicle. Although there exist many algorithms for detecting objects as well as for tracking objects, one of the major problems is integrating both. As ambiguities like detecting an already detected vehicle, many occur while tracking the vehicles. In our work, we emphasize handling such ambiguities using Deep Learning and we also provide a comparative performance analysis of our system with the already existing ones.