Abstract—The invention of computer has been a turning point in the history. It makes our day-to-day life much simpler. The way to communicate with computer i.e., Human Computer Interaction (HCI), is through input and output devices. Mouse is one of the input devices. Mouse is an important part of Human Computer Interaction (HCI). With evolving technology, the cursor can be controlled without physical mouse hardware but using tip of a finger, hand gestures, etc. These hand movements can be recorded through webcam. The movement of the cursor can also be controlled by color detection or fingertip tracking using a webcam. This new era technology is useful in many fields, in many situations. This can be accomplished using extra hardware component or developing a software system. This is achieved through various image processing techniques and gesture recognition techniques without extra hardware. In this paper we have proposed a system which uses hand gesture recognition-based Convolutional Neural Network (CNN) model for controlling the mouse. Keywords—Virtual mouse, Hand Gesture Recognition, CNN model