Stuttering or stammering assessment is one of the vital factors in speech recognition algorithms. To reconstruct the stuttered speech into spontaneous speech it is necessary to detect and correct the features influencing the speech signal. In this paper the speech signal is processed based on the disturbances created by acoustic effects like pauses and noises made both externally and internally. To eliminate the effects of noise on speech signal an Enhanced Kalman Filter is introduced here and its performance along with various filters are studied and compared based on the parameters like Mean Square Error (MSE), Mean Absolute Error (MAE), SNR ratio, Peak Signal to Noise ratio and Cross correlation. Then based on the extracted features classification of the speech signal is carried out using Convolutional Neural Network (CNN) algorithm of Deep learning technique.