The effect of the different mixture in a high volumetric concentration of oxidizer-(AP), with least percentage of binder-(HTPB+TDI), for improving the propellant burn rate was investigated. The combustion experiment is performed using a window bomb setup and the high-speed camera is utilized to capture the flame images. An image processing approach is used to measure the burn rate and intrinsic instability of flame by discrete wavelet transform method. Region growing algorithm technique is used for image segmentation. The morphological operation is implemented with Euclidean distance measurement for the identification of flame height in configuring with dependent parameters (burning rate, diffusion flame height). The qualitative analysis (signal characterization) and quantitative analysis (mean, kurtosis, skewness, standard deviation, and frequency) were used to study the intrinsic instability characteristics of the flame diffusion. A result obtained from the analysis proves that the instability in fuel combustion occurs at higher mix and pressure level.