Geo-referenced data is required in many Geographic Information System (GIS) applications and one source of such data being maps. Meteorological maps are color-coded with different regions corresponding to different values of a parameter, parsing the image to convert into data is not very difficult. However, text and different planimetric elements overlaid on the regions in the map makes accurate image to data conversion a challenging problem, because it is almost impossible to exactly replace what was underneath the text or icons i.e., the need for inpainting. In this study, we propose a probabilistic technique that uses the probability of occurrence of colors present in the map along with the occurrence of those colors in the spatial neighborhood of the pixel (corresponding to text) whose color is required to be replaced. We test the limits of our proposed technique using simulated data and compare results with a popular image editing tool using public domain data with promising results.