The detection of manhole's in a road by using Unmanned Aerial System or Vehicle (UAS) offers significant precision, however detection of manholes in above ground level images is a difficult task, with the help of Digital Single Lens Reflex (DSLR) camera and by using an image processing technique it is quite easy to detect small objects from the aerial view. Previously many methods is used to detect the manhole's covers, and many of them is quite accurate about 90% accuracy rate. But all the method applied for the detection of a covered manhole, but no one detects if there is no manhole cover lie in a manhole frame. This manuscript we present Speeded-Up Robust Feature's (SURF) algorithm for the key point detection of the manhole cover is present or not, and by marking many key points detects the manhole. After that cross checking our algorithm accuracy by using another method Learning Automata (L.A) with color gradient vector Red, Green, Blue (RGB) for the detection of a circular shape of manhole cover and color recognition for the cover is present or not.Index Terms-Manhole detection, speeded-up robust feature, Learning automata, RGB vector space.