Many fragile watermark methods have been proposed for image recovery and their performance has been greatly improved. However, jagged edges and confusion still exist in the restored areas and these problems need to be solved to achieve a better visual effect. In this paper, a method for improving recovery quality is proposed that adopts singular value decomposition (SVD) and edge detection for tamper detection and then uses a median filter for image recovery. Variable watermark information can be generated that corresponds to block classifications. With mapping and neighborhood adjustment, the area that has been tampered can be correctly detected. Subsequently, we adopt a filtering operation for the restored image obtained after the inverse watermark embedding process. During the filtering operation, a median filter is used to smooth and remove noise, followed by minimum, maximum and threshold operations to balance the image intensity. Finally, the corresponding pixels of the restored image are replaced with the filtered results. The experimental results of six different tampering attacks conducted on eight test images show that tamper detection method with the edge detection can identify the tampered region correctly but has a higher false alarm rate than other methods. In addition, compared with the other three similar methods previously, using a median filter during image recovery not only improves the visual effect of the restored image but also enhances its quality objectively under most tampering attack conditions.