This paper presents a methodology to automatically detect and remove the shadows in high-resolution urban aerial images for urban GIS applications. The image is generated by ADS40, which is a multiple line scanner. The system includes shadow simulation, ray tracing, integrated shadow detection, and shadow removal. The geodetic shadow is simulated from a digital surface model (DSM) and the sun attitude. Using an ADS40 camera model and collinearity equations, the corresponding scan line of the shadow is obtained. Ray tracing is used to determine the visibility of the shadow in the image. A building boundary driven height field ray tracing method is proposed to boost the tracing efficiency. Shadow segmentation is taken from the RGB image at the base of the traced image shadows. The segmentation threshold is derived from the histogram of the traced image shadow area. Shadow removal includes the shadow region and companion region labelling, the histogram processing, and intensity mapping. Some experiment results are presented to show the effectiveness of the methods.