Most of text line and character segmentation methods for handwritten document image basically still depend on the binary image of the document. Unfortunately, for palm leaf manuscript images, the binarization process is a real challenge. We proposed a new binarization free scheme for text line and character segmentation for palm leaf manuscript images. Our scheme consists of 4 sub-tasks: brushing character area of gray level images with minimum filtering, average block projection profile of gray level images, selection of candidate area for segmentation path, and construction of nonlinear segmentation path. For evaluation, we compared our method with the shredding method which is applied in three different schemes of experiment. The experimental results showed that the proposed method performed optimal on the palm leaf manuscript images which contain discolored parts, with low intensity variations or poor contrast, random noises, and fading.