Color is an important feature in applications like the detection of plants and diseases in plants. Deep learning networks utilize optimizers towards improving the accuracy of classification. Color space is treated as an extra dimension with which the image could be better classified. Hence a particular blending of classifier, optimizer and color space is expected to provide enhanced accuracy of classification. There are very rare cases of studies having examined the effect of color space with deep learning networks. Hence, it is motivated to study the role of color spaces. Leaf datasets available in literature have been utilized. Of the few tried networks, Inception V3 is found to perform better with optimizer Adam. Color space XYZ performed better than RGB in the above combination. It has also been tried to obtain majority voting among various optimizer combinations. This solution is also better with XYZ color space. Among the various datasets utilized, consistent performance has been observed with Flavia data set yielding superior classification accuracy.