Objectives: Designing optical character recognition systems for Kannada character is challenging due to higher self-similarity in characters and higher number of character classes. This work addresses the two major problems of reduced accuracy and higher false positives due to higher self-similarity in characters. Methods: This work proposes a two stage multi modal deep learning technique to handle the complexity in Kannada character recognition. The characters are first grouped based on morphological and structural similarity. A novel morphological/structural difference maximization convolution kernel based deep learning modal is trained for each of character group to recognize characters in that group. This divide and conquer strategy reduce the complexity of deep learning model in learning discriminative features for Kannada character recognition. Findings: The proposed two stage multi modal deep learning provides 89% recognition accuracy which is at least 6% higher compared to existing works. The false positives in proposed solution are at least10% lower compared to existing works. Novelty: A novel sector intensity distribution feature specific to curve structure of Kannada characters for deciding the number of groups of characters. Classifier is designed for each group of characters. Classification is done using convolutional neural network with a novel morphological/structural difference maximization convolution kernel to solve the structural similarity problem in Kannada character recognition.