Rice is a staple food for around 3.5 billion people in eastern, southern and south-east Asia. Prior to being rice, the rice-grain (grain) is previously husked and/or milled by the milling machine. Relevantly, the grain quality depends on its pureness of particular grain specie (without the mixing between different grain species). For the demand of grain purity inspection by an image, many researchers have proposed the grain classification (sometimes with localization) methods based on convolutional neural network (CNN). However, those papers are necessary to have a large number of labeling that was too expensive to be manually collected. In this paper, the image augmentation (rotation, brightness adjustment and horizontal flipping) is appiled to generate more number of grain images from the less data. From the results, image augmentation improves the performance in CNN and bagof-words model. For the future moving forward, the grain recognition can be easily done by less number of images.