Soft material can undergo non-uniform deformation or change of shape upon processing. Identifying shape and its change is nevertheless not straightforward. In this study, novel image-based algorithm that can be used to identify shapes of input images and at the same time classify non-uniform deformation into various patterns, i.e., swelling/shrinkage, horizontal and vertical elongations/contractions as well as convexity and concavity, is proposed. The algorithm was first tested with computer-generated images and later applied to agar cubes, which were used as model shrinkable soft material, undergoing drying at different temperatures. Shape parameters and shape-parameter based algorithm as well as convolutional neural networks (CNNs) either incorrectly identified some complicated shapes or could only identify the point where non-uniform deformation started to take place; CNNs lacked ability to describe non-uniform deformation evolution. Shape identification accuracy of the newly developed algorithm against computer-generated images was 65.88%, while those of the other tested algorithms ranged from 34.76 to 97.88%. However, when being applied to the deformation of agar cubes, the developed algorithm performed superiorly to the others. The proposed algorithm could both identify the shapes and describe their changes. The interpretation agreed well with that via visual observation.