The variable physical property of deformable objects, which are very flexible, soft and viscoelastic, causes the design of reliable automated handling system relatively difficult. In fact, most of these objects tend to be handled manually during the handling process. Therefore, a new optical tactile sensor for an intelligent handling of the non-rigid materials is presented in this paper. Mathematical modelling and control algorithm are developed and the tactile sensor is calibrated in this research. Based on the results that have been recorded, the surface characterization with the respect to normal force applied to the object is attained. A gripper handling system is used to accommodate variable physical properties of the deformable materials, which are very flexible, soft and viscoelastic. In addition to that, the gripper needs to handle the materials with the minimum deformation so that less distortion, and higher accuracy of manipulation can be achieved. Efficient and accurate modelling of deformations is crucial for grasping analysis.