Soft grippers perform various handling tasks using passive conformability. This article reviews the improvements in their capabilities of dexterous manipulations, including how they achieve dexterity and how their performance could be evaluated. This article particularly points out the correlations between potential and real dexterity, and the relationship between the space of degrees of freedom and the corresponding dexterity level. Our main contribution is the proposition of a uniform framework for the characterization of soft grippers and their performance. We first present an introduction to soft grippers and those capable of in-hand manipulation. We emphasize their hybridization by combining soft and rigid materials or using several active materials. Next, we define and discuss the manipulation tasks and how to achieve dexterity, making a distinction between stable grasping and stable in-hand manipulation. We finally discuss the means to achieve assessment and how the performance can be evaluated, and we develop a general exploitable approach for characterizing soft grippers and their dexterous performance based on their architecture, DOF space, and physical performance.