Work on the problem of handwritten text recognition in Urdu script has been an active research area. A significant progress is made in this interesting and challenging field in the last few years. In this study, the authors presented a comprehensive survey for a number of offline and online handwritten text recognition systems for Urdu script written in Nastaliq font style from 2004 to 2019. Following features make their contribution worthwhile and unique among the reviews of a similar kind: (i) their review classifies the existing studies based on types of recognition systems used for Urdu handwritten text, (ii) it covers a very different outlook of the recognition process of the Urdu handwritten text at different granularity levels (e.g. character, word, ligature, or sentence level), (iii) this review article also presents each of surveyed articles in following dimensions: the task performed, its granularity level, dataset used, results obtained, and future dimensions, and (iv) lastly it gives the summary of the surveyed articles according to the granularity levels, publishing years, related tasks or subtasks, and types of classifiers used. In the end, major challenges and tasks related to Urdu handwritten text recognition approaches are also discussed in detail.