The automated segmentation of photogrammetric or LiDAR PC data offers the basis for further post-processing. One example is the use of enriched data for simulating wind, heat, and rain to increase the resilience of cities against stronger and more frequent weather events due to climate change. The semantic information can be used to distinguish tree species or surface types within simulations for more precise results. Apart from that, many other applications like autonomous driving or disaster management can benefit. However, the quality of the segmentation highly depends on the amount and quality of annotated training data and the applied segmentation method. This paper gives an overview and compares state-of-the-art PC segmentation methods including both 2D, 3D and combined approaches. Furthermore, current annotation tools and methods are evaluated regarding the accuracy, the number of classes, and time effort. Moreover, current strategies and methods to extend training data with augmented or synthetic data are listed and assessed. Besides, we present a concept for a specific segmentation task based on a trade-off of the previously listed methods. The task includes the segmentation and partial annotation of colourized PC data of two urban areas in Freiburg Germany, acquired from a driven and flying platform. Lastly, we present the required steps to extend the segmentation task from one city section to an overall city with diverse areas and to adapt to different sensor systems leading to different PC specifications. We conclude this paper by identifying challenges and required research in the field of PC segmentation and annotation.