In recent years, advances in cell segmentation techniques have played a critical role in the analysis of biological images, especially for quantitative studies. Deep learning models have shown remarkable performance in segmenting cell and nucleus boundaries, but are often designed for specific modalities or require human intervention to select hyper-parameters, and are limited in generalizing to out-of-sample data. Building universal cell segmentation models can address the above challenges, but requires a large amount of multimodal training data. Here, we present CellBinDB, a large-scale multimodal annotated dataset established for cell segmentation. CellBinDB contains more than 1,000 annotated images of DAPI, ssDNA, H&E, and mIF staining, covering more than 30 normal and diseased tissue types from human and mouse samples. Based on CellBinDB, we benchmarked six state-of-the-art cell segmentation models and a widely used software. Evaluations were performed on the entire dataset and on each staining type, with Cellpose performed outstandingly. In addition, we analyzed the effects of four cell morphology indicators and image gradient on the segmentation results.