Ship detection has been an area of focus for high-frequency surface wave radar (HFSWR). The detection and identification of ship formation have proven significant in early warning, while studies on the formation identification are limited due to the complex background and low resolution of HFSWR. In this paper, we first establish a spatial distribution model of ship formation in HFSWR. Then, we propose a cascade identification algorithm of ship formation in the clutter edge. The proposed algorithm includes a preprocessing stage and a two-stage formation identification stage. The Faster R-CNN is introduced in the preprocessing stage to locate the clutter regions. In the first stage, we propose an extremum detector based on connected regions to extract suspicious regions. The suspicious regions contain ship formations, single-ship targets, and false targets. In the second stage, we design a network connected by a convolutional neural network (CNN) and an extreme learning machine (ELM) to identify two densely distributed ship formations from inhomogeneous clutter and single-ship targets. The experimental results based on the factual HFSWR background demonstrate that the proposed cascade identification algorithm is superior to the extremum detector combined with the classical CNN algorithm for ship formation identification. Meanwhile, the proposed algorithm performs well in weak formation and deformed formation identification.