This paper presents a new object-based segmentation technique which exploits a large temporal context in order to get coherent and robust segmentation results. The segmentation process is seen as a problem of minimization of an energy function. This energy function takes into account a data attach term and spatial and temporal regularization terms. The proposed technique used to minimize this energy function is decomposed into three main steps: 1) definition of a technique for retrieving potential objects (referenced as seed extraction), 2) motion estimation for each seed, and 3) final classification performed by minimizing the energy function using a clustering-like technique. The proposed segmentation technique has been validated on real video sequences.