In Source Separation research, "cocktail party problem" is a challenging problem that research into source separation aims to solve. Many attempts have been made to solve this complex problem. A logical approach would be to break down this complex problem into several smaller problems which are solved in different stages -each considering various aspects. In this paper, we are providing a robust solution to a part of the problem by localizing and tracking multiple moving speech sources in a room environment. Here we study the separation problem for unknown number of moving sources. The DUET-CBMeMBer method we outline is capable of estimating the number of sound sources as well as tracking and labelling them. This paper proposes a track management technique that identifies sound sources based on their trajectory as an extension to the DUET-CBMeMBer technique.