For robot manipulators, it is nowadays necessary to know their surroundings. This knowledge consists at least of a world representation with recognized objects. During the reconstruction of scene objects from multiple views, changes, like positioning of the objects, or additional unwanted signals, like parts of a human co-worker, may occur. In this paper, we classify the possible changes for a specific type of representation (boundary representation models). Afterwards, we present an approach to detect and handle these changes to maintain a valid world model. To achieve this, we compare what should be visible in the world model reconstructed thus far with the actual information from the current view. The detected change is handled by using object hypotheses as well as geometric information from the world representation. Based on an evaluation, we show a proof of concept and the usefulness of our approach and suggest future work.