This paper presents an effective, novel and robust framework for semantic analysis of a soccer video possessing varying illumination conditions. The proposed algorithm works in two phases. The proposed framework effectively detects and gathers important events in the first phase, and a later phase carries out the task of event classification. The proposed system aims to identify high-level semantics of a soccer video like card event, goal event, goal attack and other classes of events. The proposed framework effectively exploits optical flow and colour features to detect and classify the events. The use of event filtration and categorization features successfully makes the system effective over various conditions. Simulations have been performed on a large number of video datasets of different conditions of various soccer leagues. Simulation results reflect the efficiency and robustness of the proposed framework.