Content-based sports video analysis and composition

Abstract

This thesis proposes solutions for content-based sports video analysis, including multimodal feature extraction, middle-level representation and semantic event detection. In addition, solutions for sports video composition and personalization are also examined. The first part of the thesis describes our methodology to detect semantic events and event boundaries from both broadcast sports video and non-broadcast sports video. Specifically, to process broadcast sports video, our approach not only uses visual/audio features, we also analyze the web-casting text information associated with the video and synchronize it with the visual/audio features to detect event, locate event boundaries and identify involved players/teams; To process non-broadcast sports video, we select the raw unedited main-camera soccer video as the input and use visual, audio and motion features extraction with multi-level modeling to detect event and event boundaries. Our proposed techniques are evaluated using large sports video data set and achieved satisfactory user acceptance score. The second part of the thesis introduces three novel applications based on our proposed sports video analysis techniques. The first application is a live sports highlight generation system which can automatically detect multiple events from a live sports game and extract a suitable video segment for each event to provide live and personalized game viewership via various mediums. The second application attempts to automatically generate broadcast soccer video composition from multiple raw video captures by detecting events from unedited soccer video and mimicking human director's practice to control the replay insertion and camera view switching operations. The third application is a personalized music sports video generation system to automatically select and align desired sports video scenes with music clips to generate music video clips. The three proposed systems are tested using objective evaluations and subjective user studies. i ATTENTION: The Singapore Copyright Act applies to the use of this document. Nanyang Technological University Library Many people have contributed their ideas, time, and energy in my pursuit of this PhD research. I wish it was possible to thank them all. First thanks must go to my supervisor, Dr. Chng Eng Siong, and co-supervisor, Dr. Xu Chang Sheng, who have been truly inspirational throughout my candidature. I would like to thank them for all their guidance, rewarding discussions, cooperation, encouragements, and lasting support to my study and my life. I have been blessed to find in them all the good qualities of a supervisor. I am indebted to the School of Computer Engineering, Nanyang Technology University (NTU), for offering me the PhD scholarship. I am also thankful to the Institute for Infocomm Research (I 2 R) and the Centre for Multimedia and Network Technology, NTU for providing me an excellent working environment with all the much needed facilities and services, as well as financial supports including travel allowances....

Audio pattern discovery and retrieval

“…Most sports video consumers are usually more interested in the highlights of a sports video than the entire recording of a game, e.g., for a soccer game, the highlights are the goal scoring scenes, foul scenes, penalty shots, etc. Hence, many automatic highlights extraction techniques [65][66][67][68][69] have been examined and proposed. To detect these scenes, audio content are sometimes exploited.…”

“…Moreover, high energy segments were found to closely follow the occurrences of goals in basketball games [68]. To analyze soccer game, Wang [69] classified audio features such as Mel-Frequency Ceptral Coefficient (MFCC) and Linear Prediction Cepstral Coefficient (LPCC) into three groups: "whistle", "acclaim" and "noise". Such auditory labels are useful for high-level semantic analysis.…”