Marine monitoring systems have the requirements of a large field of view, low power consumption, real-time viewing, and economical and automatic functionality. This paper establishes an omnidirectional vision system used in marine buoys that meets these requirements. We present a framework for image stabilization, which is achieved by omnidirectional sea-skyline detection in a marine environment. We propose an optimal edge estimation method to calculate the sea-skyline ellipsis according to the sea-skyline characteristics in panoramic images. We construct a compact panoramic image stabilization model based on the sea-skyline and propose a reconstruction method for the invalid regions using the key frame. The experimental results and analysis show that the proposed approach is capable of acquiring stable video in real-time marine monitoring tasks and that the target detection is sufficiently effective, efficient, and accurate for a real-time ship target detection application.