Electroencephalogram (EEG) contains important physiological information that can reflect the activity of human brain, making it useful for epileptic seizure detection and epilepsy diagnosis. However visual inspection of large amounts of EEG by human expert is time-consuming, and meanwhile there are often inconsistences in judgement between physicians. In this paper, we develop a unified framework for early epileptic seizure detection and epilepsy diagnosis, which includes two phases. In the first phase, the signal intensity is first calculated for each data point of the given EEG, enabling the well-known autoregressive moving average (ARMA) model to characterize the dynamic behavior of the EEG time series. The residual error between the predicted value of learned ARMA model and the actually observed value is used as the anomaly score to support a null hypothesis testing for making epileptic seizure decision. The epileptic seizure detection phase can provide a quick detection for anomaly EEG patterns, but the resulting suspicious segment may include epilepsy or other disordering EEG activities thus required to be identified. Therefore, in the second phase, we use pattern recognition technique to classify the suspicious EEG segments. In particular, we propose a new and practical classifier based on a pairwise of one-class SVMs for epilepsy diagnosis. The proposed classifier requires normal and epilepsy data for training, but can recognize normal, epilepsy and even other disorders that would not be trained in the training samples. This point is practical and meaningful in real clinic scenarios as the input EEG may include other brain disordering diseases besides of epilepsy. We conducted experiments on the publicly-available Bern-Barcelona and CHB-MIT EEG database, respectively, to validate the effectiveness of the proposed framework, and our method achieved classification accuracy of 93% and 94% on them. Comprehensive experimental results, outperforming the state-of-the-arts, suggest its great potentials in real applications.