This paper presents adaptive Online learning with Regularized Kernel-based One-Class Extreme Learning Machine (ELM) classifiers for detection of outliers, and are collectively referred to as "ORK-OCELM". Two frameworks viz., boundary and reconstruction, are presented to detect the target class in ORK-OCELM. The kernel hyperplane based baseline one-class ELM model considers whole data in a single chunk, however, the proposed one-class classifiers are adapted in an online fashion from the stream of training samples.The performance of ORK-OCELM is evaluated on standard benchmark as well as synthetic datasets for both types of environment, i.e. stationary and non-stationary. While evaluating on stationary datasets, these classifiers are compared against batch learning based one-class classifiers. Similarly, while evaluating on non-stationary datasets, comparison is done with incremental learning based online one-class classifiers. The results indicate that the proposed classifiers yield better or similar outcomes for both. In the non-stationary dataset evaluation, adaptability of proposed classifiers in a changing environment is also demonstrated. It is further shown that proposed classifiers have large stream data handling capability even under limited system memory.Moreover, the proposed classifiers gain significant time improvement compared to traditional online one-class classifiers (in all aspects of training and testing). A faster learning ability of the proposed classifiers makes them more suitable for real-time anomaly detection.