Machine learning methods, such as support vector machine (SVM), have been applied to fMRI data analysis, where most studies focus on supervised detection and classification of cognitive states. In this work, we study the general fMRI activation detection using SVM in an unsupervised way instead of the classification of cognitive states. Specifically, activation detection is formulated as an outlier (activated voxels) detection problem of the one-class support vector machine (OCSVM). An OCSVM implementation, ν-SVM, is used where parameter ν controls the outlier ratio, and is usually unknown. We propose a detection method that is not sensitive to ν randomly set within a range known a priori. In cases that this range is also unknown, we consider ν estimation using geometry and texture features. Results from both synthetic and experimental data demonstrate the effectiveness of the proposed methods.