Decoding cognitive states using the bag of words model on fMRI time series

Abstract

Bag-of-words (BoW) modeling has yielded successful results in document and image classification tasks. In this study, we explore the use of BoW for cognitive state classification. We estimate a set of common patterns embedded in the Functional Magnetic Resonance Imaging (fMRI) time series recorded in three dimensional voxel coordinates by clustering the Blood Oxygen Level Dependent (BOLD) responses. We use these common patterns, called the code-words, to encode activities of both individual voxels and group of voxels, and obtain BoW representations on which we train linear classifiers. We experimented with a number of different BoW representations such as encoding spatial and functional neighbors, spatial pooling, and soft and hard encoding. Our experimental results show that, on a multiclass fMRI dataset, the hard BoW encoding, when applied to individual voxels, significantly improves the classification accuracy (an average 7.22% increase when applied on average intensity per voxel and an average 15.52% increase when applied to raw intensity time series per voxel) compared to a classical multi voxel pattern analysis (MVPA) method. This v preliminary result gives us a clue to generate a code-book for fMRI data which may be used to represent a variety of cognitive states to study the human brain.