Cell motility is essential to many biological processes as cells navigate and interact within their local microenvironments. Currently, most methods to quantify cell motility rely on the ability to follow and track individual cells. However, results from these approaches are typically reported as averaged values across cell populations. While these approaches offer biological simplicity, it limits the ability to assess cellular heterogeneity and infer generalizable patterns of single-cell behaviors at baseline or after perturbations. Here, we present CaMI, a computational framework that takes advantage of the single-cell nature of cell motility data to identify and classify distinct spatio-temporal behaviors of single cells. Using CaMI, we demonstrate the ability to robustly classify single-cell motility patterns in a large dataset (n=74,253 cells), quantify spatio-temporal heterogeneities, determine motility patterns in unclassified cells, and provide a visualization scheme for direct interpretation of dynamic cell behaviors. Furthermore, as a biological proof of concept, we investigate the biphasic spatial and temporal responses of T-cell lymphoma cells moving in Collagen-I gels of varying collagen concentrations and predict the dimensionality (2D vs. 3D) of matched cellular conditions based solely on spatio-temporal heterogeneities. Together, we present a multivariate framework to robustly classify emergent patterns of single-cell behaviors, highlighting cellular heterogeneity as a critical feature that establishes the behaviors of cellular populations.