Entomopathogenic nematodes, including Steinernema spp., play an increasingly important role as biological alternatives to chemical pesticides. The infective juveniles of these worms use nictation–a behavior in which animals stand on their tails–as a host-seeking strategy. The developmentally-equivalent dauer larvae of the free-living nematode Caenorhabditis elegans also nictate, but as a means of phoresy or "hitching a ride" to a new food source. Advanced genetic and experimental tools have been developed for C. elegans, but time-consuming manual scoring of nictation slows efforts to understand this behavior, and the textured substrates required for nictation can frustrate traditional machine vision segmentation algorithms. Here we present a Mask R-CNN-based tracker capable of segmenting C. elegans dauers and S. carpocapsae infective juveniles on a textured background suitable for nictation, and a machine learning pipeline that scores nictation behavior. We use our system to show that the nictation propensity of C. elegans from high-density liquid cultures largely mirrors their development into dauers, and to quantify nictation in S. carpocapsae infective juveniles in the presence of a potential host. This system is an improvement upon existing intensity-based tracking algorithms and human scoring which can facilitate large-scale studies of nictation and potentially other nematode behaviors.