Summary The tracking of multiple objects has gained immense interest in several applications that ranges from video surveillance and robotics. The goal of tracking multiple objects is to discover correspondences and perform matching among various objects in frames. However, occlusion is a major issue in several classical techniques, which fail to track multiple moving objects. Here, the extraction of frames is done to obtain information about an action. Thereafter, the segmentation of objects is done using the entropy weighting K‐means (EWKM) algorithm to generate optimum segments from the frame. After that, the detection of the object is done using a deep convolutional neural network (DCNN) to obtain objects from segments. Proposed Jaya‐political optimizer (Jaya‐PO) is used to train DCNN, which is devised by collaborating political optimizer (PO) and Jaya optimization algorithm. After detecting the object, the tracking of multiple objects is done using the proposed Henry gas solubility optimized unscented Kalman filtering (HGSO‐based UKF), wherein the UKF is modified with HGSO. The proposed Jaya‐PO‐based DCNN and HGSO‐based UKF outperformed other methods with high accuracy of 92%, high multiple object tracking precision (MOTP) of 87.6%, high sensitivity of 91.7%, and high specificity of 92%.