Following several reports announcing the detection or non-detection of minor species above the clouds of Venus, we have searched for other possible signatures of PH$_3$, HCN, and NH$_3$ in the infrared range. Since 2012, we have performed ground-based observations of Venus in the thermal infrared at various wavelengths to monitor the behavior of SO$_2$ and H$_2$O at the cloud top. We have identified spectral intervals where transitions of PH$_3$ (around 955 cm$^ $), HCN (around 747cm$^ $), and NH$_3$ (around 951cm$^ $) are present. From the absence of any feature at these frequencies, we derive, on the disk-integrated spectrum, a 3-sigma upper limit of 3 ppbv for the PH$_3$ mixing ratio, 0.5 ppbv for HCN, and 0.3 ppbv for NH$_3$, assuming that these species have a constant mixing ratio throughout the atmosphere. Maps of the Venus disk recorded at the center position of the lines show that there is no evidence for local detection anywhere over the Venus disk. Our results bring new constraints on the maximum abundance of these species at the cloud top and in the lower mesosphere of Venus.