We investigated the structural, vibrational, magnetic, and electronic properties of the recently synthesized CaCoVO double perovskite with the high-spin (HS) Co ions in a square-planar oxygen coordination at extreme conditions of high pressures and low temperatures. The single-crystal X-ray diffraction and Raman spectroscopy studies up to 60 GPa showed a conservation of its cubic crystal structure but indicated a crossover near 30 GPa. Above 30 GPa, we observed both an abnormally high "compressibility" of the Co-O bonds in the square-planar oxygen coordination and a huge anisotropic displacement of HS-Co ions in the direction perpendicular to the oxygen planes. Although this effect is reminiscent of a continuous HS → LS transformation of the Co ions, it did not result in the anticipated shrinkage of the cell volume because of a certain "stiffing" of the bonds of the Ca and V cations. We verified that the oxidation states of all the cations did not change across this crossover, and hence, no charge-transfer effects were involved. Consequently, we proposed that CaCoVO could undergo a phase transition at which the large HS-Co ions were pushed out of the oxygen planes because of lattice compression. The antiferromagnetic transition in CaCoVO at 100 K was investigated by neutron powder diffraction at ambient pressure. We established that the magnetic moments of the Co ions were aligned along one of the cubic axes, and the magnetic structure had a 2-fold periodicity along this axis, compared to the crystallographic one.