Ultracold molecules undergo "sticky collisions" that result in loss even for chemically nonreactive molecules. Sticking times can be enhanced by orders of magnitude by interactions that lead to non-conservation of nuclear spin or total angular momentum. We present a quantitative theory of the required strength of such symmetry-breaking interactions based on classical simulation of collision complexes. We find static electric fields as small as 10 V/cm can lead to non-conservation of angular momentum, while we find nuclear spin is conserved during collisions. We also compute loss of collision complexes due to spontaneous emission and absorption of black-body radiation, which are found to be slow.