The muon collider has recently received a great deal of attention because of its ability to achieve both high energy and high luminosity. It plays as a gauge boson collider because the vector boson scattering (VBS) becomes the dominant production topology for Standard Model processes starting from a few TeV of collision energy. In this paper, we propose that the process of μ+μ− annihilation into tri-photon is also very sensitive to the search of anomalous quartic gauge couplings (aQGCs). We investigate the projected constraints on the transverse operators contributing to aQGCs through μ+μ− → Z*/γ* → γγγ at muon colliders. For the muon collider with $$ \sqrt{s} $$ s = 3 TeV and $$ \mathcal{L} $$ L = 1 ab−1, the expected constraints are about two orders of magnitude stronger than those at the 13 TeV LHC.

The muon collider provides a unique opportunity to study the vector boson scattering processes and dimension-8 operators contributing to anomalous quartic gauge couplings (aQGCs). Because of the cleaner final state, it is easier to decode subprocess and certain operator couplings at a muon collider. We attempt to identify the anomalous WWWW coupling in the exclusive WW → WW scattering in this paper. Since one aQGC can be induced by multiple dimension-8 operators, the study of one coupling can help to confine different operators. Meanwhile, singling out the WW → WW process can help to study the unitarity bounds. The vector boson scattering process corresponding to the anomalous WWWW coupling is μ+μ− → $$ \nu \nu \overline{\nu}\overline{\nu } $$ νν ν ¯ ν ¯ ℓ+ℓ−, with four (anti-)neutrinos in the final state, which brings troubles in phenomenological studies. In this paper, the machine learning method is used to tackle this problem. We find that, the artificial neural network is helpful to extract the W+W− → W+W− contribution, and reconstruct the center of mass energy of the subprocess which is important in the study of the Standard Model effective field theory. The sensitivities and the expected constraints on the dimension-8 operators at the muon collider with $$ \sqrt{s} $$ s = 30 TeV are presented. We demonstrate that the artificial neural networks exhibit great potential in the phenomenological study of processes with multiple neutrinos in the final state.