A shortage in the supply of 3He used for thermal neutron detector makes researchers to find 3He alternatives for developing new neutron detectors. Here, we prepared a neutron-sensitive composite liposome with tributyl borate and encapsulating with Fe3O4@oleic acid nanoparticles (Fe3O4@OA NPs), methylene blue (MB), or anti-albumin from bovine serum (anti-BSA). The tributyl borate compound was characterized by Fourier transform infrared spectroscopy (FT-IR). In addition, the morphology, element compositions, and magnetic properties of the composite liposome were investigated with transmission electron microscopy (TEM), energy-dispersive X-ray spectroscopy (EDS), and vibrating sample magnetometer (VSM), respectively. The results indicated that a typical ellipsoidal magnetic liposome structure was obtained, and the lengths of the minor axis and major axis were 49 ± 1 nm and 87 ± 3 nm, respectively. Under thermal neutron irradiation, the structure of composite liposome was destroyed, and encapsulated reporter molecules were released, which was detected by ultraviolet–visible (UV–vis) spectroscopy and surface plasmon resonance (SPR) technology. The response of this sensor based on a destructive assay shows a good correlation with neutron doses. Besides, the sensor has a neutron to gamma-ray rejection ratio of 1568 at a thermal neutron flux rate of 135.6 n/cm2·s, which makes it a promising alternative to 3He.