PurposeAlthough classical techniques for image segmentation may work well for some images, they may perform poorly or not work at all for others. It often depends on the properties of the particular image segmentation task under study. The reliable segmentation of brain tumors in medical images represents a particularly challenging and essential task. For example, some brain tumors may exhibit complex so-called “bottle-neck” shapes which are essentially circles with long indistinct tapering tails, known as a “dual tail.” Such challenging conditions may not be readily segmented, particularly in the extended tail region or around the so-called “bottle-neck” area. In those cases, existing image segmentation techniques often fail to work well.MethodsExisting research on image segmentation using wormhole and entangle theory is first analyzed. Next, a random positioning search method that uses a quantum-behaved particle swarm optimization (QPSO) approach is improved by using a hyperbolic wormhole path measure for seeding and linking particles. Finally, our novel quantum and wormhole-behaved particle swarm optimization (QWPSO) is proposed.ResultsExperimental results show that our QWPSO algorithm can better cluster complex “dual tail” regions into groupings with greater adaptability than conventional QPSO. Experimental work also improves operational efficiency and segmentation accuracy compared with current competing reference methods.ConclusionOur QWPSO method appears extremely promising for isolating smeared/indistinct regions of complex shape typical of medical image segmentation tasks. The technique is especially advantageous for segmentation in the so-called “bottle-neck” and “dual tail”-shaped regions appearing in brain tumor images.