Operators in complex environments are supported by alarm-systems that indicate when to shift attention to certain tasks. As alarms are not perfectly reliable, operators have to select appropriate strategies of attention allocation in order to compensate for unreliability and maintain overall performance. This study investigates how humans adapt to differing alarm-reliabilities. Within a multi-tasking flight simulation, participants were randomly assigned to four alarm-reliability conditions (68.75%, 75%, 87.5%, 93.75%), and a manual control group. In experimental conditions, one out of three subtasks was supported by an alarmsystem. Compared to manual control, all experimental groups benefited from alarms in the supported task, with best results for the highest reliability condition. However, analyses of performance and eye-tracking data revealed that the benefit of the lowest reliability group was associated with an increased attentional effort, a more demanding attention allocation strategy, and a declined relative performance in a nonsupported task. Results are discussed in the context of recent research.