During machining, it is well-known that unstable self-excited vibrations known as regenerative chatter can limit productivity. There has been a great deal of research that has sought to understand regenerative chatter, and to avoid it through modifications to the machining process. One promising approach is the use of variable helix tools. Here, the time delay between successive tooth passes is intentionally modified, in order to improve the boundary of instability. Previous research has predicted that such tools can offer significant performance improvements whereby islands of instability occur in the stability lobe diagram. By avoiding these islands, it is possible to avoid regenerative chatter, at depths of cut that are orders of magnitude higher than for traditional tools. However, to the authors’ knowledge, these predictions have not been experimentally validated, and there is limited understanding of the parameters that can give rise to these improvements. The present study seeks to address this shortfall. A recent approach to analysing regenerative chatter stability is modified, and its numerical convergence is shown to outperform alternative methods. It is then shown that islands of instability only emerge at relatively high levels of structural damping, and that they are particularly susceptible to model convergence effects. The model predictions are validated against detailed experimental data that uses a specially designed configuration to minimise experimental error. To the authors’ knowledge, this provides the first experimentally validated study of unstable islands in variable helix milling, whilst also demonstrating the importance of structural damping and numerical convergence on the prediction accuracy.