Recent work has suggested that disorganised speech might be a powerful predictor of later psychotic illness in clinical high risk subjects. To that end, several automated measures to quantify disorganisation of transcribed speech have been proposed. However, it remains unclear which measures are most predictive of psychosis-onset, how different measures relate to each other and what the best strategies are to elicit disorganised speech from participants. Here, we assessed the ability of twelve automated Natural Language Processing markers to differentiate transcribed speech excerpts from subjects at clinical high risk for psychosis (N=25), first episode psychosis patients (N=16) and healthy control subjects (N=13; N=54 in total). In-line with previous work, several of these measures showed significant differences between groups, including semantic coherence and speech graph connectivity. We also proposed two additional measures of repetition and whether speech was on topic, the latter of which exhibited significant group differences and outperformed the prior, related measure of tangentiality. Most measures examined were only weakly related to each other, suggesting they provide complementary information and that combining different measures could provide additional power to predict the onset of psychotic illness. Finally, we compared the ability of transcribed speech generated using different tasks to differentiate the groups. Speech generated from picture descriptions of the Thematic Apperception Test and a story re-telling task outperformed free speech, suggesting that choice of speech generation method may be an important consideration. Overall, quantitative speech markers represent a promising direction for future diagnostic applications for psychosis risk.