In some countries, direct democracy is used successfully to increase legitimacy of decisions or mitigate conflict, and in other countries, authoritarian leaders seem to instrumentalize and manipulate referendums. How can referendum integrity be analyzed? This article presents an empirical instrument to evaluate the variety and integrity of referendums. This encompasses criteria for the analysis of direct democracy. First, we develop a referendum cycle model based on the electoral cycle framework, assessing referendum quality in a number of dimensions from electoral laws and electoral procedures, thematic limitations of referendums, to voter registration, the initiation of referendums, campaign and media coverage as well as campaign financing. The empirical instrument is designed to be used in expert surveys, and piloted in the Turkish constitutional referendum of 2017. The article presents the results of the pilot study, draws out opportunities and limitations of this approach and suggests avenues for its future development.