Healthcare fraud, waste and abuse are costly problems that have huge impact on society. Traditional approaches to identify non-compliant claims rely on auditing strategies requiring trained professionals, or on machine learning methods requiring labelled data and possibly lacking interpretability. We present Clais, a collaborative artificial intelligence system for claims analysis. Clais automatically extracts human-interpretable rules from healthcare policy documents (0.72 F1-score), and it enables professionals to edit and validate the extracted rules through an intuitive user interface. Clais executes the rules on claim records to identify non-compliance: on this task Clais significantly outperforms two baseline machine learning models, and its median F1-score is 1.0 (IQR = 0.83 to 1.0) when executing the extracted rules, and 1.0 (IQR = 1.0 to 1.0) when executing the same rules after human curation. Professionals confirm through a user study the usefulness of Clais in making their workflow simpler and more effective.