This article examines benefits and risks of Artificial Intelligence (AI) in education in relation to fundamental human rights. The article is based on an EU scoping study (Anonymous, 2017). The study takes into account the potential for AI and 'Big Data' to provide more effective monitoring of the education system in real-time, but also considers the implications for fundamental human rights and freedoms of both teachers and learners. The analysis highlights a need to balance the benefits and risks as AI tools are developed, marketed and deployed. We conclude with a call to embed consideration of the benefits and risks of AI in education as technology tools into the development, marketing and deployment of these tools. There are questions around who -which body or organisation -should take responsibility for regulating AI in education, particularly since AI impacts not only data protection and privacy, but on fundamental rights in general. Given AI's global impact, it should be regulated at a trans-national level, with a global organisation such as the UN taking on this role.