Learning to program is often reported as challenging. Difficulties might be connected to, e.g., acquisition of core concepts like iteration, specific language constructs, and program design. Additionally, mastery of a programming language's constructs does not consequently translate into solving new programming problems. These challenges have to be taken into account in the context of in-service teachers' professional development (PD). In this paper, we address challenges teachers face when learning to program, considering these closely related questions: What do teachers perceive as difficult and how does this impact the perception of challenges their students will face? How does this influence the perception of teaching programming? The paper is based on the analysis of 178 reflection notes delivered by in-service teachers attending a universitylevel course on programming to identify issues related to the research questions. Out of the topics that we have selected from the literature as challenging for newcomers, the most difficult ones for teachers seem to be writing code and pseudocode to solve a specific problem. The easiest ones are understanding variable initialization and if statements. Also, teachers express that pupils will struggle with many aspects of programming and point to the complexity and students' misconceptions inherent in several programming concepts such as loops. Our results suggest that it is not necessarily difficult to learn to program if in-service teachers are given paid time to learn.