Many practically relevant computing artifacts are forms of graphs, as, e.g., neural networks, mathematical expressions, finite automata. This great generality of the graph abstraction makes it desirable a way for searching in the space of graphs able to work effectively regardless of the graph form and application. In this paper, we propose GraphEA, a modular evolutionary algorithm (EA) for evolving graphs. GraphEA is modular in the sense that it can be tailored to any graph optimization task by providing components for specifying how to vary edges and nodes, and how to check the validity of a graph. We designed GraphEA by building on previous EAs dealing with particular kinds of graphs and included a speciation mechanism for the preservation of structural innovations and one for the gradual complexification of the solutions structure. To validate the generality of GraphEA, we applied it to 3 radically different tasks (regressions, in two flavors, text extraction from examples, evolution of controllers for soft robots) and compared its effectiveness against less general approaches. The results are promising and indicate some directions for further improvement of GraphEA.