For two integers r, ℓ ≥ 0, a graph G = (V, E) is an (r, ℓ)graph if V can be partitioned into r independent sets and ℓ cliques. In the parameterized (r, ℓ)-Vertex Deletion problem, given a graph G and an integer k, one has to decide whether at most k vertices can be removed from G to obtain an (r, ℓ)-graph. This problem is NP-hard if r+ℓ ≥ 1 and encompasses several relevant problems such as Vertex Cover and Odd Cycle Transversal. The parameterized complexity of (r, ℓ)-Vertex Deletion was known for all values of (r, ℓ) except for (2, 1), (1, 2), and (2, 2). We prove that each of these three cases is FPT and, furthermore, solvable in single-exponential time, which is asymptotically optimal in terms of k. We consider as well the version of (r, ℓ)-Vertex Deletion where the set of vertices to be removed has to induce an independent set, and provide also a parameterized complexity dichotomy for this problem.