We introduce a general differentiable solver for time-dependent deformation problems with contact. Our approach uses a finite element discretization with a high-order time integrator coupled with the recently proposed incremental potential contact method for handling contact and friction forces to solve PDE-and ODE-constrained optimization problems on scenes with a complex geometry. It support static and dynamic problems, it support differentiation with respect to all physical parameters involved in the physical problem description, which include shape, material parameters, friction parameters and initial conditions. Our analytically derived adjoint formulation is efficient, with an overhead of not more than 2 times the forward simulation, and shares many similarities with the forward problem, allowing reusing large parts of the code of an existing forward simulator code.We implement our approach on top of the open-source PolyFEM FE library, and demonstrate the applicability of our solver to shape design, initial condition optimization, and material estimation on both simulated results and in physical validations.