Penetration of the power grid by renewable energy sources, distributed storage, and distributed generators is becoming increasingly common. Increased utilization of these distributed energy resources (DERs) has given rise to additional protection coordination concerns, particularly where they are utilized in an unbalanced manner or where loading among phases is unbalanced. Digital relays such as the SEL-751 (produced by Schweitzer Engineering Laboratories, Pullman, WA, USA) series have the capability of being set on a per-phase basis. This capability is underutilized in common practice. Additionally, in optimization algorithms for determining relay settings, the time-overcurrent characteristics (TOCs) of relays are generally not treated as variables and are assigned before running the optimization algorithm. In this paper, TOC options themselves are treated as discrete variables to be considered in the optimization algorithm. A mixed integer nonlinear programming problem (MINLP) is set up where the goal is to minimize relay operating times. A genetic algorithm (GA) approach is implemented in MATLAB where two cases are considered. In the first case, the TOC and Time dial setting (TDS) of each relay is set on a three-phase basis. In the second case, per-phase settings are considered. Relay TDSs and TOCs are both considered as simultaneous discrete control variables. Despite the stochastic nature of using per-phase settings for unbalanced systems is found to generally allow for shorter operating times. However, for relatively balanced systems, it is best to use three-phase settings if computation time is of importance.