Abstract-In this paper, we present a technique for custom instruction (CI) extension considering process variations. Thetechnique bridges the gap between the high level custom instruction extension and chip fabrication in nanotechnologies. In particular, instead of using the conventional static timing analysis (STA), it utilizes statistical static timing analysis (SSTA). Therefore, the approach becomes probabilistic where the delay of each CI is modeled by a Probability Density Function (PDF).Using this probabilistic approach, different subsets of the CIs extension are identified to meet predefined constraints (identification phase) and eventually selected for realization to improve a given merit function (selection phase). In the identification phase, performance yield under both random and systematic variations is added as a constraint. Also, a pruning technique is proposed to decrease the runtime of the systematic variation modeling. The results show that the technique reduces the number of the CIs which need systematic variation modeling by about 24.6% for the cases studied in this work. In the selection phase, both greedy and branch-and-bound approaches are used. In the greedy approach, the conventional merit function based on the cycle saving and area is modified to include the performance yield. The results show the proposed merit function leads to about 3.2% increasing in the speedup. In the branch-and-bound method an effective pruning technique is described to reduce the runtime. The pruning technique is able to reduce the search space about 62%.