We propose a scheme that combines velocity-selective coherent population trapping (CPT) and Raman sideband cooling (RSC) for subrecoil cooling of optically trapped atoms outside the Lamb-Dicke regime. This scheme is based on an inverted Y configuration in an alkali-metal atom. It consists of a Λ formed by two Raman transitions between the ground hyperfine levels and the D transition, allowing RSC along two paths and formation of a CPT dark state. Using statedependent difference in vibration frequency of the atom in a circularly polarized trap, we can tune the Λ to make only the motional ground state a CPT dark state. We call this scheme motionselective coherent population trapping (MSCPT). We write the master equations for RSC and MSCPT and solve them numerically for a 87 Rb atom in a one-dimensional optical lattice when the Lamb-Dicke parameter is 1. Although MSCPT reaches the steady state slowly compared with RSC, the former consistently produces colder atoms than the latter. The numerical results also show that subrecoil cooling by MSCPT outside the Lamb-Dicke regime is possible under a favorable, yet experimentally feasible, condition. We explain this performance quantitatively by calculating the relative darkness of each motional state. Finally, we discuss on application of the MSCPT scheme to an optically trapped diatomic polar molecule whose Stark shift and vibration frequency exhibit large variations depending on the rotational quantum number.