Green tea is one of the most commonly consumed natural health beverages in Taiwan’s market, with the major functional component catechin being shown to possess several biological activities such as antioxidation, anticancer, and prevention of cardiovascular disease. The objectives of this study were to develop a high-performance liquid chromatography–mass spectrometry method to determine the variety and content of catechins in green tea leaf waste, a by-product obtained during processing of tea beverage. In addition, catechin nanoemulsion was prepared to study its inhibition effect on prostate cancer cell PC-3. Results showed that a total of eight catechin standards were separated within 25 minutes by using a Gemini C18 column and a gradient mobile phase of 0.1% formic acid (A) and acetonitrile (B) with flow rate at 1 mL/min, column temperature at 30°C, and detection wavelength at 280 nm. Among various extraction solvents, 50% ethanol generated the highest yield of total catechins from tea leaf waste, of which five catechins were identified and quantified. The catechin nanoemulsion was composed of catechin extract, lecithin, Tween 80, and deionized water in an appropriate proportion, with the mean particle size being 11.45 nm, encapsulation efficiency 88.1%, and zeta potential −66.3 mV. A high stability of catechin nanoemulsion was shown over a storage period of 120 days at 4°C. Both catechin extract and nanoemulsion could inhibit growth of PC-3 tumor cells, with the half maximal inhibitory concentration being 15.4 μg/mL and 8.5 μg/mL, respectively. The PC-3 cell cycle was arrested at S phase through elevation of P27 expression and decline of cyclin A, cyclin B, cyclin-dependent kinase 2, and cyclin-dependent kinase 1 expression. In addition, both catechin extract and nanoemulsion could induce apoptosis of PC-3 cells through decrease in B-cell lymphoma 2 (bcl-2) expression and increase in cytochrome c expression for activation of caspase-3, caspase-8, and caspase-9. Taken together, both caspase-dependent and caspase-independent pathways may be involved in apoptosis of PC-3 cells.