Litchi (Litchi chinensis Sonn.) and longan (Dimocarpus longan Lour.) fruits have a succulent and white aril with a brown seed and are becoming popular worldwide. The two fruits have been used in traditional Chinese medicine as popular herbs in the treatment of neural pain, swelling, and cardiovascular disease. The pericarp and seed portions as the by-products of litchi and longan fruits are estimated to be approximately 30% of the dry weight of the whole fruit and are rich in bioactive constituents. In the recent years, many biological activities, such as tyrosinase inhibitory, antioxidant, anti-inflammatory, immunomodulatory, anti-glycated, and anti-cancer activities, as well as memory-increasing effects, have been reported for the litchi and longan pericarp and seed extracts, indicating a potentially significant contribution to human health. With the increasing production of litchi and longan fruits, enhanced utilization of the two fruit by-products for their inherent bioactive constituents in relation to pharmacological effects is urgently needed. This paper reviews the current advances in the extraction, processing, identification, and biological and pharmacological activities of constituents from litchi and longan by-products. Potential utilization of litchi and longan pericarps and seeds in relation to further research is also discussed.