Banana fiber has a high potential for use in fiber composite structures due to its promise as a polymer reinforcement. However, it has poor bonding characteristics with the matrixes due to hydrophobic–hydrophilic incompatibility, inconsistency in blending weight ratio, and fiber length instability. In this study, the optimal conditions for a banana/epoxy composite as determined previously were used to fabricate a sandwich structure where carbon/Kevlar twill plies acted as the skins. The structure was evaluated based on two experimental tests: low-velocity impact and compression after impact (CAI) tests. Here, the synthetic fiber including Kevlar, carbon, and glass sandwich structures were also tested for comparison purposes. In general, the results showed a low peak load and larger damage area in the optimal banana/epoxy structures. The impact damage area, as characterized by the dye penetration, increased with increasing impact energy. The optimal banana composite and synthetic fiber systems were proven to offer a similar residual strength and normalized strength when higher impact energies were applied. Delamination and fracture behavior were dominant in the optimal banana structures subjected to CAI testing. Finally, optimization of the compounding parameters of the optimal banana fibers improved the impact and CAI properties of the structure, making them comparable to those of synthetic sandwich composites.