Polymer waveguides are considered to be good candidates for optical interconnects application due to their advantages such as flexibility, good compatibility, and versatility for realizing 3D and micro structures. However, constrained by the limited degrees of freedom provided by templatebased design approaches, there lacks a satisfying solution which achieves small footprint and low loss simultaneously. Inverse design has been proposed to improve the performance. However, most inverse design methods target at silicon-based optical devices optimization which has a relatively high refractive index difference. In this paper, we demonstrate an ultra-compact low loss polymer wavelength (de)multiplexer with spot-size conversion structure to interposing Si and optical fiber using gradient-based topology optimization and downhill simplex method. The 2-channel wavelength (de)multiplexer displays an excess propagation loss of 0.84 dB and 1.06 dB at 1310 nm and 1550 nm, respectively, with a total footprint of 245 × 35 µm 2 . The 4-channel wavelength (de)multiplexer for coarse wavelength division multiplexing applications with 20 nm spacing (1270, 1290, 1310, and 1330 nm) with an excess waveguide propagation loss of 1.72 dB within 271 × 50 µm 2 has also been demonstrated and all the output ports of the polymer wavelength demultiplexing structure achieved a conversion efficiency over 90%.