In communication systems, techniques which provide high resistance against Interchannel Interference under high data rate transmission become vital [1]. Hence, an efficient algorithm for design and implementation of non-uniform digital filter banks using optimization parameters is presented. The design problem is based on unconstrained optimization of cut-off frequency (ωc) for Finite Impulse Response (FIR) filter banks for optimum design of transmultiplexer depending on near perfect reconstruction [2]. The objective function is chosen as Interchannel Interference and amplitude distortion (Eam) with ωc, As, and roll off factor as varying parameter [3]. The proposed algorithm will be validated for real-time video processing with 40dB attenuation, which are sufficient to ensure an output of adequate quality.