Biased Mixtures of Experts: Enabling Computer Vision Inference Under Data Transfer Limitations

Abstract

We propose a novel mixture-of-experts class to optimize computer vision models in accordance with data transfer limitations at test time. Our approach postulates that the minimum acceptable amount of data allowing for highly-accurate results can vary for different input space partitions. Therefore, we consider mixtures where experts require different amounts of data, and train a sparse gating function to divide the input space for each expert. By appropriate hyperparameter selection, our approach is able to bias mixtures of experts towards selecting specific experts over others. In this way, we show that the data transfer optimization between visual sensing and processing can be solved as a convex optimization problem. To demonstrate the relation between data availability and performance, we evaluate biased mixtures on a range of mainstream computer vision problems, namely: (i) single shot detection, (ii) image super resolution, and (iii) realtime video action classification. For all cases, and when experts constitute modified baselines to meet different limits on allowed data utility, biased mixtures significantly outperform previous work optimized to meet the same constraints on available data.

Scaling Vision with Sparse Mixture of Experts

“…Finally, we add a small amount of noise with standard deviation 1 E to the activations Wx, which we find improves performance. We empirically found this performed well but that the setup was robust to this parameter.…”

“…For each MoE layer in V-MoE, we use the routing function g(x) = TOP k (softmax (Wx + )), where TOP k is an operation that sets all elements of the vector to zero except the elements with the largest k values, and is sampled independently ∼ N (0, 1 E 2 ) entry-wise. In practice, we use k = 1 or k = 2.…”

“…[40] alternatively enforced a balanced routing by solving a linear assignment problem. [21,2,25,1,63,47,64] focused on architectures whose scale is considerably smaller than that of both language models and our model.…”

“…Other approaches use shallow MoEs, learning a single router, either disjointly [25] or jointly [2], together with CNNs playing the role of experts. [1] further have a cost-aware procedure to bias the assignments of inputs across the experts. Unlike shallow MoEs, we operate with up to several tens of routing decisions per token along the depth of the model.…”