DSS: A Scalable and Efficient Stratified Sampling Algorithm for Large-Scale Datasets

Abstract

Statistical analysis of aggregated records is widely used in various domains such as market research, sociological investigation and network analysis, etc. Stratified sampling (SS), which samples the population divided into distinct groups separately, is preferred in the practice for its high effectiveness and accuracy. In this paper, we propose a scalable and efficient algorithm named DSS, for SS to process large datasets. DSS executes all the sampling operations in parallel by calculating the exact subsample size for each partition according to the data distribution. We implement DSS on Spark, a big-data processing system, and we show through large-scale experiments that it can achieve lower data-transmission cost and higher efficiency than state-of-the-art methods with high sample representativeness.

S-OHEM: Stratified Online Hard Example Mining for Object Detection

“…Stratified sampling is a sampling method involving the division of a population into distinct groups known as strata [21]. These strata are homogeneous subgroups of the original data with similar inner items.…”

“…Therefore, maintaining a sampling distribution according to this influence during hard example mining should enhance the performance of object detectors. S-OHEM exploits stratified sampling, a sampling method involving the division of a population into distinct groups known as strata [21] (homogeneous subgroups, in which the inner items are similar to each other). During each mini-batch iteration, S-OHEM firstly assigns candidate examples (in the form of Region of Interests, RoIs) to different strata by the ratio between classification and localization loss.…”