This paper introduces a "kernel-independent" interpolative decomposition butterfly factorization (IDBF) as a data-sparse approximation for matrices that satisfy a complementary lowrank property. The IDBF can be constructed in O(N log N ) operations for an N × N matrix via hierarchical interpolative decompositions (IDs), if matrix entries can be sampled individually and each sample takes O(1) operations. The resulting factorization is a product of O(log N ) sparse matrices, each with O(N ) non-zero entries. Hence, it can be applied to a vector rapidly in O(N log N ) operations. IDBF is a general framework for nearly optimal fast matvec useful in a wide range of applications, e.g., special function transformation, Fourier integral operators, high-frequency wave computation. Numerical results are provided to demonstrate the effectiveness of the butterfly factorization and its construction algorithms.We will describe IDBF in detail in this section. For the sake of simplicity, we assume that N = 2 L n 0 , where L is an even integer, and n 0 = O(1) is the number of column or row indices in a leaf in the dyadic trees of row and column spaces, i.e., T X and T Ω , respectively. Let's briefly introduce the