6d SCFTs compactified on a circle can often be studied from nonperturbative 5d super-Yang-Mills theories, using instanton solitons. However, the 5d Yang-Mills theories with 6d UV fixed points frequently have too many hypermultiplet matters, which makes it difficult to use the ADHM techniques for instantons. With the examples of 6d N = (1, 0) SCFTs with Sp(N ) gauge symmetry and 2N + 8 fundamental hypermultiplets, we show that one can still make rigorous studies of these 5d-6d relations in the 'fractional D-brane sectors'. We test the recently proposed 5d duals given by Sp(N + 1) gauge theories, and compare their instanton partition functions with the elliptic genera of 6d self-dual strings.