Physical performance is an essential factor for becoming a top elite team handball player; however, the relationship between specific and general physical performance is not well known. Consequently, the aim of the study was (1) to analyze the relationship between specific game-based and general physical performance in young elite male team handball players, and (2) to reduce the number of tests for a more practical implementation of physical performance diagnostics in team handball. Twenty young adult elite male team handball field players (18.6 ± 2.1 years) performed the team handball game-based performance test (GBPT), including specific movements in offense and defense such as catching, passing and throwing a ball, sprinting, stops, changes in direction, jumping, checking and screening, as well as general tests, including a 20 m sprinting test, a repeated sprint ability test (RSA), a modified t-test, countermovement (CMJ), squat (SJ) and drop jump test (DJ), a standing long jump test, a single-leg lateral three jumps test, a standing throw test, and the determination of the one repetition maximum (1RM) in the bench press, bench pull, front squat, and deadlift. Significant correlations were mostly found between different sprinting and jumping tests as well as between different strength tests. The principal component factor analysis revealed four components (power and speed, strength, jump shot performance, and endurance) including 21 variables of high loads (>0.60 or <−0.60). Due to the correlations between the different tests, we suggest a more practicable testing procedure including the 20 m sprinting test, CMJ test, 1RM in the bench press and front squat, as well as the GBPT for elite teams, or the t-test and the YoYo intermittent recovery test for youth and non-elite teams. Despite some correlations between specific and general tests, we suggest using the GBPT to measure specific performance in team handball.