We develop a three-dimensional flux line lattice model in the layered superconductors, taking into account both long-range magnetic interactions between pancake vortices located in different layers for a single flux line and those between vortices located in the same layer for different flux lines. Using Langevin dynamic simulations, we study the disorder and thermally driven melting transitions from a disentangled Bragg glass ͑BG͒ with quasi-long-range order to an entangled amorphous vortex glass ͑VG͒ or a vortex liquid ͑VL͒ in the disorder strength-temperature phase diagram. Owing to nonmonotonous temperature dependence of the interactions between vortices, the BG-VG melting line exhibit unusual temperature behavior, reproducing the inverse melting phenomenon observed recently on BSCCO crystals.