A modified model considering effects of density as well as conductivity of nanoparticles is used to investigate the instability of a binary nanofluid layer. It is assumed that volume fraction of nanoparticles is small and remains constant at the initial state which leads to very interesting and useful results. The perturbed equations so found are analyzed using normal modes and weighted residual method. It is found that oscillatory motions are not possible and instability is invariably through stationary mode. After solving the problem analytically, numerical solutions are found for metallic (aluminium, copper, silver, iron) and non-metallic (alumina, silica, titanium oxide, copper oxide) nanoparticles using the software Mathematica. The effects of size of nanoparticles, difference in solute concentration, volume fraction of nanoparticles, difference in temperature, conductivity and density of nanoparticles are studied on the onset of convection. The increase in density of nanoparticles destabilizes the fluid layer system where as increase in conductivity stabilizes the same. Lower density of aluminium makes it more stable than other nanoparticles in spite of having its lower conductivity. Metals are largely more stable than non-metals.