Radiation monitoring data is very important to support the safety of nuclear installation. The data should be ensured generated and supplied by an authenticated sensor. The new sensor should apply mutual authentication to server before joint the system. This research proposes a high secure authentication protocol based on Eight Prime Number of Modified RSA Algorithm (EPNR) method as certificateless mechanism to protect authenticity and confidentiality of sensor's identity. The main advantages of the protocol are designed for simpler direct point to point connection, using one-time random keys, and implementing five-barrier data protection mechanism. Its performances in context security, running time, and Quality of Services have been compared to other models. The proposed protocol is faster 29.9 times in random key generation, 7.27 times in encryption, and 11.1 times in decryption duration, compared to the others model based on RSA scheme. It has better performances in throughput and delay time. It also has enhanced resistance to statistical, replay, and MITM attack.