Internet of Drones (IoD) facilitates the autonomous operations of drones into every application (warfare, surveillance, photography, etc) across the world. The transmission of data (to and fro) related to these applications occur between the drones and the other infrastructure over wireless channels that must abide to the stringent latency restrictions. However, relaying this data to the core cloud infrastructure may lead to a higher round trip delay. Thus, we utilize the cloud close to the ground, i.e., edge computing to realize an edge-envisioned IoD ecosystem. However, as this data is relayed over an open communication channel, it is often prone to different types of attacks due to it wider attack surface. Thus, we need to find a robust solution that can maintain the confidentiality, integrity, and authenticity of the data while providing desired services. Blockchain technology is capable to handle these challenges owing to the distributed ledger that store the data immutably. However, the conventional block architecture pose several challenges because of limited computational capabilities of drones. As the size of blockchain increases, the data flow also increases and so does the associated challenges. Hence, to overcome these challenges, in this work, we have proposed a derived blockchain architecture that decouples the data part (or block ledger) from the block header and shifts it to off-chain storage. In our approach, the registration of a new drone is performed to enable legitimate access control thus ensuring identity management and traceability. Further, the interactions happen in the form of transactions of the blockchain. We propose a lightweight consensus mechanism based on the stochastic selection followed by a transaction signing process to ensure that each drone is in control of its block. The proposed scheme also handles the expanding storage requirements with the help of data compression using a shrinking block mechanism. Lastly, the problem of additional delay anticipated due to drone mobility is handled using a multi-level caching mechanism. The proposed work has been validated in a simulated Gazebo environment and the results are promising in terms of different metrics. We have also provided numerical validations in context of complexity, communication overheads and computation costs.