In this paper, we propose a scheme for 3D model construction by fusing heterogeneous sensor data. The proposed scheme is intended for use in an environment where multiple, heterogeneous sensors operate asynchronously. Surface depth, orientation, and curvature measurements obtained from multiple sensors and vantage points are incorporated to construct a computer description of the imaged object. The proposed scheme uses Kalman filter as the sensor data integration tool and hierarchical spline surface as the recording data structure. Kalman filter is used to obtain statistically optimal estimates of the imaged surface structure based on possibly noisy sensor measurements. Hierarchical spline surface is used as the representation scheme because it maintains high-order surface derivative continuity, may be adaptively refined, and is storage efficient. We show in this paper how these mathematical tools can be used in designing a modeling scheme to fuse heterogeneous sensor data.