Identification of nonlinear structural dynamics has received a significant attention during last decades. Yet, there are many aspects of the identification methods of nonlinear structural models to be improved. The main objective of this study is to introduce novel identification approaches for nonlinear structures. The first step in identifying nonlinear structural elements is to detect their exact location. Hence, the first section of this study focuses on the localization of nonlinear elements in structural dynamics utilizing base excitation measured data. To this end, a localization approach is used to find the location of nonlinear electromagnetic restoring force applied to the tip of a cantilever beam.Inferring the exact location of nonlinear elements, identification methods are utilized to identify and characterize the mathematical model of nonlinear structures. However, various sources of noise and error may affect the accuracy of the identified model. Therefore, in the second part of the thesis, the effect of various sources of inaccuracy on the results of nonlinear model identification is investigated. It is shown that measurement noise, expansion error, modelling error, and neglecting the effect of higher harmonics may lead to an erroneously identified model.An optimization-based framework for the identification of nonlinear systems is proposed in this work in order to avoid the bottlenecks mentioned above. The introduced method is applied to a test rig composed of a base-excited cantilever beam subjected to an electromagnetic force at the tip. According to the nonlinear response of the system, four different functions are assumed as candidate models for the unknown nonlinear electromagnetic force. The measured response is compared with the reconstructed response using various models and the most appropriate mathematical model is selected.Utilizing optimization-based identification method to characterize complex mathematical models with large number of unknown parameters would be computationally expensive. Therefore, this study introduces a harmonic-balance-based parameter estimation method for the identification of nonlinear structures in the presence of multi-harmonic response and force. For this purpose, a method with two different approaches are introduced: Analytical Harmonic-Balance-based (AHB) approach and the Alternating Frequency/Time approach using Harmonic Balance (AFTHB). The method is applied to five simulated examples of nonlinear systems to highlight different features of the method. The method can be applied to all forms of both smooth and non-smooth nonlinear functions. The computational cost is relatively low since a dictionary of candidate basis functions is avoided. The results illustrate that neglecting higher harmonics, particularly in systems with multi-harmonic response and force, may lead to an inaccurate identified model. The AFTHB approach benefits from including significant harmonics of the response and force. Applying this method leads to accurate algebraic equations for each harmonic, including the effect of higher harmonics without truncated error. In the last part of this study, the AFTHB method is applied to two experimental case studies and identifies the nonlinear mathematical model of the structures. The first case is composed of a cantilever beam with a nonlinear electromagnetic restoring force applied to the tip which is excited by a multi-harmonic external force. In the second experimental case study, a configuration of linear springs applies a geometric nonlinear restoring force to the tip of a cantilever beam resulting in internal resonance in the dynamics of the system. The good performance of the AFTHB approach in estimating the unknown parameters of the structure is illustrated by the results of identification.