Non-IIDness Learning in Behavioral and Social Data

Abstract

Most of the classic theoretical systems and tools in statistics, data mining and machine learning are built on the fundamental assumption of IIDness, which assumes the independence and identical distribution of underlying objects, attributes and/or values. However, complex behavioral and social problems often exhibit strong couplings and heterogeneity between values, attributes and objects (i.e., non-IIDness). This fundamentally challenges the IIDness-based learning methodologies and techniques. This paper presents a high-level overview of the needs, challenges and opportunities of non-IIDness learning for handling complex behavioral and social problems. By reviewing the nature and issues of classic IIDness-based algorithms in frequent pattern mining, clustering and classification to complex behavioral and social applications, concepts, structures, frameworks and exemplar techniques are discussed for non-IIDness learning. Case studies, related work and prospects of non-IIDness learning are presented. Non-IIDness learning is also a fundamental issue in big data analytics.

Data Science: Nature and Pitfalls

“…Many real-life problems fall into this level of complexities and challenges, as shown in the extreme data challenge, and they have not been addressed well; for example, • Understanding group behaviors by multiple actors where there are complex interactions and relationships, such as in the manipulation of large-scale cross-capital markets pool by internationally collaborative investors [12], each of whom plays a specific role by connecting information from the underlying markets, social media, other financial markets, socio-economic data and policies [7];…”

Reliability Analysis Based on Coupled Relationship

“…After obtaining the reliability life data we can analyze the data further through data mining and expect to acquire the relationship between attributes and life. L. Cao proposed the non-IIDness learning which thinks attribute values and attributes are usually coupled [1]. Wang et al put forward the coup nominal similarity in unsupervised learning [4], and analyzed coupled attribute on numerical data [5].…”

“…Then, based on the extended reliability lifetime information table and the revised correlation coefficient, we use ) ( Table 2, the correlation matrix of attribute 1 A and attribute 2 A is as follow: …”

“…However, in the practical situation, according to non-IIDness theory [1], the attributes of objects are more or less interacted and coupled. As Figure 1 shows, the different attributes impact each other.…”

Reimaging Research Methodology as Data Science

“…Since data may be structured or unstructured and may have multiple hierarchies, existing statistical methods (e.g., association, correlation, dependence, and causality theories and systems) cannot adequately handle data complexity (Cao, 2013) [9]. Working with complex data necessitates the development of new methods and analytic tools Gibson and Ifenthaler, 2017;Ifenthaler, 2016;Metzler et al, 2016) [10][11][12][13].…”