Statistical Challenges of Administrative and Transaction Data

Hand, David J.

doi:10.1111/rssa.12315

Cited by 101 publications

(99 citation statements)

References 123 publications

(134 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…(). A number of authors warn that these issues have the potential to provide misleading research outcomes and decisions; see, for example, Boyd and Crawford (), Wigan and Clarke (), Harford (), Hand (, ) and Fiebig ()…”

Section: A Statistical Toolboxmentioning

confidence: 99%

“…In particular, Harford () and Hand () make a point that gathering more and more data in an effort to achieve ‘ T = ALL’ is impossible, with potentially undesirable consequences. This also violates the notion of random sampling underlying the statistical theory of inference, as Hand (; p. 567) points out. Fiebig () argues that big data is not synonymous with better or more informative data, pointing out that it is often subject to missing observations and measurement errors.…”

Section: A Statistical Toolboxmentioning

confidence: 99%

“…A number of authors from many areas of science raise the point that large data sets often suffer from a range of problems (sampling error, selection bias, measurement error, missing information, aggregation error, etc. ), which can severely bias research outcomes and deliver poor decisions (see, for example, Boyd and Crawford, ; Wigan and Clarke, ; Harford, ; Hand, , ; and Fiebig, ). More problematic is that these biases can be magnified with increasing sample size, as Kaplan et al .…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Tackling False Positives in Business Research: A Statistical Toolbox With Applications

Kim

2018

Journal of Economic Surveys

View full text Add to dashboard Cite

Serious concerns have been raised that false positive findings are widespread in empirical research in business disciplines. This is largely because researchers almost exclusively adopt the ‘p‐value less than 0.05’ criterion for statistical significance; and they are often not fully aware of large‐sample biases which can potentially mislead their research outcomes. This paper proposes that a statistical toolbox (rather than a single hammer) be used in empirical research, which offers researchers a range of statistical instruments, including a range of alternatives to the p‐value criterion such as the Bayesian methods, optimal significance level, sample size selection, equivalence testing and exploratory data analyses. It is found that the positive results obtained under the p‐value criterion cannot stand, when the toolbox is applied to three notable studies in finance.

show abstract

Section: A Statistical Toolboxmentioning

confidence: 99%

Section: A Statistical Toolboxmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Tackling False Positives in Business Research: A Statistical Toolbox With Applications

Kim

2018

Journal of Economic Surveys

View full text Add to dashboard Cite

show abstract

“…Much, and perhaps all, that is at issue in [15] is very important for all that is under discussion here. This reference, [15], describes the problems of data quality, in the Big Data context, relating to administrative data. Hence, data curation is very relevant for reproducibility of analytics.…”

Section: Integration Of Data and Analytics: Context Of Applicationsmentioning

confidence: 99%

The Development of Data Science: Implications for Education, Employment, Research, and the Data Revolution for Sustainable Development

Murtagh

Devlin

2018

BDCC

View full text Add to dashboard Cite

Abstract:In Data Science, we are concerned with the integration of relevant sciences in observed and empirical contexts. This results in the unification of analytical methodologies, and of observed and empirical data contexts. Given the dynamic nature of convergence, the origins and many evolutions of the Data Science theme are described. The following are covered in this article: the rapidly growing post-graduate university course provisioning for Data Science; a preliminary study of employability requirements, and how past eminent work in the social sciences and other areas, certainly mathematics, can be of immediate and direct relevance and benefit for innovative methodology, and for facing and addressing the ethical aspect of Big Data analytics, relating to data aggregation and scale effects. Associated also with Data Science is how direct and indirect outcomes and consequences of Data Science include decision support and policy making, and both qualitative as well as quantitative outcomes. For such reasons, the importance is noted of how Data Science builds collaboratively on other domains, potentially with innovative methodologies and practice. Further sections point towards some of the most major current research issues.

show abstract

“…Much, and perhaps all, that is at issue in [17] is very important for all that is under discussion here. This reference, [17], describes the problems of data quality, in the Big Data context, relating to administrative data. Hence data curation is very relevant for reproducibility of analytics.…”

Section: Integration Of Data and Analytics: Context Of Applicationsmentioning

confidence: 99%

Data Science in Education, Employment, Research: Data Revolution for Sustainable Development

Murtagh¹,

Devlin²

2018

Preprint

View full text Add to dashboard Cite

In Data Science we are concerned with the integration of relevant sciences in observed and empirical contexts. This results in the unification of analytical methodologies, and of observed and empirical data contexts. Given the dynamic nature of convergence, described are the origins and many evolutions of the Data Science theme. The following are covered in this article: the rapidly growing post-graduate university course provisioning for Data Science; a preliminary study of employability requirements, and how past eminent work in the social sciences and other areas, certainly mathematics, can be of immediate and direct relevance and benefit for innovative methodology, and for facing and addressing the ethical aspect of Big Data analytics, relating to data aggregation and scale effects. Associated also with Data Science is how direct and indirect outcomes and consequences of Data Science include decision support and policy making, and both qualitative as well as quantitative outcomes. For such reasons, the importance is noted of how Data Science builds collaboratively on other domains, potentially with innovative methodologies and practice. Further sections point towards some of the most major current research issues.Keywords: Big Data training and learning; company and business requirements; ethics; impact; decision support; data engineering; open data; smart homes; smart cities; IoT Data Science as the Convergence and Bridging of DisciplinesIn [23] at issue are: parallels between astronomy and Earth science data, methodology transfer, and metadata and ontologies characterized as being crucial. The convergence or bridging of disciplines must address "non-homogeneous observables, and varied spatial, temporal coverage at different resolutions". Then, given computational support, "it is the complexity more than the data volume that proves to be a bigger challenge". Further benefits of this Data Science convergence are termed here tractability and reproducibility. There is discussion in section 2 of [23] of the complexity relating to resolution and distributions. In [26], this is also characterized in terms data of encoding. Plenty of work now emphasizes the importance of p-adic data encoding (binary or ternary when p = 2 or 3), compared with real-valued encoding (m-adic, especially when m = 10).The convergence and bridging of disciplines are emphasized in [23], as follows. "Methodology transfer can almost never be unidirectional. Diverse fields grow by learning tricks employed by other disciplines. The important thing is to abstract data -described by meaningful metadata -and the metadata in turn connected by a good ontology." Further description is at issue in regard to Data Science: "We have described here a few techniques from astroinformatics that are finding use in geoinformatics. There would be many from earth science that space science would do well to emulate. Even other disciplines like bioinformatics provide ample opportunities for methodology transfer and collaboration. With growing data volum...

show abstract

Statistical Challenges of Administrative and Transaction Data

Cited by 101 publications

References 123 publications

Tackling False Positives in Business Research: A Statistical Toolbox With Applications

Tackling False Positives in Business Research: A Statistical Toolbox With Applications

The Development of Data Science: Implications for Education, Employment, Research, and the Data Revolution for Sustainable Development

Data Science in Education, Employment, Research: Data Revolution for Sustainable Development

Contact Info

Product

Resources

About