Systematic review of research methods: the case of business instruction

Abstract

Purpose -The purpose of this paper is to assess the body of business instruction literature by academic librarians against evolving models for evidence-based research. Design/methodology/approach -The paper used systematic review and inter-rater reliability of the literature of business information research instruction to test two attributes of research quality: the evidence-based levels of evidence and the EBLIP critical analysis checklist. Findings -Intervention questions and case studies are the most popular research methods on the EBL levels of evidence scale. The majority of articles score below 75 on the EBLIP critical appraisal checklist. Prediction questions are represented by higher levels of evidence and study quality. Intervention questions paired with the cohort design and exploratory questions paired with survey design indicate strong areas of research quality. The case study method, while most popular, showes lower scores across all question types yet revealed some high-quality benchmark examples. Research limitations/implications -Error is possible when distinguishing between cohort and case study -some articles may fall into one or the other study design. Rater training was conducted only once, and best practices for inter-rater reliability recommend multiple rounds to achieve higher rater agreement. Practical implications -Recommendations are presented for ways to improve the evidence base of research articles and suggest areas for professional development opportunities for librarian researchers wishing to increase the quality of research publications. Originality/value -The paper goes beyond the narrative review of the literature of business instruction to measure the research methods employed in those publications against two evidence-based standards. The results will show where the literature stands as a maturing discipline and provide recommendations for increasing the levels of evidence for future research.

Evaluating Approaches to Quality Assessment in Library and Information Science LIS Systematic Reviews: A Methodology Review

“…Eight studies defined quality as an assessment of the methodological quality (Koufogiannakis and Wiebe, 2006;Zhang, Watson, & Banfield, 2007;Joshi & Trout, 2014;Golder & Loke, 2009Perrier et al, 2014;Gagnon et al, 2010;Divall, Camosso-Stefinovic, & Baker., 2013), and three studies (Perrier et al, 2014;Gagnon et al, 2010;Divall et al, 2013) specified quality as the risk of bias. Two studies defined quality assessment as assessing the quality of the study design, or level of evidence (Manning Fiegen, 2010;Ndabarora, Chipps, & Uys, 2014). One study assessed the quality of the reporting (Crumley, Wiebe, Cramer, Klassen, & Hartling, 2005).…”

“…The number of authors undertaking quality assessment in a systematic review ranged from 1 to 8. Three studies (Joshi & Trout, 2014;Manning Fiegen, 2010;Perrier et al, 2014) reported an assessment of the inter-rater agreement of quality assessment between reviewers. Nine studies reported using quality assessment as part of their inclusion and exclusion criteria.…”

“…Twenty-one studies were classed as inadequate. Of these, 18 studies (Bergman & Holden, 2010;Beverley et al, 2004;Booth et al, 2009;Brennan, Mattick, & Ellis, 2011;Brettle, 2003Brettle, , 2007Brown, 2008;Catalano, 2013;Crumley et al, 2005;Golder & Loke, 2010;Kelly & Sugioto, 2013;Koufogiannakis & Wiebe, 2006;Manning Fiegen, 2010;Ndabarora et al, 2014;Perrier et al, 2014;Sommestad et al, 2014;Winning & Beverley, 2003) reported summary data only. Thus, the systematic reviews were either unclear about which of the included studies met each of the criteria assessed, or the systematic reviews were unclear about what criteria was used to assess the included studies.…”

“…3 (Beverley et al, 2004;Rankin et al, 2008;Winning & Beverley, 2003) 3 (Bergman et al, 2010;Catalano, 2013;Manning Fiegen, 2010) 3 (Brettle, 2003(Brettle, a , 2007Brettle et al, 2011) Long, A. F., Godfrey, M., Randall, T., Brettle, A., & Grant, M. J. (2002b).…”

“…Only one quarter (10 of 40) of the systematic reviews included in this analysis adequately assessed and documented the scientific quality of the included studies (AMSTAR item #7; Shea et al, 2007). Twenty studies were assessed as inadequate because, although quality assessment was documented, 14 studies (Ankem, 2005;Bergman & Holden, 2010;Brennan et al, 2011;Brettle, 2003Brettle, , 2007Brown, 2008;Crumley et al, 2005;Golder & Loke, 2010;Gray et al, 2012;Koufogiannakis and Wiebe 2006;Manning Fiegen, 2010;Ndabarora et al, 2014;Rankin 2008;Winning & Beverley, 2003) failed to report "some kind of result for each study" (AMSTAR criteria #7; Shea et al, 2007, p. 5) and six studies (Ankem, 2006a;Fanner & Urquhart, 2008;Genero, 2011;, Kelly & Sugimoto, 2013;Matteson et al, 2011;Wagner & Byrd, 2004) did not report their quality assessment methods a priori. In one study (Perrier et al, 2014), determining whether quality assessment was documented and assessed in accordance with the AMSTAR item #7 was not possible because the link to the online supplementary table detailing the quality assessment was unavailable.…”