What We Don’t Test: What an Analysis of Unreleased ACS Exam Items Reveals about Content Coverage in General Chemistry Assessments

Abstract

General chemistry courses are often the foundation for the study of other science disciplines and upper-level chemistry concepts. Students who take introductory chemistry courses are more often from health and sciencerelated fields than chemistry. As such, the content taught and assessed in general chemistry courses is envisioned as building blocks for concepts in other science courses and across many science disciplines. American Chemical Society (ACS) exams are developed by committees of expert chemists and serve as representative artifacts of content valued by the chemistry community. Before an exam is released, items developed by an examination committee undergo trial testing, and items deemed too easy, too hard, or not discriminating based on item statistics are removed. Analysis of content coverage from items on multiple released general chemistry ACS exams from the past decade and their associated unreleased trial tests revealed content areas where few exam items were written. Content coverage was analyzed by aligning exam items to the ACS Anchoring Concepts Content Map for General Chemistry. By comparing released and unreleased trial items, we found that many of the content coverage gaps on released exams are the result of a lack of items developed about these concepts. These conceptual holes warrant a discussion of whether what is being assessed is what is desired to be assessed and how curricular improvement and assessment reform efforts address content coverage.

The American Chemical Society Exams Institute Undergraduate Chemistry Anchoring Concepts Content Map V: Analytical Chemistry

“…A new level 2 statement was proposed, further vetted, and edited and included in the final Analytical ACCM: [9H] “ Materials are frequently mixtures of chemical substances ”. This was included at the end of the other seven previous enduring understandings [9A–9G] in order to preserve alignments for the other subdisciplines. − This new Enduring Understanding will now be included in all existing ACCMs. The complete new Enduring Understanding with statements at the level 3 and 4 for analytical chemistry is provided in Table .…”

Adapting the Anchoring Concepts Content Map (ACCM) of ACS Exams by Incorporating a Theme: Merging Green Chemistry and Organic Chemistry

“…In order to confirm the validity of the alignment data, a process regularly utilized by ACS Exams for alignment validation was used where approximately 10% of the items were aligned by two additional researchers and the alignment data was compared for consistency. 18,19 As a final comparison of the original and revised ACCMs, a comparison of the number of statements within each level was completed.…”

“…For the final stage of the development (noted in Figure ), the alignment was conducted using two trial tests of an organic chemistry ACS Exam to compare the existing organic chemistry ACCM with the GC/OC ACCM (140 items total). In order to confirm the validity of the alignment data, a process regularly utilized by ACS Exams for alignment validation was used where approximately 10% of the items were aligned by two additional researchers and the alignment data was compared for consistency. , As a final comparison of the original and revised ACCMs, a comparison of the number of statements within each level was completed.…”

“…Second, the alignment discussions presented here both for the statements (such as those that might be part of a set of learning objectives for a curriculum) and the assessment items can provide a useful template to discuss altering a content curriculum. In the case of incorporating green chemistry into organic, this could include adding a theme into the curriculum, by adjusting some coverage of major concepts in the curriculum or identifying high concentrations or lack of coverage ,, through curriculum evaluation. Finally, because this project considered the inclusion of green chemistry into organic chemistry, it can be used as a means to identify ways that green chemistry can enhance an organic chemistry curriculum.…”

Should Chemistry Education Be Doing More to Support the Teaching of the Central Dogma of Molecular Biology?

“…As a chemist looking at this list of topics that pose conceptualization challenges for students, there are opportunities to connect commonly taught chemistry concepts. For example, the role of intermolecular forces is an underlying aspect of several of these molecular biology concepts; however, evidence suggests that the role of such noncovalent interactions in large molecules has found little traction in the content curriculum of general chemistry courses. , Thus, while the foundations of intermolecular forces remain an important learning outcome in the college chemistry classroom or lecture hall, the application of those foundations in larger systems, including molecular biology, may need to be more thoroughly considered.…”