User-centered &amp; Robust NLP OSS: Lessons Learned from Developing &amp; Maintaining RSMTool

Madnani, Nitin; Loukina, Anastassia

doi:10.18653/v1/2020.nlposs-1.20

Cited by 2 publications

(2 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While conversational, LLM-based online services have certainly grown more popular (Zhou et al, 2023), and NLP libraries and pipelines are increasingly accessible to even hobbyist programmers (Madnani & Loukina, 2020), these tools are out of reach for people who end up doing the most repetitive writing tasks. (Kinnula et al, 2021) This highlights the importance of addressing accessibility considerations in the development and deployment of NLP technology.…”

Section: No -3mentioning

confidence: 99%

Assessing Usability of Large Language Models in Education

Huovinen

2024

Preprint

View full text Add to dashboard Cite

The aim of the final year project was to investigate the potential of utilizing Large Language Models (LLMs) in automating and enhancing the process of curriculum development.The project was carried out through the development and implementation of the LLM-based web application, followed by usability testing using the cognitive walkthrough approach and user feedback sessions. Test users were presented with the tool and tasked with evaluating its effectiveness, clarity, and overall usability in the context of curriculum planning. Feedback from users highlighted the importance of clear guidance, integration of tasks into a centralized tool, and the need for human input in verifying and refining generated content, to ensure accurate and meaningful goals in curriculum design. These findings will guide the further development and refinement of the application.

show abstract

Section: No -3mentioning

confidence: 99%

Assessing Usability of Large Language Models in Education

Huovinen

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…The Rater Scoring Modeling Tool (Madnani & Loukina, 2020) is an open source Python‐based toolkit for automatic scoring. It provides ways to train scoring models using scikit‐learn (Pedregosa et al., 2011) as its backbone and evaluates the results with a wide variety of psychometric measures and evaluation metrics.…”

Section: Toolsmentioning

confidence: 99%

To Score or Not to Score: Factors Influencing Performance and Feasibility of Automatic Content Scoring of Text Responses

Zesch

Horbach

Zehner

2023

Educational Measurement

View full text Add to dashboard Cite

In this article, we systematize the factors influencing performance and feasibility of automatic content scoring methods for short text responses. We argue that performance (i.e., how well an automatic system agrees with human judgments) mainly depends on the linguistic variance seen in the responses and that this variance is indirectly influenced by other factors such as target population or input modality. Extending previous work, we distinguish conceptual, realization, and nonconformity variance, which are differentially impacted by the various factors. While conceptual variance relates to different concepts embedded in the text responses, realization variance refers to their diverse manifestation through natural language. Nonconformity variance is added by aberrant response behavior. Furthermore, besides its performance, the feasibility of using an automatic scoring system depends on external factors, such as ethical or computational constraints, which influence whether a system with a given performance is accepted by stakeholders. Our work provides (i) a framework for assessment practitioners to decide a priori whether automatic content scoring can be successfully applied in a given setup as well as (ii) new empirical findings and the integration of empirical findings from the literature on factors that influence automatic systems' performance.

show abstract

User-centered & Robust NLP OSS: Lessons Learned from Developing & Maintaining RSMTool

Cited by 2 publications

References 20 publications

Assessing Usability of Large Language Models in Education

Assessing Usability of Large Language Models in Education

To Score or Not to Score: Factors Influencing Performance and Feasibility of Automatic Content Scoring of Text Responses

Contact Info

Product

Resources

About