MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities

Maffey, Katherine R.; Kyle, Dotterrer,; Niemann, Jennifer; Cruickshank, Iain J.; Lewis, Grace A.; Kästner, Christian

doi:10.1109/icse-nier58687.2023.00012

Cited by 3 publications

(1 citation statement)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It will not be news to most readers that translating high-level and generalised principles of fairness, bias, or transparency into an actionable and, by extension, auditable obligation can prove exceptionally challenging [26,30,51,119]. Similarly, negotiating benchmarks for quality assessment internally can itself prove a laborious and demanding process as siloed teams can pose significant communication problems and inefficiencies [66,82,89,116]. Unsurprisingly, there has been a plethora of approaches and toolkits for assessing algorithmic systems, with varied yardsticks for measurement and evaluation depending on the nature of the algorithmic system in question [21,40,73,86,91,124,129].…”

Section: Whose Benchmarks? Whose Methods?mentioning

confidence: 99%

Law and the Emerging Political Economy of Algorithmic Audits

Terzis,

Veale,

Gaumann

2024

Preprint

View full text Add to dashboard Cite

Accepted in the Proceedings of the 2024 ACM Conference on Fairness, Accountability and Transparency. For almost a decade now, scholarship in and beyond the ACM FAccT community has been focusing on novel and innovative ways and methodologies to audit the functioning of algorithmic systems. Over the years, this research idea and technical project has matured enough to become a regulatory mandate. Today, the Digital Services Act (DSA) and the Online Safety Act (OSA) have established the framework within which technology corporations and (traditional) auditors will develop the ‘practice’ of algorithmic auditing thereby presaging how this ‘ecosystem’ will develop. In this paper, we systematically review the auditing provisions in the DSA and the OSA in light of observations from the emerging industry of algorithmic auditing. Who is likely to occupy this space? What are some political and ethical tensions that are likely to arise? How are the mandates of ‘independent auditing’ or ‘the evaluation of the societal context of an algorithmic function’ likely to play out in practice? By shaping the picture of the emerging political economy of algorithmic auditing, we draw attention to strategies and cultures of traditional auditors that risk eroding important regulatory pillars of the DSA and the OSA. Importantly, we warn that ambitious research ideas and technical projects of/for algorithmic auditing may end up crashed by the standardising grip of traditional auditors and/or diluted within a complex web of (sub-)contractual arrangements, diverse portfolios, and tight timelines.

show abstract