Show simple item record

dc.contributor.authorKlie, Jan-Christoph
dc.contributor.authorEckart de Castilho, Richard
dc.contributor.authorGurevych, Iryna
dc.descriptionThis is the accompanying data for the paper "Analyzing Dataset Annotation Quality Management in the Wild". Data quality is crucial for training accurate, unbiased, and trustworthy machine learning models and their correct evaluation. Recent works, however, have shown that even popular datasets used to train and evaluate state-of-the-art models contain a non-negligible amount of erroneous annotations, bias or annotation artifacts. There exist best practices and guidelines regarding annotation projects. But to the best of our knowledge, no large-scale analysis has been performed as of yet on how quality management is actually conducted when creating natural language datasets and whether these recommendations are followed. Therefore, we first survey and summarize recommended quality management practices for dataset creation as described in the literature and provide suggestions on how to apply them. Then, we compile a corpus of 591 scientific publications introducing text datasets and annotate it for quality-related aspects, such as annotator management, agreement, adjudication or data validation. Using these annotations, we then analyze how quality management is conducted in practice. We find that a majority of the annotated publications apply good or very good quality management. However, we deem the effort of 30% of the works as only subpar. Our analysis also shows common errors, especially with using inter-annotator agreement and computing annotation error rates.de_DE
dc.rightsCreative Commons Attribution-NonCommercial 4.0
dc.subjectquality managementde_DE
dc.subject.classification409-05 Interaktive und intelligente Systeme, Bild- und Sprachverarbeitung, Computergraphik und Visualisierungde_DE
dc.titleAnalyzing Dataset Annotation Quality Management in the Wildde_DE

Files in this item


This item appears in the following Collection(s)

Show simple item record

Creative Commons Attribution-NonCommercial 4.0
Except where otherwise noted, this item's license is described as Creative Commons Attribution-NonCommercial 4.0