Show simple item record

dc.contributor.authorKlie, Jan-Christoph
dc.contributor.authorWebber, Bonnie
dc.contributor.authorGurevych, Iryna
dc.descriptionThis is the accompanying data for our paper "Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future". Annotated data is an essential ingredient in natural language processing for training and evaluating machine learning models. It is therefore very desirable for the annotations to be of high quality. Recent work, however, has shown that several popular datasets contain a surprising number of annotation errors or inconsistencies. To alleviate this issue, many methods for annotation error detection have been devised over the years. While researchers show that their approaches work well on their newly introduced datasets, they rarely compare their methods to previous work or on the same datasets. This raises strong concerns on methods’ general performance and makes it difficult to assess their strengths and weaknesses. We therefore reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets for text classification as well as token and span labeling. In addition, we define a uniform evaluation setup including a new formalization of the annotation error detection task, evaluation protocol, and general best practices. To facilitate future research and reproducibility, we release our datasets and implementations in an easy-to-use and open source software package.de_DE
dc.rightsCreative Commons Attribution-NonCommercial 4.0
dc.subjectannotation error detectionde_DE
dc.subject.classification409-05 Interaktive und intelligente Systeme, Bild- und Sprachverarbeitung, Computergraphik und Visualisierungde_DE
dc.titleAnnotation Error Detection: Analyzing the Past and Present for a More Coherent Futurede_DE

Files in this item


This item appears in the following Collection(s)

Show simple item record

Creative Commons Attribution-NonCommercial 4.0
Except where otherwise noted, this item's license is described as Creative Commons Attribution-NonCommercial 4.0