Show simple item record

dc.contributor.authorKlie, Jan-Christoph
dc.contributor.authorWebber, Bonnie
dc.contributor.authorGurevych, Iryna
dc.date.accessioned2023-09-08T14:34:23Z
dc.date.available2023-09-08T14:34:23Z
dc.date.issued2023-03
dc.identifier.urihttps://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/3943
dc.descriptionThis is the accompanying data for our paper "Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future". Annotated data is an essential ingredient in natural language processing for training and evaluating machine learning models. It is therefore very desirable for the annotations to be of high quality. Recent work, however, has shown that several popular datasets contain a surprising number of annotation errors or inconsistencies. To alleviate this issue, many methods for annotation error detection have been devised over the years. While researchers show that their approaches work well on their newly introduced datasets, they rarely compare their methods to previous work or on the same datasets. This raises strong concerns on methods’ general performance and makes it difficult to assess their strengths and weaknesses. We therefore reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets for text classification as well as token and span labeling. In addition, we define a uniform evaluation setup including a new formalization of the annotation error detection task, evaluation protocol, and general best practices. To facilitate future research and reproducibility, we release our datasets and implementations in an easy-to-use and open source software package.de_DE
dc.relationIsSupplementTo;DOI;https://doi.org/10.1162/coli_a_00464
dc.rightsCreative Commons Attribution-NonCommercial 4.0
dc.rights.urihttps://creativecommons.org/licenses/by-nc/4.0/
dc.subjectnlpde_DE
dc.subjectannotation error detectionde_DE
dc.subject.classification4.43-04 Künstliche Intelligenz und Maschinelle Lernverfahrende_DE
dc.subject.classification4.43-05 Bild- und Sprachverarbeitung, Computergraphik und Visualisierung, Human Computer Interaction, Ubiquitous und Wearable Computing
dc.subject.ddc004
dc.titleAnnotation Error Detection: Analyzing the Past and Present for a More Coherent Futurede_DE
dc.typeDatasetde_DE
tud.unitTUDa
tud.history.classificationVersion=2016-2020;409-05 Interaktive und intelligente Systeme, Bild- und Sprachverarbeitung, Computergraphik und Visualisierung


Files in this item

Thumbnail
Thumbnail
Thumbnail

This item appears in the following Collection(s)

Show simple item record

Creative Commons Attribution-NonCommercial 4.0
Except where otherwise noted, this item's license is described as Creative Commons Attribution-NonCommercial 4.0