Open Access

'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions

Files

AITDNA.zip (8.33 MB)

Date

2026-06-15

Type

Other

Authors

Dycke, Nils
Sakharova, Marina

Daheim, Nico

Iryna, Gurevych

Description

Although it is generally agreed that AI-generated text poses a broad societal risk, there is no common understanding in the AI-generated text detection literature on what constitutes harmful use. Rather, existing datasets and approaches often define their own criteria and make their own assumptions, sometimes implicitly, and often only loosely related to real-world needs and applications. To address this gap, we here systematically define various notions of AI-generated text and their characteristics. To study these, we collect AITDNA - a new benchmark of human-machine co-constructed texts that is annotated with detailed genesis information, such as the entire edit and AI-interaction history. AITDNA is a dataset of human-AI interactions collected throughout a set of user studies. The dataset contains: 1. Full creation information for each text: raw user edits, model suggestions, user queries etc. 2. Representation of each text with respect to different notions (definitions) of AI-generated text described in the paper. Currently supported notions: - Document-level: one label per document (AI if >=50% of tokens are AI-generated) - Sentence-level: one label per sentence (AI if >=50% of tokens are AI-generated) - Token-level: one label per token - Boundary-level: divide text into N parts by finding most optimal split indices (default N = 5) - Span-level: character-level spans of same authorship (e.g. User: "GPUs are speci", AI: "alized processors",...) - Intent-based: sentence-level labels based on a pre-defined set of rules specifying allowed and forbidden types of user queries. - Content-based: sentence-level labels based on a pre-defined set of rules specifying allowed and forbidden types of model output. - Membership-based: token-level labels based on occurence of N-grams in reference human corpus (default N = 2, reference human corpus = human-only part of the dataset) The dataset is provided in the form of parquet files. A loader script is provided which can be used as: ``` from load_dataset import load_config ds = load_config(name="membership") ``` For more details, please go through the README file included.