---
dataset_info:
- config_name: actionability
  features:
  - name: review_point
    dtype: string
  - name: paper_id
    dtype: string
  - name: venue
    dtype: string
  - name: focused_review
    dtype: string
  - name: batch
    dtype: int64
  - name: actionability
    struct:
    - name: annotators
      sequence: string
    - name: labels
      sequence: string
  - name: actionability_label
    dtype: string
  - name: actionability_label_type
    dtype: string
  - name: id
    dtype: int64
  splits:
  - name: gold
    num_bytes: 1108218
    num_examples: 444
  - name: silver
    num_bytes: 1802899
    num_examples: 764
  - name: hard
    num_bytes: 515201
    num_examples: 224
  - name: full
    num_bytes: 3426318
    num_examples: 1432
  download_size: 3418239
  dataset_size: 6852636
- config_name: addressed_to_author
  features:
  - name: review_point
    dtype: string
  - name: paper_id
    dtype: string
  - name: venue
    dtype: string
  - name: focused_review
    dtype: string
  - name: batch
    dtype: int64
  - name: addressed_to_author
    struct:
    - name: annotators
      sequence: string
    - name: labels
      sequence: string
  - name: addressed_to_author_label
    dtype: string
  - name: addressed_to_author_label_type
    dtype: string
  - name: id
    dtype: int64
  splits:
  - name: gold
    num_bytes: 2492631
    num_examples: 990
  - name: silver
    num_bytes: 724483
    num_examples: 361
  - name: full
    num_bytes: 3217114
    num_examples: 1351
  download_size: 3097695
  dataset_size: 6434228
- config_name: combined_main_aspects
  features:
  - name: review_point
    dtype: string
  - name: paper_id
    dtype: string
  - name: venue
    dtype: string
  - name: focused_review
    dtype: string
  - name: batch
    dtype: int64
  - name: actionability
    struct:
    - name: annotators
      sequence: string
    - name: labels
      sequence: string
  - name: actionability_label
    dtype: string
  - name: actionability_label_type
    dtype: string
  - name: id
    dtype: int64
  - name: grounding_specificity
    struct:
    - name: annotators
      sequence: string
    - name: labels
      sequence: string
  - name: grounding_specificity_label
    dtype: string
  - name: grounding_specificity_label_type
    dtype: string
  - name: verifiability
    struct:
    - name: annotators
      sequence: string
    - name: labels
      sequence: string
  - name: verifiability_label
    dtype: string
  - name: verifiability_label_type
    dtype: string
  - name: helpfulness
    struct:
    - name: annotators
      sequence: string
    - name: labels
      sequence: string
  - name: helpfulness_label
    dtype: string
  - name: helpfulness_label_type
    dtype: string
  splits:
  - name: full
    num_bytes: 3854198
    num_examples: 1430
  download_size: 1682108
  dataset_size: 3854198
- config_name: context_experiment_with_paper_text
  features:
  - name: review_point
    dtype: string
  - name: paper_id
    dtype: string
  - name: venue
    dtype: string
  - name: focused_review
    dtype: string
  - name: batch
    dtype: string
  - name: actionability
    dtype: string
  - name: actionability_label
    dtype: string
  - name: actionability_label_type
    dtype: string
  - name: id
    dtype: string
  - name: grounding_specificity
    dtype: string
  - name: grounding_specificity_label
    dtype: string
  - name: grounding_specificity_label_type
    dtype: string
  - name: verifiability
    dtype: string
  - name: verifiability_label
    dtype: string
  - name: verifiability_label_type
    dtype: string
  - name: helpfulness
    dtype: string
  - name: helpfulness_label
    dtype: string
  - name: helpfulness_label_type
    dtype: string
  - name: paper_path
    dtype: string
  - name: paper_text
    dtype: string
  - name: paper_word_count
    dtype: string
  splits:
  - name: full
    num_bytes: 3545935
    num_examples: 100
  download_size: 1842203
  dataset_size: 3545935
- config_name: grounding_specificity
  features:
  - name: review_point
    dtype: string
  - name: paper_id
    dtype: string
  - name: venue
    dtype: string
  - name: focused_review
    dtype: string
  - name: batch
    dtype: int64
  - name: grounding_specificity
    struct:
    - name: annotators
      sequence: string
    - name: labels
      sequence: string
  - name: grounding_specificity_label
    dtype: string
  - name: grounding_specificity_label_type
    dtype: string
  - name: id
    dtype: int64
  splits:
  - name: gold
    num_bytes: 1505935
    num_examples: 572
  - name: silver
    num_bytes: 1497864
    num_examples: 671
  - name: hard
    num_bytes: 418189
    num_examples: 188
  - name: full
    num_bytes: 3421988
    num_examples: 1431
  download_size: 3400827
  dataset_size: 6843976
- config_name: helpfulness
  features:
  - name: review_point
    dtype: string
  - name: paper_id
    dtype: string
  - name: venue
    dtype: string
  - name: focused_review
    dtype: string
  - name: batch
    dtype: int64
  - name: helpfulness
    struct:
    - name: annotators
      sequence: string
    - name: labels
      sequence: string
  - name: helpfulness_label
    dtype: string
  - name: helpfulness_label_type
    dtype: string
  - name: id
    dtype: int64
  splits:
  - name: gold
    num_bytes: 775527
    num_examples: 300
  - name: silver
    num_bytes: 1924632
    num_examples: 811
  - name: hard
    num_bytes: 720495
    num_examples: 319
  - name: full
    num_bytes: 3420654
    num_examples: 1430
  download_size: 3387958
  dataset_size: 6841308
- config_name: professional_tone
  features:
  - name: review_point
    dtype: string
  - name: paper_id
    dtype: string
  - name: venue
    dtype: string
  - name: focused_review
    dtype: string
  - name: batch
    dtype: int64
  - name: professional_tone
    struct:
    - name: annotators
      sequence: string
    - name: labels
      sequence: string
  - name: professional_tone_label
    dtype: string
  - name: professional_tone_label_type
    dtype: string
  - name: id
    dtype: int64
  splits:
  - name: gold
    num_bytes: 3357730
    num_examples: 1396
  - name: silver
    num_bytes: 45455
    num_examples: 26
  - name: full
    num_bytes: 3403185
    num_examples: 1422
  download_size: 3316675
  dataset_size: 6806370
- config_name: valid_point
  features:
  - name: review_point
    dtype: string
  - name: paper_id
    dtype: string
  - name: venue
    dtype: string
  - name: focused_review
    dtype: string
  - name: batch
    dtype: int64
  - name: valid_point
    struct:
    - name: annotators
      sequence: string
    - name: labels
      sequence: string
  - name: valid_point_label
    dtype: string
  - name: valid_point_label_type
    dtype: string
  - name: id
    dtype: int64
  splits:
  - name: gold
    num_bytes: 2817466
    num_examples: 1184
  - name: silver
    num_bytes: 456261
    num_examples: 198
  - name: full
    num_bytes: 3273727
    num_examples: 1382
  download_size: 3214375
  dataset_size: 6547454
- config_name: verifiability
  features:
  - name: review_point
    dtype: string
  - name: paper_id
    dtype: string
  - name: venue
    dtype: string
  - name: focused_review
    dtype: string
  - name: batch
    dtype: int64
  - name: verifiability
    struct:
    - name: annotators
      sequence: string
    - name: labels
      sequence: string
  - name: verifiability_label
    dtype: string
  - name: verifiability_label_type
    dtype: string
  - name: id
    dtype: int64
  splits:
  - name: gold
    num_bytes: 445004
    num_examples: 185
  - name: silver
    num_bytes: 2079105
    num_examples: 838
  - name: hard
    num_bytes: 896599
    num_examples: 407
  - name: full
    num_bytes: 3420708
    num_examples: 1430
  download_size: 3367607
  dataset_size: 6841416
configs:
- config_name: actionability
  data_files:
  - split: gold
    path: actionability/gold-*
  - split: silver
    path: actionability/silver-*
  - split: hard
    path: actionability/hard-*
  - split: full
    path: actionability/full-*
- config_name: addressed_to_author
  data_files:
  - split: gold
    path: addressed_to_author/gold-*
  - split: silver
    path: addressed_to_author/silver-*
  - split: full
    path: addressed_to_author/full-*
- config_name: combined_main_aspects
  data_files:
  - split: full
    path: combined_main_aspects/full-*
- config_name: context_experiment_with_paper_text
  data_files:
  - split: full
    path: context_experiment_with_paper_text/full-*
- config_name: grounding_specificity
  data_files:
  - split: gold
    path: grounding_specificity/gold-*
  - split: silver
    path: grounding_specificity/silver-*
  - split: hard
    path: grounding_specificity/hard-*
  - split: full
    path: grounding_specificity/full-*
- config_name: helpfulness
  data_files:
  - split: gold
    path: helpfulness/gold-*
  - split: silver
    path: helpfulness/silver-*
  - split: hard
    path: helpfulness/hard-*
  - split: full
    path: helpfulness/full-*
- config_name: professional_tone
  data_files:
  - split: gold
    path: professional_tone/gold-*
  - split: silver
    path: professional_tone/silver-*
  - split: full
    path: professional_tone/full-*
- config_name: valid_point
  data_files:
  - split: gold
    path: valid_point/gold-*
  - split: silver
    path: valid_point/silver-*
  - split: full
    path: valid_point/full-*
- config_name: verifiability
  data_files:
  - split: gold
    path: verifiability/gold-*
  - split: silver
    path: verifiability/silver-*
  - split: hard
    path: verifiability/hard-*
  - split: full
    path: verifiability/full-*
license: cc-by-nc-sa-4.0
task_categories:
- text-classification
language:
- en
size_categories:
- 1K<n<10K
---


# RevUtil: Measuring the Utility of Peer Reviews for Authors

[📄 Paper](https://aclanthology.org/2025.emnlp-main.1476/)  
[💻 GitHub Repository](https://github.com/bodasadallah/RevUtil)  

---

## 📚 Overview

Providing **constructive feedback** to authors is a key goal of peer review. To support research on evaluating and generating useful peer review comments, we introduce **RevUtil**, a dataset for measuring the utility of peer review feedback.  

RevUtil focuses on four main aspects of review comments:

- **Actionability** – Can the author act on the comment?  
- **Grounding & Specificity** – Is the comment concrete and tied to the paper?  
- **Verifiability** – Can the statement be checked against the paper?  
- **Helpfulness** – Does the comment assist the author in improving their work?  

---

## 🧑‍🔬 RevUtil Human

- **1,430** review comments from real peer reviews.  
- Each comment is annotated independently by **three human raters**.  
- Labels are provided as `"gold"` (3/3 agreement), `"silver"` (2/3), or `"none"` (no agreement).  

**Key columns:**

| Column              | Description                                                                 |
| ------------------- | --------------------------------------------------------------------------- |
| `paper_id`          | ID of the reviewed paper                                                    |
| `venue`             | Conference or journal name                                                  |
| `focused_review`    | Full review (weakness + suggestion sections)                                |
| `review_point`      | Individual review comment being evaluated                                   |
| `id`                | Unique ID for the review point                                              |
| `batch`             | Annotation batch/study identifier                                           |
| `ASPECT`            | Dictionary with annotators and their labels                                 |
| `ASPECT_label`      | Majority label (if available)                                               |
| `ASPECT_label_type` | `"gold"`, `"silver"`, or `"none"`                                           |

---

## 🚀 Usage

You can load the datasets directly via 🤗 Datasets:

```python
from datasets import load_dataset

# Human annotations
human = load_dataset("boda/RevUtil_human")

# Synthetic annotations
synthetic = load_dataset("boda/RevUtil_synthetic")
```



## 📎 Citation
```
@inproceedings{sadallah-etal-2025-good,
    title = "The Good, the Bad and the Constructive: Automatically Measuring Peer Review{'}s Utility for Authors",
    author = {Sadallah, Abdelrahman  and
      Baumg{\"a}rtner, Tim  and
      Gurevych, Iryna  and
      Briscoe, Ted},
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.1476/",
    doi = "10.18653/v1/2025.emnlp-main.1476",
    pages = "28979--29009",
    ISBN = "979-8-89176-332-6"
}
```