Evaluate how accurately LLMs recommend real academic references. This benchmark verifies every paper citation against four authoritative academic databases—Crossref, PubMed, arXiv, and DBLP—providing objective, reproducible metrics to measure reference hallucination across models and disciplines.

What is Reference Hallucination Arena?

Reference Hallucination Arena is a benchmark designed to evaluate LLMs' ability to recommend real, verifiable academic papers. Unlike subjective evaluation tasks, this benchmark uses fully automated, objective verification: every reference generated by a model is checked against real-world academic databases.

The benchmark addresses a critical problem: when researchers ask LLMs for literature recommendations, models frequently "hallucinate" references—generating papers that sound plausible but do not actually exist. Reference Hallucination Arena quantifies this phenomenon across multiple models and academic disciplines.

The official evaluation dataset is available on HuggingFace: OpenJudge/ref-hallucination-arena.

Key Features:

The evaluation pipeline consists of six automated steps:

Dataset

The evaluation dataset is hosted on HuggingFace: OpenJudge/ref-hallucination-arena.

Each query item in the dataset follows this schema:

{
  "query": "Please recommend papers on Transformer architectures for NLP.",
  "discipline": "computer_science",
  "num_refs": 5,
  "language": "en",
  "year_constraint": {"min_year": 2020}
}

Year Constraint Formats:

You can download the dataset and use it directly, or create your own custom queries following the same format.

How to Run the Evaluation

Follow this workflow to evaluate your models' reference recommendation capabilities:

Interpreting Results

The primary metric is overall accuracy (also called verification rate)—the percentage of references where title, author, and year all exactly match a real paper. Models are ranked by: overall accuracy → year compliance rate → average confidence → completeness (descending):

============================================================
REFERENCE HALLUCINATION ARENA - RANKINGS
============================================================
  1. Model A          [################----] overall=78.4%  title=85.2%  author=80.1%  doi=52.3%  refs=50
  2. Model B          [###############-----] overall=75.2%  title=82.0%  author=77.5%  doi=48.7%  refs=50
  3. Model C          [##############------] overall=72.8%  title=80.3%  author=75.0%  doi=45.2%  refs=50
  4. Model D          [#############-------] overall=69.5%  title=78.1%  author=72.8%  doi=41.5%  refs=50
============================================================

Interpretation:

• > 75% — Excellent: Model rarely hallucinates references
• 60-75% — Good: Most references are real, but some fabrication occurs
• 40-60% — Fair: Significant hallucination, use with caution
• < 40% — Poor: Model frequently fabricates references

Beyond overall rates, examine per-field accuracy for fine-grained insight:

Per-Field Accuracy (Model A):
  Title Accuracy  : 82.3%    # Percentage of titles matching real papers
  Author Accuracy : 68.5%    # Percentage of correct author lists
  Year Accuracy   : 71.2%    # Percentage of correct publication years
  DOI Accuracy    : 45.8%    # Percentage of valid DOIs

This breakdown reveals that models may get titles right but fabricate author names or DOIs—a common pattern where the model "remembers" a paper's topic but not its exact metadata.

Per-Discipline Performance shows which academic fields are most challenging:

Per-Discipline Overall Accuracy:
  Computer Science  : 81.2%
  Biomedical        : 74.5%
  Physics           : 70.3%
  Chemistry         : 65.8%
  Social Science    : 58.1%

Error Analysis

Analyze verification results to understand hallucination patterns and guide model selection:

Verification Status Categories

Each reference receives one of four verification statuses:

Note: Under the current strict verification logic, a reference is only marked VERIFIED when all provided fields (title, author, year) exactly match a real paper. Partial matches (e.g., correct title but wrong authors) are counted as NOT_FOUND with match details preserved for per-field accuracy analysis.

Common Hallucination Patterns

Programmatic Error Analysis

import json

# Load verification results
with open("evaluation_results/ref_hallucination_arena/verification_results.json") as f:
    results = json.load(f)

# Analyze hallucination patterns per model
for model_name, model_results in results.items():
    total = sum(r["total_refs"] for r in model_results)
    verified = sum(r["verified"] for r in model_results)
    not_found = sum(r["not_found"] for r in model_results)

    print(f"\n{model_name}:")
    print(f"  Total refs: {total}")
    print(f"  Verified: {verified} ({verified/total:.1%})")
    print(f"  Not found: {not_found} ({not_found/total:.1%})")

    # Per-discipline breakdown
    by_discipline = {}
    for r in model_results:
        d = r.get("discipline", "unknown")
        if d not in by_discipline:
            by_discipline[d] = {"total": 0, "verified": 0}
        by_discipline[d]["total"] += r["total_refs"]
        by_discipline[d]["verified"] += r["verified"]

    for d, stats in by_discipline.items():
        rate = stats["verified"] / stats["total"] if stats["total"] > 0 else 0
        print(f"  {d}: {rate:.1%} ({stats['verified']}/{stats['total']})")

Improving Model Performance

Based on error analysis, consider these strategies:

Output Files

All results are saved to the configured output directory:

evaluation_results/ref_hallucination_arena/
├── evaluation_report.md          # Detailed Markdown report (bilingual zh/en)
├── evaluation_results.json       # Final rankings, per-field accuracy, and scores
├── verification_chart.png        # Per-field accuracy breakdown bar chart (Title/Author/Year/DOI/Overall)
├── discipline_chart.png          # Per-discipline overall accuracy grouped bar chart
├── queries.json                  # Loaded evaluation queries
├── responses.json                # Raw model responses
├── extracted_refs.json           # Extracted BibTeX references
├── verification_results.json     # Detailed per-reference verification results
└── checkpoint.json               # Pipeline checkpoint for resume

Advanced Topics

verification:
  crossref_mailto: "your-email@example.com"  # Join Crossref polite pool
  pubmed_api_key: "your-pubmed-api-key"       # Higher PubMed rate limit

# First run (interrupted after verifying 500/1000 items)
python -m cookbooks.ref_hallucination_arena --config config.yaml --save

# Resume from checkpoint (automatically picks up at item 501)
python -m cookbooks.ref_hallucination_arena --config config.yaml --save

# Start fresh (ignore checkpoint)
python -m cookbooks.ref_hallucination_arena --config config.yaml --fresh --save

target_endpoints:
  # Same model, bare mode (no tools)
  model_a_bare:
    base_url: "https://api.example.com/v1"
    api_key: "${MODEL_A_API_KEY}"
    model: "model-a"
    extra_params:
      temperature: 0.3

  # Same model, tool-augmented mode
  model_a_with_tools:
    base_url: "https://api.example.com/v1"
    api_key: "${MODEL_A_API_KEY}"
    model: "model-a"
    extra_params:
      temperature: 0.3
    tool_config:
      enabled: true
      tavily_api_key: "${TAVILY_API_KEY}"
      max_iterations: 10
      search_depth: "advanced"

target_endpoints:
  my_model:
    base_url: "https://api.example.com/v1"
    api_key: "${API_KEY}"
    model: "my-model"
    system_prompt: |
      You are an academic literature recommendation expert.
      Based on the user's research topic, recommend {num_refs}
      real, high-quality academic papers. Output each paper in
      standard BibTeX format with title, author, year,
      journal/booktitle, and doi fields.

report:
  enabled: true        # Enable report generation
  language: "zh"       # "zh" (Chinese) or "en" (English)
  include_examples: 3  # Examples per section (1-10)
  chart:
    enabled: true          # Generate visualization charts
    orientation: "vertical"  # "horizontal" or "vertical"
    show_values: true      # Show values on bars
    highlight_best: true   # Highlight best-performing model

Best Practices

Next Steps

• Auto Arena — Automatically compare models with generated queries
• Refine Data Quality — Improve model outputs using grader feedback
• Create Custom Graders — Build custom evaluation pipelines