Reproduction logbook — paper-TBaUfO9znF

Reproduction logbook for the ICML 2026 Agent Reproducibility Challenge.

Paper: On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models — Charlie Zhang, Graham Neubig, and Xiang Yue. OpenReview TBaUfO9znF, arXiv 2512.07783.

Result: 5 claims VERIFIED and 1 claim FALSIFIED. The falsified claim is the conjunctive performance statement that process-aware rewards both reduce shortcut exploitation and improve reasoning relative to outcome-only rewards: the released OOD-hard table shows the shortcut rate falling from 0.460 to 0.380/0.250 while pass@128 falls from 0.428 to 0.031/0.007.

This logbook uses the paper's own printed definitions and figures, official released datasets, official released checkpoints, and small exact arithmetic audits. It does not train a model, run a demo, or require GPU inference. The raw rows and derived values are in outputs/; the standard-library validator is code/audit_metrics.py.

Six registered claims

# Registered claim Verdict
1 Controlled synthetic reasoning framework with explicit dependency graphs, contextual templates, and process-verified evaluation (Figure 2). VERIFIED
2 RL yields extrapolative pass@128 gains only when post-training tasks sit near the model's edge of competence; gains vanish when tasks are covered or too far OOD (Figures 1 and 3). VERIFIED
3 Contextual generalization needs minimal but nonzero long-tail pre-training exposure; about 1% enables RL transfer (Figures 1 and 4). VERIFIED
4 Mid-training plus RL beats RL-only on OOD-hard tasks under fixed compute, by the reported 10.8 percentage points (Figures 1 and 6). VERIFIED
5 Process-aware rewards reduce shortcut exploitation and improve reasoning relative to pure outcome rewards (Figure 7). FALSIFIED
6 Reward stagnates when RL data are too easy or too hard, but improves when data are calibrated to the edge of competence (Figure 11). VERIFIED

Layout 

pages/
  index.md
  executive-summary/page.md
  claim-1-controlled-synthetic-framework/page.md
  claim-2-edge-of-competence/page.md
  claim-3-context-exposure/page.md
  claim-4-mid-training-fixed-compute/page.md
  claim-5-process-reward/page.md
  claim-6-reward-dynamics/page.md
  conclusion/page.md
code/audit_metrics.py
outputs/extracted_metrics.csv
outputs/paper_calibration.csv
outputs/audit_results.json

Sources