Introduction. Using AI as a “synthetic peer review” — meaning letting large language models (LLMs) generate full or near-full peer reviews, act as stand-in reviewers, or produce autonomous evaluations — is one of the more realistic near-future disruptions in academic publishing. By early 2026, it’s already happening in fragmented, semi-covert ways, and the data shows both real utility and serious landmines.

The promising side (what actually works well)

Current frontier LLMs (GPT-4o, Claude 3.5/4, o1 variants, etc.) can already deliver surprisingly useful feedback in several narrow but important dimensions:

• They catch structural, methodological checklist items, reporting guideline adherence (CONSORT, PRISMA, etc.), and basic statistical red flags at a level comparable to — or occasionally better than — average busy human reviewers.
• Overlap between points raised by GPT-4 and human reviewers has been measured at ~30–39% in large-scale studies of Nature-family journals and ICLR submissions (roughly the same as the overlap between two independent human reviewers).
• In user studies, a majority of researchers found LLM feedback helpful, and many rated it more useful than at least some human reviews they’ve received.
• They excel at triage, summarizing dense manuscripts, checking reference consistency, spotting plagiarism patterns, and drafting polished review language — reducing cognitive load on humans.

In short: as a first-pass filter, sparring partner, or supplement to human review, synthetic peer review already adds net value in 2025–2026.

The serious problems

Why full synthetic replacement is dangerous right now.

Several failure modes are well-documented and recurring:

1. Over-acceptance of garbage — When tested on fully AI-fabricated nonsense papers, current LLMs recommend acceptance >80% of the time in multiple independent experiments. They lack the lived disciplinary intuition to smell sophisticated fraud.
2. Easy manipulation & jailbreaking — Invisible text injection, prompt leaking, or cleverly worded “adversarial” sections in manuscripts can systematically bias LLM reviews toward acceptance. This has been reproducibly shown across major commercial models.
3. Bias preservation/amplification — LLMs inherit whatever biases exist in their training corpora (which include decades of unevenly distributed scientific literature). They can quietly disadvantage work from underrepresented regions, methods, or author demographics unless very carefully prompted and audited.
4. Lack of real accountability & taste — AI reviews rarely display the subtle priority judgments, field-specific “taste”, or ethical boundary-setting that experienced humans provide. They are polite, verbose, and frequently shallow when the question moves beyond pattern-matching.
5. Confidentiality & misuse risk — Uploading unpublished work to public-facing models creates leak and intellectual-property hazards. Paper mills already exploit this.
6. Detection is hard but rising — By 2025, roughly 12–20% of reviews at major venues (ICLR, Nature Communications) show strong signs of being AI-generated or heavily AI-assisted, and detection tools are lagging.

Realistic 2026–2028 picture

Full synthetic peer review (AI alone decides accept/revise/reject) is not ready and probably shouldn’t be for high-stakes archival journals. But a hybrid model is already emerging and likely inevitable:

• AI does first-pass technical checks, completeness audits, language polishing, reference validation, basic statistical sanity.
• Human experts do the final integrative judgment, novelty assessment, methodological deep-dive, and fraud smelling.
• Transparent disclosure of AI usage becomes mandatory (many journals already require or strongly encourage this).
• Meta-review layer where another AI (or human) summarizes & reconciles multiple human + synthetic reports.

This hybrid is probably superior to the status quo of exhausted volunteers, 3–6 month delays, and reviewer scarcity.

Bottom line

I think synthetic peer review is useful today as an amplifier and accelerator, but dangerous as a replacement. The technology is moving so fast that anything I say in February 2026 will look conservative by late 2027. (Link goes to overview)

The winning path is probably mandatory disclosure + human veto power + ongoing red-teaming of LLM reviewers against deliberately nasty adversarial papers. If done carefully, we could end up with faster, fairer, more consistent peer review than we have now. If done carelessly, we accelerate the flood of polished garbage into the literature. (Link goes to “Official Statement.”)

###