Good point

Who knows if this is real or more apocryphal stuff trying to raise awareness

Apparently the Guardian reported that researchers have been hiding LLM prompts in their papers to get favorable reviews from an LLM

EDIT. Sorry. A link https://www.theguardian.com/technology/2025/jul/14/scientists-reportedly-hiding-ai-text-prompts-in-academic-papers-to-receive-positive-peer-reviews

Chris Howden wrote:

I think LLM’s could be very useful in picking up common methodological and statistical mistakes as well as deviations from the journals style. Made even more useful if each journal trained up their own and made it available to people submitting papers.

That's a really interesting idea! I'd be fascinated to see how journals would implement that - would they run a pre-review through the LLM and provide the pre-reviewed copy to the reviewers? Or get the reviewer to do it themselves? It sounds almost like a proofing service.

I totally agree that the current peer review system is poor, but I'm not convinced that LLMs are the solution. Maybe for things like formatting your review, fixing typos, changing the tone - but I'd draw the line at completing a full peer review.

If LLMs are the only ones performing peer review, it feels like a slippery slope toward LLMs also being the only ones generating evidence syntheses - at which point I'd wonder what's the point of even writing scientific papers any more! 

Using LLMs to provide peer review defeats the purpose of peer review IMO, the point is to get expert human review of your work in the context of broader scientific knowledge. Also, LLMs aren't our peers. I'd be disappointed if a reviewer used an LLM to review my work (and alarmed if they uploaded my unpublished work).

That said, without it being blindingly obvious that an LLM was used (such as the review including text like "Yes I can do that, here's an improved version of your review"), I would be very hesitant to claim the reviewers used LLMs in their reviews. It feels difficult to prove, especially when the reviews are not independent - they are both reviews of the same piece of work. The intro text where they describe what your paper is about feels robotic but seems to be an academic norm. If the work does indeed have the issues identified by one reviewer, it shouldn't be surprising that the other reviewer also picks up on the same issues.

Whether the things like critiquing your work for issues that are either non-existent in it or already addressed by the text are the result of LLM usage or human error/laziness is also difficult to say. Humans make mistakes too and I find it more useful to be generous in your assumptions. Replying under the assumption that a mistake was made, it gives the reviewer/editor an 'out' and hopefully leads to a productive conversation where maybe you can get to your desired outcome.