AI Slop Comes for Science
Balancing breakthroughs with BS
“[I]t’s my guess that powerful AI could at least 10x the rate of these discoveries, giving us the next 50-100 years of biological progress in 5-10 years.”
“…[AI] will likely make drugs cheaper and more accessible for everybody on the planet.”
“[AI] is going to transform the scientific method itself and usher in a whole new era of scientific inventions and discoveries”
These quotes1 from scientists promise a utopian future where artificial intelligence discovers new cures for terminal diseases, creates technology to mitigate climate change, and unravels the mysteries of the universe in physics and math. The first one is from the essay “Machines of Loving Grace”2 by Anthropic AI CEO Dario Amodei (who has a PhD in electrophysiology). Amodei’s essay is provocative, and while I don’t agree with everything he says, I recommend you read it to understand the optimistic Silicon Valley perspective. In the first section on AI in biological research, he writes:
…a surprisingly large fraction of the progress in biology has come from a truly tiny number of discoveries, often related to broad measurement tools or techniques that allow precise but generalized or programmable intervention in biological systems. There’s perhaps ~1 of these major discoveries per year and collectively they arguably drive >50% of progress in biology.
[…]
I think their rate of discovery could be increased by 10x or more if there were a lot more talented, creative researchers. Or, put another way, I think the returns to intelligence are high for these discoveries, and that everything else in biology and medicine mostly follows from them.
His argument is essentially that rare discoveries change the trajectory of science, the main bottleneck is human talent and time, and that by massively increasing the amount of research capacity, we dramatically improve our odds of finding those needles in the proverbial haystack. He asks us to imagine what medicine could look like if we discovered things like CRISPR, mRNA vaccines, and CAR-T cell cancer therapies all within a year or two, instead of spread out over decades.
At the same time, people have raised questions about the accuracy of AI, and whether or not it will accelerate the problem of low-quality or fraudulent studies. Today we’ll dig deeper into this phenomenon and ask: is this really a revolution, or just a bunch of BS?
—Eric
The Good
One of the early examples that seemed to validate AI’s potential was Google DeepMind’s AlphaFold, a machine learning algorithm that was able to solve the 3D crystal structure of hundreds of thousands of proteins. Two DeepMind researchers won the Nobel Prize in Chemistry last year for their work on the software, and Time Magazine named it one of the best inventions of 2024. An explosion of studies based on AlphaFold predictably followed:
In that vein, last January I wrote about how researchers used generative AI to discover new classes of antibiotics that could fight resistant superbugs and how it was being used to predict personalized cancer therapies 👇
Given these big wins, it should not be surprising that scientists have gotten the memo that AI is the hot new thing, and are increasingly publishing on the topic:
In a Nature survey of more than 1,600 scientists, respondents provided their thoughts on the benefits and drawbacks of using AI in their work:
The perceived positives line up closely with the Amodei essay, even though I’d wager the vast majority of scientists have not read it. You can see more than two thirds of respondents thought AI could accelerate data analysis, which is one of the main challenges in biomedical research, especially in new, data-intensive fields like genomics and proteomics.
The Bad
Let’s return to the example of DeepMind, the team behind AlphaFold. They also claimed to have discovered the crystal structure for over 2 million inorganic compounds, which would represent a massive leap in the field of materials science. However, when other groups independently checked the data, they found: “Most of the materials produced were misidentified, and the rest were already known.” A less charitable view is that the predictions were “mostly junk.” Even among the Nobel prize-winning AlphaFold, the predictions range from dead-on to, well, garbage:
For an example from the world of physics, Dr.
recently wrote about his disappointing experience with AI during his PhD dissertation on the Substack newsletter Understanding AI (which I recommend everyone check out). While much of his work is over my head, the take home point is a lot of promising papers that used fancy neural networks to solve complex equations didn’t pan out and underperformed older, simpler methods. He writes:Every field of science is experiencing AI differently, so we should be cautious about making generalizations. I’m convinced, however, that some of the lessons from my experience are broadly applicable across science:
AI adoption is exploding among scientists less because it benefits science and more because it benefits the scientists themselves.
Because AI researchers almost never publish negative results, AI-for-science is experiencing survivorship bias.
The positive results that get published tend to be overly optimistic about AI’s potential.
As a result, I’ve come to believe that AI has generally been less successful and revolutionary in science than it appears to be… While the causes of this tendency towards overoptimism are complex, the core issue appears to be a conflict of interest in which the same people who evaluate AI models also benefit from those evaluations.
These issues seem to be bad enough that I encourage people to treat impressive results in AI-for-science the same way we treat surprising results in nutrition science: with instinctive skepticism.
His whole essay is worth reading and can be found below:
The Ugly
Finally, I want to close with my biggest concern about the use of AI in science: “slop.” If you aren’t familiar with the phrase, “AI slop” is a term coined in the last couple years to describe the low-quality, often non-sensical images and web pages created by generative AI like ChatGPT. Sometimes slop is mindless fun, while other instances are linked to click-bait spam or other nefarious motivations.
Making it easier to quickly produce nonsense is a serious risk for science, which has already seen the number of retracted articles increase 10-fold over the past two decades, driven in part by AI. This is complicated by the fact that even peer-review may be “contaminated” by large language models.
We’re now seeing this “slop” show up in the research literature, as reported by Nature. A recent study found an exponential increase in poorly-done papers that appear to use AI published in low-quality journals (I.e. for-profit “paper mills”). This validates one of the top concerns in the survey above: scientific fraud.
Some of these fraudulent papers are comically obvious to spot, like the one chronicled in this VICE story: “Study Featuring AI-Generated Giant Rat Penis Retracted, Journal Apologizes.” However, in other another case, researchers showed it was trivial to quickly generate a fake but plausible neurosurgery study that might pass expert peer review.
What Do We Do?
Whatever you think about AI, it is already here and disrupting scientific research. Even if the big companies like OpenAI and Anthropic went out of business, there are many open source models available free on the internet like DeepSeek (text generation), Stable Diffusion (image generation), and others. How do we adapt to this quasi-dystopian future? In my opinion, the answers to a machine-made problem are the very soft skills that make us human.
First: Caveat emptor! A healthy dose of skepticism is warranted with any new technology, but especially artificial intelligence, because it is programmed to convincingly tell us what we want to hear. And Dr. McGreivy’s experience shows that AI failures are rarely published, so all we see are the experiments that worked. As always, if something sounds too good to be true, it probably is.
A second critical component is transparency. Many scientific journals are starting to craft AI policies for submissions. Some outright ban it, others take a more moderate stance, but the trend is moving towards mandatory disclosure at minimum. Other institutions ranging from government to universities and private companies will have to grapple with whether and how to use AI.
Finally, we need to apply critical thinking. AI, like any technology, is not a monolith: There are huge differences in the underlying workings and capabilities of traditional machine learning, large language models, computer vision, and natural language processing algorithms. AI is also not magic! These companies want us to think their products are all powerful because that is in their financial interest. In reality, AI is merely a tool that can be put to positive or negative ends, just like the Internet, smart phones, and social media. It’s a false choice to say we must pick between “AI is smarter than us, so let’s turn our brains off and hand it all over” or “AI is evil and anyone using it should be shamed.” The truth is more nuanced, and now is the time to develop ethical standards and thoughtful regulations around AI in science.
What do YOU think about the growing use of AI in science? Leave a comment below!
First quote: Dario Amodei, CEO of generative AI company Anthropic
Second quote: Alex Zhavoronkov, CEO of biotech company Insilico Medicine
Third quote: Anima Anandkumar, Bren Professor of Computing and Mathematical Sciences at Caltech
The name is a reference to the 1967 poem by Richard Brautigan, “All Watched Over By Machines Of Loving Grace”
Great post. Thank you!
Rather than taking the wheel, I find that AI is helpful for generating ideas, brainstorming, rough drafts and the like. I will sometimes use while seeing patients and I need some answers/ideas that would take 10X the time to find in reference books or studies. But having experience, a good speciality specific knowledge base, and a scalpel to take to much of the responses is needed.
I don’t see LLMs pushing the needle of progress forward that fast… more like productivity tools that end up costing us a lot of time to clean up, cross reference, and check sources.