Brain-imaging data sets had been swelling in size.
Article continues below
In 2019, neuroscientist Scott Marek was asked to contribute a paper to a journal that focuses on child development. Previous studies had shown that differences in brain function between children were linked with performance in intelligence tests. So Marek decided to examine this trend in 2,000 kids.
To show that this growth was making studies more reliable, Marek, based at Washington University in St. Louis, Missouri (WashU), and his colleagues split the data in two and ran the same analysis on each subset, expecting the results to match. Instead, they found the opposite.
“I was shocked. I thought it was going to look exactly the same in both sets,†says Marek. “I stared out of my apartment window in depression, taking in what it meant for the field.â€
Now, in a bombshell Nature study, Marek and his colleagues show that even large brain-imaging studies, such as his, are still too small to reliably detect most links between brain function and behaviour.
As a result, the conclusions of most published ‘brain-wide association studies’ — typically involving dozens to hundreds of participants — might be wrong.
Such studies link variations in brain structure and activity to differences in cognitive ability, mental health and other behavioural traits.
For instance, numerous studies have identified brain anatomy or activity patterns that, the studies say, can distinguish people who have been diagnosed with depression from those who have not. Studies also often seek biomarkers for behavioural traits.
“There’s a lot of investigators who have committed their careers to doing the kind of science that this paper says is basically junk,†says Russell Poldrack, a cognitive neuroscientist at Stanford University in California, who was one of the paper’s peer reviewers. “It really forces a rethink.â€
After his botched replication, Marek set out to understand the reasons for the failure together with Nico Dosenbach, a neuroscientist at WashU, and their colleagues. That work resulted in the latest study, in which they analysed magnetic resonance imaging (MRI) brain scans and behavioural data from 50,000 participants in several large brain-imaging efforts, such as the UK Biobank’s collection of brain scans.
The researchers then used subsets drawn from these large databases to simulate billions of smaller studies. These analyses looked for associations between MRI scans and various cognitive, behavioural and demographic traits, in samples ranging from 25 people to more than 32,000.
In simulated studies involving thousands of people, the researchers identified reliable correlations between brain structure and activity in particular regions and different behavioural traits — associations that they could replicate in different subsets of the data. However, these links tended to be much weaker than those typically reported by most other studies. ■