"hellhole" Romanian orphanages.
They grew up with brains 8.6% smaller than other adoptees.
The researchers said it was the "most compelling" evidence of the impact on the adult brain.
The appalling care at the orphanages came to light after the fall of Romania's communist dictator Nicolae Ceausescu in 1989.
"I remember TV pictures of those institutions, they were shocking," Prof Edmund Sonuga-Barke, who now leads the study following those children, told the BBC.
He described the institutions as "hellholes" where children were "chained into their cots, rocking, filthy and emaciated".
The children were physically and psychologically deprived with little social contact, no toys and often ravaged by disease.
The children studied had spent between two weeks and nearly four years in such institutions.
Previous studies on children who were later adopted by loving families in the UK showed they were still experiencing mental health problems in adulthood.
Higher levels of traits including autism, attention deficit hyperactivity disorder (ADHD) and a lack of fear of strangers (disinhibited social engagement disorder) have all been documented.
Adopted Romanian orphans 'still suffering in adulthood'
The latest study, published in Proceedings of the National Academy of Sciences, is the first to scan the brains for answers.
There were 67 Romanian adoptees in the study and their brains were compared to 21 adoptees who did not suffer early life deprivation.
"What we found is really quite striking," Prof Sonuga-Barke told the BBC.
First the total brain volume - the size of the brain - was 8.6% smaller in the Romanian adoptees on average.
And the longer they spent in the Romanian orphanages, the greater the reduction in brain size.
However, the impact on the brain was not uniform.
Prof Mitul Mehta, one of the researchers, said: "We found structural differences between the two groups in three regions of the brain.
"These regions are linked to functions such as organisation, motivation, integration of information and memory."
The researchers say these findings could help explain lower IQ and higher rates of ADHD in these adults.
What the study cannot explain is what exactly about early life neglect and deprivation has this effect on the brain.
It means it is hard to work out the effect of other early life traumas such as abuse or being a refugee.
However, the study is clear that the impact on the developing brain goes far beyond just poor nutrition.
Prof Sonuga-Barke said: "This study is important because it highlights for the first time, in a compelling way, the power of the early environment and early adversity to shape brain development.
"It drives impairments over this long period of time - over 20 years - even when children have received top-notch care in loving adoptive families."