Prince William and Prince Harry‘s early childhood is examined closely in historian Robert Lacey‘s Battle of Brothers, and the picture that emerges is not always flattering. According to Lacey, their mother Princess Diana fired their beloved nanny Barbara “Baba” Barnes, believing her to be “too possessive” with the boys.

She was forbidden from saying goodbye to them or contacting them ever again. “So far as the boys were concerned, she just vanished into thin air,” Lacey writes.

Once Barnes was hired, Lacey claims, she “stepped in firmly to assert control.” One Kensington Palace staff member reflected, “Barbara guarded the nursery floor like the Vatican… It was her kingdom.”

Nicknamed “Baba” by the boys, she “became something of a surrogate mother” to them as they spent an increasing amount of time in her care, with the nanny teaching them to “walk, talk and read.”

Barnes also “comforted them when they awoke crying in the night,” and went on to take the princes “away on their own “family” holiday without parents—to Scotland and the Isles of Scilly—where she set the agenda every day as any mother would.”

The last straw for Diana was when she was photographed in 1986 at a party of her former employer Lord Glenconner, alongside Raquel Welch and Princess Margaret.

Diana, who felt Barnes had “got above herself,” “brusquely informed her that it would be 'better,' as she put it, if Barnes departed.”

Lacey also theorizes that Diana’s rocky marriage to Prince Charles led to other distressing changes in the boys. While a “once-rambunctious” William grew “more reflective” with a “noticeably quiet character — definitely introverted” as he headed off to boarding school, Harry's personality was also changing.

“Both brothers have been damaged by their upbringings; both have reacted by finding different solutions,” Lacey, who is also the historical consultant for Netflix‘s The Crown, tells People. “There is so much pain and trauma in this story, going right back to the beginning.”