"I said to her, 'Oh, Andrew's really good-looking,'" recalls the Duchess, 61, whose debut novel, the sweeping coming-of-age story Her Heart for a Compass, will be published on August 3. "And she said, 'Duh, Fergs!'" (The pair wed in 1986 and divorced a decade later but remain close co-parents of daughters Princess Eugenie, 31, and Princess Beatrice, 32.)
David Burton Sarah Ferguson, Duchess of York on the cover of PEOPLE.
But when both women became global superstars following their royal weddings, the media often pitted them against each other.
"In the '80s, it was Diana looking beautiful, and there was fat, frumpy Fergie," says the Duchess, who was labeled the "Duchess of Pork" in the tabloid press. "We were just there for people to make a lot of money. At the time we both didn't realize that."
Mike Forster/Daily Mail/Shutterstock Princes Diana and Sarah, the Duchess of York in 1991 with Prince William (left) and Prince Harry (center).
The two women, who dominated the front page as the "wives of Windsor," would compare notes — and offer each other support.
"Diana and I both had our own mental health issues, and she and I used to talk," recalls the Duchess. "She said, 'Fergie, remember one thing: When you're at the top of the pedestal, it's so easy to fall off. And you're at the bottom. You just climb up.'"
David Burton Sarah Ferguson, the Duchess of York, photographed for PEOPLE.
For the full exclusive interview with Sarah Ferguson, Duchess of York, pick up this week's issue of PEOPLE, on newsstands now
The Duchess continues, "We were positioned as saint and sinner. And the most important thing was to remain robust together, and we did, no matter what anyone wrote."
Listen below to our daily podcast PEOPLE Every Day for more on PEOPLE's cover story interview with Sarah 'Fergie' Ferguson.