Hugh Hefner: The Ups and Downs of a Spectacular Life

While Hugh Hefner had seen his reputation somewhat tarnished by Playmate Holly Madison‘s tell-all about life in the Playboy Mansion, Down the Rabbit Hole, and the subsequent sale of the Playboy Mansion, his life was about far more than what people tuning into The Girls Next Door or readers of Madison’s book might suspected.

Hefner’s death of natural causes throws the Playboy empire into a state of flux. For one thing, in a 2011 interview with The Hollywood Reporter Scott Flanders, CEO of Playboy, admitted there was no succession plan for leadership of the company. And for another, it’s far more successful overseas than it is domestically, with the U.S. editions continually operating at a loss. But Hefner’s legacy doesn’t seem tied to any of those uncertainties. Rather, it’s been sealed by his remarkable life, which has had its share of ups and downs, as you’ll see below.

Hefner was born in Chicago, Illinois, picking up the nickname “Hef” while in high school. After what he described to PEOPLE as “a happy childhood,” he spent two years in the Army, working as a clerk and drawing cartoons for several newspapers. He graduated from the University of Illinois at Urbana Champaign with a bachelor’s degree in psychology and a double minor in creative writing and art in 1949. That same year, he married Mildred Williams. Before they were married, Williams confessed that she’d had an affair while he was in the Army; she later allowed him to carry on extramarital affairs out of guilt and in the hopes that it would preserve their marriage.

Hefner quit Esquire, reportedly after being denied a $5 raise. The magazine moved to New York and Hefner stayed in Chicago, opting to raise money for the first issue of Playboy. With a combined $8,000 raised from 45 investors, including his mother and brother, he published the first issue of Playboy, which featured as its centerfold a nude photo of Marilyn Monroe taken years earlier, in December 1953. It sold over 50,000 copies and launched Hefner’s career.

Work on the magazine consumed most of Hefner’s life through the 1950s. He admitted to PEOPLE earlier this year that he was “an absent dad” to his two children with Mildred, Christie and David. The two divorced in 1959, and the modern-day, image of “Hef” as a swinging, urbane man-about-town was born. Part of this were the Playboy Clubs, nightclubs staffed by the famously dressed Bunnies. However, as Hefner told PEOPLE, “The magazine was about breaking barriers. It wasn’t just a sexual thing.”

To that end, the Playboy Clubs were some of the earliest to break racial barriers with regards to booking; Dick Gregory and Sonny Rollins were among the black entertainers the club lifted from the then-seedy worlds of jazz clubs and the chitlin’ circuit. (Notably, only one woman, actress and singer Lainie Kazan, ever ran a Playboy Club, though it was an important one – the Los Angeles branch.)

Another way Hefner was ahead of his time was Playboy’s Penthouse. A sort of talk show disguised as a bachelor party, Playboy’s Penthouse, launched in 1959, was an early showcase for prominent black musicians like Ella Fitzgerald, Nat “King” Cole, Sarah Vaughn and Harry Belfonte, notably sitting down next to each other and conversing as equals. It also booked Lenny Bruce for an appearance, something unheard of at the time. The show ran for two seasons and was eventually followed by Playboy After Dark, a similar version of the show, this time in color, that ran for two seasons.

Through the ’60s, Playboy the magazine attracted an equal mix of compliments and controversy. Author Alex Haley inaugurated the Playboy interview in 1962 with jazz legend Miles Davis, three years after the latter’s definitive album Kind of Blue and 14 years before Haley published Roots. The year after, Hefner was arrested on obscenity charges following a nude pictorial of actress Jayne Mansfield. (The charges were eventually dropped.) Also in 1963, Gloria Steinem went undercover as a bunny at a Playboy Club for an article exposing what the women who worked there endured.

The magazine’s momentum continued through the early 1970s, and Hefner was able to purchase the L.A. Playboy mansion (as well as the Big Bunny, a converted DC-30 jet he used to traverse the globe) in 1971. It was at the L.A. mansion that he began throwing the wild parties with Hollywood elite like Jack Nicholson and Warren Beatty that further bolstered his reputation as a party animal for the ages.

But the magazine’s fortunes were on a downward trend, helped along by competitors like the more explicit Penthouse and national recession. Then, in 1975, Hefner’s former secretary, Bobbie Arnstein, committed suicide. She was being pressured by the FBI to inform on Hefner for drugs; the agency had been investigating him since 1972.

In 1985, Hefner had a minor stroke, with many citing The Killing of the Unicorn, Peter Bogdanovich’s book about the murder of a former Playmate and the magazine’s declining fortunes, as the cause. (Around the same time, he was being sued for palimony by Playmate Carrie Leigh, which probably didn’t help things either.) Hefner scaled back on the partying, gave up smoking and eventually married a longtime girlfriend, Kimberly Conrad, in 1989. “I was ready to be a dad again,” he told PEOPLE. (The pair had two sons, Cooper and Marston, and remained together for 10 years before separating in 1999.) In 1986, the last Playboy Club closed and in 1988, he turned over control of Playboy Enterprises to his daughter Christie.

Despite this, Hefner has always been generous with his money, spending $27,000 in 1978 to symbolically purchase the “y” in the Hollywood Sign after organizing fundraising profits to spur its restoration. (He paid a further $900,000 in 2010 to a conservation group attempting to stop the development of the land around the sign. Hef really liked the Hollywood sign.) He’s also contributed $2.1 million to the University of Southern California’s School of Cinematic Arts.

The Girls Next Door introduced a whole new generation to Hefner and his lifestyle when it hit the airwaves in 2005, though after the 2015 publication of Madison’s book, that lifestyle lost quite a bit of its shine, as she described the controlling nature of the relationship between Hefner and the Playmates, the disrepair the mansion had fallen into, and the nightly sexual routine that Hefner’s closest of “girlfriends” was forced to endure.

And he continued to lose people along the way: His secretary of 50 years, Mary O’Connor, passed away in 2013 and his brother Keith died this year, just a day before Hefner’s 90th birthday. Hefner continued to cycle through girlfriends as well, telling The Hollywood Reporter that, “The truth of the matter is, I should be single. I’m better served that way. Maybe I don’t pick the right women. Or maybe I’m just too complicated.” (He married third wife, Crystal Harris, in 2012.)

“I lost some very dear friends,” he told THR in 2011. “If I could have figured a magic way to save their lives…” But, an agnostic with no faith in any afterlife, he said in the same interview he had no fear of death. “My mother lived to 101.”

Hefner remained sanguine and positive about his life into his old age: “Playboy is a fantasy life for a lot of people,” he told PEOPLE earlier this year. “Including me.”