It wasn’t that he told a woman there was something wrong with her for wearing a hijab in America. It wasn’t that he encouraged people to “Purge the Illegals” and gave out ICE’s hotline number at a presentation. It wasn’t that he mocked a transgender college student in front of a crowd, saying he’d still almost bang her because she looked like a man. Instead, it was his discussion of the complexities of his sexual experiences with adults as a gay teenager that caused Milo Yiannopoulos to lose his $250,000 book deal with Simon and Schuster.

The swift recent reversal of Yiannopoulos’s fortunes is in many ways illuminating. The Breitbart editor had spent the last year building a public profile by going around American college campuses giving “lectures” with titles like “Why Do Lesbians Fake So Many Hate Crimes?” and “Why Ugly People Hate Me.” At these events, he would tell people why “feminism is cancer,” refer to various people as “cunts” and “retards,” and make jokes about how Muslims were probably terrorists. When appalled students tried to have the talks canceled, he would insist that the PC left was simply afraid to deal with arguments, facts, and statistics. (The more obvious explanation is that the PC left doesn’t think a person whose idea of elevated political discourse is “100% of fat people are fucking gross”—and who gigglingly posts pictures of the overweight people at his gym—is sincere about wanting to improve political dialogue on campus.)

As Yiannopoulos would continue to bait students with outrageous and cruel remarks, and students would continue to take the bait by giving Yiannopoulos publicity and fueling his persecution narrative, he managed to bring himself mainstream attention. For God only knows what reason, a major publishing house decided to reward him with a six-figure advance. (Actually, we know full well the reason: $) Bill Maher invited Yiannopoulous on Real Time, where the two enjoyed a pleasant back-and-forth about how the left were the real intolerant ones, before agreeing that transgender people were a bunch of sex criminals who couldn’t be trusted in women’s bathrooms. (The only relief during Yiannopoulos’s otherwise unendurable Real Time appearance was provided by Larry Wilmore, who enthusiastically told Yiannopoulos to go fuck himself after Yiannopoulos speculated that his black co-panelists must have low IQs.) Finally, the Conservative Political Action Conference placed a gleaming maraschino atop Yiannopoulos’s recent success by offering him a speaking slot.

Until a few days ago, then, Milo Yiannopoulos was doing quite well for himself. Then the pedophilia tapes surfaced. It turned out that Yiannopoulos had once made a few remarks that were difficult to interpret as anything other than a defense of sex between older men and young boys:

“We get hung up on this sort of child abuse stuff, to the point where we are heavily policing consensual adults. In the homosexual world, particularly, some of those relationships between younger boys and older men — the sort of ‘coming of age’ relationship — those relationships in which those older men help those young boys discover who they are and give them security and safety and provide them with love and a reliable, sort of rock, where they can’t speak to their parents.”

When the interviewer pointed out that this sounded like “Catholic priest molestation,” Yiannopoulos replied: “You know what? I’m grateful for Father Michael. I wouldn’t give nearly such good head if it wasn’t for him…” In another interview, Yiannopoulos confirmed that age 14 he had had sexual interactions with a priest, but said that this “wasn’t molestation,” nor was it pedophilia, because “pedophilia is not a sexual attraction to somebody 13 years old who is sexually mature. Pedophilia is attraction to children who have not reached puberty.”

Conservatives were scandalized. Bill Kristol called the remarks “despicable” and CPAC rapidly rescinded Yiannopoulos’s invitation to speak. Soon after, Simon & Schuster canceled his book deal, and there were reports that Breitbart editors were threatening to resign if he wasn’t fired. During Friday’s Real Time, Bill Maher had said that Yiannopoulos was “only at the beginning of [his] career.” By Monday, it seemed like he was at the end of it.

The rapid undoing of Yiannopoulos was interesting for several reasons. It served as an instructive illustration of what conservatives were and were not willing to tolerate. All the hateful filth about women, Muslims, and transgender people actually made a conservative publishing imprint want to publish his book. These things evidently do not cross a moral line. (To his credit, National Review editor Jonah Goldberg deplored this lack of principle, commenting that “apparently the racism and anti-Semitism wasn’t a deal breaker.”)

As far as CPAC goes, Yiannopoulos’s invitation and dis-invitation shows where the standards lie. For Simon and Schuster, on the other hand, dropping Yiannopoulos may have been strictly business. As Roxane Gay, who withdrew her book from the publishing house in protest of their decision to offer Yiannopoulos a contract, explained: “Simon and Schuster realized it would cost them more money to do business with Milo than he could earn for them. They did not finally ‘do the right thing.’ They were fine with his racist and xenophobic and sexist ideologies.” Indeed, like most publishers, S&S is far more concerned with what they can sell than with whether it’s moral or immoral.

But just as interesting as what didn’t make Yiannopoulos toxic is what did. Ironically, the remarks that finally got him expelled from the mainstream were among his less indefensible. He has been condemned by almost everybody for “defending pedophilia.” But this is not quite fair. In fact, while his comments are shocking, the arguments he is making are not unfamiliar in LGBT discourse. As Current Affairs editor Yasmin Nair explained in a thoughtful and provocative essay in 2005, the intensity of feelings around child abuse often prevent people from appreciating nuanced arguments. Nair was writing about a publisher’s decision to exclude an article on pederasty from a book on the history of same-sex relationships, after right-wing complaints that it would condone “rampant child molestation.” As she writes, there is a long tradition of the right using fears about pedophilia “to condemn all queers, particularly gay men, as predators of children.” It is often impossible to have a discussion about the reality of queer people’s lives, because anyone who speaks of their neutral or positive experiences with older people as a youth is perceived as endorsing pedophilia.

Yiannopoulos says that gay men’s experiences as teens with older men are often complicated, not always easily captured by the available terminology. He says that his own teenage sexual encounters with men did not fit the labels “molestation” and “pedophilia,” especially since pedophilia refers to attraction to the pre-pubescent. He offered a further clarification on Facebook:

I do not support pedophilia. Period. It is a vile and disgusting crime, perhaps the very worst…If I choose to deal in an edgy way on an internet livestream with a crime I was the victim of that’s my prerogative. It’s no different to gallows humor from AIDS sufferers…I did say that there are relationships between younger men and older men that can help a young gay man escape from a lack of support or understanding at home. That’s perfectly true and every gay man knows it. But I was not talking about anything illegal and I was not referring to pre-pubescent boys.

It’s not, on the face of it, an unreasonable explanation. Yiannopoulos may not have made his point very well. But there’s something nuanced and defensible here. First, he’s saying that the relationships between gay men and teenage boys (according to their own accounts) have historically been messier than simple categories allow for. And second, it’s absurd to say that he can’t make dark or crass jokes about his priest if it’s his way of dealing with what happened to him.

Unfortunately for Yiannopoulos, there is no possibility of complexity where it comes to discussions around age, sex, and consent. Fears of pedophilia have made it so that even the slightest hint that one is condoning it brings instant total ostracism. (These same sentiments have also made it so that no punishment is considered too severe when it comes to those convicted of sex crimes against minors. Nobody wants to speak out on behalf of society’s most loathed group of criminals, thus they get shunted under bridges and denied housing rather than given treatment.)

Yiannopoulos has quickly found out which ideas will actually get you booted from the public square, and they’re left-wing ones rather than right-wing ones. It turned out the real people you can’t offend are the conservatives whose latent homophobia make them instantly pounce on a gay man as a defender of pedophilia when he tries to explain his world to them. How fitting that Yiannopoulos should end up subjected to the very kind of vicious misrepresentation of LGBT people that he has spent his time encouraging. How appropriate for him to discover that his friends on the right only supported him so long as he nurtured their prejudices; they loved their campy gay mascot until the moment he challenged them. Then he was a pervert.

You can learn a lot about society’s values and the allocation of power by examining what people get exiled for. Yiannopoulos is not the only heinous individual who was punished for his lesser crime rather than his greater ones. Trump’s former National Security Adviser Michael Flynn was a bigoted nutcase who called Islam a “cancer” and said that “fear of Muslims is rational.” He promoted the works of alt-right conspiracist and rape apologist Mike Cernovich, as well as a host of other figures from the far-right fringes. Yet all of this was, if anything, a qualification for his position. Instead, what did him in was his dissembling about a chat with the Russian ambassador. Tell scurrilous lies about the weak and excluded, and you’re fine. Tell minor lies about diplomacy to the Vice President, and you’re toast.

It is interesting watching people turn on Yiannopoulos over the more innocuous thing, instead of the more insidious things. Newsweek‘s Kurt Eichenwald, for example, said that while Yiannopoulos had “good points on PC culture,” anyone who stuck with him now was “evil.” Thus in the phase when Milo was bashing gays and transgender people, he was simply “making good points.” Now that he’s been caught sticking up for gays, he’s a monster. When he invited conservatives to titter at LGBT people, he was fun. When he tried to speak earnestly and sincerely on behalf of those people, he was reprehensible.

Of course, Yiannopoulos is a monster. Personally, I find him totally odious and lacking in any appealing traits. (Though my opinion on this is not universal; New Statesman editor Laurie Penny has called him “sweet,” “charming,” and “kind.”) The fact that he received a major book deal, and was paid any mainstream attention at all, sadly shows how the amorality of the market can allow those who pander to the ugliest human instincts to be handsomely remunerated. But even those of us on the left who have hated nearly every word he has spoken should be disturbed that his attempts to defend gay sexuality, rather than his attacks on it, are what got his $250,000 taken away.

I doubt anything could make Milo Yiannopoulos feel even a faint pang of conscience or regret over his long record of cruelty and unpleasantness. He seems, in both his public and private dealings, a proud sociopath. But ideally this would teach Yiannopoulos an important lesson. He believed he was a martyr for free speech when he toured college campuses making cracks about how feminists are ugly. In fact, he was steadily growing more popular, and offered limitless money and television appearances. He believed that people found his ideas “dangerous,” but a mainstream publisher seemed to think they belonged in bookshops everywhere.

In fact, it turned out that there was nothing “dangerous” at all in picking on women and refugees. People will pay you good money for that. The dangerous ideas are the ones they don’t pay you for, the ones that don’t get you on HBO. You’re actually dangerous when you do what Yiannopoulos did in the “pedophile” tapes: defend society’s most hated outcasts, and tell the truth about the complexities of gay men’s sexuality. You’re dangerous when you stick up for those on the fringes rather than kicking them. There’s nothing courageous or edgy in bullying the despised and excluded. But it might be dangerous if you dared to empathize with them.