All because, supposedly, he was too political. Too partisan.
"Is Stephen Colbert too liberal for his own (ratings') good?" asked The Washington Post. The New York Post took that further: "Colbert's Late Show has become propaganda for Democrats."
And then Trump came to Colbert's rescue. Seriously.
With Trump in the White House and Colbert going after him pretty much all the time, "The Late Show's" ratings have surged. For the week ending March 10 (the most recent available), Colbert averaged 3.23 million viewers, his largest weekly audience since he moved to CBS. That's up 33 percent from a year ago.
Fallon's "Tonight Show" (which was in repeats) averaged 2.47 million. He continues to lead in the lucrative 18-to-49 demographic, but even that lead shrank to a tenth of a rating point. Colbert has beaten him in viewers for six straight weeks — CBS' longest winning streak in seven years.
Yes, Fallon, Kimmel, Corden, Seth Meyers and Trevor Noah all make Trump jokes. But they haven't gone at it the way Colbert has. There are nights when his entire opening monologue consists of nothing but jabs at Trump.
He isn't telling polite jokes, he's delivering body slams.
He likened Trump's inaugural address to "Lincoln huffing paint thinner." When Trump claimed he inherited "a mess," Colbert responded, "No, you inherited a fortune — we ELECTED a mess."
Colbert hit him twice with this one: "Trump aides were, quote, 'in constant touch with senior Russian officials during the campaign.' Constant Touch, by the way, is also Trump's Secret Service code name."
Three months ago, the right-wing website American Outlook declared, "Colbert's numbers would probably improve if he dropped the hard left stuff but he likely won't. Liberal celebs still don't seem to understand that the ground has shifted beneath them."
Well, Trump supporters probably aren't tuning in. But that hardly seems to matter, because the numbers are in Colbert's favor. Yes, Trump lost the popular vote to Hillary Clinton by 2.86 million votes — but add all the votes that went to other candidates and 10 million more people voted against than for Trump.
You'd have to think that the 54 percent of voters who voted against him would be open to a late-night show that regularly makes fun of him.
And this week's Gallup tracking poll has Trump's approval rating down to 39 percent; his disapproval rating is up to 55 percent.
That's a huge potential audience for Colbert to cultivate.