Valve.
One of the problems with watching comp TF2 is that it's way too much of a mess and since there is no reliable auto director like CSGO has there is no coordination with what's on stream and what the casters are saying, which makes it really hard for a TF2 newbie to understand what's going on.
The other problem is all the bugs/issues with the game itself, pause breaks the HUD and player models, no real spectator tools (there is so much the community can do), random lags, etc.
But the worst has to be the lack of people that actually know about comp TF2, I can guarantee 80% of your normal pubbers don't know what rollout is, something like Dota's main page stream and tournaments information would bump up those viewers count up.