I have the same problem, it's with the Source game browser. For whatever reason, all ~3000 of the tf2 servers which show up in the browser are pinged at the exact same time, and every time you refresh, it does it again. This effectively DDOS's your router, which bottlenecks the shit out of your net, to the point where you basically can't use it. Luckily, in response to this, valve made a limiter to how often servers can ping you which is here:
http://puu.sh/3zAsv.jpg
http://puu.sh/3zAoK.png
http://puu.sh/3zApU.png
Set it to something like 1000 and it you won't see any noticeable change in net speed when refreshing the browser. It still loads servers reasonably fast.