You got to download everything once, after that it's a fast connection. You can also turn custom downloads off. You will download a ton of music, skins and skyboxes I think.

It's a great server, and once you've downloaded everything needed, it'll connect you pretty fast.
Writing everything in bold is pretty fun.

Pretty much what Dr_Heinz said, enable custom downloads and let it download for a bit. After that you should connect to it fine.

If for whatever reason it's not downloading maps try the jump.tf map list: http://jump.tf/forum/index.php?topic=22.0 or Gaugey von Snugglebutt's archive: http://thephlognoobs.tf/tf/maps/ or even
the Whisker Biscuit's one: http://jumptf.site.nfoservers.com/server/maps/

I would recommend other jump servers like TPN or GFM Palestra. Chillouts massive amount of unnecessary plugins and downloads causes problems joining and jumpin on that server. But if you really wanna play there then do what heinz said.

And btw, whenever you join that server it will appear to download the same thing each time, but its apparently just checking if you have the files for that server or something (ive heard this from drexen i think). This will add another ~5 seconds tobyour join time every time you connect.