It seems the IRC part doesn't really check for OAUTH refusal, i.e. the IRC connection will be fine and valid but the twitch server will refuse to let you use the account.
I'd suggest you try connecting to the twitch IRC using the username and oauth key you're trying to use to see if something is going wrong, or even you could go as far as modifying the code to show you all messages received and sent (using log.Print and then recompiling).
Or you could just not bother at all and use http://tf.gg/42187