As some people have said above, make sure the TGA folder is wherever TF2 is set to store your TGA files. You can also uncheck "delete frames after processing" to see if it's actually recording anything.
I personally haven't had issues with the program, but whenever I record stuff I also tend to use HLAE, on top of having my own recording config, so I never actually launch the game/use the built-in hotkeys to record stuff.
Here's what I normally do:
Just launch the game how you would regularly, load up the demo, find where you wanna record, all that stuff. Then in AVI recorder, (make sure "Start game and create recording configs" is unchecked), hit "Start recording."
I have these two binds to start and stop recording:
bind [ "engine_no_focus_sleep 0; r_cleardecals; snd_soundmixer Default_Mix; demo_resume; gameui_hide; startmovie frames_ raw"
bind ] "endmovie; engine_no_focus_sleep 50; demo_pause; demoui"
(All of those commands are what the program normally uses anyway when you hit F9, but if you don't care to fuck with that you can just do this:)
bind [ "demo_resume; startmovie frames_ raw"
bind ] "endmovie; demo_pause"
Make sure you have "host_framerate" set to what framerate you want to record at, and check that it matches in AVI Recorder. Then, just hit [ to start recording, and then hit ] to stop it. Whenever you're done recording whatever make sure to hit "Stop recording" before closing the program to avoid losing anything.
At this point, you're basically just using the program to cut out having to stitch the frames together, but I still use it so I don't have to worry about filling up my SSD with uncompressed picture files, or have to worry about accidentally overriding frames because I forgot to change the name of the shit in the startmovie command. (Which I've done a lot rofl)
I'm not sure if this is 100% necessary or not to get the thing to work, but this is what I've always done and its always worked fine.
If you don't want to do any of that shit, at the very least try having your own recording binds and test it out that way, without using F9 to stop and start the recording.