What I learned:
1) OBS has overall higher performance than Xsplit.
2) Turning Aero off isn't as great as people think, and in fact OBS has significantly increased performance when using screen regions if Aero is on.
3) OBS's built-in Window Capture is more efficient than Dxtory.
NOTE: OBS can function EXACTLY like Dxtory (only showing your game window) if you play in a window, use "Window Capture", and have Aero enabled.

Caveats:[/b]
These results are specific to my computer and my setup. Your results may vary.
TF2 was the only game tested. Other games may perform better or worse in these configurations.
Window Capture and Monitor Capture in OBS are NOT the same thing, and can behave very differently. It is NOT recommended to use Aero if using Monitor Capture.
Even if using Aero results in higher performance, it may also result in higher input lag. Use caution.

Conclusion:
The data speaks for itself, OBS w/ Software Capture & Aero enabled has the best performance. I am going to stop using Dxtory to capture TF2, and stop using Xsplit altogether. OBS is only in an early beta phase, and it is already a superior program. OBS, in my experience, also has better image quality than Xsplit at similar bitrates. After further testing, OBS and XSplit appear to have nearly identical quality

Other tests:
There was a rumor that streaming at 29.97 fps would result in significantly higher performance than streaming at 30 fps. In my tests, this was not the case. Compare the result below to the "Xsplit w/ Dxtory & No Aero" result above.

Xsplit w/ Dxtory & No Aero (29.97 FPS)
AVERAGE: 41486 frames 211.896 seconds 195.81 fps ( 5.11 ms/f) 49.1 fps variability

Image Quality:
It's really hard to tell XSplit and OBS apart in terms of IQ.

Comparison 1: OBS | XSplit
Comparison 2: OBS | XSplit

If you would like to conduct your own tests and compare them against mine, here is the demo file I used for the timedemos: https://dl.dropbox.com/u/6089084/Research/the_timedemo.dem

That's what "Software Capture" is.

When FFsplit has more features than a rock, sure. As it is now I wouldn't use it, so I don't care about testing it.

OBS can do almost anything Xsplit can, and the things it can't do are planned features for future updates.

I believe you can specify a custom RTMP url, so yeah. (Though TBH I think TCP relay is kind of bogus)
Also: http://obsproject.com/forum/viewtopic.php?f=5&t=56

I only tested it for a little bit, but using OBS's screen capture did seem to alleviate the input lag problems I was having. I still need to play with it more though and see how it feels in a 6v6.

As far as I can tell this input lag is a strange problem specific to me, so you guys shouldn't worry about it.

EDIT: I've been told that reducing the "Send Buffer Size" to a low value (such as 4096) can eliminate input lag in some cases. If you have input lag please try this fix.

There are three main ways you can capture video for OBS:
1) Raw screen capture, like the "screen regions" in XSplit everyone knows and loves.
2) Window Capture WITH AERO ENABLED (!!) will ONLY capture the specified window, and no windows layered on top of it.
3) Dxtory

There is no reason that OBS shouldn't be satisfactory for professional casts, though at this moment in time XSplit may be slightly better suited for it. However, I foresee that changing quickly.

placeholder

I got some more data points and didn't have a way to elegantly put them in the original thread (character limit exceeded). New thread ¯\_(ツ)_/¯

45 more FPS by enabling aero (for OBS screen capture)? Wow that's crazy.

Seems great but it limits my fps in tf2 to 60 :\

Try it with Aero off. I've heard that if you have an ATI card, Aero results in forced VSync.

I posted on thier fourms and they said you have to stream with dxtory if you dont wnat it limiting your fps

It's not limiting my FPS when I stream. I have an NVidia card with VSync forced off in my drivers. In fact, I actually can't force it on if I try for windowed applications.

EDIT: I just read your thread and I think you are confused. The "FPS" option you set in OBS is not your desired in-game FPS, and it has no bearing on your in-game FPS. It is the FPS that your stream will run at.

Updated "Caveats" with warning about Aero and input lag.

Yes i know, yet it makes my in game fps go to 60

Played around with it for a while.
I was unable to stream tf2 AT all with xsplit or dxtory but this has just enough to get me playing.

Only problem is it feels like my monitor isn't doing 120hz when I play, like certain things feel laggier.
I am getting above 120fps however though so i dont know if thats the issue. :/

EDIT: it felt significantly smoother if i launched OBS after tf2 was launched.

also here is a small quality sample.

http://www.twitch.tv/withadancenumber/b/334941966

would people consider this watchable?

what about the audio quality between obs and xsplit?

Identical at identical bitrates/codecs.

why is it that OBS has better performance with Aero on? seems strange

I have no idea, and it may be an anomaly local to my system. You should do some timedemos with it on and off and see if you get the same change.

Is there any way that I could bring my transparent stream chat overlay and my "Currently Playing" scrolling title to OBS?

I'd love a performance increase, but I'd hate to sacrifice these things form my stream.

I've tested this, and OBS seems to increase my scoreboard ping (latency?) by 20ms, while Xsplit appears to have no effect. Actual ping stays about the same though.

Edit: I do seem to get better local performance though. It probably has a lot to do with my computer at the moment.

My fps is capped at 121, and while using OBS, tf2 tells me I'm getting 120 fps, but it feels like every other frame is being dropped. It's a lot more sluggish than when I'm playing normally. My computer is pretty shitty though. I get 30 fps with the dxtory/xsplit setup.

obs seems to be worse for me in terms of similar settings than xsplit - i don't use dxtory for either one

i was trying to stream dota but video quality seemed a lot more laggy despite having very similar settings. any idea what could have been happening?

input lag seemed similar(it's not really much of an issue, very very minimal)

amd 955, 8 gb ram, 275 nvidia card

edit/note: this lag is only there when i play, not when i spectate?

If you're having latency problems"

This worked for me. My scoreboard ping now hovers around 40 like it usually does.

I did testing with OBS as well. 1090T just set affinity to the last 3 cores while leave the first 3 to your game ect... Didn't notice any game degradation and the stream was clear.

For affinity settings.
Task manager > Processes > Right click OBS** process > Set Affinity > set to desired processors.

yeah i get mouse input lag with OBS...stopped using it

If you have problems with OBS please post about them on the OBS forums.

http://obsproject.com/forum/viewforum.php?f=6

To anyone using OBS: Is your ping higher when using OBS vs xsplit? For me its about 20 more with the same settings and even lower. Thats why I made the post about tcp relay on their forums.

See

Very nice write up Lange

I am going to check out obs but I doubt I will be able to live without the delay server from xsplit, although I imagine I can do something locally - anyone know of a way?

Really nice program this. I don't get nearly as much framedrop using this as I do with xsplit.

Seem to have a problem where it disconnects me quite a lot while I'm playing, though. Anyone else had that yet? Already made a topic about it on their forums.

It's a planned feature.