Upvote Upvoted 29 Downvote Downvoted
TF2's hidden stats part 2
1
#1
0 Frags +

Last time I introduced the concept of Net Kills per Minute. This time I'll be using it to inform our understanding of healing and medic performance.

Medics are very difficult to understand or rank in TF2. They're completely at the mercy of their team's death match performance and team fight coordination, if they can't be protected they won't get good healing rates. Although uber drop rates can be a useful indicator, there are times when dropping uber is the right play, and over any large dataset uber rates correspond closely with healing rates.

Healing rates have also fairly recently started to become less reliable because of the rise of the crossbow and the fact that crossbow healing isn't logged. But even so, it seems some patterns can still be discerned.

So how does Net Kills per Minute play into medic healing rates? Taking the concept that medics are subject to their team's performance, and that NKPM is a superior stat for fragging classes, we can look at how healing is related to this.

The relationship between raw team NKPM and healing is actually quite weak, and the obvious reason is that many of the kills are taking place away from the medic by players on the flank. Although the success or failure of these players can affect the medic by allowing flanks, etc, it's the performance of the combo players that receive the bulk of the healing and whose duties specifically include protecting the medic that matters most.

So we can come up with yet another stat, Healing adjusted Net Kills Per Minute (HNKPM). The purpose of this stat, probably against everybody's expectation, is not to judge those players performances by the amount of healing they receive - this isn't a particularly useful exercise. It's to judge their influence on the medic.

The form of HNKPM is the player's NKPM mulitplied by their healing received percentage, so if a player has +1.5 NKPM and they received 40% of the heals, their HNKPM is +1.5 * 0.4, or +0.6. This way we scale down the influence of players that receive little healing in comparison to their combo counterparts. It would be possible to do something that doesn't scale down combo players as well, like multiplying by their healing percentage over team average, but that wouldn't affect the maths and adds an unnecessary complication.

Once we have player HNKPM we can calculate the team value, THNKPM. The comparison between the two team stats in a given match produces something interesting, the expected difference of healing rates between the two teams. The formula:

165 * T1HNKPM - 165 * T2HNKPM

Gives the expected healing rate difference between the two teams. The r squared is around 0.5, which is pretty good considering how arcane the process is to get here.

165 is a constant arrived at from performing a regression of a few hundred top level competitive logs. In this case it represents something to do with the meta game, if crossbow usage went down for example this would start to rise because more medigun healing would be logged increasing the difference. If Medics somehow became much more vulnerable it would go down as there would be less healing overall, and therefore less difference.

But this doesn't tell us anything about an individual medic's healing rate in particular. In fact the link between either team's kill stats and either one of the medic's expected performance is very weak. The reason I would postulate for this is that kill rates don't tell us anything particularly interesting about the type of game and it's effects on the medics.

It seems games where the medics are all-in and risking everything, being put under constant pressure and having their healing rates suppressed aren't indicated by player kill stats. Players will always get kills it seems even when medics are having a fairly easy time of things.

The only stat I could find that could potentially inform one medic of the style of the game is the other medic's healing rate. This may seem a cop out at first, but remember that when your medic dies, a priority is often to kill or force the other medic to neutralise uber advantage. When your team is fully committed and your medic is in danger, so is theirs. And the fact that comparing fragging rates above gives a reasonable prediction of the difference in healing rates, but are of little use when predicting the specific rate of healing tells us that the relationship must work this way.

I also looked at DPM in this area, and performed that same heal adjustments on it. My logic in including it was that DPM is something of a proxy for aggressive play, and a medic on an aggressive team will have plenty of healing to do regardless of success. I found including them somewhat improved the accuracy of the model otherwise I wouldn't have kept it. THDPM is a team's heal adjusted DPM.

Performing a regression on this data we get the following formula:

Expected Team 1 Heals Per Minute = 380 + 145 * T1HNKPM - 255 * T2HNKPM + 85 * T1HDPM + 0.67 * T2HPM

Which can give us an expectation of the medic's performance, and getting a healing rate above or below this level could be considered a better or worse performance.

T2HPM is the other team's healing rate. To work out the expected healing rate for the team 2 just reverse the references. The r squared for this model is 0.6, which probably leaves too much that isn't predicted to be reasonably accounted for by skill differences, but without crossbow healing it would be impossible to become more accurate.

There is also the risk of overfitting - the constants in this model are obviously subject to changes in the metagame but more importantly just biases in the dataset that may not hold more generally. In terms of material available though this is a fairly simple model that reflects what has happened to a reasonable degree. It's possible to get a more accurate formula, but it's incredibly complex and arcane and almost certainly matched too closely to the dataset.

This is probably much too complicated for anyone to actually make use of, but it at least shows what is possible.

Last time I introduced the concept of Net Kills per Minute. This time I'll be using it to inform our understanding of healing and medic performance.

Medics are very difficult to understand or rank in TF2. They're completely at the mercy of their team's death match performance and team fight coordination, if they can't be protected they won't get good healing rates. Although uber drop rates can be a useful indicator, there are times when dropping uber is the right play, and over any large dataset uber rates correspond closely with healing rates.

Healing rates have also fairly recently started to become less reliable because of the rise of the crossbow and the fact that crossbow healing isn't logged. But even so, it seems some patterns can still be discerned.

So how does Net Kills per Minute play into medic healing rates? Taking the concept that medics are subject to their team's performance, and that NKPM is a superior stat for fragging classes, we can look at how healing is related to this.

The relationship between raw team NKPM and healing is actually quite weak, and the obvious reason is that many of the kills are taking place away from the medic by players on the flank. Although the success or failure of these players can affect the medic by allowing flanks, etc, it's the performance of the combo players that receive the bulk of the healing and whose duties specifically include protecting the medic that matters most.

So we can come up with yet another stat, Healing adjusted Net Kills Per Minute (HNKPM). The purpose of this stat, probably against everybody's expectation, is not to judge those players performances by the amount of healing they receive - this isn't a particularly useful exercise. It's to judge their influence on the medic.

The form of HNKPM is the player's NKPM mulitplied by their healing received percentage, so if a player has +1.5 NKPM and they received 40% of the heals, their HNKPM is +1.5 * 0.4, or +0.6. This way we scale down the influence of players that receive little healing in comparison to their combo counterparts. It would be possible to do something that doesn't scale down combo players as well, like multiplying by their healing percentage over team average, but that wouldn't affect the maths and adds an unnecessary complication.

Once we have player HNKPM we can calculate the team value, THNKPM. The comparison between the two team stats in a given match produces something interesting, the expected difference of healing rates between the two teams. The formula:

165 * T1HNKPM - 165 * T2HNKPM

Gives the expected healing rate difference between the two teams. The r squared is around 0.5, which is pretty good considering how arcane the process is to get here.

165 is a constant arrived at from performing a regression of a few hundred top level competitive logs. In this case it represents something to do with the meta game, if crossbow usage went down for example this would start to rise because more medigun healing would be logged increasing the difference. If Medics somehow became much more vulnerable it would go down as there would be less healing overall, and therefore less difference.

But this doesn't tell us anything about an individual medic's healing rate in particular. In fact the link between either team's kill stats and either one of the medic's expected performance is very weak. The reason I would postulate for this is that kill rates don't tell us anything particularly interesting about the type of game and it's effects on the medics.

It seems games where the medics are all-in and risking everything, being put under constant pressure and having their healing rates suppressed aren't indicated by player kill stats. Players will always get kills it seems even when medics are having a fairly easy time of things.

The only stat I could find that could potentially inform one medic of the style of the game is the other medic's healing rate. This may seem a cop out at first, but remember that when your medic dies, a priority is often to kill or force the other medic to neutralise uber advantage. When your team is fully committed and your medic is in danger, so is theirs. And the fact that comparing fragging rates above gives a reasonable prediction of the difference in healing rates, but are of little use when predicting the specific rate of healing tells us that the relationship must work this way.

I also looked at DPM in this area, and performed that same heal adjustments on it. My logic in including it was that DPM is something of a proxy for aggressive play, and a medic on an aggressive team will have plenty of healing to do regardless of success. I found including them somewhat improved the accuracy of the model otherwise I wouldn't have kept it. THDPM is a team's heal adjusted DPM.

Performing a regression on this data we get the following formula:

Expected Team 1 Heals Per Minute = 380 + 145 * T1HNKPM - 255 * T2HNKPM + 85 * T1HDPM + 0.67 * T2HPM

Which can give us an expectation of the medic's performance, and getting a healing rate above or below this level could be considered a better or worse performance.

T2HPM is the other team's healing rate. To work out the expected healing rate for the team 2 just reverse the references. The r squared for this model is 0.6, which probably leaves too much that isn't predicted to be reasonably accounted for by skill differences, but without crossbow healing it would be impossible to become more accurate.

There is also the risk of overfitting - the constants in this model are obviously subject to changes in the metagame but more importantly just biases in the dataset that may not hold more generally. In terms of material available though this is a fairly simple model that reflects what has happened to a reasonable degree. It's possible to get a more accurate formula, but it's incredibly complex and arcane and almost certainly matched too closely to the dataset.

This is probably much too complicated for anyone to actually make use of, but it at least shows what is possible.
2
#2
4 Frags +

How well does this correlate with, say, mid and low tier games, given that your model currently only reflects higher level play (presumably Prem and Invite)?

How well does this correlate with, say, mid and low tier games, given that your model currently only reflects higher level play (presumably Prem and Invite)?
3
#3
6 Frags +
KiNGHow well does this correlate with, say, mid and low tier games, given that your model currently only reflects higher level play (presumably Prem and Invite)?

I forgot to mention this doesn't reflect ESEA, they still don't even log overheal so there's less relevance there. Also the post only refers to 6v6.

I'm not sure what mid tier TF2 looks like, my options are between high level datasets that are recorded for the stream stats you see around the leagues and a huge blob of lobby data that has insanely high variations in skill level and coordination.

[quote=KiNG]How well does this correlate with, say, mid and low tier games, given that your model currently only reflects higher level play (presumably Prem and Invite)?[/quote]
I forgot to mention this doesn't reflect ESEA, they still don't even log overheal so there's less relevance there. Also the post only refers to 6v6.

I'm not sure what mid tier TF2 looks like, my options are between high level datasets that are recorded for the stream stats you see around the leagues and a huge blob of lobby data that has insanely high variations in skill level and coordination.
4
#4
8 Frags +

jon you should teach a statistics class on tf2 logs

jon you should teach a statistics class on tf2 logs
5
#5
1 Frags +

very interesting just like your last post, makes me wish tf2 had people paid to analyse these kind of statistics.

very interesting just like your last post, makes me wish tf2 had people paid to analyse these kind of statistics.
6
#6
16 Frags +
Moist_Penguinvery interesting just like your last post, makes me wish tf2 had people paid to analyse these kind of statistics.

Do you want my paypal?

[quote=Moist_Penguin]very interesting just like your last post, makes me wish tf2 had people paid to analyse these kind of statistics.[/quote]
Do you want my paypal?
7
#7
2 Frags +

Is it always the case that logs don't count crossbow heals? I think in ESEA it tracks them because I remember seeing some absurd pocket heal numbers that only made sense because they were arrow building, maybe logs doesn't though.

Is it always the case that logs don't count crossbow heals? I think in ESEA it tracks them because I remember seeing some absurd pocket heal numbers that only made sense because they were arrow building, maybe logs doesn't though.
8
#8
0 Frags +
bearodactylIs it always the case that logs don't count crossbow heals? I think in ESEA it tracks them because I remember seeing some absurd pocket heal numbers that only made sense because they were arrow building, maybe logs doesn't though.

I'm honestly not sure, I thought it was commonly accepted that they don't but maybe it's just not possible to tell how much comes from crossbow and how much from medigun. I'd have to go through a file looking for the healing events to tell which I haven't.

If they are logged as normal healing events than those would appear in ESEA, and it changes the interpretation of the analysis a little but not to an enormous degree. In that case I might have taken some of the more advanced regressions a bit more seriously but they produce really complicated formulae anyway.

[quote=bearodactyl]Is it always the case that logs don't count crossbow heals? I think in ESEA it tracks them because I remember seeing some absurd pocket heal numbers that only made sense because they were arrow building, maybe logs doesn't though.[/quote]
I'm honestly not sure, I thought it was commonly accepted that they don't but maybe it's just not possible to tell how much comes from crossbow and how much from medigun. I'd have to go through a file looking for the healing events to tell which I haven't.

If they are logged as normal healing events than those would appear in ESEA, and it changes the interpretation of the analysis a little but not to an enormous degree. In that case I might have taken some of the more advanced regressions a bit more seriously but they produce really complicated formulae anyway.
9
#9
5 Frags +

I am pretty sure that logs count crossbow healing up to a certain point, but after a certain threshold logs.tf counts it separately in which case it sometimes will not appear on the logs, varying by server. My current theory behind it is that non medigun healing can be counted in logs (such as mad milk or dispensers in highlander) so crossbows might be affected in some form by that mechanism. In ultiduo logs the effect can be rather obvious when crossbow and medigun healing is pooled into a medics total healing but the heal distribution shows something like 70% healing into the soldier with an unaccounted for 30%. Presumably that 30% is the healing done by arrows, but I haven't actually checked through healing events in logs to try and figure out the specifics.

Edit: here is an example log-> http://logs.tf/1760443#76561198066923067
Delpo has an unaccounted for 25% healing, and my healing to percy roughly matches delpo's accounted for 75% healing. This leads me to believe that delpo had 4555 crossbow healing, and my presumed crossbow healing never showed in logs. I don't have the patience to parse through the log file to confirm this, but it makes the most logical sense as I don't think percy would be winning a game where she takes more damage and airshots while also getting healed for nearly 200 hp/minute less.

The fact that it also appears to vary by server makes it more of a headache, as it should then theoretically be a problem with the MedicStats plugin? I honestly have no idea.

I am pretty sure that logs count crossbow healing up to a certain point, but after a certain threshold logs.tf counts it separately in which case it sometimes will not appear on the logs, varying by server. My current theory behind it is that non medigun healing can be counted in logs (such as mad milk or dispensers in highlander) so crossbows might be affected in some form by that mechanism. In ultiduo logs the effect can be rather obvious when crossbow and medigun healing is pooled into a medics total healing but the heal distribution shows something like 70% healing into the soldier with an unaccounted for 30%. Presumably that 30% is the healing done by arrows, but I haven't actually checked through healing events in logs to try and figure out the specifics.

Edit: here is an example log-> http://logs.tf/1760443#76561198066923067
Delpo has an unaccounted for 25% healing, and my healing to percy roughly matches delpo's accounted for 75% healing. This leads me to believe that delpo had 4555 crossbow healing, and my presumed crossbow healing never showed in logs. I don't have the patience to parse through the log file to confirm this, but it makes the most logical sense as I don't think percy would be winning a game where she takes more damage and airshots while also getting healed for nearly 200 hp/minute less.

The fact that it also appears to vary by server makes it more of a headache, as it should then theoretically be a problem with the MedicStats plugin? I honestly have no idea.
10
#10
2 Frags +
Daffodil-The fact that it also appears to vary by server makes it more of a headache, as it should then theoretically be a problem with the MedicStats plugin? I honestly have no idea.

It might be that there is a difference between the stats plugins and how things are broken down, tftrue might operate differently to the sourcemod plugin. These kinds of differences might account for some of the variability in the study.

[quote=Daffodil-]The fact that it also appears to vary by server makes it more of a headache, as it should then theoretically be a problem with the MedicStats plugin? I honestly have no idea.[/quote]
It might be that there is a difference between the stats plugins and how things are broken down, tftrue might operate differently to the sourcemod plugin. These kinds of differences might account for some of the variability in the study.
Please sign in through STEAM to post a comment.