Upvote Upvoted 22 Downvote Downvoted
Python Script to Average All DPM / KD from Logs.TF
posted in Projects
1
#1
0 Frags +

Last night I started writing a python script that would iterate through all logs.tf files linked to an inputted steam ID. It grabs all the important details from your logs and averages them out to give you a good idea of your logs.tf profile as a whole. It runs in a CMD window and has been compiled to .exe using PyInstaller.

Features:
Average DPM
Average KD Ratio

Average Kills/Game
Average Deaths/Game

It only runs at the speed of which the requests module in python can harvest a logs.tf page source code. I've found that it takes on average 0.795s to iterate through an entire log file and isolate a specific steam ID's elements. The Average KD ratio grabbed from logs.tf will be slightly different than dividing Average Kills by Average Deaths due to rounding error.

Download

Source Code

Version 2 is now available here: https://www.teamfortress.tv/57870/python-logs-tf-stats-calculator-v2

Last night I started writing a python script that would iterate through all logs.tf files linked to an inputted steam ID. It grabs all the important details from your logs and averages them out to give you a good idea of your logs.tf profile as a whole. It runs in a CMD window and has been compiled to .exe using PyInstaller.

[u][b]Features:[/b][/u]
Average DPM
Average KD Ratio

Average Kills/Game
Average Deaths/Game

It only runs at the speed of which the requests module in python can harvest a logs.tf page source code. I've found that it takes on average 0.795s to iterate through an entire log file and isolate a specific steam ID's elements. The Average KD ratio grabbed from logs.tf will be slightly different than dividing Average Kills by Average Deaths due to rounding error.


[url=https://mega.nz/file/vH5B3I4L#k7Av6TnEARHN7F0DPZlYmIGIX4kLFQOSFqEmBL2blqQ]Download[/url]

[url=https://pastebin.com/VhSWJ0vs]Source Code[/url]

Version 2 is now available here: https://www.teamfortress.tv/57870/python-logs-tf-stats-calculator-v2
2
#2
3 Frags +

Finally people will see my TF2Center prowess!

Seriously though, this is kinda beast and would be nice if you could filter tf2center/pugchamp logs.

Finally people will see my TF2Center prowess!

Seriously though, this is kinda beast and would be nice if you could filter tf2center/pugchamp logs.
3
#3
11 Frags +

hey i,m posting this in place of an anonymous friend they would like to request average caps par game added to this plugin

hey i,m posting this in place of an anonymous friend they would like to request average caps par game added to this plugin
4
#4
16 Frags +

https://i.imgur.com/3kdXjY4.png

maybe i play too much tf2

[img]https://i.imgur.com/3kdXjY4.png[/img]

maybe i play too much tf2
5
#5
-4 Frags +

https://i.imgur.com/9sQqIC7.png

[img]https://i.imgur.com/9sQqIC7.png[/img]
6
#6
3 Frags +

This is very cool, shared it with my team and we had some feedback. It'd be cool if we could filter by a date range so we don't have to include our noob logs from years ago when we went 2:20 with 100 DPM, and filter by 6s/highlander. I'm not much of a programmer so I hope this stuff isn't hard to implement, assuming you're interested in doing so. Thanks for sharing this.

This is very cool, shared it with my team and we had some feedback. It'd be cool if we could filter by a date range so we don't have to include our noob logs from years ago when we went 2:20 with 100 DPM, and filter by 6s/highlander. I'm not much of a programmer so I hope this stuff isn't hard to implement, assuming you're interested in doing so. Thanks for sharing this.
7
#7
7 Frags +

either im incredibly dumb or it doesnt work with my steam id
EDIT: you should specify which SteamID you need. Turns out its Steam64 ID

either im incredibly dumb or it doesnt work with my steam id
EDIT: you should specify which SteamID you need. Turns out its Steam64 ID
8
#8
whitelist.tf
8 Frags +

Should make it save the log files locally in a folder with the associated steamid, that way you can just run the script again and only grab the new logs that haven't been downloaded since last time you ran it.

Should make it save the log files locally in a folder with the associated steamid, that way you can just run the script again and only grab the new logs that haven't been downloaded since last time you ran it.
9
#9
4 Frags +

Im assuming this collates logs regardless of class? So med DPM and K/D would be included too

Im assuming this collates logs regardless of class? So med DPM and K/D would be included too
10
#10
21 Frags +

I don't want to be that annoying guy but I think you should be using logs.tf API for this. Most definitely if you want to continue working on this.
This way you can get the last 10000 logs from a player in a JSON format (which is the limit):
http://logs.tf/api/v1/log?player=<Steam64ID>&limit=10000
And this way you can get the information from each log:
http://logs.tf/api/v1/log/<logid>;
More on this here: logs.tf

I've been working on a project for a year now which uses that api and some other stuff to return the ticks of certain events that happen in game (e.g. killstreaks, airshots etc,) so one can easily record a frag movie or something of the sorts. It's pretty much finished I've just been too scared to release it ^^"

If you have any questions about this stuff I'd be glad to help!

I don't want to be that annoying guy but I think you should be using logs.tf API for this. Most definitely if you want to continue working on this.
This way you can get the last 10000 logs from a player in a JSON format (which is the limit):
http://logs.tf/api/v1/log?player=<Steam64ID>&limit=10000
And this way you can get the information from each log:
http://logs.tf/api/v1/log/<logid>
More on this here: [url=https://logs.tf/about#json]logs.tf[/url]

I've been working on a project for a year now which uses that api and some other stuff to return the ticks of certain events that happen in game (e.g. killstreaks, airshots etc,) so one can easily record a frag movie or something of the sorts. It's pretty much finished I've just been too scared to release it ^^"

If you have any questions about this stuff I'd be glad to help!
11
#11
2 Frags +

This started as a small project but I can see a lot of people are enjoying it! Thanks for the feedback everyone, I'm taking all of your ideas into consideration and will be expanding upon this project soon :)

This started as a small project but I can see a lot of people are enjoying it! Thanks for the feedback everyone, I'm taking all of your ideas into consideration and will be expanding upon this project soon :)
12
#12
7 Frags +
BvI don't want to be that annoying guy but I think you should be using logs.tf API for this. Most definitely if you want to continue working on this.
This way you can get the last 10000 logs from a player in a JSON format (which is the limit):
http://logs.tf/api/v1/log?player=<Steam64ID>&limit=10000
And this way you can get the information from each log:
http://logs.tf/api/v1/log/<logid>;
More on this here: logs.tf

I've been working on a project for a year now which uses that api and some other stuff to return the ticks of certain events that happen in game (e.g. killstreaks, airshots etc,) so one can easily record a frag movie or something of the sorts. It's pretty much finished I've just been too scared to release it ^^"

If you have any questions about this stuff I'd be glad to help!

Thanks for introducing me to this, search times will be significantly sped up using this method.

Edit: After testing out this method a little bit I've noticed a couple things. The API / json version of log files are grabbed nearly just as fast as the original page source code. This is due to the limited internet speed of the python requests module, which can access most websites like logs.tf/<log-id>; just as fast as it's API counterpart. I'll definitely still be using the API / json method for grabbing log files purely because it's much easier to read and iterate through, however don't expect a major speed increase. The API for the profile page seems very handy, as it completely eliminates the "Preparing for Iteration" you see when you first run the script by quickly grabbing the entire list of a user's log IDs. It'll also be a big help when I start developing custom searches, as it shows the date and how many players participated.

[quote=Bv]I don't want to be that annoying guy but I think you should be using logs.tf API for this. Most definitely if you want to continue working on this.
This way you can get the last 10000 logs from a player in a JSON format (which is the limit):
http://logs.tf/api/v1/log?player=<Steam64ID>&limit=10000
And this way you can get the information from each log:
http://logs.tf/api/v1/log/<logid>
More on this here: [url=https://logs.tf/about#json]logs.tf[/url]

I've been working on a project for a year now which uses that api and some other stuff to return the ticks of certain events that happen in game (e.g. killstreaks, airshots etc,) so one can easily record a frag movie or something of the sorts. It's pretty much finished I've just been too scared to release it ^^"

If you have any questions about this stuff I'd be glad to help![/quote]

Thanks for introducing me to this, search times will be significantly sped up using this method.

[u]Edit:[/u] After testing out this method a little bit I've noticed a couple things. The API / json version of log files are grabbed nearly just as fast as the original page source code. This is due to the limited internet speed of the python requests module, which can access most websites like logs.tf/<log-id> just as fast as it's API counterpart. I'll definitely still be using the API / json method for grabbing log files purely because it's much easier to read and iterate through, however don't expect a major speed increase. The API for the profile page seems very handy, as it completely eliminates the "Preparing for Iteration" you see when you first run the script by quickly grabbing the entire list of a user's log IDs. It'll also be a big help when I start developing custom searches, as it shows the date and how many players participated.
13
#13
2 Frags +

i want to be able to see what logs i hit the most airshots in

i want to be able to see what logs i hit the most airshots in
14
#14
6 Frags +

Nice work!

If you're curious, I did something similar for pugs ( https://github.com/TheAntecedent/Quintessence ), which was based on grabbing logs uploaded by a single uploader. You could swap out the intake piece with grabbing all logs for a given player and use the rest of the code, if that's helpful.

Zesty also wrote some log analysis code a little while back to predict which team would win a given match, and 1Stirling ran a long-term metrics project that ranked both players & teams for about 3 years ( https://tf2metrics.wordpress.com/ ).

Nice work!

If you're curious, I did something similar for pugs ( https://github.com/TheAntecedent/Quintessence ), which was based on grabbing logs uploaded by a single uploader. You could swap out the intake piece with grabbing all logs for a given player and use the rest of the code, if that's helpful.

Zesty also wrote some log analysis code a little while back to predict which team would win a given match, and 1Stirling ran a long-term metrics project that ranked both players & teams for about 3 years ( https://tf2metrics.wordpress.com/ ).
15
#15
2 Frags +
AntecedentNice work!

If you're curious, I did something similar for pugs ( https://github.com/TheAntecedent/Quintessence ), which was based on grabbing logs uploaded by a single uploader. You could swap out the intake piece with grabbing all logs for a given player and use the rest of the code, if that's helpful.

Zesty also wrote some log analysis code a little while back to predict which team would win a given match, and 1Stirling ran a long-term metrics project that ranked both players & teams for about 3 years ( https://tf2metrics.wordpress.com/ ).

Very cool stuff, I've only been using python for about a year so I'm always looking forward to learning more. I'll definitely take a look at these and see what I can use in my project.

[quote=Antecedent]Nice work!

If you're curious, I did something similar for pugs ( https://github.com/TheAntecedent/Quintessence ), which was based on grabbing logs uploaded by a single uploader. You could swap out the intake piece with grabbing all logs for a given player and use the rest of the code, if that's helpful.

Zesty also wrote some log analysis code a little while back to predict which team would win a given match, and 1Stirling ran a long-term metrics project that ranked both players & teams for about 3 years ( https://tf2metrics.wordpress.com/ ).[/quote]

Very cool stuff, I've only been using python for about a year so I'm always looking forward to learning more. I'll definitely take a look at these and see what I can use in my project.
16
#16
0 Frags +
Nathanhttps://i.imgur.com/3kdXjY4.png

maybe i play too much tf2

not sure what causes it but about 90 minutes into parsing these logs (85% completion) the program crashes. too many logs maybe?

[quote=Nathan][img]https://i.imgur.com/3kdXjY4.png[/img]

maybe i play too much tf2[/quote]

not sure what causes it but about 90 minutes into parsing these logs (85% completion) the program crashes. too many logs maybe?
17
#17
3 Frags +
NathanNathanhttps://i.imgur.com/3kdXjY4.png

maybe i play too much tf2

not sure what causes it but about 90 minutes into parsing these logs (85% completion) the program crashes. too many logs maybe?

Must've been an error while iterating through a specific log. Most of these issues will be fixed when I upload v2 of the program in a couple days.

[quote=Nathan][quote=Nathan][img]https://i.imgur.com/3kdXjY4.png[/img]

maybe i play too much tf2[/quote]

not sure what causes it but about 90 minutes into parsing these logs (85% completion) the program crashes. too many logs maybe?[/quote]

Must've been an error while iterating through a specific log. Most of these issues will be fixed when I upload v2 of the program in a couple days.
18
#18
1 Frags +

tried it yesterday and got this error

Show Content

it was an ok log and ran fine through the other people in the log, this first error then resolved itself and it now crashes on an actually broken log (https://logs.tf/2627689#76561198071882999) with this

Show Content

the second one actually goes through the try except block lol and crashes later. from what i can see in the source code, you don't skip broken logs?

also a tip, there are many filter settings suggested, so you could make a very basic gui, download and store log files locally (maybe in folders named after steam ids?) and filter logs locally

tried it yesterday and got this error [spoiler][img]https://i.imgur.com/ob0Zs1L.png[/img][/spoiler]
it was an ok log and ran fine through the other people in the log, this first error then resolved itself and it now crashes on an actually broken log (https://logs.tf/2627689#76561198071882999) with this [spoiler][img]https://i.imgur.com/BgDx3Ci.png[/img][/spoiler]
the second one actually goes through the try except block lol and crashes later. from what i can see in the source code, you don't skip broken logs?

also a tip, there are many filter settings suggested, so you could make a very basic gui, download and store log files locally (maybe in folders named after steam ids?) and filter logs locally
19
#19
10 Frags +

So I had a look at your source code (with the caveat that I only just understand python) and you're averaging DPM numbers from logs, which gets you the wrong result - you need to sum damage and time played, then divide the one by the other. The same goes with averaging K/D rather than dividing avg kills by avg deaths.
It might also be worth averaging kills by time instead of by game, since if someone always plays longer games they'd naturally have more kills per game than someone who doesn't, even if they both had the same kills per 30 mins

So I had a look at your source code (with the caveat that I only just understand python) and you're averaging DPM numbers from logs, which gets you the wrong result - you need to sum damage and time played, then divide the one by the other. The same goes with averaging K/D rather than dividing avg kills by avg deaths.
It might also be worth averaging kills by time instead of by game, since if someone always plays longer games they'd naturally have more kills per [i]game[/i] than someone who doesn't, even if they both had the same kills per [i]30 mins[/i]
20
#20
-1 Frags +

will u be importing sklearn?

will u be importing sklearn?
21
#21
0 Frags +

(assuming you only care about tracking averages and aggregate stuff for single players,) rather than saving all of the logs locally, you can just keep running sums/averages for w/e stats you care about, and some data on the most recent log in a single yaml or json file for each steamid. When searching, you just get everything from one file. If new logs have been added, you'll just update the running averages and stuff and set the most recent log. Lots cleaner and easier than storing every log imo!

(assuming you only care about tracking averages and aggregate stuff for single players,) rather than saving all of the logs locally, you can just keep running sums/averages for w/e stats you care about, and some data on the most recent log in a single yaml or json file for each steamid. When searching, you just get everything from one file. If new logs have been added, you'll just update the running averages and stuff and set the most recent log. Lots cleaner and easier than storing every log imo!
22
#22
tf2pickup.org
2 Frags +

you may also want to do a github repo to store all the data so you can easily get some pull requests :o

you may also want to do a github repo to store all the data so you can easily get some pull requests :o
23
#23
12 Frags +

wat happens when u play a season of medic

wat happens when u play a season of medic
Please sign in through STEAM to post a comment.