Upvote Upvoted 68 Downvote Downvoted
1 2
trends.tf - TF2 stats collected from logs.tf
posted in Projects
31
#31
5 Frags +

Hey just wanted to comment and say thanks for putting in the effort to create and maintain this site. It's really cool

Hey just wanted to comment and say thanks for putting in the effort to create and maintain this site. It's really cool
32
#32
trends.tf
4 Frags +

It's been a while since the last update. I'm still working on the site, but recently a lot of that has gone into stuff behind-the-scenes.

  • The whole site configuration got added to version control. I can now spin up a new server in around 30 minutes. I can also track configuration changes with the rest of the site. This has been particularly helpful for working with nginx.
  • I switched to Digital Ocean from Azure, since my student discount ran out. I'm pretty happy with them. I may upgrade to a larger instance if I ever see an uptick in traffic. The database could really use more ram.
  • I added tracing/error reporting using Sentry. This should help me address slow queries and determine if the changes I made are having the affect I wanted. It also has helped me quickly identify errors. I've pushed out several bug fixes recently using errors I found with Sentry.

In addition to the above, there are also some feature updates

  • There is now a recent logs page where you can see all logs uploaded to the site.
  • You can now filter by player. The search is for aliases like the search box, but it only shows their current name. I may try and fix this at some point.
  • Weapons now use their in-game names. I'm using a semi-automated process for this, so let me know if you see any weird weapon names.
  • The combined log limit has been increased to 10. This is primarily so that you can do combined logs for long series like the RGL s7 6s grand finals
  • There is now an API for querying for players and logs. It is pretty basic at the moment because it is just for internal use. If anyone is interested in using the data from trends.tf, please send me patches for improving the API :)

Once again, please send me any bugs or suggestions you have either on Discord, Steam, or as a reply in this thread.

BYS_ShadowFenixHey just wanted to comment and say thanks for putting in the effort to create and maintain this site. It's really cool

Thanks :)

It's been a while since the last update. I'm still working on the site, but recently a lot of that has gone into stuff behind-the-scenes.

[list]
[*] The whole site configuration got added to version control. I can now spin up a new server in around 30 minutes. I can also track configuration changes with the rest of the site. This has been particularly helpful for working with nginx.
[*] I switched to Digital Ocean from Azure, since my student discount ran out. I'm pretty happy with them. I may upgrade to a larger instance if I ever see an uptick in traffic. The database could really use more ram.
[*] I added tracing/error reporting using Sentry. This should help me address slow queries and determine if the changes I made are having the affect I wanted. It also has helped me quickly identify errors. I've pushed out several bug fixes recently using errors I found with Sentry.
[/list]

In addition to the above, there are also some feature updates

[list]
[*] There is now a recent logs page where you can see all logs uploaded to the site.
[*] You can now filter by player. The search is for aliases like the search box, but it only shows their current name. I may try and fix this at some point.
[*] Weapons now use their in-game names. I'm using a semi-automated process for this, so let me know if you see any weird weapon names.
[*] The combined log limit has been increased to 10. This is primarily so that you can do combined logs for long series [url=https://trends.tf/log/3090487+3090485+3090479+3090475+3090469+3090462+3090454]like the RGL s7 6s grand finals[/url]
[*] There is now an API for querying for players and logs. It is pretty basic at the moment because it is just for internal use. If anyone is interested in using the data from trends.tf, please send me patches for improving the API :)
[/list]

Once again, please send me any bugs or suggestions you have either on Discord, Steam, or as a reply in this thread.

[quote=BYS_ShadowFenix]Hey just wanted to comment and say thanks for putting in the effort to create and maintain this site. It's really cool[/quote]

Thanks :)
33
#33
trends.tf
14 Frags +

Long time no update. I's been over two years since I started this website, and almost a year since the last update, so I figure a post is long overdue. I've been doing mostly incremental changes over the past year. Similar to last time, a lot of stuff is on the backend

  • In general, performance should be better. We now keep track of when the backing data for a page changes and only regenerate it when we have to. The header for each player is cached in memcached. More data is denormalized so we don't have to do as many joins. And a lot of queries were rewritten to work aroung postgres's query planner. The overview page in particular has been getting a lot of hatelove.
  • A lot of bugs got quashed. The site should now crash less often and be more robust when things fail.
  • There are now some basic tests, mostly for the various fetchers. There are also some smoke tests which try to access every page on the site. Much better than pushing directly to prod.
  • We now have backups(!)

In addition, there have also been some feature updates

  • You can now sort almost every table on the site.
  • Logs are now linked to their demos on demos.tf. I'm hoping to use the same techniques to link to ETF2L and RGL match pages.
  • There is now a maps page where you can view your stats broken down by map.
  • The "averages" table on the overview has been moved to the totals page. It's been replaced with format stats.
  • Duplicates are now used to ignore later logs rather than earlier ones. This should increase the accuracy by exluding most combined logs.

Once again, please send me any bugs or suggestions you have either on Discord, Steam, or as a reply in this thread.

Long time no update. I's been over two years since I started this website, and almost a year since the last update, so I figure a post is long overdue. I've been doing mostly incremental changes over the past year. Similar to last time, a lot of stuff is on the backend

[list]
[*] In general, performance should be better. We now keep track of when the backing data for a page changes and only regenerate it when we have to. The header for each player is cached in memcached. More data is denormalized so we don't have to do as many joins. And a lot of queries were rewritten to work aroung postgres's query planner. The overview page in particular has been getting a lot of [s]hate[/s]love.
[*] A lot of bugs got quashed. The site should now crash less often and be more robust when things fail.
[*] There are now some basic tests, mostly for the various fetchers. There are also some smoke tests which try to access every page on the site. Much better than pushing directly to prod.
[*] We now have backups(!)
[/list]

In addition, there have also been some feature updates

[list]
[*] You can now sort almost every table on the site.
[*] Logs are now linked to their demos on [url=https://demos.tf/]demos.tf[/url]. I'm hoping to use the same techniques to link to ETF2L and RGL match pages.
[*] There is now a maps page where you can view your stats broken down by map.
[*] The "averages" table on the overview has been moved to the totals page. It's been replaced with format stats.
[*] Duplicates are now used to ignore later logs rather than earlier ones. This should increase the accuracy by exluding most combined logs.
[/list]

Once again, please send me any bugs or suggestions you have either on [url=https://discord.gg/szeZmthBa4]Discord[/url], Steam, or as a reply in this thread.
34
#34
trends.tf
10 Frags +

This one's been a long time coming, but logs are now linked to ETF2L matches. That means you can go to your team and see all your matches and their associated logs, automatically combine all logs in a match, filter your logs by whether they were part of a match, and see which players had the best DPM this season (or all time).

I'm pretty happy with how this turned out. 97% of matches are linked automatically, but I will be continuing to improve this over the coming months.

I did ETF2L first because they have a proper API. RGL has been working on theirs, but they are still lacking in some major areas. I hope to add RGL support soon, once they iron out their issues.

This one's been a long time coming, but logs are now linked to ETF2L matches. That means you can [url=https://trends.tf/league/etf2l/team/32593/matches]go to your team and see all your matches and their associated logs[/url], [url=https://trends.tf/log/3401981+3401911+3401950+3401956]automatically combine all logs in a match[/url], [url=https://trends.tf/player/76561198075312767/logs?league=etf2l]filter your logs by whether they were part of a match[/url], and [url=https://trends.tf/league/etf2l/comp/844/players?sort=dpm&sort_dir=desc]see which players had the best DPM this season[/url] (or [url=https://trends.tf/leaderboard?class=&league=etf2l&format=&map=&sort=dpm&sort_dir=desc]all time[/url]).

I'm pretty happy with how this turned out. 97% of matches are linked automatically, but I will be continuing to improve this over the coming months.

I did ETF2L first because they have a proper API. RGL has been working on theirs, but they are still lacking in some major areas. I hope to add RGL support soon, once they iron out their issues.
35
#35
whitelist.tf
0 Frags +
Forty-Botautomatically combine all logs in a match

Kinda neat feature, does this have some API output if the parsing is better than logs.tf? (properly combining team scores on stopwatch maps for instance isn't flawless: 6-0 score when team names are known in the 2-1 series)

Also noticed a thing with testing the combining, and noticed that because of the serveme + pickup site duplicate uploads your detection for that at least works, but then still combines them in the output, e.g. this.

(and in case when combined, chronologically order them?)

---

Logs might now also support pause duration in the output, is this parsed yet and taken into account for DPM calculations?

---

Also ozfortress should have an API, could link their matches and logs too!

[quote=Forty-Bot][url=https://trends.tf/log/3401981+3401911+3401950+3401956]automatically combine all logs in a match[/url][/quote]
Kinda neat feature, does this have some API output if the parsing is better than logs.tf? (properly combining team scores on stopwatch maps for instance isn't flawless: [url=https://trends.tf/log/3375490+3375517+3375534]6-0 score when team names are known in the 2-1 series[/url])

Also noticed a thing with testing the combining, and noticed that because of the serveme + pickup site duplicate uploads your detection for that at least works, but then still combines them in the output, e.g. [url=https://trends.tf/log/3403758+3403757]this[/url].

(and in case when combined, chronologically order them?)

---

Logs might now also support pause duration in the output, is this parsed yet and taken into account for DPM calculations?

---

Also ozfortress should have an API, could link their matches and logs too!
36
#36
trends.tf
2 Frags +
WiethoofdForty-Botautomatically combine all logs in a matchKinda neat feature, does this have some API output if the parsing is better than logs.tf? (properly combining team scores on stopwatch maps for instance isn't flawless: 6-0 score when team names are known in the 2-1 series)

The scores are combined properly. The scores in ETF2L don't necessarily match the round scores. As I understand it, there are always 6 points to be won. So if you win a map you get 6 points.

Also noticed a thing with testing the combining, and noticed that because of the serveme + pickup site duplicate uploads your detection for that at least works, but then still combines them in the output, e.g. this.

If you explicitly request to combine two logs, they will be combined. No matter if one is marked as a duplicate or not.

(and in case when combined, chronologically order them?)

Combined logs are ordered in the order you check the boxes (I think).

Logs might now also support pause duration in the output, is this parsed yet and taken into account for DPM calculations?

Pauses should be accounted for when calculating durations. That said, I don't re-parse logs (yet) so the stats are all the same as on logs.tf.

Also ozfortress should have an API, could link their matches and logs too!

Yeah, I'm interested in doing ozfortress as well. I couldn't find their API, if it exists.

[quote=Wiethoofd][quote=Forty-Bot][url=https://trends.tf/log/3401981+3401911+3401950+3401956]automatically combine all logs in a match[/url][/quote]
Kinda neat feature, does this have some API output if the parsing is better than logs.tf? (properly combining team scores on stopwatch maps for instance isn't flawless: [url=https://trends.tf/log/3375490+3375517+3375534]6-0 score when team names are known in the 2-1 series[/url])
[/quote]

The scores are combined properly. The scores in ETF2L don't necessarily match the round scores. As I understand it, there are always 6 points to be won. So if you win a map you get 6 points.

[quote]
Also noticed a thing with testing the combining, and noticed that because of the serveme + pickup site duplicate uploads your detection for that at least works, but then still combines them in the output, e.g. [url=https://trends.tf/log/3403758+3403757]this[/url].
[/quote]

If you explicitly request to combine two logs, they will be combined. No matter if one is marked as a duplicate or not.

[quote]
(and in case when combined, chronologically order them?)
[/quote]

Combined logs are ordered in the order you check the boxes (I think).

[quote]
Logs might now also support pause duration in the output, is this parsed yet and taken into account for DPM calculations?
[/quote]

Pauses should be accounted for when calculating durations. That said, I don't re-parse logs (yet) so the stats are all the same as on logs.tf.

[quote]
Also ozfortress should have an API, could link their matches and logs too![/quote]

Yeah, I'm interested in doing ozfortress as well. I couldn't find their API, if it exists.
37
#37
2 Frags +
Forty-BotAlso ozfortress should have an API, could link their matches and logs too!
Yeah, I'm interested in doing ozfortress as well. I couldn't find their API, if it exists.

There is most certainly one, hit up core in the OZF Discord and he should be able to hook you up.

[quote=Forty-Bot]
[quote]
Also ozfortress should have an API, could link their matches and logs too![/quote]

Yeah, I'm interested in doing ozfortress as well. I couldn't find their API, if it exists.[/quote]

There is most certainly one, hit up core in the [url=https://discord.gg/dyudcxB]OZF Discord[/url] and he should be able to hook you up.
38
#38
trends.tf
4 Frags +

Small update: RGL matches are now also tracked. They haven't really fixed their API (and there are a lot of data issues in older seasons), but it's good enough now to use. I'm still working on improving the UI, especially regarding roster dates.

Regarding OZF, I have no plans to implement support atm. When linking logs to matches, I look for logs within 12 hours of the scheduled match time with enough players from both teams. However, OZF doesn't store the scheduled match time anywhere. All they have is the last time the match page was updated, which includes comments from way after the match.

No plans for UGC, CEVO, etc. since I don't have anything from before 2019 imported at the moment. Maybe in the future.

Small update: RGL matches are now also tracked. They haven't really fixed their API (and there are a lot of data issues in older seasons), but it's good enough now to use. I'm still working on improving the UI, especially regarding roster dates.

Regarding OZF, I have no plans to implement support atm. When linking logs to matches, I look for logs within 12 hours of the scheduled match time with enough players from both teams. However, OZF doesn't store the scheduled match time anywhere. All they have is the last time the match page was updated, which includes comments from way after the match.

No plans for UGC, CEVO, etc. since I don't have anything from before 2019 imported at the moment. Maybe in the future.
1 2
Please sign in through STEAM to post a comment.