Reliability ratings and locking changes

Members can make suggestions for improving the site and improving the forum as well as submit bug reports to be reviewed by our support team here.
Post Reply
Message
Author
User avatar
kestasjk
Developer
Developer
Posts: 534
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Reliability ratings and locking changes

#1 Post by kestasjk » Tue Oct 11, 2022 4:46 pm

I've made some changes to reliability ratings; the previous optimizations meant you couldn't really see what was going on with the calcs if they went wrong, but now what's shown on your profile is tied to the actual numbers used in the new calcs.
It seemed like the way it was calculated was built on over time so I tried to simplify the explanation down so it's easier to see where your RR comes from:
Screenshot 2022-10-12 004250.png
Let me know if it's not straightforward or if you notice your numbers seem off. It definitely works when doing a global refresh, but whether it'll work as it goes we'll have to wait and see over time.

I also noticed that deadlocks have been coming back, and it was 90+% due to order submission via the new API that the bots and new UI used. It was locking the entire game instead of just the user record so I've reduced that down. Hopefully there won't be anything to notice here, but I'm interested if anyone got any errors during order submission recently.

User avatar
JECE
Posts: 1704
Joined: Sun Dec 31, 2017 4:35 pm
Contact:

Re: Reliability ratings and locking changes

#2 Post by JECE » Tue Oct 11, 2022 10:31 pm

I just checked and my reliability rating is 0%. I haven't played a game on webDip in a few years, but that seems a little harsh! Could this be an error with the recent changes?
See my full Profile:
http://www.webdiplomacy.net/profile.php?userID=17421

User avatar
kestasjk
Developer
Developer
Posts: 534
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Reliability ratings and locking changes

#3 Post by kestasjk » Tue Oct 11, 2022 10:52 pm

Oops a divide by 0, always forget something.. Thanks that's fixed now:
Screenshot 2022-10-12 065144.png

PinguPingu
Posts: 5
Joined: Wed Sep 28, 2022 9:53 pm
Contact:

Re: Reliability ratings and locking changes

#4 Post by PinguPingu » Thu Dec 22, 2022 12:56 pm

How long dos it take to shed the non-live penalties?

Crazy Anglican
Posts: 383
Joined: Fri Oct 06, 2017 12:04 am
Contact:

Re: Reliability ratings and locking changes

#5 Post by Crazy Anglican » Thu Dec 22, 2022 6:02 pm

In looking at the RR chart, excused misses count against you?

A_Tin_Can
Lifetime Site Contributor
Posts: 283
Joined: Fri Sep 29, 2017 9:18 pm
Contact:

Re: Reliability ratings and locking changes

#6 Post by A_Tin_Can » Sat Dec 24, 2022 8:48 am

While we're on the subject of ratings, my ghost rating seems to have gone down in the last two years, despite me not having played a single game. Is there perhaps something wrong with the calculation?

Navel_Reservist
Posts: 5
Joined: Wed Dec 14, 2022 2:57 pm
Contact:

Re: Reliability ratings and locking changes

#7 Post by Navel_Reservist » Fri Dec 30, 2022 5:41 am

How long does it take for base rate improvements to be reflected in the reliability rating? My base rate has been increasing, and the calculation implies that my reliability rating should be increasing as a result, but it's still stuck at its low point. Is this intentional? Right now the math on my reliability rating page shows this:

"Total:
Reliability Rating: 100% - Base/Total reduction (16.7%) - Non-live penalty (11%) - Live penalty (0%) = Reliability rating (64%) "

Doug7878
Gold Donator
Gold Donator
Posts: 577
Joined: Wed Feb 07, 2018 11:05 am
Location: Massachusetts, USA
Contact:

Re: Reliability ratings and locking changes

#8 Post by Doug7878 » Sat Dec 31, 2022 12:46 am

@ATC -
I had noticed that the GB and Peak GR ratings changed also for some players when the GR calculations were incorporated into the site at the beginning of 2021. I understand that there were some calculation errors in the old ratings that were corrected at that time. I watch the GB GR changes pretty closely and have seen no unusual changes in gunboat ratings since FWIW.

A_Tin_Can
Lifetime Site Contributor
Posts: 283
Joined: Fri Sep 29, 2017 9:18 pm
Contact:

Re: Reliability ratings and locking changes

#9 Post by A_Tin_Can » Sun Jan 01, 2023 1:45 pm

Thanks Doug, but I'm not sure how your reply is related. I'm talking about the stats reported on the site only.

User avatar
kestasjk
Developer
Developer
Posts: 534
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Reliability ratings and locking changes

#10 Post by kestasjk » Tue Jan 03, 2023 11:26 am

A_Tin_Can wrote:
Sat Dec 24, 2022 8:48 am
While we're on the subject of ratings, my ghost rating seems to have gone down in the last two years, despite me not having played a single game. Is there perhaps something wrong with the calculation?
Hmm, there may well be, I've never looked into how GRs are calculated. Definitely it could be something similar to RRs where they were implemented in a way that was simple and easy to understand but required reading every players entire game history turn by turn for the last year every time a game they were involved in got processed, which contributed to those deadlocking issues that were a big problem for site performance/reliability this time last year.

The new RR calcs are a lot more complex because they put turns into different buckets, and there may still be a bug or two lurking there, but it's O(1) instead of O(N).
Navel_Reservist wrote:
Fri Dec 30, 2022 5:41 am
How long does it take for base rate improvements to be reflected in the reliability rating? My base rate has been increasing, and the calculation implies that my reliability rating should be increasing as a result, but it's still stuck at its low point. Is this intentional? Right now the math on my reliability rating page shows this:

"Total:
Reliability Rating: 100% - Base/Total reduction (16.7%) - Non-live penalty (11%) - Live penalty (0%) = Reliability rating (64%) "
I'll take a look asap

User avatar
kestasjk
Developer
Developer
Posts: 534
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Reliability ratings and locking changes

#11 Post by kestasjk » Tue Jan 03, 2023 11:57 am

kestasjk wrote:
Tue Jan 03, 2023 11:26 am
Navel_Reservist wrote:
Fri Dec 30, 2022 5:41 am
How long does it take for base rate improvements to be reflected in the reliability rating? My base rate has been increasing, and the calculation implies that my reliability rating should be increasing as a result, but it's still stuck at its low point. Is this intentional? Right now the math on my reliability rating page shows this:

"Total:
Reliability Rating: 100% - Base/Total reduction (16.7%) - Non-live penalty (11%) - Live penalty (0%) = Reliability rating (64%) "
I'll take a look asap
So I did a global RR recalc and you went from 72% to 74%:
RR1.png
to
RR2.png
Looks like when you last checked the RR should've been 72% (2 missed turns/12 yearly turns = 16.7%), and when I checked before the global recalc the RR should've been 74% (2 missed turns / 14 yearly turns = 14.3%) which it now is.

Looks like when the yearly turn counter gets incremented it wouldn't trigger a refresh, so it would only update on missed turns. I've fixed that now, thanks for reporting:
https://github.com/kestasjk/webDiplomac ... 6ee7c7f79e

learnedSloth
Posts: 206
Joined: Mon Sep 12, 2022 10:20 pm
Contact:

Re: Reliability ratings and locking changes

#12 Post by learnedSloth » Thu Jan 05, 2023 12:50 pm

kestasjk wrote:
Tue Jan 03, 2023 11:26 am
The new RR calcs are a lot more complex because they put turns into different buckets, and there may still be a bug or two lurking there, but it's O(1) instead of O(N).
Couldn't you just

Code: Select all

newRR = 99% * oldRR + 1% * current_turn
?
¶ Keep thy heart with all diligence; for out of it are the issues of life.
-- Proverbs of Solomon, chapter 4, verse 23

User avatar
kestasjk
Developer
Developer
Posts: 534
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Reliability ratings and locking changes

#13 Post by kestasjk » Fri Jan 06, 2023 1:22 pm

You could do a lot of things.. I wanted to preserve the system jmo put in place as it seemed to be working well. The current system adds some forgiveness for consecutive turns, lets mods forgive turns, differentiates between live games and non-live, etc.
I’m not saying you couldn’t make something like that have similar features.. but you definitely come to learn that any system that tries to sum up any metric like reliability, Diplomacy skill, account-suspiciousness, user-relationship likelihood, etc, it’s always going to be a balance between loads of different trade offs and compromises, and there’ll always be suggestions on how it could be improved in some way or other.

PinguPingu
Posts: 5
Joined: Wed Sep 28, 2022 9:53 pm
Contact:

Re: Reliability ratings and locking changes

#14 Post by PinguPingu » Fri Jan 06, 2023 2:13 pm

Sorry - how long does it take for non-live penalties to stop contributing to your RR? (I really like the RR system for what it's worth)

Post Reply

Who is online

Users browsing this forum: No registered users