Site Loading Incredibly Slow

[Chaqa]

For the past few days the site has been loading painfully slowly. I know at least a few others have noted it.

[Wattsthematter]

This thread hasn't loaded yet, but I stg if it mentions how slow Webdip has been loading in the last few days...

[dargorygel]

Working on it

[gramblor]

I didn't want to be "that guy" to post it in the forums, but totally slow. The AI based games are also taking 4-6 hrs to process even a retreat. I think there must be some server maintenance or some such going on...

[Wattsthematter]

Seems better at this moment

[kestasjk]

Hi all, Yep I had heard that bot games were running slow, haven't had time to look into it but am going to do so now. First thing I notice is that the server load is very high but not sure exactly why at the moment, the database process is doing a lot of work just not sure what it's working on atm. Will keep you posted and do some maintenance while Im here

[kestasjk]

Whenever I looked at the running queries it was running several queries counting the number of liked posts users have.. Looks like every time you open a forum thread, for each post in the thread, the system was looking for ever other post made by that user and counting the number of times that post was liked. A little wasteful..

Looking into it though we've also got tens of thousands of guest sessions open in the forum.. They're doing things like looking at the members list starting with different letters, opening multiple report post screens, searching for posts, downloading files..

Looking into it further .. I think we're being DDoSed:

The majority of our user base is in the US, and in the last 7 days we've had ~60k forum search requests from the US.. but 1,810k forum search requests from Hong Kong.. A search request is a typical sort of request intended to use resources on the server since anyone can do a search but it requires the server to do a lot of work, so this isn't about spamming it's purely about using server resources.
You can also see that thoughout a whole day on the 6th the request rate was static, which you'll never see from real users who sleep, and how the request rate suddenly dropped for a while then resumed, indicating the requests are coming from a single source.

To be honest I'm surprised the server was usable at all and didn't totally grind to a halt.. Millions of requests to search, view user profiles, submit reports, view posts, and for each request for each post there was a lookup for the posting user of all their other posts to count the number of likes..

I wonder if anyone has a grudge against the site or has some reason to want to cause trouble.. hmmm..


Anyway I'm going to cache user like count against the user record, so that the count will be delayed but will only need to be counted each day, and more importantly block everything in the forum except the index, view forum and view post pages for users without an account.

It is a concern though as .. I hesitate to say this but .. if whoever is doing this knew what they were doing they would target play.webdiplomacy.net and create anonymous bot games instead.
I'll also add something to limit anonymous bot games by IP, and aggressively delete anonymous bot games that aren't being actively played.


What a pain having to use very limited time to work on webDip on nonsense like this..

[kestasjk]

Looks like they were also targeting play.webdiplomacy.net as there were thousands of unattended games.

Anyway I've made changes, let's see how it goes.
- Unattended bot games are culled more frequently,
- per-user anonymous game limits reduced,
- only basic functions are allowed for forum users that aren't logged on, and
- anonymous users don't see like info.

If you notice any issues let us know.

[wintergreen]

I think you disabled all like info.

[kestasjk]

Oops, forgot to turn the setting back on, it's showing now.

[Pennsta7]

Three out of 3 AI bot games turned out "Finished" almost moments after starting.

[kestasjk]

Three out of 3 AI bot games turned out "Finished" almost moments after starting.

Hmm, I can't reproduce this but I can see how it may be possible if you logged on a long time ago and the last session end time hasn't updated. I've made the unattended game check check whether the user is currently logged in, let me know if you still have issues

[learnedSloth]

I wonder if anyone has a grudge against the site or has some reason to want to cause trouble.. hmmm..

I think Hong Kong is controlled by China now, and a DDoS attack could be a crude attempt to censor the site by blocking requests. Of course I'm just guessing, and there's a very narrow possibility that the forum has become viral in Hong Kong...

[Jamiet99uk]

I wonder if anyone has a grudge against the site or has some reason to want to cause trouble.. hmmm..

I could mention someone, but you can't handle the truth...

[Chaqa]

Three out of 3 AI bot games turned out "Finished" almost moments after starting.

Hmm, I can't reproduce this but I can see how it may be possible if you logged on a long time ago and the last session end time hasn't updated. I've made the unattended game check check whether the user is currently logged in, let me know if you still have issues

What about people who have logged in in the past ~48 hours?

[Pennsta7]

@kestajk:

In reply to your question I get only:

No-press bot game limit 600/600 reached: Apologies, the no-press bot game limit has been reached. To conserve server resources we have to limit the number of anonymous games. Please try again later.

[kestasjk]

Hmm.. I'm not getting it, I'm in two bot games at the moment one which I just started.

Btw I'm playing in a Cicero game atm.. damn those bots are so neat, they always blow me away how smart they are

[kestasjk]

I wonder if anyone has a grudge against the site or has some reason to want to cause trouble.. hmmm..

I could mention someone, but you can't handle the truth...

I'm sure it wouldn't be a big surprise.. I don't know how I can accumulate enemies running a free to play DIplomacy website but somehow I pull it off

I wonder if anyone has a grudge against the site or has some reason to want to cause trouble.. hmmm..

I think Hong Kong is controlled by China now, and a DDoS attack could be a crude attempt to censor the site by blocking requests. Of course I'm just guessing, and there's a very narrow possibility that the forum has become viral in Hong Kong...

Yeah I expect the government has bigger things to worry about, though my partner is from HK and apparently they can be very petty.. I'm really not sure what's behind this, but I enjoy writing code for a Diplomacy website in my free time and even I think this person needs to get a life..

[kestasjk]

Three out of 3 AI bot games turned out "Finished" almost moments after starting.

Hmm, I can't reproduce this but I can see how it may be possible if you logged on a long time ago and the last session end time hasn't updated. I've made the unattended game check check whether the user is currently logged in, let me know if you still have issues

What about people who have logged in in the past ~48 hours?

Hmm, not sure what you mean it previously cancelled games where an account's last session ended over 48 hours ago, but after hearing that games were closing instantly I changed it to the last session ended over 48 hours ago and they're not logged on at the moment. I think that'll do the trick but let's see

[Pengwinja]

My bot game stopped and didn’t refund me my so now I have 90.

I’m so bad I can’t even have the minimum amount of points. Do I get a record for lowest points?