Bots are broken tonight

Members can make suggestions for improving the site and improving the forum as well as submit bug reports to be reviewed by our support team here.
Post Reply
Message
Author
ShipOfTheseus
Posts: 10
Joined: Thu Dec 22, 2022 9:03 am
Karma: 4
Contact:

Bots are broken tonight

#1 Post by ShipOfTheseus » Sat Feb 04, 2023 7:35 am

Bots don't seem to be entering moves tonight. I played an all-bots AI game, and nothing happened (with 3-day turns.)

I then started a game with five-minute turns, invited humans, and used "fill with bots". The bots used up all their missed-turns and then crashed into civil disorder.

(Kinda fun playing against one other guy with a bunch of sleeping units, but not what I wanted to play, of course!)

User avatar
kestasjk
Developer
Developer
Posts: 522
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Karma: 788
Contact:

Re: Bots are broken tonight

#2 Post by kestasjk » Sat Feb 04, 2023 8:45 am

Thanks; we had to update the server and the docker bot container can't connect out, trying to resolve it now

User avatar
kestasjk
Developer
Developer
Posts: 522
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Karma: 788
Contact:

Re: Bots are broken tonight

#3 Post by kestasjk » Sat Feb 04, 2023 9:02 am

Yeah some weird firewall issue, but once it starts it's okay so the bots are now running again. Looks like they're catching up now

ShipOfTheseus
Posts: 10
Joined: Thu Dec 22, 2022 9:03 am
Karma: 4
Contact:

Re: Bots are broken tonight

#4 Post by ShipOfTheseus » Sat Feb 04, 2023 6:11 pm

Yup, so far so good! Also, thank you for your work on making bots that play pretty well. Super fun and helpful to be able to play many, many rounds of Dip. (Looking forward to playing against Cicero when that finally happens. Don't discount a paid edition of WebDip -- it'll be worth it!)

User avatar
kestasjk
Developer
Developer
Posts: 522
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Karma: 788
Contact:

Re: Bots are broken tonight

#5 Post by kestasjk » Mon Feb 06, 2023 8:00 am

ShipOfTheseus wrote:
Sat Feb 04, 2023 6:11 pm
Yup, so far so good! Also, thank you for your work on making bots that play pretty well. Super fun and helpful to be able to play many, many rounds of Dip. (Looking forward to playing against Cicero when that finally happens. Don't discount a paid edition of WebDip -- it'll be worth it!)
Yeah I really like having bots on the site to play against, and as a software dev it's good to learn more about Docker and AI in a hands-on way.

Based on the recent diplomacygames interview with Noam I'm less hopeful Cicero will be able to be run at scale on the site. He said that even running the model takes 56 GPUs, and I was previously hoping based on the code that that was just the requirement for training. (Usually training takes crazy processing power, but running the model is okay)
I think the best reasonable outcome is that we have a Cicero instance that only plays a small number of games then caps out, and only a limited number of messages per game. That way it won't be able to play live, it'll be 56x slower with 1/56th the processing power, but at least people will be able to play against it.
Edit: Hmm, I get this when running a single Cicero bot:

Code: Select all

RuntimeError: CUDA out of memory. Tried to allocate 32.00 MiB (GPU 0; 7.79 GiB total capacity; 5.54 GiB already allocated; 30.25 MiB free; 6.06 GiB reserved in total by PyTorch)
So looks like a consumer grade graphics card won't do for Cicero, even if running slowly :( Hopefully the no-press bots are much less memory intensive

Edit: Yep looks like searchbot runs quickly enough that at 1-2 bots should be able to run on a GPU, not sure if I can multiplex it because it uses ~3GB GPU memory and my card only has 8GB.
I tried diplodocus and it does work, but takes minutes to generate orders.

Making progress though.. I've got a new bot account FairBot2 that is running from my machine doing FvA games. It's using searchbot, which I'm not sure if that's the right one.
Also I noticed there's no GvI support in the current repo; did bots ever play GvI?

Note this won't be super-stable since it's running from my home machine in Australia, this is just for testing an evaluation. But please give it a go and let me know your thoughts. I'm also interested to replace one or two of the MILA bots to try and tweak the difficulties etc (both higher and lower).
1

User avatar
kestasjk
Developer
Developer
Posts: 522
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Karma: 788
Contact:

Re: Bots are broken tonight

#6 Post by kestasjk » Mon Feb 06, 2023 2:22 pm

Tried my luck running a searchbot in a 7 player game while running the FvA bot but no it needs more memory than I've got available, need to look at the options

Trigfea63
Gold Donator
Gold Donator
Posts: 330
Joined: Fri Apr 02, 2021 11:17 pm
Karma: 77
Contact:

Re: Bots are broken tonight

#7 Post by Trigfea63 » Tue Feb 07, 2023 8:49 am

Kestas, this is excellent, thank you for spending the time to get this up and running again!

I've started a game against Fairbot 2, and it seems like the old "FB" so far. It's only 1902, so a bit hard to tell. One difference I've noticed already, it seems to want to play a strange opening as Austria, to ALB, RUM, and GAL. Not just in my game. I've seen MerryBBuck use this opening with some success against the old FB, but it only works if you catch your opponent off guard.

I'll update you on this thread once I've had a chance to play a few games against Fairbot 2. It might be a few days. I will say, it's much faster in processing moves than the old FB.

The advanced Meta AI never played GvI. GvI has always been the old Univ. of Montreal bots. I know, because when I got tired of repeated whippings at the hands of FB, I would play a game of GvI where I could usually win. :-D

User avatar
kestasjk
Developer
Developer
Posts: 522
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Karma: 788
Contact:

Re: Bots are broken tonight

#8 Post by kestasjk » Tue Feb 07, 2023 10:56 am

Ah great news, glad to hear it’s at least an improvement. Interested to hear if you think it’s definitely behaving differently or not over time, if so I can try to look into what I might be doing different.
Interesting/concerning to hear it’s faster too.. I was not expecting that, I can’t think why as they had a supercomputer and I’ve got a 2080..

There’s a lot to look into now that I can get them running locally; getting tuneable difficulty, different personalities, what the performance bottlenecks are, what would the ideal Diplomacy AI machine be and what can you run at what cost.

I’m looking into getting a dedicated data processing Tesla GPU that’s designed for AI workloads/datacenters. I’d like to have MILA with easy/medium/hard for live 1v6 games, searchbot for FvA, searchbot/diplodocus for non-live 1v6 games, diplodocus for civil disorders, and one Cicero that perhaps joins one long-phase anonymous game at any time that people can know it was Cicero after the game ends (but I don’t want to neglect human games)

User avatar
JECE
Posts: 1508
Joined: Sun Dec 31, 2017 4:35 pm
Karma: 399
Contact:

Re: Bots are broken tonight

#9 Post by JECE » Tue Feb 07, 2023 5:45 pm

Trigfea63 wrote:
Tue Feb 07, 2023 8:49 am
One difference I've noticed already, it seems to want to play a strange opening as Austria, to ALB, RUM, and GAL. Not just in my game. I've seen MerryBBuck use this opening with some success against the old FB, but it only works if you catch your opponent off guard.
At least that opening has a name! Maybe Fairbot 2 knows something about the "Warthog" that we don't, ha ha.
The Library of Diplomacy Openings wrote: RUMANIAN GAMBIT, BALKAN ROADHOG VARIANT
The variant of the Rumanian Gambit which sends Vienna to Galicia.
Tony Swinnerton's Diplomatic Pouch article, "Playing the Key Lepanto," recommends it in conjunction with an unusual Italian/Austrian alliance opening.

RUMANIAN GAMBIT, WARTHOG VARIANT
The Warthog is Richard Sharp's name for the rarely-used Austrian opening (0.87%): F Tri-Alb, A Vie-Gal and, A Bud-Rum. Richard writes that it is "a new name for something resembling a Hedgehog, but not as pretty."

[RUSSIAN FROLIC]
Allen Wells' name for the Austro-Turkish opening based on A Vie-Gal, A Bud-Rum, A Con H, A Smy-Arm, F Ank-Bla. The key here is A Con H which, by eliminating Turkey's 1901 risk to Serbia, permits Austria to move safely to Rum. Discussed in issue 29 of Diplomacy World.
http://uk.diplom.org/pouch/Online/Openings/
See my full Profile:
http://www.webdiplomacy.net/profile.php?userID=17421

captainmeme
Posts: 617
Joined: Fri Sep 29, 2017 3:06 pm
Location: Manchester, UK
Karma: 768
Contact:

Re: Bots are broken tonight

#10 Post by captainmeme » Tue Feb 07, 2023 6:24 pm

This is called the Russian Opening (Albanian Variation) in FvA. Both my 2017 openings tier list and RoganJosh's (better) 2020 tier list put it at the very lowest tier that's still considered viable in high level play.

Very surprised to hear that the bot uses it, or even that Merry managed to use it with success against the old bot - it seemed just straight up suboptimal to me.

Trigfea63
Gold Donator
Gold Donator
Posts: 330
Joined: Fri Apr 02, 2021 11:17 pm
Karma: 77
Contact:

Re: Bots are broken tonight

#11 Post by Trigfea63 » Wed Feb 08, 2023 6:50 am

captainmeme wrote:
Tue Feb 07, 2023 6:24 pm
This is called the Russian Opening (Albanian Variation) in FvA. Both my 2017 openings tier list and RoganJosh's (better) 2020 tier list put it at the very lowest tier that's still considered viable in high level play.
Very interesting. In your 2017 article, you say, "This opening is strictly worse than the standard Russian opening because it immediately loses the game against any Piedmont opening." And that is what happened here. I played the Alpine. Now the game is at 1902 builds, and I feel like I've already won (although, if this is the real Fairbot, it's dangerous to speak too soon):
https://webdiplomacy.net/board.php?gameID=533605
captainmeme wrote:
Tue Feb 07, 2023 6:24 pm
Very surprised to hear that the bot uses it, or even that Merry managed to use it with success against the old bot - it seemed just straight up suboptimal to me.
Here is an example, with Merry as Austria securing the victory in 1903:
https://webdiplomacy.net/board.php?game ... #gamePanel

User avatar
kestasjk
Developer
Developer
Posts: 522
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Karma: 788
Contact:

Re: Bots are broken tonight

#12 Post by kestasjk » Fri Feb 10, 2023 5:03 am

FYI the FvA bot will be down for a few hours as I need to do some work I can only do on my Windows OS and currently I can only run the bots on Ubuntu. (Going to try to run in WSL on Windows now that I know how on Ubuntu)

I've got a graphics card upgrade that was able to run searchbot, but even running one searchbot is a lot slower than the MILA bots. Really starting to appreciate how fast the MILA bots are, and understanding why FAIR turned the bots off so quickly after the research was complete. Going to have to explore some options and try different things out to see what we can get running sustainably

Edit: I've got it running under my main Windows OS now, also I've found the old DORA bot and the original FvA bot as it was when FAIR were running it, I can see that they made it search a lot deeper since it's just 1v1 which might explain some of the different behavior

Definitely the FAIR bots are going to be too slow for live play, but if we rotate the same bot instance across multiple bot accounts the FAIR bots will be able to play longer term games. Exciting though that with the new card I'm just barely able to fit an unmodified Cicero 1v6 into memory, and it's generating text:

Code: Select all

I0210 20:09:01 [parlai_message_handler:621] Pseudo orders for TURKEY: PseudoOrders({'S1901M': {'TURKEY': ('F ANK - BLA', 'A SMY - CON', 'A CON - BUL'), 'RUSSIA': ('F STP/SC - BOT', 'A MOS - UKR', 'A WAR - GAL', 'F SEV - BLA')}, 'F1901M': {'TURKEY': ('F ANK - BLA', 'A BUL - GRE', 'A CON - BUL'), 'RUSSIA': ('F BOT - SWE', 'F SEV S A UKR - RUM', 'A UKR - RUM', 'A GAL - VIE')}})
I0210 20:09:01 [parlai_message_handler:1055] Applying initial message prompt for [82301] TURKEY -> RUSSIA: [0] TURKEY -> ITALY: Hey Italy!
I think the IT can be a really effective alliance when it’s done well, specifically when Turkey focuses on armies and after dealing with Austria, Italy builds fleets and heads west. Would you be down for something like that?
I0210 20:09:01 [parlai_message_handler:1055] Applying initial message prompt for [82301] TURKEY -> RUSSIA: [1500] TURKEY -> ENGLAND: Hey England, just reaching out to establish comms! I’d love to share intel as the game goes on if you’d be down to do the same, I reckon we could help one another out quite a bit!
I0210 20:09:02 [parlai_message_handler:556] Running message filtering on message "Hey Russia, hoping we can come to an arrangement about the Black Sea. Would you be open to an arranged bounce?".
I0210 20:09:02 [classifiers:667] Skipping nonsense classifier hvm_with_no_message_history for message proposal Hey Russia, hoping we can come to an arrangement about the Black Sea. Would you be open to an arranged bounce?, due to it not having order history to rely on.
I0210 20:09:02 [classifiers:652] Skipping nonsense classifier hvm_singleseed_2Mphasesorderhistory_150000steps for message proposal Hey Russia, hoping we can come to an arrangement about the Black Sea. Would you be open to an arranged bounce?, due to it not having message history to rely on.
I0210 20:09:02 [classifiers:652] Skipping nonsense classifier hvm_singleseed_seed2_2Mphasesorderhistory_2Mphasesstatehistory_150000steps for message proposal Hey Russia, hoping we can come to an arrangement about the Black Sea. Would you be open to an arranged bounce?, due to it not having message history to rely on.
I0210 20:09:02 [classifiers:652] Skipping nonsense classifier hvm_no_speaker_dialogue for message proposal Hey Russia, hoping we can come to an arrangement about the Black Sea. Would you be open to an arranged bounce?, due to it not having message history to rely on.
I0210 20:09:02 [classifiers:652] Skipping nonsense classifier hvm_4MPhasesOrderHistory_1epoch_seed3 for message proposal Hey Russia, hoping we can come to an arrangement about the Black Sea. Would you be open to an arranged bounce?, due to it not having message history to rely on.
I0210 20:09:02 [classifiers:652] Skipping nonsense classifier hvm_no_speaker_dialogue_bilateral_fixed for message proposal Hey Russia, hoping we can come to an arrangement about the Black Sea. Would you be open to an arranged bounce?, due to it not having message history to rely on.
I0210 20:09:02 [classifiers:652] Skipping nonsense classifier hvwm for message proposal Hey Russia, hoping we can come to an arrangement about the Black Sea. Would you be open to an arranged bounce?, due to it not having message history to rely on.
I0210 20:09:02 [classifiers:652] Skipping nonsense classifier hvm_NEW_justifications_2seeds for message proposal Hey Russia, hoping we can come to an arrangement about the Black Sea. Would you be open to an arranged bounce?, due to it not having message history to rely on.
I0210 20:09:02 [classifiers:652] Skipping nonsense classifier hvm_3seeds for message proposal Hey Russia, hoping we can come to an arrangement about the Black Sea. Would you be open to an arranged bounce?, due to it not having message history to rely on.
Which means we'll probably be able to run full press AI here before long :o
1

Trigfea63
Gold Donator
Gold Donator
Posts: 330
Joined: Fri Apr 02, 2021 11:17 pm
Karma: 77
Contact:

Re: Bots are broken tonight

#13 Post by Trigfea63 » Sat Feb 11, 2023 8:38 pm

Kestas, I don't think Fairbot2 is the same AI as the old (2021-22) Fairbot (now "FB"). There's the opening. Fairbot2 has played that Russian/Albanian opening in all 21 of its games as Austria. Even beyond that, it's not super-difficult to predict Fairbot2's moves and outguess it. FB was brutal on the guesses. And there's the win percentage. FB was kind of unbeatable. It played 5,592 games and won 80% of them. So far, Fairbot2 is winning 37%. It's only finished 30 games. Still, I think it's statistically impossible that an 80%-win-percentage AI could fare so poorly against mere humans. :-D

User avatar
kestasjk
Developer
Developer
Posts: 522
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Karma: 788
Contact:

Re: Bots are broken tonight

#14 Post by kestasjk » Sun Feb 12, 2023 7:47 am

Trigfea63 wrote:
Sat Feb 11, 2023 8:38 pm
Kestas, I don't think Fairbot2 is the same AI as the old (2021-22) Fairbot (now "FB"). There's the opening. Fairbot2 has played that Russian/Albanian opening in all 21 of its games as Austria. Even beyond that, it's not super-difficult to predict Fairbot2's moves and outguess it. FB was brutal on the guesses. And there's the win percentage. FB was kind of unbeatable. It played 5,592 games and won 80% of them. So far, Fairbot2 is winning 37%. It's only finished 30 games. Still, I think it's statistically impossible that an 80%-win-percentage AI could fare so poorly against mere humans. :-D
Got it, really like the direct feedback, we’ll need lots of that to refine things. I’ll try switching it to running the original FvA bot that does a lot deeper search and we’ll see what happens

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests