Austria vs Bots

Use this forum to discuss Diplomacy strategy.
Forum rules
This forum is limited to topics relating to the game Diplomacy only. Other posts or topics will be relocated to the correct forum category or deleted. Please be respectful and follow our normal site rules at http://www.webdiplomacy.net/rules.php.
Message
Author
User avatar
kestasjk
Developer
Developer
Posts: 532
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Austria vs Bots

#501 Post by kestasjk » Mon Feb 06, 2023 7:35 am

Bladerunners wrote:
Sun Feb 05, 2023 9:26 am
#1 development flaw with the bots is designers programming Italy to attack Austria in extremely unrealistic percentage of games. Zero way the bots have ‘learned this’ as no real Italy diplomacy player plays like that….
Yep when the FAIR research ended unfortunately so did the bots. We were a bit spoiled by Philip Paquette who ran his MILA bots himself for us on his system for years. Also when his hardware couldn't process orders fast enough anymore he had a docker image that made it pretty straightforward to get the bots running (even just running on CPU, no GPU needed).

With the FAIR bots it's quite different; their bots are available but the models aren't public. I do have access to them but it's set up to run CICERO on a supercomputer, and needs a powerful graphics card.
I've spent the weekend and took today off work to try and get the FAIR bots running on my desktop machine at home where I have a 2080 GPU; this way I'll have a better understanding of the requirements for the various models and how it'll integrate in, but getting it set up is much more complicated.

I have got it to install and compile, the unit tests have passed, now I'm porting the model data over and will try and spin it up. Fingers crossed.
If it does work I'll get my machine to submit orders to the production site until we host it in the cloud.

Bladerunners
Posts: 49
Joined: Sat Dec 30, 2017 11:53 pm
Contact:

Re: Austria vs Bots

#502 Post by Bladerunners » Wed Feb 15, 2023 12:03 am

Nice job! Austria very hard to play as majority of starts has Italy very very strangely programmed to attack Austria in most games (which is not common diplomacy play / really bad opening normally for Italy)

User avatar
kestasjk
Developer
Developer
Posts: 532
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Austria vs Bots

#503 Post by kestasjk » Fri Feb 17, 2023 1:48 am

FYI the AvF bot is currently down, going to replace with the DORA configuration over the weekend

Trigfea63
Gold Donator
Gold Donator
Posts: 334
Joined: Fri Apr 02, 2021 11:17 pm
Contact:

Re: Austria vs Bots

#504 Post by Trigfea63 » Fri Feb 17, 2023 3:59 am

kestasjk wrote:
Fri Feb 17, 2023 1:48 am
FYI the AvF bot is currently down, going to replace with the DORA configuration over the weekend
That's great news, Kestas. I'm already looking forward to my thrashing(s) at DORA's hands!

User avatar
kestasjk
Developer
Developer
Posts: 532
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Austria vs Bots

#505 Post by kestasjk » Mon Feb 20, 2023 3:33 am

Trigfea63 wrote:
Fri Feb 17, 2023 3:59 am
kestasjk wrote:
Fri Feb 17, 2023 1:48 am
FYI the AvF bot is currently down, going to replace with the DORA configuration over the weekend
That's great news, Kestas. I'm already looking forward to my thrashing(s) at DORA's hands!
Well let's not count our chickens.. but I'm running the new config now, interested to see if you notice a difference

Trigfea63
Gold Donator
Gold Donator
Posts: 334
Joined: Fri Apr 02, 2021 11:17 pm
Contact:

Re: Austria vs Bots

#506 Post by Trigfea63 » Mon Feb 20, 2023 7:06 am

kestasjk wrote:
Mon Feb 20, 2023 3:33 am
Trigfea63 wrote:
Fri Feb 17, 2023 3:59 am
kestasjk wrote:
Fri Feb 17, 2023 1:48 am
FYI the AvF bot is currently down, going to replace with the DORA configuration over the weekend
That's great news, Kestas. I'm already looking forward to my thrashing(s) at DORA's hands!
Well let's not count our chickens.. but I'm running the new config now, interested to see if you notice a difference
Yes, clearly this is a superior AI. I only played one game, but the difference was unmistakable. For some reason, it still opened with the Russian opening, Albanian variation (TRI->ALB, BUD->RUM, VIE->GAL). Unlike the previous AI, it actually followed through on those opening moves in a more sensible way, to GRE, SEV, and WAR in Autumn 1901. It made another major blunder in 1902, though, sending the SEV army to Turkey instead of north to StP. Those early, unforced errors put it too far behind, and I was able to secure the stalemate lines and win.

https://webdiplomacy.net/board.php?gameID=546552

Except for the weird opening errors, this AI was highly effective throughout the game, similar to "FB" (the old Fairbot from 2021-22). The MILA AIs have difficulty once the game position gets too complex in the mid-game. This AI was tough to outwit. I expect I would have been hard pressed to beat it if it had just played a normal opening.

Also, the speed was great. FB used to take a long time to enter orders during the mid-game, sometimes as long as 15 minutes. This AI entered orders immediately every time. No different from the MILA AIs in terms of speed.

I'll play a few more games and provide more feedback in another week or so.

User avatar
kestasjk
Developer
Developer
Posts: 532
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Austria vs Bots

#507 Post by kestasjk » Tue Feb 21, 2023 1:52 am

Hmm, interesting. Really like that we can start playing with the bot difficulties now. Will try to see what other FvA options there are for some clues about the opening, worst case scenario I'll reach out to the FAIR team but they're pretty hard to get hold of

MerryBBuck
Gold Donator
Gold Donator
Posts: 56
Joined: Sat Feb 29, 2020 9:08 pm
Contact:

Re: Austria vs Bots

#508 Post by MerryBBuck » Fri Feb 24, 2023 12:23 pm

When you know your opponent's opening: https://webdiplomacy.net/board.php?gameID=550660

Trigfea63
Gold Donator
Gold Donator
Posts: 334
Joined: Fri Apr 02, 2021 11:17 pm
Contact:

Re: Austria vs Bots

#509 Post by Trigfea63 » Sat Feb 25, 2023 5:01 am

Looks like you knew more than its opening.

This and other data are leading me to doubt my original conclusion about the new AI. I don't see how a sophisticated AI on the level of DORA gets smoked like that. Or like this:
https://webdiplomacy.net/board.php?game ... #gamePanel
https://webdiplomacy.net/board.php?game ... #gamePanel

Yeah, I know, not as impressive as Merry's. Still, easy wins for the human player. I didn't have to think about my moves at all, and won both games in 1905.

Kestas, Fairbot2's overall record since you made the AI switch is only 15 wins and 20 losses. Plus a bunch of draws (mostly incomplete games). Is it possible for DORA to perform that badly?

Trigfea63
Gold Donator
Gold Donator
Posts: 334
Joined: Fri Apr 02, 2021 11:17 pm
Contact:

Re: Austria vs Bots

#510 Post by Trigfea63 » Sat Feb 25, 2023 6:10 am

Trigfea63 wrote:
Sat Feb 25, 2023 5:01 am
Kestas, Fairbot2's overall record since you made the AI switch is only 15 wins and 20 losses. Plus a bunch of draws (mostly incomplete games). Is it possible for DORA to perform that badly?
The DORA AI paper (Bakhtin et al. 2021) says DORA won all 13 games as Austria and 73% as France against top FvA players on webdip. See Table 1 on page 7:
https://arxiv.org/pdf/2110.02924

ShipOfTheseus
Posts: 10
Joined: Thu Dec 22, 2022 9:03 am
Contact:

Re: Austria vs Bots

#511 Post by ShipOfTheseus » Sat Feb 25, 2023 6:31 am

MerryBBuck wrote:
Fri Feb 24, 2023 12:23 pm
When you know your opponent's opening: https://webdiplomacy.net/board.php?gameID=550660
Another great example of why we need a better final screen on the Beta edition -- you should be able to screenshot the all-34 blue map.

User avatar
kestasjk
Developer
Developer
Posts: 532
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Austria vs Bots

#512 Post by kestasjk » Sat Mar 04, 2023 4:52 am

Trigfea63 wrote:
Sat Feb 25, 2023 5:01 am
Looks like you knew more than its opening.

This and other data are leading me to doubt my original conclusion about the new AI. I don't see how a sophisticated AI on the level of DORA gets smoked like that. Or like this:
https://webdiplomacy.net/board.php?game ... #gamePanel
https://webdiplomacy.net/board.php?game ... #gamePanel

Yeah, I know, not as impressive as Merry's. Still, easy wins for the human player. I didn't have to think about my moves at all, and won both games in 1905.

Kestas, Fairbot2's overall record since you made the AI switch is only 15 wins and 20 losses. Plus a bunch of draws (mostly incomplete games). Is it possible for DORA to perform that badly?
It's definitely not unlikely that it's a different bot, I just loaded the model that looked like it was probably the FvA DORA model. Definitely need people's feedback to get this right.
Do you think there was any difference at all in the performance before / after?

Also I've got the cicero codebase set up and running and I loaded the DORA config and model using the cicero codebase, assuming that the bots would run the same it was just configuration, but I may need to get the original searchbot repo running. I'll take a closer look into the exact model used in that paper you listed. (I'm hoping I'm not going to find they used some crazy supercluster of GPUs with a crazy set of parameters I can't match)

By the way the FvA bot will be up and down today; I've added code that will hopefully allow a single running bot instance to play as multiple accounts and want to test that, and also want to try and get some functionality so the bots will be more reliable.
I may need to get a server machine for running the bots here, as running these bots under Linux under Windows on my main desktop is always going to be pretty flaky, but I'll see how far I can go with the current setup first

User avatar
kestasjk
Developer
Developer
Posts: 532
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Austria vs Bots

#513 Post by kestasjk » Sat Mar 04, 2023 5:13 am

Interested to see how you go now; I think I've matched it up to the exact config they referred to in that paper but let's see

Trigfea63
Gold Donator
Gold Donator
Posts: 334
Joined: Fri Apr 02, 2021 11:17 pm
Contact:

Re: Austria vs Bots

#514 Post by Trigfea63 » Sat Mar 04, 2023 6:46 am

[/quote]
Do you think there was any difference at all in the performance before / after?
[/quote]

At first I thought so. The after version seemed improved. But then, after 2 more games, I'm not sure. It's not easy to separate out the poor opening from sub-par play generally. In 1v1, if one player gets ahead by too much, it tends to snowball.
kestasjk wrote:
Sat Mar 04, 2023 5:13 am
Interested to see how you go now; I think I've matched it up to the exact config they referred to in that paper but let's see
That's the old Fairbot alright! Knows what moves I'm gonna make before I do. I'm not quite dead yet, but it's a struggle to stay afloat :sick:

https://webdiplomacy.net/board.php?gameID=558795

User avatar
kestasjk
Developer
Developer
Posts: 532
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Austria vs Bots

#515 Post by kestasjk » Sat Mar 04, 2023 2:55 pm

Trigfea63 wrote:
Sat Mar 04, 2023 6:46 am
kestasjk wrote:
Sat Mar 04, 2023 5:13 am
Interested to see how you go now; I think I've matched it up to the exact config they referred to in that paper but let's see
That's the old Fairbot alright! Knows what moves I'm gonna make before I do. I'm not quite dead yet, but it's a struggle to stay afloat :sick:

https://webdiplomacy.net/board.php?gameID=558795
Ah good stuff, that's that sorted.

Since the Searchbot / Cicero / Diplodicus / DORA bots are so resource intensive I've changed the API so that one bot can handle multiple bot accounts.

Here's a game of 7 Cicero's all playing against each other, and the one Cicero bot thinks it's in 7 different games:
Screenshot 2023-03-04 222744.png
Screenshot 2023-03-04 222821.png
Some of the things it says that you see before it gets filtered out :lol: it can be quite sassy..
There are so many ways it can be adjusted.. I understand it well enough now that I know we can have a single bot with multiple difficulty levels, press and no-press, carefully filtered or blunt, quick and sloppy or slow and careful, all running from a single bot instance.

Exciting stuff.. I thought getting Cicero/full-press bots running here was years away.

User avatar
JECE
Posts: 1625
Joined: Sun Dec 31, 2017 4:35 pm
Contact:

Re: Austria vs Bots

#516 Post by JECE » Sat Mar 04, 2023 8:25 pm

There is something not right about seeing an AI send the message "Nah I'm gonna kill you". :shock:
See my full Profile:
http://www.webdiplomacy.net/profile.php?userID=17421

User avatar
kestasjk
Developer
Developer
Posts: 532
Joined: Tue Mar 14, 2017 8:13 pm
Location: Perth, Australia
Contact:

Re: Austria vs Bots

#517 Post by kestasjk » Sun Mar 05, 2023 1:57 am

JECE wrote:
Sat Mar 04, 2023 8:25 pm
There is something not right about seeing an AI send the message "Nah I'm gonna kill you". :shock:
Oh trust me.. that made it through the filter :shock: The AI says the darndest things that don't make it through. jmo would ban an unfiltered cicero in a heartbeat!

mongster_03
Posts: 4
Joined: Tue Jan 23, 2018 3:43 am
Contact:

Re: Austria vs Bots

#518 Post by mongster_03 » Thu Mar 16, 2023 3:06 am

Quick question, why is it that every game I play as Turkey, I get Lepantoed, while every game I play as Austria, Italy attacks me? This is starting to feel rigged to give the most difficult possible scenarios.

User avatar
DougJoe
Posts: 913
Joined: Thu Aug 15, 2019 9:02 pm
Location: Alto, MI, USA
Contact:

Re: Austria vs Bots

#519 Post by DougJoe » Thu Mar 16, 2023 9:31 pm

mongster_03 wrote:
Thu Mar 16, 2023 3:06 am
Quick question, why is it that every game I play as Turkey, I get Lepantoed, while every game I play as Austria, Italy attacks me? This is starting to feel rigged to give the most difficult possible scenarios.
It's not. How many games have you played?

Bladerunners
Posts: 49
Joined: Sat Dec 30, 2017 11:53 pm
Contact:

Re: Austria vs Bots

#520 Post by Bladerunners » Fri Mar 17, 2023 7:32 am

He's right re. Italy being "rigged -- the developers mistakenly programmed Italy to attack Austria in greater than 75% of games rather than real life percentages....Italy Bots programmed not to learn from optimal play..

Post Reply

Who is online

Users browsing this forum: No registered users