@imperious-leader said in đ˛đ˘ PRNG dice support group â and ranting:
@aardvarkpepper Since you donât own the program and may be unaware that they installed 2 dice systemsâŚperhaps you may have a more informed opinion of their efforts?
But I do own the program.
https://www.axisandallies.org/forums/topic/36399/how-to-punish-early-japanese-industrial-complex-aka-how-do-you-even-kjf-in-online/8
You can see I have an excellent grasp of 1942 Onlineâs applications. It would be hard to comment to the degree I did regarding the specific issues without firsthand knowledge.
Weâve even played multiple multiplayer 1942 Online games together. Youâre the first and only player (other than myself) that Iâve seen build a German Mediterranean cruiser.
As to the matter of PRNGs and dice, for over a year I posted extensively on Steam forums, and Iâve made comments on these forums as well regarding the system, the implementation, what I say should have been done and which was not done (even allowing for budgetary constraints).
As to having a âmore informedâ opinion, I flatter myself to think that I probably have one of the most informed opinions.
Some of the things that stick out to my mind -
Over a year ago, a developer (more correctly, someone working for the developerâs company) commented to the effect it was âprovedâ that the PRNG had no issues. But that simply isnât how things work. Proper statistical analysis does not prove things, it merely provides get calculated probabilities that for a given dataset gathered under a given specific set of conditions, thereâs a 99.998% (or such) that that particular dataset is in accordance with a particular predictive model.
Well, that was their first mistake, but what of it? People make mistakes. Especially when people get told things by other people, there are misunderstandings, no matter how small, and just because I thought it was a very bad look, eh.
At that point I didnât think that players would make a big thing of PRNG. Iâd noted in my google document (link below) that I thought the developers ought to make data importable and exportable, but that was a matter of general utility.
https://docs.google.com/document/d/17F3TotY7HEKeiLv3ewlfYotQv_hWXqh5PDo7B0exXpY/edit?usp=sharing
Youâll see one of the first comments was dated September 11, 2019, well over a year ago. Well, all thatâs an aside, though it comes up later.
But back to the story of PRNG and the developer response. Some players started saying that the PRNG was suspect, which personally Iâm indifferent to (as I consider the rules changes far more an issue, which is something that I think Panther commented on, on these forums, more than a year ago? He wrote about how the differences were not advertised which is coincidentally one of the things I personally rather dislike, and in fact Iâd go so far as to say the gameâs features are falsely advertised, but thatâs another matter.
So more than a year ago, some players said there was a PRNG issue, and some other players said there wasnât a PRNG issue, itâs all âin your headâ (though fancier terms were used). The developers COULD have responded acknowledging the legitimacy over concerns over the PRNG, but that didnât happen. At the time.
That was the developersâ second mistake. Which, okay, again, letâs be fair here, people make mistakes. You get a program, you think you need programmers, thatâs what you hire for, you hire marketers, support staff. If you donât think you need statisticians then you donât put them in the budget, if you donât think you need a decent public relations team then you donât budget for that either. Well, sometimes you get lucky and you get something good you didnât pay for. In this case, not so lucky. But okay.
The developers were very dismissive, and about that time I wrote a bunch of posts on Steam forums about how exactly PRNGs work, how you get emergent behaviors that are not easily detected, and why concerns over PRNG were legitimate. I wrote that it could be the case that there was an issue with the PRNG, but that they wouldnât know, as they hadnât run the tests.
The developer response? First, they said theyâd looked at whatever many number of dice rolls. Just a really very basic test, which granted should have been done as it would detect the most egregious of issues, but nowhere near what was needed or appropriate. I wrote that isnât what was needed, you need to look at groups of dice. I wrote more posts, months passed.
The developers released another statement, stating they had run a Dickey-Fuller test, then I forget if they said they ran chi-squared or whatever. So I looked into the tests, I really looked. And the fact is, the developers were still making the same mistake (making this their third) in that they were not testing what needed to be tested. Same mistake, so why count it again? Because they had the opportunity to look at and consider their methodology and they did the same thing over again. Only this time, instead of âeyeballingâ it they tried using technical gobbledygook. Which actually ISNâT gobbledygook if it was being used correctly which it wasnât.
And all this time, there was just the most dismissive attitude by the developers, with various posters getting banned or censured - which, really, I wouldnât complain about, except only those with criticisms of developers were clamped down on. Those that âsupportedâ the developers were allowed to repeatedly post the most egregiously nasty sort of nonsense, and their behavior passed uncommented (and certainly uncensured).
I had quite a number of unpleasant interactions on Discord (which were probably unpleasant for others as well, considering I was pointing out their faults), and as usual, when I made specific points or asked for specific answers, what I wrote was blatantly ignored. Not in the context of âwe donât have the time to look at this right now but weâll get back to it laterâ. I mean I made the same points in public and private repeatedly, and the points were IGNORED, there is no doubt about it.
Well, you could see how this was going. I knew the developers had the point made to them, and how? Because I had made the points, personally! And if it was said (fairly enough) that they didnât work for me, that they had budgetary concerns, that they werenât necessarily professionals in statistics or even in the scientific method, well, still, things were what they were. There were any number of good responses they could have made that werenât made, and they chose bad responses so many times.
(And I want to be clear, sometimes there are chain of command and information flow strictures that others are not aware of. Or sometimes there are multiple âbossesâ that give conflicting orders that are left for others to resolve, with the whole mess covered up by nondisclosure agreements. Iâm used to that sort of thing. But the sort of pushback I was receiving, I really felt that it was on a personal level. Like itâs not someone just going along because they have to, they want to do what theyâre doing.)
But about at that point, a few months ago, the developers MOSTLY stopped trying to make boasts like âitâs proved that thereâs nothing wrong with the PRNGâ (still popped up now and then). Maybe theyâre back to their old shenanigans, since I was wrongfully banned on Steam forums I havenât checked, and I donât bother with the Discord I see one of their representatives touting either. But if the developers arenât false virtue signaling about the PRNG in my face, then good enough for me.
But the developers made a fourth mistake! Iâd made the point on Discord that resolving player concerns over the PRNG was a public relations issue. But by that Iâd only meant they needed to have it properly looked at instead of doing a patch job, I meant they needed to be understanding of customers rather than clamping down on them and denying / attacking them. But apparently âpublic relationsâ didnât mean to them what it means to me, and I suppose if someoneâs used to an authoritarian regime and not from the West then maybe thatâs natural for them, I donât know.
But anyways, they released a statement about stabilized dice - or was that the one? Anyways at the end, they said theyâd switched up their PRNG a bunch of times and they hadnât noticed any differences in complaints. And I was like really now. Theyâd always been very reticent to release the details of their studies (which frankly I donât know that they ever made a study worth publishing), and here they said they âdidnât notice any differenceâ - using what metrics, specifically? I say they didnât know what the issue was, they didnât know what to look for, then they shouted âvictory!â from the rooftops without looking down, without even checking, the situation on the ground. Which was precisely what they had been doing right along, but making a point of it and attacking their customers that way and saying they were trying to run games behind the scenes and being all âtee hee!â about it was . . . I donât even know what to say. Itâs just outside my understanding of how you would ever want professional to act. I mean, itâs very normal for âattack politicsâ, I expect it there, but here I just donât even know.
So would it be fair to say I have an informed opinion of their efforts? Or not? And letâs not say that I decided to close the door on the developers. They banned me, not the other way around. And the criticisms I make are not just about what happened in the past but what is happening now, what continues to happen, and what continues to NOT happen.
Do the developers need to own up to past mistakes? I donât insist on it. Do they need to âfixâ their PRNG? I donât even insist on that. At this point, Iâm only really saying that they should either implement the game as it is in the board game for the âfull experienceâ, OR they should correct the description they have on Steamâs front page, openly acknowledging the differences between their program and Axis and Allies 1942 Second Edition. (Oh, and also I say the developers shouldnât say things HAVE been fixed and/or properly analyzed when they have not been. I donât think thatâs too much to ask. And if players push on their claims that things have been âprovedâ or âdisprovedâ, then Iâll often say something about that too.)
To close this out - I know a lot gets taken out of context, and maybe some readers will look at this and get the takeaway that I think things were done wrongly (fair enough). But they wonât understand the methodology that Iâm saying should have been done. So Iâll repeat here what Iâve written I donât even know how many times between Steam, Discord, and I think Iâve made some allusions on these forums but not a proper address. Heck, even the following isnât a proper address.
Iâll use an analogy, as in my experience, when I write exactly how it works mathematically a lot of times readers just donât understand. Nor do readers even understand when I explain exactly why itâs important in actual game terms. They need something to relate to.
So, cards, not dice. Think about what Iâm writing, then try to understand how it applies to Axis and Allies.
Four suits, thirteen ranks, deck has fifty-two cards. Now suppose I were to say ânobody knows what a single random card draw will be!â. Well, you donât know, precisely. But assuming conditions to be equal and the deck to have been fairly randomly shuffled then you have a one in fifty-two chance of drawing any particular card.
But already we have our first set of assumptions - that the deck is shuffled fairly and conditions are equal. A lot of people dismiss the details, but those details are important. Whatâs the first card youâll draw off a fifty-two card deck that you just bought from the store? I donât know, King of Hearts every time from a particular manufacturer? And if you shuffled and cut a deck of cards, if one of the cards has a little bend in it - no matter how slight, one that you wonât even see if you visually inspect the deck as a whole - the deck will cut at that place.
And something that a lot of readers do with dice is they assume they know how it all works, but they donât want to be bothered with those pesky details, you know, the ones that challenge the paradigm theyâve built for themselves? But the more you know about mathematics and practical data sciences, the more you know about programming, the more you realize there MAY be issues.
Such as? Letâs say that in the LHTR setup a USSR player opens with 12 units to West Russia, 9 to Ukraine. Yes? That is the meta, is it not? But then what happens?
PRNGs do NOT generate a series of random numbers. Hence the name, âPseudo Random Number Generatorâ. They take a time seed and generate a sequence of numbers and use that. And the issues that result come from how and when those time seeds are taken, and how the PRNG output is applied.
So what then happens is, without anything needing to have been deliberately programmed in (though for those that claim things could have been deliberately programmed in thatâs a possibility) - but even WITHOUT deliberate programming, you get these sequences of calculated numbers with the methods of generation that result in - what? Emergent behavior.
To use another analogy, when you go to the ocean, why are there waves? Why isnât the ocean as flat as glass? There are currents and winds that bunch up and move water in different ways, there are thresholds that are surpassed, and though each individual molecule of water may not have a predicable pattern as such, groups of water molecules do*. That is why there are waves.
So a lot of what we see is people look at the interactions between various agencies and individual molecules of water, then claim, because they can detect no pattern, that things must be ârandomâ. Some may even go so far as to say we will see no waves in the ocean because analysis of collective and emergent behaviors is not a âthingâ - itâs confusing, and it just makes sense to say very loudly and repeatedly that waves are nonsense, you have a glass of water and IT doesnât have waves, and if you drop a pebble in your glass of water and it causes ripples, well, that isnât a wave, itâs a ripple, and how dare you throw pebbles in my glass of water thatâs just impertinent etc.
Right. But back to this comfortable simple model. You have a randomly shuffled deck, nobodyâs put bends in the cards or dropped any sawdust, just a single draw.
So then we proceed to hands of cards, in, say, poker. Suppose someone were to claim that they are just âluckyâ, they just HAPPEN to get royal flushes all the time, or if theyâve picked up on card sharping, maybe they just so happen to have gotten high pair a lot.
Which is an interesting story, sometimes you have people that refuse to believe that thereâs anything wrong with their play or with the deck, and they end up losing all their money. And the one that takes the money is so sympathetic, youâre good at cards! really! just a run of bad luck!
In cards, you have to understand, thereâs probability, then thereâs being able to read and predict opponents. How does this relate to Axis and Allies?
On both sides of the argument, you have players saying there are problems or there are not problems, but who is doing the statistical studies? And I mean PROPER statistics, itâs a lot of real work.
But back to poker. So letâs say you have a player thatâs âluckyâ a lot. Then they say âI canât help it! Iâm just lucky!â
But you have to really think about whatâs going on there. Suppose a player is ACTUALLY just lucky on a PARTICULAR night. Well, that happens. Everyoneâs number comes up once in a while. But suppose someone is CONSISTENTLY âluckyâ, repeatedly. Suppose itâs not just that itâs casually THOUGHT that the player is lucky, suppose you can demonstrate the truth of this luck through mathematics. Can we say âthatâs just how luck works!â But that isnât how luck works. One night sure, but repeatedly? No, that player probably has an edge. And you just have to figure out how that edge works.
And Iâve seen all sorts of players try to figure out what the âedgeâ is. Mirrors? Angles? And often they overlook something. Like a bit of sawdust. Switching out decks. Confederates. No matter how clever you are at cheat detection, thereâs a cheat out there thatâs figured out what youâre going to try, and figured out a way around it.
Well, thatâs another topic. But what Iâm trying to impart is the sense of coldness about it all. You have a very sharp sort of game in cards, where you have to think about things very coldly and rationally and using calculations, whether you are cheating or trying to detect cheating, and at some point if a lot of players are fed up they donât care about calculations and thereâs heck to pay. Which is another topic.
But what Iâm getting at is in that cold and sharp rational game, suppose you have a player that doesnât understand mathematics. They start saying things like âyou can draw any one of fifty two cards, but nobody can know what the probability of a given hand isâ (even though you can CALCULATE the probability of a given hand). Iâm not trying to shame anyone here, but readers have got to understand. Thereâs entire different levels of understanding, you have the very sharp knowledgeable players that know what to look out for, whether theyâre cheating or trying to make sure others donât cheat, then you have players that understand the vague concepts and why a straight beats a pair (and maybe even the probabilities of each), then you have absolute rubes that hardly even understand mathematics at all, they know a straight beats a pair because everyone knows that, but the question of probability does not enter their mind. And itâs these rubes that proclaim thereâs nothing wrong with a system they donât understand, that itâs impossible that anyoneâs cheating (which they wouldnât begin to understand how to do or prevent), and though some are well-intentioned, still. You just donât want to be taking their advice, and if they âcanât be botheredâ with the details, then you want to think about that. You really do.
But how do you calculate probabilities for combinations of cards? Unlike with dice, thereâs a load of literature about calculating probabilities for meaningful combinations of cards. Feel free to read up on it.
Again - itâs not that I want to jump around between dice and cards and the movement of the tides. But Iâve found again and again, when I explain the literal case and the mathematics, readers sometimes just canât relate their everyday experience to the points I"m making. When thereâs nothing for them to relate to, they think it just doesnât apply to them for some reason. Though it does, of course.
So, hands of cards. If a particular player is drawing a lot of straight flushes, thatâs going to be weird right? But why? Are we seriously going to argue that any player can draw any hand, and everythingâs random and everything defies analysis? Even players that donât understand numbers donât hold with such truck. If you see the same player winning over and over, well, letâs just say that if there arenât other factors involved, that player isnât going to be with the group for long. But you understand, the question is not about a single card, itâs not about the fact hands exist, itâs about WHO gets WHAT hand. Yes?
So now we have posts by some posters - not singling anyone out in particular, there are a LOT of posts by DIFFERENT posters - that CLAIM they are all about that data and proof yet they refuse to understand the conceptual argument and even say things indicating they do not understand the argument. Then the same posters often dismiss these things that they do not understand as âemotionalâ or âcognitive dissonanceâ and sometimes they drag up fancy words to belittle those they disagree with - but what you notice is they do NOT themselves have a reasoned approach, they do NOT present DATA, or PROOFS, nor do they even extend the benefit of a reasonable doubt to others. No, they are DISMISSIVE, ABSENT PROOFS. And that just isnât right.
(Isnât it right? Well if youâre accepting of every point then you end up with people coming to you with weird propositions and expecting you to swallow whatever story theyâre selling, so you have to be dismissive sometime. Right? Thatâs just how people live. Right?)
Sure. But Iâm saying even so thereâs a line. If it affects you personally, then be dismissive as you like. But if you have a professional obligation, then you canât just be dismissive. And if youâre being dismissive of others, without reason then youâre on shaky ground. When people refuse to discuss things, what sort of discussion or community does that create? We donât have to discuss everything, sure. But when posters just say âyouâre wrong! fancy term saying Iâm right! no proofs or explanations! youâre emotional! go home!â I think we need a lot less of that.
Back to Axis and Allies and dice.
You need a lot of data if you want to look for patterns. Thatâs just how it is. That isnât me trying to push the developers to do anything, it isnât me trying to dictate to others what is right or what is wrong, thatâs just how it is.
And when you look at that dice, you need what I call âcomplete dataâ, because you need to be able to look for patterns of emergent behavior.
And now Iâm going to get into the theory of the ACTUAL case. Well, briefly, anyways, Iâve written much more elsewhere.
We know that PRNGs do NOT actually generate random numbers. Thatâs why theyâre called PRNGs, âpseudoâ means fake. Not random. Get it? Like how âI canât believe itâs not butter!â is NOT BUTTER.
Depending on the implementation thereâs broadly two ways it can go, assuming a simplistic implementation (which, considering 1942 Online doesnât even implement the board game rules, I would think a reasonable assumption.) Either a single time seed is generated for an entire game, in which itâs susceptible to reverse engineering and manipulation by players picking the order of combats and the units allocated to battles, and in which case you can get emergent behaviors that lead to clustering that creates anomalous dice results. OR you get multiple time seeds that are generated individually based on players sending requests, in which case cheaters have finer control over how they can manipulate the time seeds, and which ALSO can result in emergent behaviors.
Emergent behaviors? Remember again, PRNGs are not truly random. Remember Axis and Allies games involve a single setup - and even if you account for LHTR setup and even for preplaced bids, there would still be convergent points for any particular player.
So what happens when you have the same player doing 12 to West Russia and 9 to Ukraine, over and over again? If the dice were actually random, then the results would be whatever they would be. We would see some clustering of results that player would interpret as âgoodâ, we would see some clustering of results that player would interpret as âbadâ. But if it is a given that the dice are really random then thatâs just how it is.
But how is that different to clustering observed with PRNGs? Because PRNG clustering is simply different. Suppose a player assumes that the dice are fair (and without a huge dataset and proper analysis, thatâs the going assumption that players would reasonably make). But suppose the PRNG implementation and clustering of outputs is such that instead of 85% favorable results at Ukraine (including attack and retreat actions), the player instead has only 65%? Depending on the PRNG implementation we are going to possibly see such skewed results because you are relying on a PRNG that outputs a sequence of numbers on a time seed for a game that has a single static setup that encourages groupings of the same moves that results in clustering. It is NOT actually RNG, that is the POINT.
Dismissive readers will say things like â85%, 65%, whatâs the difference?â Or theyâll say things like âyou can still get good and bad results, so whatâs the difference?â
The difference is that players assume theyâre playing the game according to their model, but actually the game works on a different model, one thatâs never explained to the player.
Letâs say I have a coin. I flip it. Heads you give me a dollar, tails I give you a dollar. Okay. Right? Letâs play this game a bunch of times, maybe you win, maybe you lose, whatever.
But now letâs say my coin has two heads. You could say itâs less egregious, like maybe we pull a coin out of a small jar each time, and out of ten coins in that jar only four of them have two heads. But do you think thatâs the sort of information you would want to have about the game before playing? I would hope so.
Returning to the actual case, what Iâm saying is you need to be able to differentiate each dice so you can put them in groups, then you need to look at the behavior of those groups. Is one player getting a load of âluckyâ rolls? How do antiaircraft guns work?
And some responders say things like âthey work, sometimes AA shoots bombers downâ. But that totally misses the point. If youâre not reverse engineering the program to look at the specific PRNG implementation, how do you know how, exactly, how that PRNG implementation works? What if the PRNG has different implementations? Even if itâs not programmed for bias, can you guarantee thereâs no bugs in the code? Even if there are no bugs and itâs not programmed for bias, have you accounted for emergent behaviors? You canât, unless you look at the data.
Anyways in closing:
Whatâs in the past is in the past. But for the present, the developers should REALLY change the description on 1942 Onlineâs Steam page to reflect the fact 1942 Online is NOT like the board game in terms of gameplay. Or better, they could put in live play. Ooo yeah. They should do THAT. But if not, at least change the store description.
As to the PRNG, I think itâs perfectly reasonable to believe the PRNG may have issues, because as far as I know it might really have issues. Iâve seen a lot of bad âstudiesâ and a lot of specious arguments arguing there arenât any issues, but the fact is itâs the groupings of dice that need to be looked at, not the individual dice, or even the aggregate of individual dice taken all together, but sub-groupings, thatâs just how it is, and nobodyâs even attempted to do that properly that Iâve ever seen.
And if anyone wants to undertake it, better believe thereâs a lot more to it than just what Iâm writing here. If a proper job is to be made of it, itâll require a lot of work.