@imperious-leader said in š²š¢ PRNG dice support group ā and ranting:
@aardvarkpepper Since you donāt own the program and may be unaware that they installed 2 dice systemsā¦perhaps you may have a more informed opinion of their efforts?
But I do own the program.
https://www.axisandallies.org/forums/topic/36399/how-to-punish-early-japanese-industrial-complex-aka-how-do-you-even-kjf-in-online/8
You can see I have an excellent grasp of 1942 Onlineās applications. It would be hard to comment to the degree I did regarding the specific issues without firsthand knowledge.
Weāve even played multiple multiplayer 1942 Online games together. Youāre the first and only player (other than myself) that Iāve seen build a German Mediterranean cruiser.
As to the matter of PRNGs and dice, for over a year I posted extensively on Steam forums, and Iāve made comments on these forums as well regarding the system, the implementation, what I say should have been done and which was not done (even allowing for budgetary constraints).
As to having a āmore informedā opinion, I flatter myself to think that I probably have one of the most informed opinions.
Some of the things that stick out to my mind -
Over a year ago, a developer (more correctly, someone working for the developerās company) commented to the effect it was āprovedā that the PRNG had no issues. But that simply isnāt how things work. Proper statistical analysis does not prove things, it merely provides get calculated probabilities that for a given dataset gathered under a given specific set of conditions, thereās a 99.998% (or such) that that particular dataset is in accordance with a particular predictive model.
Well, that was their first mistake, but what of it? People make mistakes. Especially when people get told things by other people, there are misunderstandings, no matter how small, and just because I thought it was a very bad look, eh.
At that point I didnāt think that players would make a big thing of PRNG. Iād noted in my google document (link below) that I thought the developers ought to make data importable and exportable, but that was a matter of general utility.
https://docs.google.com/document/d/17F3TotY7HEKeiLv3ewlfYotQv_hWXqh5PDo7B0exXpY/edit?usp=sharing
Youāll see one of the first comments was dated September 11, 2019, well over a year ago. Well, all thatās an aside, though it comes up later.
But back to the story of PRNG and the developer response. Some players started saying that the PRNG was suspect, which personally Iām indifferent to (as I consider the rules changes far more an issue, which is something that I think Panther commented on, on these forums, more than a year ago? He wrote about how the differences were not advertised which is coincidentally one of the things I personally rather dislike, and in fact Iād go so far as to say the gameās features are falsely advertised, but thatās another matter.
So more than a year ago, some players said there was a PRNG issue, and some other players said there wasnāt a PRNG issue, itās all āin your headā (though fancier terms were used). The developers COULD have responded acknowledging the legitimacy over concerns over the PRNG, but that didnāt happen. At the time.
That was the developersā second mistake. Which, okay, again, letās be fair here, people make mistakes. You get a program, you think you need programmers, thatās what you hire for, you hire marketers, support staff. If you donāt think you need statisticians then you donāt put them in the budget, if you donāt think you need a decent public relations team then you donāt budget for that either. Well, sometimes you get lucky and you get something good you didnāt pay for. In this case, not so lucky. But okay.
The developers were very dismissive, and about that time I wrote a bunch of posts on Steam forums about how exactly PRNGs work, how you get emergent behaviors that are not easily detected, and why concerns over PRNG were legitimate. I wrote that it could be the case that there was an issue with the PRNG, but that they wouldnāt know, as they hadnāt run the tests.
The developer response? First, they said theyād looked at whatever many number of dice rolls. Just a really very basic test, which granted should have been done as it would detect the most egregious of issues, but nowhere near what was needed or appropriate. I wrote that isnāt what was needed, you need to look at groups of dice. I wrote more posts, months passed.
The developers released another statement, stating they had run a Dickey-Fuller test, then I forget if they said they ran chi-squared or whatever. So I looked into the tests, I really looked. And the fact is, the developers were still making the same mistake (making this their third) in that they were not testing what needed to be tested. Same mistake, so why count it again? Because they had the opportunity to look at and consider their methodology and they did the same thing over again. Only this time, instead of āeyeballingā it they tried using technical gobbledygook. Which actually ISNāT gobbledygook if it was being used correctly which it wasnāt.
And all this time, there was just the most dismissive attitude by the developers, with various posters getting banned or censured - which, really, I wouldnāt complain about, except only those with criticisms of developers were clamped down on. Those that āsupportedā the developers were allowed to repeatedly post the most egregiously nasty sort of nonsense, and their behavior passed uncommented (and certainly uncensured).
I had quite a number of unpleasant interactions on Discord (which were probably unpleasant for others as well, considering I was pointing out their faults), and as usual, when I made specific points or asked for specific answers, what I wrote was blatantly ignored. Not in the context of āwe donāt have the time to look at this right now but weāll get back to it laterā. I mean I made the same points in public and private repeatedly, and the points were IGNORED, there is no doubt about it.
Well, you could see how this was going. I knew the developers had the point made to them, and how? Because I had made the points, personally! And if it was said (fairly enough) that they didnāt work for me, that they had budgetary concerns, that they werenāt necessarily professionals in statistics or even in the scientific method, well, still, things were what they were. There were any number of good responses they could have made that werenāt made, and they chose bad responses so many times.
(And I want to be clear, sometimes there are chain of command and information flow strictures that others are not aware of. Or sometimes there are multiple ābossesā that give conflicting orders that are left for others to resolve, with the whole mess covered up by nondisclosure agreements. Iām used to that sort of thing. But the sort of pushback I was receiving, I really felt that it was on a personal level. Like itās not someone just going along because they have to, they want to do what theyāre doing.)
But about at that point, a few months ago, the developers MOSTLY stopped trying to make boasts like āitās proved that thereās nothing wrong with the PRNGā (still popped up now and then). Maybe theyāre back to their old shenanigans, since I was wrongfully banned on Steam forums I havenāt checked, and I donāt bother with the Discord I see one of their representatives touting either. But if the developers arenāt false virtue signaling about the PRNG in my face, then good enough for me.
But the developers made a fourth mistake! Iād made the point on Discord that resolving player concerns over the PRNG was a public relations issue. But by that Iād only meant they needed to have it properly looked at instead of doing a patch job, I meant they needed to be understanding of customers rather than clamping down on them and denying / attacking them. But apparently āpublic relationsā didnāt mean to them what it means to me, and I suppose if someoneās used to an authoritarian regime and not from the West then maybe thatās natural for them, I donāt know.
But anyways, they released a statement about stabilized dice - or was that the one? Anyways at the end, they said theyād switched up their PRNG a bunch of times and they hadnāt noticed any differences in complaints. And I was like really now. Theyād always been very reticent to release the details of their studies (which frankly I donāt know that they ever made a study worth publishing), and here they said they ādidnāt notice any differenceā - using what metrics, specifically? I say they didnāt know what the issue was, they didnāt know what to look for, then they shouted āvictory!ā from the rooftops without looking down, without even checking, the situation on the ground. Which was precisely what they had been doing right along, but making a point of it and attacking their customers that way and saying they were trying to run games behind the scenes and being all ātee hee!ā about it was . . . I donāt even know what to say. Itās just outside my understanding of how you would ever want professional to act. I mean, itās very normal for āattack politicsā, I expect it there, but here I just donāt even know.
So would it be fair to say I have an informed opinion of their efforts? Or not? And letās not say that I decided to close the door on the developers. They banned me, not the other way around. And the criticisms I make are not just about what happened in the past but what is happening now, what continues to happen, and what continues to NOT happen.
Do the developers need to own up to past mistakes? I donāt insist on it. Do they need to āfixā their PRNG? I donāt even insist on that. At this point, Iām only really saying that they should either implement the game as it is in the board game for the āfull experienceā, OR they should correct the description they have on Steamās front page, openly acknowledging the differences between their program and Axis and Allies 1942 Second Edition. (Oh, and also I say the developers shouldnāt say things HAVE been fixed and/or properly analyzed when they have not been. I donāt think thatās too much to ask. And if players push on their claims that things have been āprovedā or ādisprovedā, then Iāll often say something about that too.)
To close this out - I know a lot gets taken out of context, and maybe some readers will look at this and get the takeaway that I think things were done wrongly (fair enough). But they wonāt understand the methodology that Iām saying should have been done. So Iāll repeat here what Iāve written I donāt even know how many times between Steam, Discord, and I think Iāve made some allusions on these forums but not a proper address. Heck, even the following isnāt a proper address.
Iāll use an analogy, as in my experience, when I write exactly how it works mathematically a lot of times readers just donāt understand. Nor do readers even understand when I explain exactly why itās important in actual game terms. They need something to relate to.
So, cards, not dice. Think about what Iām writing, then try to understand how it applies to Axis and Allies.
Four suits, thirteen ranks, deck has fifty-two cards. Now suppose I were to say ānobody knows what a single random card draw will be!ā. Well, you donāt know, precisely. But assuming conditions to be equal and the deck to have been fairly randomly shuffled then you have a one in fifty-two chance of drawing any particular card.
But already we have our first set of assumptions - that the deck is shuffled fairly and conditions are equal. A lot of people dismiss the details, but those details are important. Whatās the first card youāll draw off a fifty-two card deck that you just bought from the store? I donāt know, King of Hearts every time from a particular manufacturer? And if you shuffled and cut a deck of cards, if one of the cards has a little bend in it - no matter how slight, one that you wonāt even see if you visually inspect the deck as a whole - the deck will cut at that place.
And something that a lot of readers do with dice is they assume they know how it all works, but they donāt want to be bothered with those pesky details, you know, the ones that challenge the paradigm theyāve built for themselves? But the more you know about mathematics and practical data sciences, the more you know about programming, the more you realize there MAY be issues.
Such as? Letās say that in the LHTR setup a USSR player opens with 12 units to West Russia, 9 to Ukraine. Yes? That is the meta, is it not? But then what happens?
PRNGs do NOT generate a series of random numbers. Hence the name, āPseudo Random Number Generatorā. They take a time seed and generate a sequence of numbers and use that. And the issues that result come from how and when those time seeds are taken, and how the PRNG output is applied.
So what then happens is, without anything needing to have been deliberately programmed in (though for those that claim things could have been deliberately programmed in thatās a possibility) - but even WITHOUT deliberate programming, you get these sequences of calculated numbers with the methods of generation that result in - what? Emergent behavior.
To use another analogy, when you go to the ocean, why are there waves? Why isnāt the ocean as flat as glass? There are currents and winds that bunch up and move water in different ways, there are thresholds that are surpassed, and though each individual molecule of water may not have a predicable pattern as such, groups of water molecules do*. That is why there are waves.
So a lot of what we see is people look at the interactions between various agencies and individual molecules of water, then claim, because they can detect no pattern, that things must be ārandomā. Some may even go so far as to say we will see no waves in the ocean because analysis of collective and emergent behaviors is not a āthingā - itās confusing, and it just makes sense to say very loudly and repeatedly that waves are nonsense, you have a glass of water and IT doesnāt have waves, and if you drop a pebble in your glass of water and it causes ripples, well, that isnāt a wave, itās a ripple, and how dare you throw pebbles in my glass of water thatās just impertinent etc.
Right. But back to this comfortable simple model. You have a randomly shuffled deck, nobodyās put bends in the cards or dropped any sawdust, just a single draw.
So then we proceed to hands of cards, in, say, poker. Suppose someone were to claim that they are just āluckyā, they just HAPPEN to get royal flushes all the time, or if theyāve picked up on card sharping, maybe they just so happen to have gotten high pair a lot.
Which is an interesting story, sometimes you have people that refuse to believe that thereās anything wrong with their play or with the deck, and they end up losing all their money. And the one that takes the money is so sympathetic, youāre good at cards! really! just a run of bad luck!
In cards, you have to understand, thereās probability, then thereās being able to read and predict opponents. How does this relate to Axis and Allies?
On both sides of the argument, you have players saying there are problems or there are not problems, but who is doing the statistical studies? And I mean PROPER statistics, itās a lot of real work.
But back to poker. So letās say you have a player thatās āluckyā a lot. Then they say āI canāt help it! Iām just lucky!ā
But you have to really think about whatās going on there. Suppose a player is ACTUALLY just lucky on a PARTICULAR night. Well, that happens. Everyoneās number comes up once in a while. But suppose someone is CONSISTENTLY āluckyā, repeatedly. Suppose itās not just that itās casually THOUGHT that the player is lucky, suppose you can demonstrate the truth of this luck through mathematics. Can we say āthatās just how luck works!ā But that isnāt how luck works. One night sure, but repeatedly? No, that player probably has an edge. And you just have to figure out how that edge works.
And Iāve seen all sorts of players try to figure out what the āedgeā is. Mirrors? Angles? And often they overlook something. Like a bit of sawdust. Switching out decks. Confederates. No matter how clever you are at cheat detection, thereās a cheat out there thatās figured out what youāre going to try, and figured out a way around it.
Well, thatās another topic. But what Iām trying to impart is the sense of coldness about it all. You have a very sharp sort of game in cards, where you have to think about things very coldly and rationally and using calculations, whether you are cheating or trying to detect cheating, and at some point if a lot of players are fed up they donāt care about calculations and thereās heck to pay. Which is another topic.
But what Iām getting at is in that cold and sharp rational game, suppose you have a player that doesnāt understand mathematics. They start saying things like āyou can draw any one of fifty two cards, but nobody can know what the probability of a given hand isā (even though you can CALCULATE the probability of a given hand). Iām not trying to shame anyone here, but readers have got to understand. Thereās entire different levels of understanding, you have the very sharp knowledgeable players that know what to look out for, whether theyāre cheating or trying to make sure others donāt cheat, then you have players that understand the vague concepts and why a straight beats a pair (and maybe even the probabilities of each), then you have absolute rubes that hardly even understand mathematics at all, they know a straight beats a pair because everyone knows that, but the question of probability does not enter their mind. And itās these rubes that proclaim thereās nothing wrong with a system they donāt understand, that itās impossible that anyoneās cheating (which they wouldnāt begin to understand how to do or prevent), and though some are well-intentioned, still. You just donāt want to be taking their advice, and if they ācanāt be botheredā with the details, then you want to think about that. You really do.
But how do you calculate probabilities for combinations of cards? Unlike with dice, thereās a load of literature about calculating probabilities for meaningful combinations of cards. Feel free to read up on it.
Again - itās not that I want to jump around between dice and cards and the movement of the tides. But Iāve found again and again, when I explain the literal case and the mathematics, readers sometimes just canāt relate their everyday experience to the points I"m making. When thereās nothing for them to relate to, they think it just doesnāt apply to them for some reason. Though it does, of course.
So, hands of cards. If a particular player is drawing a lot of straight flushes, thatās going to be weird right? But why? Are we seriously going to argue that any player can draw any hand, and everythingās random and everything defies analysis? Even players that donāt understand numbers donāt hold with such truck. If you see the same player winning over and over, well, letās just say that if there arenāt other factors involved, that player isnāt going to be with the group for long. But you understand, the question is not about a single card, itās not about the fact hands exist, itās about WHO gets WHAT hand. Yes?
So now we have posts by some posters - not singling anyone out in particular, there are a LOT of posts by DIFFERENT posters - that CLAIM they are all about that data and proof yet they refuse to understand the conceptual argument and even say things indicating they do not understand the argument. Then the same posters often dismiss these things that they do not understand as āemotionalā or ācognitive dissonanceā and sometimes they drag up fancy words to belittle those they disagree with - but what you notice is they do NOT themselves have a reasoned approach, they do NOT present DATA, or PROOFS, nor do they even extend the benefit of a reasonable doubt to others. No, they are DISMISSIVE, ABSENT PROOFS. And that just isnāt right.
(Isnāt it right? Well if youāre accepting of every point then you end up with people coming to you with weird propositions and expecting you to swallow whatever story theyāre selling, so you have to be dismissive sometime. Right? Thatās just how people live. Right?)
Sure. But Iām saying even so thereās a line. If it affects you personally, then be dismissive as you like. But if you have a professional obligation, then you canāt just be dismissive. And if youāre being dismissive of others, without reason then youāre on shaky ground. When people refuse to discuss things, what sort of discussion or community does that create? We donāt have to discuss everything, sure. But when posters just say āyouāre wrong! fancy term saying Iām right! no proofs or explanations! youāre emotional! go home!ā I think we need a lot less of that.
Back to Axis and Allies and dice.
You need a lot of data if you want to look for patterns. Thatās just how it is. That isnāt me trying to push the developers to do anything, it isnāt me trying to dictate to others what is right or what is wrong, thatās just how it is.
And when you look at that dice, you need what I call ācomplete dataā, because you need to be able to look for patterns of emergent behavior.
And now Iām going to get into the theory of the ACTUAL case. Well, briefly, anyways, Iāve written much more elsewhere.
We know that PRNGs do NOT actually generate random numbers. Thatās why theyāre called PRNGs, āpseudoā means fake. Not random. Get it? Like how āI canāt believe itās not butter!ā is NOT BUTTER.
Depending on the implementation thereās broadly two ways it can go, assuming a simplistic implementation (which, considering 1942 Online doesnāt even implement the board game rules, I would think a reasonable assumption.) Either a single time seed is generated for an entire game, in which itās susceptible to reverse engineering and manipulation by players picking the order of combats and the units allocated to battles, and in which case you can get emergent behaviors that lead to clustering that creates anomalous dice results. OR you get multiple time seeds that are generated individually based on players sending requests, in which case cheaters have finer control over how they can manipulate the time seeds, and which ALSO can result in emergent behaviors.
Emergent behaviors? Remember again, PRNGs are not truly random. Remember Axis and Allies games involve a single setup - and even if you account for LHTR setup and even for preplaced bids, there would still be convergent points for any particular player.
So what happens when you have the same player doing 12 to West Russia and 9 to Ukraine, over and over again? If the dice were actually random, then the results would be whatever they would be. We would see some clustering of results that player would interpret as āgoodā, we would see some clustering of results that player would interpret as ābadā. But if it is a given that the dice are really random then thatās just how it is.
But how is that different to clustering observed with PRNGs? Because PRNG clustering is simply different. Suppose a player assumes that the dice are fair (and without a huge dataset and proper analysis, thatās the going assumption that players would reasonably make). But suppose the PRNG implementation and clustering of outputs is such that instead of 85% favorable results at Ukraine (including attack and retreat actions), the player instead has only 65%? Depending on the PRNG implementation we are going to possibly see such skewed results because you are relying on a PRNG that outputs a sequence of numbers on a time seed for a game that has a single static setup that encourages groupings of the same moves that results in clustering. It is NOT actually RNG, that is the POINT.
Dismissive readers will say things like ā85%, 65%, whatās the difference?ā Or theyāll say things like āyou can still get good and bad results, so whatās the difference?ā
The difference is that players assume theyāre playing the game according to their model, but actually the game works on a different model, one thatās never explained to the player.
Letās say I have a coin. I flip it. Heads you give me a dollar, tails I give you a dollar. Okay. Right? Letās play this game a bunch of times, maybe you win, maybe you lose, whatever.
But now letās say my coin has two heads. You could say itās less egregious, like maybe we pull a coin out of a small jar each time, and out of ten coins in that jar only four of them have two heads. But do you think thatās the sort of information you would want to have about the game before playing? I would hope so.
Returning to the actual case, what Iām saying is you need to be able to differentiate each dice so you can put them in groups, then you need to look at the behavior of those groups. Is one player getting a load of āluckyā rolls? How do antiaircraft guns work?
And some responders say things like āthey work, sometimes AA shoots bombers downā. But that totally misses the point. If youāre not reverse engineering the program to look at the specific PRNG implementation, how do you know how, exactly, how that PRNG implementation works? What if the PRNG has different implementations? Even if itās not programmed for bias, can you guarantee thereās no bugs in the code? Even if there are no bugs and itās not programmed for bias, have you accounted for emergent behaviors? You canāt, unless you look at the data.
Anyways in closing:
Whatās in the past is in the past. But for the present, the developers should REALLY change the description on 1942 Onlineās Steam page to reflect the fact 1942 Online is NOT like the board game in terms of gameplay. Or better, they could put in live play. Ooo yeah. They should do THAT. But if not, at least change the store description.
As to the PRNG, I think itās perfectly reasonable to believe the PRNG may have issues, because as far as I know it might really have issues. Iāve seen a lot of bad āstudiesā and a lot of specious arguments arguing there arenāt any issues, but the fact is itās the groupings of dice that need to be looked at, not the individual dice, or even the aggregate of individual dice taken all together, but sub-groupings, thatās just how it is, and nobodyās even attempted to do that properly that Iāve ever seen.
And if anyone wants to undertake it, better believe thereās a lot more to it than just what Iām writing here. If a proper job is to be made of it, itāll require a lot of work.