@imperious-leader said in 🎲😢 PRNG dice support group — and ranting:
@aardvarkpepper Since you don’t own the program and may be unaware that they installed 2 dice systems…perhaps you may have a more informed opinion of their efforts?
But I do own the program.
https://www.axisandallies.org/forums/topic/36399/how-to-punish-early-japanese-industrial-complex-aka-how-do-you-even-kjf-in-online/8
You can see I have an excellent grasp of 1942 Online’s applications. It would be hard to comment to the degree I did regarding the specific issues without firsthand knowledge.
We’ve even played multiple multiplayer 1942 Online games together. You’re the first and only player (other than myself) that I’ve seen build a German Mediterranean cruiser.
As to the matter of PRNGs and dice, for over a year I posted extensively on Steam forums, and I’ve made comments on these forums as well regarding the system, the implementation, what I say should have been done and which was not done (even allowing for budgetary constraints).
As to having a “more informed” opinion, I flatter myself to think that I probably have one of the most informed opinions.
Some of the things that stick out to my mind -
Over a year ago, a developer (more correctly, someone working for the developer’s company) commented to the effect it was “proved” that the PRNG had no issues. But that simply isn’t how things work. Proper statistical analysis does not prove things, it merely provides get calculated probabilities that for a given dataset gathered under a given specific set of conditions, there’s a 99.998% (or such) that that particular dataset is in accordance with a particular predictive model.
Well, that was their first mistake, but what of it? People make mistakes. Especially when people get told things by other people, there are misunderstandings, no matter how small, and just because I thought it was a very bad look, eh.
At that point I didn’t think that players would make a big thing of PRNG. I’d noted in my google document (link below) that I thought the developers ought to make data importable and exportable, but that was a matter of general utility.
https://docs.google.com/document/d/17F3TotY7HEKeiLv3ewlfYotQv_hWXqh5PDo7B0exXpY/edit?usp=sharing
You’ll see one of the first comments was dated September 11, 2019, well over a year ago. Well, all that’s an aside, though it comes up later.
But back to the story of PRNG and the developer response. Some players started saying that the PRNG was suspect, which personally I’m indifferent to (as I consider the rules changes far more an issue, which is something that I think Panther commented on, on these forums, more than a year ago? He wrote about how the differences were not advertised which is coincidentally one of the things I personally rather dislike, and in fact I’d go so far as to say the game’s features are falsely advertised, but that’s another matter.
So more than a year ago, some players said there was a PRNG issue, and some other players said there wasn’t a PRNG issue, it’s all “in your head” (though fancier terms were used). The developers COULD have responded acknowledging the legitimacy over concerns over the PRNG, but that didn’t happen. At the time.
That was the developers’ second mistake. Which, okay, again, let’s be fair here, people make mistakes. You get a program, you think you need programmers, that’s what you hire for, you hire marketers, support staff. If you don’t think you need statisticians then you don’t put them in the budget, if you don’t think you need a decent public relations team then you don’t budget for that either. Well, sometimes you get lucky and you get something good you didn’t pay for. In this case, not so lucky. But okay.
The developers were very dismissive, and about that time I wrote a bunch of posts on Steam forums about how exactly PRNGs work, how you get emergent behaviors that are not easily detected, and why concerns over PRNG were legitimate. I wrote that it could be the case that there was an issue with the PRNG, but that they wouldn’t know, as they hadn’t run the tests.
The developer response? First, they said they’d looked at whatever many number of dice rolls. Just a really very basic test, which granted should have been done as it would detect the most egregious of issues, but nowhere near what was needed or appropriate. I wrote that isn’t what was needed, you need to look at groups of dice. I wrote more posts, months passed.
The developers released another statement, stating they had run a Dickey-Fuller test, then I forget if they said they ran chi-squared or whatever. So I looked into the tests, I really looked. And the fact is, the developers were still making the same mistake (making this their third) in that they were not testing what needed to be tested. Same mistake, so why count it again? Because they had the opportunity to look at and consider their methodology and they did the same thing over again. Only this time, instead of “eyeballing” it they tried using technical gobbledygook. Which actually ISN’T gobbledygook if it was being used correctly which it wasn’t.
And all this time, there was just the most dismissive attitude by the developers, with various posters getting banned or censured - which, really, I wouldn’t complain about, except only those with criticisms of developers were clamped down on. Those that “supported” the developers were allowed to repeatedly post the most egregiously nasty sort of nonsense, and their behavior passed uncommented (and certainly uncensured).
I had quite a number of unpleasant interactions on Discord (which were probably unpleasant for others as well, considering I was pointing out their faults), and as usual, when I made specific points or asked for specific answers, what I wrote was blatantly ignored. Not in the context of “we don’t have the time to look at this right now but we’ll get back to it later”. I mean I made the same points in public and private repeatedly, and the points were IGNORED, there is no doubt about it.
Well, you could see how this was going. I knew the developers had the point made to them, and how? Because I had made the points, personally! And if it was said (fairly enough) that they didn’t work for me, that they had budgetary concerns, that they weren’t necessarily professionals in statistics or even in the scientific method, well, still, things were what they were. There were any number of good responses they could have made that weren’t made, and they chose bad responses so many times.
(And I want to be clear, sometimes there are chain of command and information flow strictures that others are not aware of. Or sometimes there are multiple “bosses” that give conflicting orders that are left for others to resolve, with the whole mess covered up by nondisclosure agreements. I’m used to that sort of thing. But the sort of pushback I was receiving, I really felt that it was on a personal level. Like it’s not someone just going along because they have to, they want to do what they’re doing.)
But about at that point, a few months ago, the developers MOSTLY stopped trying to make boasts like “it’s proved that there’s nothing wrong with the PRNG” (still popped up now and then). Maybe they’re back to their old shenanigans, since I was wrongfully banned on Steam forums I haven’t checked, and I don’t bother with the Discord I see one of their representatives touting either. But if the developers aren’t false virtue signaling about the PRNG in my face, then good enough for me.
But the developers made a fourth mistake! I’d made the point on Discord that resolving player concerns over the PRNG was a public relations issue. But by that I’d only meant they needed to have it properly looked at instead of doing a patch job, I meant they needed to be understanding of customers rather than clamping down on them and denying / attacking them. But apparently “public relations” didn’t mean to them what it means to me, and I suppose if someone’s used to an authoritarian regime and not from the West then maybe that’s natural for them, I don’t know.
But anyways, they released a statement about stabilized dice - or was that the one? Anyways at the end, they said they’d switched up their PRNG a bunch of times and they hadn’t noticed any differences in complaints. And I was like really now. They’d always been very reticent to release the details of their studies (which frankly I don’t know that they ever made a study worth publishing), and here they said they “didn’t notice any difference” - using what metrics, specifically? I say they didn’t know what the issue was, they didn’t know what to look for, then they shouted “victory!” from the rooftops without looking down, without even checking, the situation on the ground. Which was precisely what they had been doing right along, but making a point of it and attacking their customers that way and saying they were trying to run games behind the scenes and being all “tee hee!” about it was . . . I don’t even know what to say. It’s just outside my understanding of how you would ever want professional to act. I mean, it’s very normal for “attack politics”, I expect it there, but here I just don’t even know.
So would it be fair to say I have an informed opinion of their efforts? Or not? And let’s not say that I decided to close the door on the developers. They banned me, not the other way around. And the criticisms I make are not just about what happened in the past but what is happening now, what continues to happen, and what continues to NOT happen.
Do the developers need to own up to past mistakes? I don’t insist on it. Do they need to “fix” their PRNG? I don’t even insist on that. At this point, I’m only really saying that they should either implement the game as it is in the board game for the “full experience”, OR they should correct the description they have on Steam’s front page, openly acknowledging the differences between their program and Axis and Allies 1942 Second Edition. (Oh, and also I say the developers shouldn’t say things HAVE been fixed and/or properly analyzed when they have not been. I don’t think that’s too much to ask. And if players push on their claims that things have been “proved” or “disproved”, then I’ll often say something about that too.)
To close this out - I know a lot gets taken out of context, and maybe some readers will look at this and get the takeaway that I think things were done wrongly (fair enough). But they won’t understand the methodology that I’m saying should have been done. So I’ll repeat here what I’ve written I don’t even know how many times between Steam, Discord, and I think I’ve made some allusions on these forums but not a proper address. Heck, even the following isn’t a proper address.
I’ll use an analogy, as in my experience, when I write exactly how it works mathematically a lot of times readers just don’t understand. Nor do readers even understand when I explain exactly why it’s important in actual game terms. They need something to relate to.
So, cards, not dice. Think about what I’m writing, then try to understand how it applies to Axis and Allies.
Four suits, thirteen ranks, deck has fifty-two cards. Now suppose I were to say “nobody knows what a single random card draw will be!”. Well, you don’t know, precisely. But assuming conditions to be equal and the deck to have been fairly randomly shuffled then you have a one in fifty-two chance of drawing any particular card.
But already we have our first set of assumptions - that the deck is shuffled fairly and conditions are equal. A lot of people dismiss the details, but those details are important. What’s the first card you’ll draw off a fifty-two card deck that you just bought from the store? I don’t know, King of Hearts every time from a particular manufacturer? And if you shuffled and cut a deck of cards, if one of the cards has a little bend in it - no matter how slight, one that you won’t even see if you visually inspect the deck as a whole - the deck will cut at that place.
And something that a lot of readers do with dice is they assume they know how it all works, but they don’t want to be bothered with those pesky details, you know, the ones that challenge the paradigm they’ve built for themselves? But the more you know about mathematics and practical data sciences, the more you know about programming, the more you realize there MAY be issues.
Such as? Let’s say that in the LHTR setup a USSR player opens with 12 units to West Russia, 9 to Ukraine. Yes? That is the meta, is it not? But then what happens?
PRNGs do NOT generate a series of random numbers. Hence the name, “Pseudo Random Number Generator”. They take a time seed and generate a sequence of numbers and use that. And the issues that result come from how and when those time seeds are taken, and how the PRNG output is applied.
So what then happens is, without anything needing to have been deliberately programmed in (though for those that claim things could have been deliberately programmed in that’s a possibility) - but even WITHOUT deliberate programming, you get these sequences of calculated numbers with the methods of generation that result in - what? Emergent behavior.
To use another analogy, when you go to the ocean, why are there waves? Why isn’t the ocean as flat as glass? There are currents and winds that bunch up and move water in different ways, there are thresholds that are surpassed, and though each individual molecule of water may not have a predicable pattern as such, groups of water molecules do*. That is why there are waves.
So a lot of what we see is people look at the interactions between various agencies and individual molecules of water, then claim, because they can detect no pattern, that things must be “random”. Some may even go so far as to say we will see no waves in the ocean because analysis of collective and emergent behaviors is not a “thing” - it’s confusing, and it just makes sense to say very loudly and repeatedly that waves are nonsense, you have a glass of water and IT doesn’t have waves, and if you drop a pebble in your glass of water and it causes ripples, well, that isn’t a wave, it’s a ripple, and how dare you throw pebbles in my glass of water that’s just impertinent etc.
Right. But back to this comfortable simple model. You have a randomly shuffled deck, nobody’s put bends in the cards or dropped any sawdust, just a single draw.
So then we proceed to hands of cards, in, say, poker. Suppose someone were to claim that they are just “lucky”, they just HAPPEN to get royal flushes all the time, or if they’ve picked up on card sharping, maybe they just so happen to have gotten high pair a lot.
Which is an interesting story, sometimes you have people that refuse to believe that there’s anything wrong with their play or with the deck, and they end up losing all their money. And the one that takes the money is so sympathetic, you’re good at cards! really! just a run of bad luck!
In cards, you have to understand, there’s probability, then there’s being able to read and predict opponents. How does this relate to Axis and Allies?
On both sides of the argument, you have players saying there are problems or there are not problems, but who is doing the statistical studies? And I mean PROPER statistics, it’s a lot of real work.
But back to poker. So let’s say you have a player that’s “lucky” a lot. Then they say “I can’t help it! I’m just lucky!”
But you have to really think about what’s going on there. Suppose a player is ACTUALLY just lucky on a PARTICULAR night. Well, that happens. Everyone’s number comes up once in a while. But suppose someone is CONSISTENTLY “lucky”, repeatedly. Suppose it’s not just that it’s casually THOUGHT that the player is lucky, suppose you can demonstrate the truth of this luck through mathematics. Can we say “that’s just how luck works!” But that isn’t how luck works. One night sure, but repeatedly? No, that player probably has an edge. And you just have to figure out how that edge works.
And I’ve seen all sorts of players try to figure out what the “edge” is. Mirrors? Angles? And often they overlook something. Like a bit of sawdust. Switching out decks. Confederates. No matter how clever you are at cheat detection, there’s a cheat out there that’s figured out what you’re going to try, and figured out a way around it.
Well, that’s another topic. But what I’m trying to impart is the sense of coldness about it all. You have a very sharp sort of game in cards, where you have to think about things very coldly and rationally and using calculations, whether you are cheating or trying to detect cheating, and at some point if a lot of players are fed up they don’t care about calculations and there’s heck to pay. Which is another topic.
But what I’m getting at is in that cold and sharp rational game, suppose you have a player that doesn’t understand mathematics. They start saying things like “you can draw any one of fifty two cards, but nobody can know what the probability of a given hand is” (even though you can CALCULATE the probability of a given hand). I’m not trying to shame anyone here, but readers have got to understand. There’s entire different levels of understanding, you have the very sharp knowledgeable players that know what to look out for, whether they’re cheating or trying to make sure others don’t cheat, then you have players that understand the vague concepts and why a straight beats a pair (and maybe even the probabilities of each), then you have absolute rubes that hardly even understand mathematics at all, they know a straight beats a pair because everyone knows that, but the question of probability does not enter their mind. And it’s these rubes that proclaim there’s nothing wrong with a system they don’t understand, that it’s impossible that anyone’s cheating (which they wouldn’t begin to understand how to do or prevent), and though some are well-intentioned, still. You just don’t want to be taking their advice, and if they “can’t be bothered” with the details, then you want to think about that. You really do.
But how do you calculate probabilities for combinations of cards? Unlike with dice, there’s a load of literature about calculating probabilities for meaningful combinations of cards. Feel free to read up on it.
Again - it’s not that I want to jump around between dice and cards and the movement of the tides. But I’ve found again and again, when I explain the literal case and the mathematics, readers sometimes just can’t relate their everyday experience to the points I"m making. When there’s nothing for them to relate to, they think it just doesn’t apply to them for some reason. Though it does, of course.
So, hands of cards. If a particular player is drawing a lot of straight flushes, that’s going to be weird right? But why? Are we seriously going to argue that any player can draw any hand, and everything’s random and everything defies analysis? Even players that don’t understand numbers don’t hold with such truck. If you see the same player winning over and over, well, let’s just say that if there aren’t other factors involved, that player isn’t going to be with the group for long. But you understand, the question is not about a single card, it’s not about the fact hands exist, it’s about WHO gets WHAT hand. Yes?
So now we have posts by some posters - not singling anyone out in particular, there are a LOT of posts by DIFFERENT posters - that CLAIM they are all about that data and proof yet they refuse to understand the conceptual argument and even say things indicating they do not understand the argument. Then the same posters often dismiss these things that they do not understand as “emotional” or “cognitive dissonance” and sometimes they drag up fancy words to belittle those they disagree with - but what you notice is they do NOT themselves have a reasoned approach, they do NOT present DATA, or PROOFS, nor do they even extend the benefit of a reasonable doubt to others. No, they are DISMISSIVE, ABSENT PROOFS. And that just isn’t right.
(Isn’t it right? Well if you’re accepting of every point then you end up with people coming to you with weird propositions and expecting you to swallow whatever story they’re selling, so you have to be dismissive sometime. Right? That’s just how people live. Right?)
Sure. But I’m saying even so there’s a line. If it affects you personally, then be dismissive as you like. But if you have a professional obligation, then you can’t just be dismissive. And if you’re being dismissive of others, without reason then you’re on shaky ground. When people refuse to discuss things, what sort of discussion or community does that create? We don’t have to discuss everything, sure. But when posters just say “you’re wrong! fancy term saying I’m right! no proofs or explanations! you’re emotional! go home!” I think we need a lot less of that.
Back to Axis and Allies and dice.
You need a lot of data if you want to look for patterns. That’s just how it is. That isn’t me trying to push the developers to do anything, it isn’t me trying to dictate to others what is right or what is wrong, that’s just how it is.
And when you look at that dice, you need what I call “complete data”, because you need to be able to look for patterns of emergent behavior.
And now I’m going to get into the theory of the ACTUAL case. Well, briefly, anyways, I’ve written much more elsewhere.
We know that PRNGs do NOT actually generate random numbers. That’s why they’re called PRNGs, “pseudo” means fake. Not random. Get it? Like how “I can’t believe it’s not butter!” is NOT BUTTER.
Depending on the implementation there’s broadly two ways it can go, assuming a simplistic implementation (which, considering 1942 Online doesn’t even implement the board game rules, I would think a reasonable assumption.) Either a single time seed is generated for an entire game, in which it’s susceptible to reverse engineering and manipulation by players picking the order of combats and the units allocated to battles, and in which case you can get emergent behaviors that lead to clustering that creates anomalous dice results. OR you get multiple time seeds that are generated individually based on players sending requests, in which case cheaters have finer control over how they can manipulate the time seeds, and which ALSO can result in emergent behaviors.
Emergent behaviors? Remember again, PRNGs are not truly random. Remember Axis and Allies games involve a single setup - and even if you account for LHTR setup and even for preplaced bids, there would still be convergent points for any particular player.
So what happens when you have the same player doing 12 to West Russia and 9 to Ukraine, over and over again? If the dice were actually random, then the results would be whatever they would be. We would see some clustering of results that player would interpret as “good”, we would see some clustering of results that player would interpret as “bad”. But if it is a given that the dice are really random then that’s just how it is.
But how is that different to clustering observed with PRNGs? Because PRNG clustering is simply different. Suppose a player assumes that the dice are fair (and without a huge dataset and proper analysis, that’s the going assumption that players would reasonably make). But suppose the PRNG implementation and clustering of outputs is such that instead of 85% favorable results at Ukraine (including attack and retreat actions), the player instead has only 65%? Depending on the PRNG implementation we are going to possibly see such skewed results because you are relying on a PRNG that outputs a sequence of numbers on a time seed for a game that has a single static setup that encourages groupings of the same moves that results in clustering. It is NOT actually RNG, that is the POINT.
Dismissive readers will say things like “85%, 65%, what’s the difference?” Or they’ll say things like “you can still get good and bad results, so what’s the difference?”
The difference is that players assume they’re playing the game according to their model, but actually the game works on a different model, one that’s never explained to the player.
Let’s say I have a coin. I flip it. Heads you give me a dollar, tails I give you a dollar. Okay. Right? Let’s play this game a bunch of times, maybe you win, maybe you lose, whatever.
But now let’s say my coin has two heads. You could say it’s less egregious, like maybe we pull a coin out of a small jar each time, and out of ten coins in that jar only four of them have two heads. But do you think that’s the sort of information you would want to have about the game before playing? I would hope so.
Returning to the actual case, what I’m saying is you need to be able to differentiate each dice so you can put them in groups, then you need to look at the behavior of those groups. Is one player getting a load of “lucky” rolls? How do antiaircraft guns work?
And some responders say things like “they work, sometimes AA shoots bombers down”. But that totally misses the point. If you’re not reverse engineering the program to look at the specific PRNG implementation, how do you know how, exactly, how that PRNG implementation works? What if the PRNG has different implementations? Even if it’s not programmed for bias, can you guarantee there’s no bugs in the code? Even if there are no bugs and it’s not programmed for bias, have you accounted for emergent behaviors? You can’t, unless you look at the data.
Anyways in closing:
What’s in the past is in the past. But for the present, the developers should REALLY change the description on 1942 Online’s Steam page to reflect the fact 1942 Online is NOT like the board game in terms of gameplay. Or better, they could put in live play. Ooo yeah. They should do THAT. But if not, at least change the store description.
As to the PRNG, I think it’s perfectly reasonable to believe the PRNG may have issues, because as far as I know it might really have issues. I’ve seen a lot of bad “studies” and a lot of specious arguments arguing there aren’t any issues, but the fact is it’s the groupings of dice that need to be looked at, not the individual dice, or even the aggregate of individual dice taken all together, but sub-groupings, that’s just how it is, and nobody’s even attempted to do that properly that I’ve ever seen.
And if anyone wants to undertake it, better believe there’s a lot more to it than just what I’m writing here. If a proper job is to be made of it, it’ll require a lot of work.