NHacker Next
login
▲Doom crash after 2.5 years of real-world runtime confirmed on real hardwarelenowo.org
358 points by minki_the_avali 17 hours ago | 122 comments
Loading comments...
jbreckmckye 15 hours ago [-]
About a year ago I was looking at Crash Bandicoot timer systems and I found that Crash 3 has a constantly incrementing int32. It only resets if you die.

Left for 2.26 years, it will overflow.

When it does finally overflow, we get "minus" time and the game breaks in funny ways. I did a video about it: https://youtu.be/f7ZzoyVLu58

jsheard 14 hours ago [-]
There's a weapon in Final Fantasy 9 which can only be obtained by reaching a lategame area in less than 12 hours of play time, or 10 hours on the PAL version due to an oversight. Alternatively you can just leave the game running for two years until the timer wraps around. Slow and steady wins the race.

https://finalfantasy.fandom.com/wiki/Excalibur_II_(Final_Fan...

Gravityloss 3 hours ago [-]
Am reminded by this quote from Ferdinand Porsche:

"The perfect racing car crosses the finish line first and subsequently falls into its component parts."

Games fit this philosophy, compared to many other pieces of software that are expected to be long-lived and receiving a lot of maintenance and changes and evolve.

WJW 2 hours ago [-]
The Porsche quote reflects a wider design philosophy that says "Ideally, all components of a system lasts as long as the design life of the entire system and there should be no component that lives significantly longer. If there is such a component, it has been overengineered and thus the system will be more expensive to the end consumer than it needs to be.". It kinda skips over maintenance, but overall most people find it unobjectionable when stated like this.

But plenty of people will find complaints when they try to drive their car beyond its design specs and more or less everything starts failing at once.

creaturemachine 3 minutes ago [-]
Porsche was talking about racing, where the primary focus is reaching the finish line faster an anyone else, and over-engineering can easily get in the way of that goal. Back in the real world, no race team would agree that their cars should disintegrate after one race.
ortusdux 13 minutes ago [-]
Anyone can build a bridge, but it takes an engineer to barely build a bridge.
signalToNose 2 hours ago [-]
Consumer protection laws prevents businesses following this to it’s extreme. For many businesses the ideal would be to just sell stuff that immediately breaks down as soon as it’s sold. It has the fulfilled its purpose from their point of view
delichon 1 hours ago [-]
I run sous vide cookers 24*7, and they uniformly break within 90 days or less. But they don't like to admit their smaller duty cycle, so they don't, and keep sending me warranty replacements instead. I keep buying different brands looking for one with a longer life. I'll bet most people do that when their gadgets die, and purposely making products that die as soon as sold isn't often a successful business model.
cestith 8 minutes ago [-]
A friend of mine gets new headphones/headsets every six to eighteen months, and hasn’t bought a pair entirely out of pocket in years. For him it’s all down to buying the Microcenter protection plan every time they’re replaced. They fail, he takes them back, he gets store credit for the purchase price, and he buys a new set and a new plan. He doesn’t even care about the manufacturer’s warranty anymore.

Personally, most of my headphones I look for metal mechanical connections instead of plastic and I buy refurbished when I can. I think I pay about as much as he does or less, but we haven’t really hashed out the numbers together. I’m typing this while wearing a HyperX gaming headset I bought refurbished that’s old enough that I’ve replaced the earpads while everything else continues to work.

Computers and computer parts often have, in my experience, a better reliability record competently refurbished than when they first leave the factory too. I wonder if sous vide cookers would.

rlander 57 minutes ago [-]
That’s not a small cycle count for a normal household. 90 × 24 = 2,160 total hours.

I sous vide now and then, about twice a week for 6 hours each, so around 12 hours a week. That works out to roughly 15 years of usable machine time for the average person.

Not bad at all.

plywoodShadow 23 minutes ago [-]
2160/12 is 180 weeks, or roughly 3.5 years, not 15 years
josephg 46 minutes ago [-]
Photography is the same way. Most SLR / DSLR / mirrorless cameras have a mechanical shutter which is expected to last around 200k-1m activations. I've had a camera for a bit over a year. I've used it quite heavily, and my shutter count is at about 13k photos. At this rate, the shutter will probably last for 20+ years - which seems fine. If I'm still using the camera by then, spending a few hundred dollars to replace the shutter mechanism sounds totally reasonable.
47282847 51 minutes ago [-]
Assuming linearity, which I doubt is the case.
account42 48 minutes ago [-]
You think a measly 360 uses at your 6 hours typical operation is even remotely acceptable for a glorified heating element?

And yes, 15 years is bad. I don't want to replace my entire household every 15 years FFS.

account42 52 minutes ago [-]
Well from an evil business perspective their options are either

- the product doesn't break and you don't buy a replacement from them because you still have a working product

- the product breaks and there is a greater than 0% chance that you will buy a replacement product from them

Of course in practice it's more complicated but I wouldn't be so quick to declare that the math doesn't work out.

hnuser123456 49 minutes ago [-]
Are there not industrial ones meant to last longer? Maybe you can buy a used but good condition one of those.
WJW 35 minutes ago [-]
There are, and if you really have the workload that you need to cook stuff 24/7 (what in gods name is OP cooking btw?) then you should definitely get one of those. Maybe not even secondhand but just a new one. The cheap consumer grade ones are meant for people who use them once or twice a year.

This is a fine example of what I meant about people complaining when they use products beyond their design parameters.

elzbardico 1 minutes ago [-]
It is easy to have to run a bunch of sous vide cooker 24/7 if you have a small restaurant or food delivery business.
doubled112 2 hours ago [-]
When the design spec seems to be a 3 year long lease I can see why people get bothered.
lelandfe 13 hours ago [-]
So the invisible 12h timer runs during cutscenes. During Excalibur 2 runs, I used to open and close the PS1 disc tray to skip (normally unskippable) cutscenes. Never knew why that worked.

(I also never managed to get it)

jonhohle 13 hours ago [-]
I’m going to wager that the cutscenes are all XA audio/video DMA’d from the disc. Opening the disc kills the DMA and the error recovery is just to end the cutscene and continue. The program is in RAM, so a little interruption on reading doesn’t hurt unless you need to time it to avoid an error reading the file for the next section of gameplay.
ad133 5 hours ago [-]
This is a significantly better handling than the previous game (final fantasy viii). My disk 1 (it had four disks) got scratched over time (I was a child after all), and the failure mode was just to crash - thus the game was unplayable. The game had a lot of cutscenes.
Insanity 12 hours ago [-]
That’s a solid guess. And if that’s the case, that’s actually pretty good error handling!
Jare 8 hours ago [-]
I recall that handling disc eject was an explicit part of the Tech Requirements Doc (things the console manufacturer requires you to comply with). They'd typically check while playing, while loading and while streaming.
p1necone 13 hours ago [-]
> Never knew why that worked.

I'm guessing the game probably streams FMV cutscenes of the disc as they play, and the fallback behaviour if it can't find them is to skip rather than crash.

elcritch 55 minutes ago [-]
We should rally together to force game companies to use 32 bit timers rather than 64bit ones so we can keep finding these fun little glitches. The time to protect overflows is now! ;)
jbreckmckye 14 hours ago [-]
Oh yeah. The sword you pick up in Memoria. The problem there is that the PAL version runs slower; the way PSX games "translated" between the two video systems was just to have longer VSync pauses for PAL. So the game is actually slower, not interpolated
reactordev 14 hours ago [-]
Longer vsync pauses but larger frame time deltas so it’s basically the same speed of play. The only thing that was even noticeable was the UI lag.
fredoralive 9 hours ago [-]
Erm. No, like lots of games during the era quite a lot of stuff is tied to the frame rate, so the 50Hz region game just runs slower than the 60Hz one as next to nobody bothers to adjust for it. The clock for the hidden weapon does run at the same rate for both unfortunately, hence it being harder to get in 50Hz regions.
reactordev 6 hours ago [-]
Incorrect. I’m looking at the source code. It’s not perfect but it’s not just “slowed down to 50hz” like people claim.
jbreckmckye 5 hours ago [-]
When you say looking at the source code, what do you mean here?

AFAIK the source for FF9 PSX (and all the PSX ff games) has been lost as Square just used short term archives

Also, FF9 does not run at a constant framerate. Like all the PSX FF games it runs at various rates, sometimes multiple at a time (example: model animations are 15fps vs 30 for the UI)

In terms of timers, the bios does grant you access to root timers, but these are largely modulated by a hardware oscillator

(Incidentally, the hardware timing component is the reason a chipped PAL console cannot produce good NTSC video. Only a Yaroze can support full multiregion play)

anthk 2 hours ago [-]
FF VII-IX were reimplemented under a custom engine.
reactordev 3 hours ago [-]
It’s definitely not lost…
jbreckmckye 2 hours ago [-]
What code are you looking at?

FFIX for PSX would have been written in C (or possibly C++) with PSY-Q. It will not be one program - those games were composed of multiple overlays that are banked in / out over the PlayStation's limited memory.

From what I know the PC release was a port to a new framework, which supports the same script engines, but otherwise is fresh code. This is how it can support mobile, widescreen, Steam achievements etc.

mungoman2 8 hours ago [-]
Wouldn't a slower tick make it easier as you get more wall time to do the same challenge.
fredoralive 8 hours ago [-]
No? Wall time (that the challenge runs on) is unchanged, game time (Vsync) is running at 83% of full speed (50Hz vs 60Hz), so if something tied to frame rate (animation, walking speed etc.) takes 1 second to do on NTSC, it'll take 1.2 seconds to do on PAL etc.
7 hours ago [-]
BolexNOLA 3 hours ago [-]
Lord have mercy fandom really has become unbearable with the ads and pop ups.
coldpie 1 hours ago [-]
Install an ad blocker.
BolexNOLA 1 hours ago [-]
I opened this on an iPhone which has fewer adblock options. Desktop is better locked down.

Regardless I can still complain about how intrusive the ads are.

account42 4 minutes ago [-]
Don't accept devices that limit your ad blocker options.
coldpie 57 minutes ago [-]
There are many ad block options on iPhone. I currently use Wipr 2, but in the past I've used both 1Blocker and AdBlock Pro with success.
JustExAWS 43 minutes ago [-]
I just opened this my iPhone with 1Blocker installed. I saw no ads. It’s been around since iOS 8
debo_ 14 hours ago [-]
So that's why it's called Excalibur 2!
xhrpost 24 minutes ago [-]
Is it common to default to a signed integer for tracking a timer? I realize being unsigned it would still overflow but at least you'd get twice the time, no?
stevage 14 hours ago [-]
You really managed to make the whole video without making a single "crash" pun? (Those freezes come close enough that you could call them crashes...)
jonhohle 13 hours ago [-]
I think many games were that way. SotN definitely has a global timer. On a native 32-bit system it makes sense, especially when the life of a game was a few months to a few years on the retail shelf. No player is going to leave their system running for 2.27 years so what’s the point of even tesing it?

Who knew at the time they were creating games that would be disassembled, deconstructed, reverse engineered. Do any of us think about that regarding any program we write?

account42 2 minutes ago [-]
For some games the timer is stored is save files so it doesn't even have to be continuous play time. 2 years is still longer than anyone is expected to spend on a game.
Gamemaster1379 12 hours ago [-]
Can be more than timers too. There's a funny one in Paper Mario where a block technically can be hit so many times it'll reset and award items again. Hit enough times it'll eventually crash. Of course it'd take around 30 years for the first rollover and 400 or so for the crash. https://n64squid.com/paper-mario-reward-block-glitch/
technion 6 hours ago [-]
Let's say youre pedantic with code. Ive been trying to be lately - clippy has an ovefflow lint for rust i try to use.

Error: game running for two years, rebooting so you cant cheese a timer.

Does this make the bug any better handled? Bugs like this annoy me because they arent easily answered.

jraph 8 hours ago [-]
Isn't this common in the computer game scene? Shouldn't you asume your game will be disassembled, deconstructed, reverse engineered?

Although for old games released before internet was widespread in the general population, it might have not been this obvious.

sim7c00 6 hours ago [-]
aslong as it doesnt lead to online cheats having such code is fine. if someone wants to reverse the game find an obscure almost untriggerable bug and then trigger it or play with it. 2.6 year game session is crazy if its not a server, and if its a server, thats still really crazy even for some open-world open-ended game... its a long time to keep a server up w/o restarts or anything (updates?).

looking at the various comments, there might be even some kind of weird appeal to leave such things in your game :D for people to find and chuckle about. it doesnt really disrupt the game normally does it?

lentil_soup 6 hours ago [-]
they're still made like this. Just now I made a frame counter that just increments every frame on a int64. It would eventually wrap around but doubt anyone will still be around to see it happen :|
ThrowawayTestr 11 hours ago [-]
Great video, just subscribed
glitchc 16 minutes ago [-]
I love the post, but your blurry text is hurting my eyes. Looks like it's intentionally blurry but I can't figure out why. This can't be a holdover from older systems, they had razor-sharp text rendering on CRTs.
patchtopic 13 minutes ago [-]
I haven't opened my DOOM software box, it's still in the shrinkwrap. I guess I can take it back and ask for a refund now?
12 minutes ago [-]
shultays 6 hours ago [-]
Does that hardware traps overflows or something?

  I had read an article about how DOOMs engine works and noticed how a variable for tracking the demo kept being incremented even after the next demo started. This variable was compared with a second one storing its previous value

Doesn't sound like something that would crash, I wonder what was the actual crash
Sharlin 3 hours ago [-]
Signed overflow is undefined behavior in C, so pretty much anything could happen. Though this crash seems to be deterministic between platforms and compilers, so probably not about that. TFA says the variable is being compared to its previous value, and that comparison presumably assumes new < old cannot happen. And when it does, it could easily lead to eg. stack corruption. C after all happily goes to UB land if, for example, some execution path doesn’t return a value in a function that’s supposed to return a value.
phkahler 2 hours ago [-]
That doesn't make sense. If new < old cant happen there is no need to make a comparison. Stack corruption? Nah, its a counter not an index or pointer or it would fail sooner. But then what is the failure? IDK
jraph 2 hours ago [-]
Assuming new > old doesn't mean you actually make the comparison, but rather that the code is written with the belief that new > old. This code behaves correctly under this assumption, but might be doing something very bad that leads to a crash if the new < old.

An actual analysis would be needed to understand the actual cause of the crash.

Sharlin 54 minutes ago [-]
Um, there are the cases new == old and new > old. And all the more specific cases new == old + n. I haven’t seen the code so this is just speculation, but there are plenty of ways how an unexpected, "can never happen" comparison result causes immediate UB because there’s no execution path to handle it, causing garbage to be returned from a function (and if that garbage was supposed to be a pointer, well…) or even execution never hitting a `ret` and just proceeding to execute whatever is next in memory.

Another super easy way to enter UB land by assuming an integer is nonnegative is array indexing.

  int foo[5] = { … }
  foo[i % 5] = bar;
Everything is fine as long as i isn’t negative. But if it is… (note that negative % positive == negative in C)
Insanity 13 hours ago [-]
Literally unplayable, someone should fix that.

Doom is actually such a good game, I always go back to it every few years. The 2016 reboot is also pretty fun, but the later two in the series didn’t do it for me.

jama211 9 hours ago [-]
Same. Something about the metroidvania design with the home hub of the later ones didn’t give the same feeling. It should be run, kill, find secrets, end, next level.
jeffwask 20 minutes ago [-]
I just finished Robocop: Rogue City and it was exactly this a linear level by level shooter that felt like a pure Robocop power fantasy movie. I played new game plus it was so much fun and I never do that.

It's like the game industry got a fake memo saying no one wanted linear story-based games anymore. I ended up buying two more Teyon games because I was so happy with their formula and they are playable in a dozen or so hours. Tight, compact, linear, fun story and game play... No MTX or always online BS and they don't waste my time with busy work.

jjbinx007 7 hours ago [-]
This caters for people who prefer the classic Doom style of gameplay in FPS games:

https://www.reddit.com/r/boomershooters/

Insanity 1 hours ago [-]
Ahh yes, I'm quite happy that a few years ago this has become a trend!
xmonkee 12 hours ago [-]
Same. And love those brutality mods.
shpongled 11 hours ago [-]
2016 remains one the greatest single player FPS games I've played (Titan Fall 2 is the other)
bitwize 12 hours ago [-]
Fun fact: Doom is now a Microsoft property, along with Quake, StarCraft, WarCraft, Overwatch, all of the adventure games from Infocom and Sierra, and of course Halo. Microsoft pretty much owns most of PC gaming. Which is what they've wanted since 1996 or so.
kodarna 8 hours ago [-]
They own the past of PC gaming, as well as Call of Duty but that is more popular on consoles than PC nowadays. Those listed are small time compared to Counter-Strike 2, Dota 2, League of Legends, Valorant, Roblox, Apex Legends, Marvel Rivals and a number of hard-hitting games every year such as Witcher 3, Elden Ring, Baldur's Gate 3 etc.
Novosell 5 hours ago [-]
They own Minecraft as well.
nurettin 9 hours ago [-]
> Microsoft pretty much owns most of PC gaming.

So valve next?

Lightkey 8 hours ago [-]
They missed that window when Sierra was still the publisher for Half-Life. Besides, Valve is not a publicly traded company and Gabe Newell as former manager at Microsoft has no interest in getting back together. Valve is betting everything on Linux right now to be more independent from Microsoft.
lukan 4 hours ago [-]
"Valve is betting everything on Linux right now"

Not everything, but they do invest in it.

tomwojcik 8 hours ago [-]
As long as Gabe is alive, no way.
HeckFeck 6 hours ago [-]
We must find a way to extend his life indefinitely.
pizza234 5 hours ago [-]
I'm under the impression that since Doom Eternal (the first after Doom 2016), the gameplay has considerably shifted to an "interconnected arenas" style, and with more sophisticated combat mechanics. Many games have started adopting this design, for example, Shadow Warrior 3.

I also dislike this trend. As a sibling comment noted, boomer shooters are generally closer to the old-school Doom gameplay, although some are adopting the newer design too.

billyp-rva 3 minutes ago [-]
The enemy cap all but forces the arena style gameplay. Doom 2016 tried to hide it more, but it still felt very stifling.
spjt 11 hours ago [-]
Just be glad you knew what the bug was before you started. After 2.5 years... "Shit, I forgot to enable debug logging"
jraph 8 hours ago [-]
Notably, DOOM crashed before Windows CE.
wingi 6 hours ago [-]
Yes, great achivement!
qiine 45 minutes ago [-]
In games I worked on I use time to pan textures for animated FX.

After a few hours precision errors accumulate and the texture become stretched and noisy, but since explosions are generally short-lived its never a problem.

Yet this keep bothering me..

LorenDB 12 hours ago [-]
Since we've hugged the site to death, have an archive.org link: https://web.archive.org/web/20250916234009/https://lenowo.or...

Sadly it appears that archive.org didn't capture all of the site formatting, but at least the text is there.

JoshGlazebrook 15 hours ago [-]
2038 is going to be a fun year.
jonhohle 13 hours ago [-]
That seems much closer than it did in y2k.
aaronbrethorst 8 hours ago [-]

    [ 25 ] Now [ 13 ]
yep
kevin_thibedeau 13 hours ago [-]
Everybody is sleeping on 2036 for NTP. That's when the fun begins.
wiredpancake 12 hours ago [-]
Assuming correct implementation of the NTP spec and adherence to the "eras" functions, NTP should be resistant to this failure in 2036.

The problem being so many micro-controllers, non-interfaceable or cheaply designed computers/devices/machines might not follow the standards and therefore be susceptible although your iPhone, Laptop and Fridge should all be fine.

pjc50 5 hours ago [-]
Fixing that is my retirement plan.
piker 6 hours ago [-]
Props again to the id team. No doubt something like that engineered by most folks today would have died long before the 2 year mark due to memory fragmentation if not outright leaks.
Zobat 7 hours ago [-]
This is a level of testing that exceeds what the testers I know commit to. I myself was annoyed the five or so times yesterday we had to sit and wait to check the error handling after a 30 second timeout in the system I work on.
ustad 8 hours ago [-]
Was this specific to the PDA port or the core doom code?

@ID_AA_Carmack Are you going to write a patch to fix this?

otikik 4 hours ago [-]
Quick! John Carmack needs to be brought into this immediately.
0cf8612b2e1e 15 hours ago [-]
I am going to need to see this replicated before I can believe.
serf 15 hours ago [-]
The easy way to e-Nostradamus predictions:

"See this crash?

I predicted it years ago.

Don't ask me how, I couldn't tell you."

p.s. I had an old iPaq that I wouldn't have trusted to run for longer than a day and stay stable, kudos for that at the very minimum.

prmoustache 8 hours ago [-]
I had an iPaq for a while and I don't remember seeing OS/hardware crashes.
kwertyoowiyop 2 hours ago [-]
CNR. Please attach video.
jeffrallen 10 hours ago [-]
This headline gave me a heart attack... I misread the site's name as Lenovo, and as I'm responsible for a whole lot of their servers running for years in a critical role... heart attack.

Maybe I need my morning coffee. :)

minki_the_avali 8 hours ago [-]
I mean I wouldn't mind getting a subdomain there but I do like lenowo more :3
ranger_danger 15 hours ago [-]
Seems to be a PocketPC port of Doom, with no source given or even a snippet of the relevant code/variable name/etc. shown at all.
unixhero 15 hours ago [-]
Yes. I think it it seems like it was the os that overflowed, and not Doom in this case.
nomel 15 hours ago [-]
It's also running on very old hardware, potentially with some electrolytic capacitors that have dried up. And, there's always the possibility that it's a gamma ray [1]!

[1] https://www.bbc.com/future/article/20221011-how-space-weathe...

jama211 9 hours ago [-]
They explained it was in the game code though?
unixhero 6 hours ago [-]
To me, that error message was caused by some panic, and then the OS began gracefully shutting down the application in this case DooM - which would not have been done by the program itself. Therefore I conclude it was the OS.

I am not an OS developer, so I take my own conclusion with a grain of salt.

15 hours ago [-]
johnjames87 6 hours ago [-]
Literally unplayable
17 hours ago [-]
DeathArrow 9 hours ago [-]
It's good it didn't took a billion years to overflow. That would have been quite a long wait.
moomin 7 hours ago [-]
Literally unplayable.
casey2 12 hours ago [-]
Has this ever come up in a TAS of custom levels?
ZsoltT 10 hours ago [-]
glitchless?
sunrunner 15 hours ago [-]
Not a comment on the post, but I sure wish Jira would load even half as quickly as this site.
antsar 15 hours ago [-]
It takes serious hardware investment [0] to pull that off.

[0] https://lenowo.org/viewtopic.php?t=28

fifteen1506 6 hours ago [-]
Meta-Meta-Meta:

Update:

After the recent hacker news "invasion", I have now determined that the page can handle up to 1536 users before running out of RAM, meaning that the IP camera surprisingly is fully sufficient for its purpose. In other words, I will not be moving the forum in the near future as 32 MB of RAM seem to be enough to run it

Source: https://lenowo.org/viewtopic.php?t=28

skilled 8 hours ago [-]
> Host it on the Fritzbox 7950 instead?

It's a router.. oh my god that made me laugh

andrewinardeer 14 hours ago [-]
Perhaps it's hosted on a disposable vape?
gmane 13 hours ago [-]
Commenting on my Epic from an LG Fridge.
minki_the_avali 5 hours ago [-]
[dead]
stevage 14 hours ago [-]
It's not loading for me at all.
9dev 11 hours ago [-]
We recently moved to Linear and couldn’t be happier, can recommend!
hughes 14 hours ago [-]
Is this a joke because the site isn't loading at all?
sunrunner 14 hours ago [-]
At the time of writing the comment it was practically instantaneous for me and the comment was genuine. Now it seems to be having trouble and I'm choosing to retroactively make the comment a joke about Jira ;)
SpicyUme 14 hours ago [-]
Came back to check this since the tab never loaded. I'm guessing traffic caused some issues?
Insanity 12 hours ago [-]
I’m guessing HN hug of death. Probably smarter than just auto scaling to handle any surge traffic and then get swamped by crawlers & higher bills.
minki_the_avali 8 hours ago [-]
You folks overflowed the 32 MB of RAM that my forum is running on and caused it to restart a few times due to the high amount of simultaneous connections. It has recovered now though
fifteen1506 6 hours ago [-]
It just supports 1536 concurrent users [0].

Which is fine unless you get to HN frontpage.

[0] https://lenowo.org/viewtopic.php?t=28

14 hours ago [-]