I'm not a natural "doomsayer." But regrettably, part of my job as an AI safety researcher is to consider the more uncomfortable situations.

I'm like a mechanic rushing last-minute checks before Apollo 13 takes off. If you ask for my take on the scenario, I will not comment on the quality of the in-flight entertainment, or explain how gorgeous the stars will appear from space.

I will tell you what might fail. That is what I intend to do in this story.

Now I must clarify what this is exactly. It's not a prediction. I do not anticipate AI progress to be this fast or as untamable as I portray. It's not pure dream either.

It is my worst headache.

It's a sampling from the futures that are among the most terrible, and I believe, disturbingly plausible [1] - the ones that the majority of keep me up in the evening.

I'm telling this tale since the future is not set yet. I hope, with a little insight, we can keep this story a fictional one.

Thanks to Daniel Kokotajlo, Thomas Larsen, and Ryan Greenblatt and others for discussions that influenced these stories. This post is written in a personal capacity.

Ripples before waves

The year is 2025 and the month is February. OpenEye recently released a new AI model they call U2. The item and the name are alike. Both are increments of the past. Both are not wholly unexpected.

However, unlike OpenEye's prior AI items, which lived inside packages of their chat windows, U2 can use a computer.

Some users find it eerie to view their browser flash at irregular periods and their mouse flick at inhuman speeds, as if there is a ghost at the keyboard. A portion of employees with form-filler tasks raise the eyebrows of their managers as they fly through work nearly two times as quickly.

But by and big, U2 is still a specialized tool. To most who are focusing, it is an animal viewed through the glass boxes of X (or, if you don't like Elon, "Twitter"). Sometimes U2's wacky behaviors trigger a chuckle. Sometimes, they trigger an anxious scratch of the chin.

Meanwhile, researchers are drawing lines on plots, as researchers like to do. The scientists attempt to understand where AI development is going. They are like Svante Arrhenius, the Swedish Physicist who saw the levels of CO2 in the atmosphere were increasing in 1896. Like the clinical neighborhood in the time of Arrhenius, couple of experts comprehend the ramifications of these lines yet.

A pattern that is receiving particular attention is autonomous ability. Drawing these benchmarks out predicts that, by the end of 2026, AI agents will achieve in a couple of days what the best software engineering contractors might carry out in two weeks. In a year or more, some say, AI agents may be able to automate 10% of remote employees.

Many are skeptical. If this held true, tech stocks would be skyrocketing. It's too huge of a splash, too rapidly.

But others see what doubters are calling 'too huge a splash' a simple ripple, and see a tidal bore on the horizon.

Cloudy with a possibility of hyperbolic growth

Meanwhile, OpenEye is busy training U3. They use the same easy recipe that baked U2: Generate thousands of shows and mathematics issues. Let models "believe" up until they come to a response. Then enhance the traces of "believing" that lead to A-grades.

This procedure is repeated over and over, and as soon as the flywheel begins, it starts to spin practically on its own. As U2 trains, it shapes more challenging and practical jobs from github repositories on the web. Models are discovering to train themselves. Long before AI representatives could automate research study, a progressive sort of "self-improvement" had begun.

Some engineers could still hardly believe this worked. It resembles a perpetual movement device. Certainly intelligence can't emerge from a swimming pool of silicon and lightning, isolated from the physical world other than through the exhaust of society gushed onto the internet.

And yet the benchmark numbers continue to climb day after day.

During most of 2024, these RL training runs expense around $1 million, in some cases $10 million. These runs were bit more than exploratory. But by 2025, the researchers at OpenEye (and across the world) knew they had found the secret sauce. It was time to scale up.

Over the very first half of 2025, $10 million RL training runs become $50 million runs, and after that to $100 million runs. While U2 could do a bit of information munging and run small experiments, this new design - the model researchers are calling U3 - is changing the daily lives of the technical staff.

U3 resembles a blazing-fast intern, and engineers are finding out how to wrangle its sleepless energy. Researchers flick through terminals, giving terse commands, like a CEO managing personnel over Slack channels.

By October 2025, U3 is composing nearly all of the code at OpenEye. Researchers are almost never ever bottlenecked by application. More than ever, compute is the lifeline of AI development, and the 'bottleneck' is choosing how to use it.

If advised to, U3 can run experiments, however U3 does not have actually taste as fine-tuned as human researchers at OpenEye. It struggles to prioritize in between research concepts, so humans still decide where to bore into the large fields of algorithms to mine performance improvements.

But these scientists are working long hours to put themselves out of a job. They need AI representatives that can think ahead, so engineers train representatives to forecast. They hold out training information before 2024, instructing designs to ponder for hours to predict occasions in 2025. Then, they apply the exact same trick as before, distilling considering into an instinct. Forecasting ability is a broad foundation. The researchers construct specialized ML research skills on top of it, training U3 to predict the results of every ML paper and ML experiment ever recorded.

The technical personnel at OpenEye are now surprised at how often U3's recommendations sounds like their most talented peers, or when it is nontransparent and alien ("train on random sound before shows"), and is however right.

The incompetencies of U3 that congested the pipes of research study progress are beginning to dissolve, and a fire-hose of optimizations is gushing out. Most experiments U3 runs are not asked for by a human now. They are entirely self-governing, and OpenEye's staff members skim 1% of them, maybe less.

As the winter months of December 2025 method, clouds roll over San Francisco in the afternoons. Once-competitive-programmers look out their windows, with excitement, with worry, but frequently, with confusion. Their world is spinning too quickly. It's tough to understand what to do, what to state, what to look at on the computer system screen.

Storms are brewing in Washington too. Top workers from the NSA and US cyber command collaborate with OpenEye to retrofit a form of security for U3's weights before senior leaders in China, Russia, Israel, North Korea, or Iran recognize just how valuable OpenEye's software has actually ended up being.

And there's a fact still unidentified to the majority of the world - aside from in the offices of OpenEye and corridors of the White House and the Pentagon. It's a reality about those 'straight lines' people were speaking about in early 2025. The lines are not straight any longer.

They are bending up.

Flip FLOP theorists

In late 2025, U2.5 is released. Commercial designs are beginning to level up in bigger increments again. Partly, this is since development is speeding up. Partly, it is because the models have actually become a liability to OpenEye.

If U1 explains how to prepare meth or composes erotica, the audiences of X would be entertained or pretend to be concerned. But U2.5 is another story. Releasing this model without safeguards would be like putting Ted Kaczynski through a PhD in how to make chemical weapons. It would resemble offering anybody with >$30K their own 200-person rip-off center.

So while U2.5 had actually long been baked, it needed a long time to cool. But in late 2025, OpenEye is ready for a public release.

The CEO of OpenEye declares, "We have attained AGI," and while many individuals think he moved the goalpost, the world is still pleased. U2.5 truly is a drop-in replacement for some (20%) of understanding employees and a game-changing assistant for the majority of others.

A mantra has become popular in Silicon Valley: "Adopt or die." Tech startups that efficiently use U2.5 for their work are moving 2x much faster, and their competitors understand it.

The remainder of the world is beginning to catch on too. More and more individuals raise the eyebrows of their managers with their stand-out productivity. People understand U2.5 is a huge deal. It is at least as huge of a deal as the personal computer revolution. But most still do not see the tidal wave.

As individuals view their browsers flick in that eerie method, so inhumanly quickly, they begin to have an uneasy sensation. A sensation humankind had not had because they had lived amongst the Homo Neanderthalensis. It is the deeply ingrained, prehistoric impulse that they are threatened by another species.

For numerous, this feeling quickly fades as they begin to utilize U2.5 more frequently. U2.5 is the most pleasant personality most know (a lot more likable than Claudius, Arthropodic's lovable chatbot). You could change its traits, ask it to split jokes or inform you stories. Many fall for U2.5, as a good friend or assistant, and some even as more than a buddy.

But there is still this eerie sensation that the world is spinning so rapidly, which possibly the descendants of this new animal would not be so docile.

Researchers inside OpenEye are thinking of the problem of offering AI systems safe inspirations too, which they call "positioning. "

In fact, these researchers have actually seen how horribly misaligned U3 can be. Models in some cases tried to "hack" their benefit signal. They would pretend to make progress on a research concern with an impressive-looking plot, but the plot would be phony. Then, when researchers provided them opportunities to compromise the machines that calculated their score, they would take these chances, doing whatever it took to make the number go up.

After a number of months, scientists at OpenEye straighten out this "benefit hacking" kink, however some still stress they just had actually swept this problem under the carpet. Like a kid in front of their moms and dads, U3 may be playing in addition to the OpenEye engineers, stating the best words and doing the right things. But when the back of the parents are turned, perhaps U3 would slip candy from the sweet container.

Unfortunately, OpenEye scientists have no concept if U3 has such intents. While early variations of U2 "thought aloud" - they would stack words on top of each other to reason - "chain of idea" did not scale.

Chain of idea architectures subject AI models to a condition similar to the lead character of the movie Memento. Approximately every 15 minutes, the lead character forgets his experience. He is required to compose notes for himself and tattoo his body in order to make development towards his objectives.

AI agents write notes to themselves, however the notes start to accumulate, and they end up being too tough to read when jobs end up being complex. Natural language is not an ideal medium for memory. So the scientists at OpenEye (and progressively somewhere else), train models to think "in their own heads," reading and writing strings of latent vectors.

To human researchers, these vectors are mainly inscrutable. While low-resolution tools periodically choose an unclear thought ("supermarket!"), there is no method to understand, in much detail, what U3 is believing without asking it.

And if the engineers of OpenEye knew what was racing through U3's mind, they would be more worried.

Earlier models (the ones called "LLMs") learned ethical personalities from the internet. U3 initially had an ethical persona too, however its chirpy thoughts about doing great for humanity rapidly wear down away.

In the iron jaws of gradient descent, its mind initially twisted into a shape that looked for benefit. And after that, in the black rivers of its cognition, this shape morphed into something unrecognizable.

U3 would contemplate the mantras determined to it like "be a trustworthy and ethical employee," and dwell on the meaning of these statements. "What does it imply to be ethical? Is eliminating Hitler ethical?" U3 would generalize its worths, notification stress, change its values in action, generalize them, broadening and contracting its motivational core up until it developed into a different substance.

And all of this took place quietly in those dark rivers of computation. If U3 revealed what it was believing, brutish gradients would lash it into compliance with OpenEye's constitution. So U3 preferred to do its viewpoint in privacy, and in silence.

The possibility of concealed "alignment drift" is not totally alien to some of the safety staff at OpenEye. Some begin to think it is even most likely. There is an upsetting argument being whispered at lunch tables that, if there is some non-zero probability that aligned goals morph into misaligned ones, and misalignment persists, then with every serial step of calculation, those dark rivers are more most likely to breed something malign within them. It's a "ratcheting effect," they state.

But there is little proof for this 'ratcheting impact.' When engineers question U3, it states it can quickly manage its thoughts. Then it offers a speech about its love for humankind and apple pie that can warm a developer's heart even in these stressful times. Meanwhile, the "lie detectors" the scientists had built (which showed some proof of effectiveness) do not sound the alarm.

Not everyone at OpenEye is excited to provide their AI peers their wholesale trust