There's a lot of talk about AI going bad at an existential level. Sci-fi takeover. Self-improving systems racing past us. Daniel Miessler recently made a quieter point I'd echo: the less competent lean on the more competent, and eventually the most competent thing in the room is AI.
I'd add another risk. Eventually, "just pull the plug" stops being a realistic option.
Think about the Internet.
In the early 2000s, an Internet outage was inconvenient. For most people, life went on. You could still work. Businesses could still operate. You could buy things, travel, go to school, communicate. Plenty of people might not even notice.
Today, turn off the Internet and see what happens.
Or look at what happens when CrowdStrike, Cloudflare, AWS, or another critical piece of digital infrastructure has a major failure. Flights are disrupted. Payments stop working. Employees can't access systems. Students can't turn in papers. Businesses can't serve customers.
At some point, a useful technology quietly became infrastructure.
I suspect AI will follow the same path, only much faster.
Right now, we can still talk about turning AI off as though it were an application sitting on top of society. But what happens when it is writing and maintaining software, operating networks, detecting fraud, managing logistics, allocating resources, assisting doctors, running security systems, and making millions of small decisions embedded inside other systems?
What happens when humans no longer understand some of those systems well enough to operate them without it?
Then "turn it off" isn't a safety plan. Turning it off may cause as much disruption as leaving it on.
Then "turn it off" isn't a safety plan. Turning it off may cause as much disruption as leaving it on.
And that makes the slow-path argument especially interesting. The path to AI running everything may not look like a takeover at all. It may look like a thousand perfectly rational decisions to let the more capable system handle one more thing.
Then another.
And another.
Until one day we discover that delegation has become dependency.
Delegation became dependency.
The concentration of that capability makes the problem sharper. If much of this intelligence ultimately depends on models and infrastructure controlled by a handful of companies, we aren't just creating technological dependency. We are creating enormous systemic concentration risk.
That is one reason open-source models, private AI, local inference, interoperability, and genuinely distributed AI infrastructure are going to matter so much.
Not because centralized AI is inherently bad, but because anything important enough that society cannot afford to turn it off is too important to have only a few points of failure.
AI becoming extraordinarily capable is one risk.
AI becoming indispensable may be the one that sneaks up on us.