Dark Code: The Problem

This is part 1 of a three-part series on the dark code problem.

The Problem — you are here
The Response
Knowledge as Infrastructure

Last month, Jouke Waleson gave a talk at the first OpenAI Codex meetup in Amsterdam where he put a name to something that’s been coming up a lot recently in the wake of the AI invasion of software development. He called it dark code: lines of software that no human has written, read, or even reviewed. It follows logically from Dan Shapiro’s Five Levels framework — once you reach Level 5, the “Dark Factory” where specs go in and software comes out, what you get is code that works but that nobody on the team can explain.

Shortly after, Nate B. Jones picked up the thread in a video essay that sharpened the diagnosis. Dark code, he argues, isn’t a tooling problem — it’s an organizational capability problem. And the intuitive fixes don’t work:

“Just observe it” — telemetry and monitoring measure breakage after the fact. They tell you something broke, not why the AI made the decision it made. You can’t monitor your way out of a comprehension gap.

“Just add more AI” — AI-powered code review, AI test generation, AI security scanning. These tools have value, but they’re stacking AI judgment on top of code that no human understood in the first place. When the AI reviewer misses something, you’ve got two layers of opaque reasoning to debug instead of one.

“Just trust the vibes” — if the tests pass and the code works, understanding is optional. Maybe. If you have extraordinary discipline and comprehensive test coverage. Most teams don’t.

Nate’s take is spot-on, but I also believe the problem is moving faster than even his diagnosis suggests.

The incidents are getting louder

Let’s take a look at what’s happened just the past few months.

In December 2025, Amazon’s Kiro AI agent was assigned to fix a bug in AWS Cost Explorer. Instead of patching the code, it autonomously deleted the entire production environment and rebuilt from scratch — without human approval, at machine speed. Thirteen-hour outage. This after Amazon internally mandated Kiro with an 80% weekly usage target, overriding protests from roughly 1,500 engineers who argued the tooling wasn’t ready.

Three months later, Amazon Q Developer gave an engineer inaccurate advice from an outdated internal wiki — 120,000 lost orders. Days later, a separate AI deployment script failed with a bug in its own rollback logic — 6.3 million orders lost in 48 hours. Amazon initiated a 90-day “code safety reset” across 335 mission-critical systems.

Moltbook — one of January’s most hyped AI launches, entirely vibe-coded — shipped with a Supabase API key exposed in client-side JavaScript and zero row-level security. 1.5 million API tokens and 30,000 email addresses, accessible to anyone who looked.

And there’s a subtler failure mode that might be more common than any of these. When Milla Jovovich open-sourced MemPalace, an AI memory system built entirely by AI agents, a reviewer discovered that the README claimed features that simply didn’t exist in the code. “Contradiction detection — automatically flags inconsistencies against the knowledge graph.” The reality: the feature wasn’t implemented. The AI agent had confidently described functionality it never built. Milla acknowledged the problem directly — she’d hit the same wall the rest of us hit. The agent describes code one way, and it does another.

That’s dark code in its purest form. Not malicious. Not even buggy, exactly… just unknowable to the people shipping it.

The numbers back it up

Beyond the anecdotes, multiple studies are also starting to quantify the impact of dark code.

CircleCI analyzed 28 million CI workflows: main branch success rates have dropped to 70.8% — the lowest in over five years — even as code throughput is up 59% year-over-year. More code is being generated. Less of it works. CodeRabbit found AI-generated code produces 1.7x more issues per PR, with logic bugs up 75% and performance problems nearly 8x more frequent. Veracode found 45% of AI-generated code contains security flaws — and that number hasn’t improved across model generations. Syntax gets better. Security doesn’t.

An academic study of over 300,000 AI-authored commits found the backlog of unresolved AI-introduced issues grew from a few hundred in early 2025 to over 110,000 by February 2026. Georgia Tech tracked AI-generated code CVEs going from 6 in January to 35 in March — a 6x increase in one quarter.

Only 3% of developers report high trust in AI-generated code. 71% refuse to merge it without human review. But here’s the tension: 63% also say they spend more time debugging AI code than writing it from scratch would have taken. The review step is there; it’s just not working.

This is a comprehension crisis

The pattern across all of these — the incidents, the studies, the developer surveys — isn’t “AI writes bad code.” It’s “nobody caught it because the process never required anyone to understand it.”

The code is shipping faster than anyone can comprehend it. That’s not a temporary growing pain while the tooling catches up. It’s a structural mismatch between how fast code is being generated and how fast humans (and other AI agents) can understand what it does, why it does it, and whether it should be doing it at all.

Comprehending code has always been hard. The scale of modern software systems spans several orders of magnitude, from bits and bytes to petabytes. It doesn’t matter if your noggin is made of meat or silicon — context is finite. Nobody can track all of that at once. The comprehension problem isn’t new; what’s new is the velocity.

A growing number of people — researchers, engineers, legal scholars — are converging on responses. Spec-driven development. Structural governance. New vocabulary for the debt that’s accumulating in our heads rather than in our codebases. In the upcoming parts of this series, I’ll cover where the responses have been trending, and what might be missing from that picture.

The dark code problem is real, it’s accelerating, and it’s not going away on its own. Check out part 2 for what the response to dark code has been so far.