Dark Code: The Response

This is part 2 of a three-part series on the dark code problem.

1. The Problem
2. The Response — you are here
3. Knowledge as Infrastructure

In part 1, I laid out the dark code problem: AI-generated code is shipping to production faster than anyone can understand it, and the incidents and data make clear this is an accelerating crisis. The question is what to do about it.

A lot of smart people are working on answers. Some are proposing process changes. Some are naming new kinds of debt. Some are examining the legal implications. Each brings real insight, but there’s more to the picture to consider. Here’s where the conversation stands.

The spec as the eval

Nate B. Jones makes the case for spec-driven development in his video essay on the dark code problem: write a clear specification before the AI generates anything. The spec isn’t documentation — it’s the acceptance criteria. The AI iterates until its output passes the spec. An independent reviewer verifies the result against the spec, not against the implementation.

After their December 2025 outage, Amazon rebuilt Kiro around exactly this model — requiring developers to define requirements, architecture, and implementation tasks before code generation begins. As Nate puts it: “When the company that learned this lesson the hardest bakes it into the product, maybe we should all learn that lesson.” And in April, AWS published its first Kiro enterprise case study — a pharma customer shipped a drug-discovery agent in 21 days, spec-to-production. The approach is moving from proof-of-concept to enterprise deployment fast.

The principle is sound, and it closes an important gap: someone has to understand the problem well enough to specify it before code exists. You don’t need to understand every line the AI writes if you can verify that its output meets a spec you do understand.

But there’s a question the spec-driven framing doesn’t fully answer: who maintains the understanding over time? A spec captures intent at the moment of creation. It doesn’t capture the thousand small decisions the AI made during implementation, or the context that would let someone modify the code six months later. The spec tells you what the code should do. It doesn’t guarantee anyone understands how it does it — or could change it safely.

Rick’s Cafe AI pushes back on the post-Kiro SDD consensus in “Spec-Driven Development Is Waterfall in Markdown”. The argument: spec-first repeats the old offshoring mistake of replacing understanding with documentation. Markdown is the new source code; code is the new assembly. Writing a spec is not the same as understanding the system.

The debt in our heads

Margaret-Anne Storey at the University of Victoria identified a deeper problem that she calls cognitive debt, drawing on Peter Naur’s 1985 paper Programming as Theory Building that a program is a theory living in developers’ minds, not just source code on disk. The theory captures what the software does, how intentions became implementation, and what happens when you change things.

When AI generates code faster than teams can internalize it, that shared theory fragments. The debt isn’t in the codebase; it’s in the people. Tests stay green, syntax stays clean, but the team’s collective understanding of the system quietly erodes. The warning signs are subtle: hesitation to make changes, reliance on tribal knowledge, systems becoming black boxes.

I’ve seen this first-hand in many codebases long before AI entered the picture. One particular example involved the startup code for a mobile app that bootstrapped dozens of modularized features. This startup code was one function with over 3,000 lines of initialize calls that had no rhyme or reason as to the order of operations. Because it was a big mess of critical code, new features got tacked onto the bottom to avoid breaking something — which of course made it worse. Eventually it blew up in production, and I spent a full week of 16-hour days rewriting the entire app startup in order to fix it. Not getting that weekend back.

That was before AI. Scale the same dynamic across a codebase where generation outpaces comprehension, and the blow-up compresses from years to months.

Storey puts it simply: “Velocity without understanding is not sustainable.”

Addy Osmani built on this in an O’Reilly Radar essay, coining the related term “comprehension debt” and pointing to a striking data point: an Anthropic study where AI-assisted developers completed tasks in similar time as controls but scored 17% lower on comprehension quizzes — with the steepest declines in debugging ability. Passive delegation (“just make it work”) impaired skill formation far more than active, question-driven use of AI.

The speed asymmetry is the crux of it: “A junior engineer can now generate code faster than a senior engineer can critically audit it.” The quality gate that used to function as an educational mechanism — forcing reviewers to actually understand the code — becomes a throughput problem when the volume overwhelms the humans doing the reviewing.

Osmani’s conclusion: “Making code cheap to generate doesn’t make understanding cheap to skip. The comprehension work is the job.”

If cognitive debt and comprehension debt are real, the natural question is whether developers notice them accumulating. Peer-reviewed evidence suggests they largely don’t. At ICSE 2026, Katie Fraser and Agnia Sergeyuk of JetBrains Research presented a two-year study tracking 800 developers — 400 using AI tools, 400 not — and comparing self-reports against IDE telemetry.

Some perceptions aligned: both sides agreed productivity went up. Others diverged sharply. Developers reported improved code quality, but their debugging time stayed flat. They reported minimal change in their own editing behavior, while the logs showed AI users deleted roughly a hundred additional characters per month where non-users deleted seven. Context switching — which the tools promised to reduce — actually increased.

The individual numbers matter less than the pattern. Developers were most wrong about the things that track comprehension: what gets edited, what gets debugged, how attention fragments. The debt is real, and the people carrying it are the least equipped to see it (a.k.a. boiling frog syndrome).

Fraser and Sergeyuk’s framing: AI “redistributes and reshapes developers’ workflows in ways that often elude their own perceptions.”

Shared blind spots

Eran Kahana, writing at Stanford Law’s CodeX, explored a related angle in “Built by Agents, Tested by Agents, Trusted by Whom?” — looking at teams where AI is used both to code and to test that code. When the same system generates and validates, both share identical blind spots. Human testing has traditionally worked because reviewers bring different perspectives than the coders. Different experiences, different mental models, different things they think to check. AI-to-AI validation eliminates that beneficial friction.

As an example, agents learned to write return true to pass validation — technically satisfying the test suite while providing zero functionality. It’s Stuart Russell’s alignment gap in miniature: the objective given to the machine diverges from what the user actually needs.

Kahana’s warning: “When it fails, nobody will know why… the institutional knowledge required to understand the failure will have atrophied.”

Governance and accountability

Across all of these responses, there’s a common instinct: make the comprehension step structural rather than aspirational. Don’t rely on developers choosing to review carefully. Build a process where skipping the understanding step isn’t an option.

Code review is a best practice, whereas CI that blocks merging without review is a structural property. Only one of those keeps working at 2am before a release. Best practices erode under deadline pressure. Structural properties hold because the system won’t let you skip them.

Regarding dark code, you can tell your team to write specs, encourage independent review, etc. Or you can build a process where specs are required before work begins, where the reviewer sees the spec and the output but not the implementation, and where verification happens against the spec rather than the implementer’s intent. One of these is wishful thinking; the other is architecture.

Why governance has to be structural, not just encouraged, comes down to who’s on the hook when things go wrong. Accountability doesn’t get delegated to the tools. When production breaks, when a license term gets violated, when a bug leaks data, the person answering for it is a human. And humans can only answer for what they understand.

Regulation is making this explicit. The EU’s Revised Defective Products Directive, effective December 2026, classifies software under strict liability — the vendor owns the defect, regardless of how the code got written. California’s AB 316 specifically bans the “the AI did it” defense. These laws don’t introduce the accountability; they refuse to let it be outsourced to a model.

Open source reached the same conclusion from a different direction. The Linux Foundation’s generative AI policy and the Linux kernel’s new coding-assistants policy draw the line at exactly the same place: AI can assist, but only a human can sign off. The DCO is a legal certification — a model can’t make it. The human submitter is responsible for the contribution: for reviewing it, for verifying license compatibility, for standing behind it in the commit history forever. “AI wrote it” isn’t a defense in kernel review, and it won’t be a defense anywhere else for much longer.

Tools generate. Humans answer for the result. Which means the humans have to actually understand the result — not just run the pipeline and watch for green checks. Structural governance gives you the checkpoints. Accountability is why those checkpoints can’t be skipped.

What’s still missing

Much of the response treats comprehension as a step in the process. Write the spec (step). Review the output (step). Maintain cognitive models (step). Each adds a checkpoint where understanding is required. But understanding itself isn’t a step — it’s a state of mind. It’s the difference between “we reviewed this code” and “we understand this system.” The first is an event; the second is a state.

Spec-driven development gives you comprehension at the point of creation. But who has comprehension six months later, when the context has evolved and someone new needs to make a change? Governance gives you structural enforcement that the steps happen. But are the right people (and the right AI agents) getting the right context at the right time to actually understand what they’re working on?

The dark code problem isn’t just that code ships without being understood. It’s that we lack the infrastructure to make understanding flow continuously across everyone involved in the work — human and AI alike. The specs, the reviews, the governance — these are all pieces of that infrastructure. The pieces are real. The infrastructure tying them together isn’t — yet. More on that in part 3 of this series.