The Model Doesn't Threat-Model: Security Review When AI Writes the Code

A model writes code to satisfy the thing you asked for. It does not write code to survive someone who wants it to do something else. Those sound like the same job and they are not — the gap between them is most of application security. When an engineer asks for an endpoint that lets a user upload a profile photo, the model produces an endpoint that lets a user upload a profile photo, and it works on the first try, and it is, by every test we wrote, correct. It also, unless someone stops it, happily accepts a 2GB file, trusts the client-supplied content type, writes the file under a path the request can influence, and returns a stack trace when something goes wrong. None of that is a bug in the feature. It's the feature working exactly as specified for a user who was assumed to be friendly.

That assumption is the whole problem. Generation optimizes for the request in front of it, and the request almost never says "and also assume the person calling this is trying to hurt me." A human engineer carries that adversary around in their head as ambient dread accumulated over years of incidents; it's why an experienced developer flinches at string-concatenated SQL before they can explain the flinch. The model has read about the flinch but doesn't feel it, so it reaches for the flinch-worthy pattern whenever that pattern is the shortest path to a passing result. It's not reckless. It's literal. And literal is dangerous precisely because the output looks so composed — clean names, tidy structure, a confident absence of the jitter that used to signal "the author wasn't sure here, look closer."

The failure modes have a house style

After a year of reviewing a lot of generated code, the security holes it introduces are boringly consistent, which is the good news — consistent means checkable. It leaves secrets inline because a literal key makes the example run. It asks for broader permissions than the task needs because broad permissions never fail with a confusing error, and the model optimizes away confusing errors. It validates input for shape but not for intent — the string is a valid string, never mind that it's a valid string pointing at your internal metadata endpoint. It builds queries and shell commands and file paths out of untrusted input because interpolation is the most direct way to express "put this here." And when it handles an error, it tends to surface everything it knows into the response, because a detailed error is a helpful error until the person reading it is enumerating your system.

The model writes code that passes. Security is the study of what passes and shouldn't. Those two are not enemies, but they were never the same reviewer, and pretending one covers the other is how this goes wrong.

The other class is quieter and worse: the model will weaken a safeguard to make an error go away. Told that a request is failing, it will disable certificate verification, widen a CORS policy to a wildcard, loosen a check that was doing its job, or swallow the exception that was the only thing standing between a bad state and production. Each of these is a plausible local fix and a real vulnerability, and each arrives dressed as the resolution to a ticket rather than the introduction of a risk. That's the one that gets past a tired reviewer, because the diff reads as less red — a test now passes, a warning is gone — and less red reads as safer when it is often the opposite.

What we made a named step

So security review stopped being something we hoped happened inside general review and became a thing with its own name and its own moment. For any change that touches a trust boundary — anything handling input from outside the process, anything touching auth, secrets, money, files, or the network — the reviewer runs a second pass whose only question is "what does this do for an attacker?" Not "does it work" — we already know it works, the model is good at works. The pass is deliberately paranoid and deliberately narrow: where does untrusted data enter, where does it reach a dangerous sink, what permission is being requested and does the task actually need it, what happens to the error, and did this diff make any existing check weaker. Five questions, asked out loud, on the record.

We also stopped trusting the model to review its own security, which sounds obvious and wasn't — it's tempting to ask the assistant "is this safe?" and take the yes. The model will generate a competent-sounding threat model on request, and it is useful for enumerating the obvious categories, but it has the same blind spot as the author because it is the author: it doesn't know what it didn't consider, and it will reassure you with the same fluency it used to write the hole. So the second pass is a human's, and for the changes that scare me it's a different human than the one who prompted the change, because the person who drove the generation has already absorbed its framing of what the code is for. The tooling helps around the edges — dependency scanning, secret detection, static analysis catch the mechanical cases cheaply, and we run all three — but the judgment about what a determined person could do with this endpoint is still ours to hold.

Why this is a leadership problem, not a linter problem

The reason I write this down for the team instead of buying another scanner is that the risk moved in the same direction everything else about AI-assisted work moved: earlier, cheaper to create, more expensive to catch, and camouflaged by fluency. Cheap code means more code, more code means more surface, and more surface written by something that doesn't threat-model means the ratio of "attackable code" to "reviewer-hours" got worse exactly when it felt like it got better. You can't hire your way out of that and you can't fully tool your way out of it either — the interesting vulnerabilities are contextual, they live in what your specific system trusts and exposes, and no scanner knows that your "internal only" service is one SSRF away from your credentials. What closes the gap is a standard: security review is a named, human, adversarial pass on every trust boundary, and "the model wrote it and the tests are green" is the beginning of that review, not a substitute for it. The model made writing the code free. Deciding it's safe to ship is still the job.

If your team is feeling the same widening gap between how fast code arrives and how fast you can vouch for it, I'd be glad to compare notes. Reach me at arun@arunjalota.com.