For most of my career, and for most of the engineers I’ve ever managed, the job was the application. If you’re at a SaaS company, you build the software. Front end, back end, the CI, the tooling, all the stuff that props the application up. Somebody asks for a feature, or wants a number on a screen to be different, or wants it faster, and you go make that happen. That’s working at the application layer. It’s what we’ve always done.
Here’s the thing I’ve had to accept, and it was not comfortable, AI can do that layer faster and cheaper than we can, and the output is good enough to ship, possibly better. And sure, there are articles and research out there that says, the output is not “better” exactly. But it is good enough, as I said, to ship. And, especially if you’re in a startup, good enough to ship is pretty freaking good.
So that leaves us with a question, what do we do? If AI can write code and deliver it, what does that mean for us, as software engineers. It’s a serious question, and one that I didn’t invent the answer for, and hell, I don’t even think everything I’m going to say below is an attempt at ensuring we have something to do for the sake of having something to do. I genuinely believe it’s a great piece of work, it’s engineering, it’s problem solving, and I think it’s a pattern that’s been repeated before in history.
The application layer only needs a product owner now
Sure, we could keep living at the application layer. Somebody asks for a feature, we build it. That world is still possible. But in a world where the model can produce that layer well enough, a product owner is really all you need there. Somebody who knows what the feature should do, writes it down properly, and checks that it does it.
Once the factory exists. That’s the catch, and I’ll come back to it.
What is also still true, and I’ve seen this with my own eyes, is that a product owner or a prototyper mostly can’t get a thing into production. Escape scanned about 1,400 apps built on the vibe-coding platforms and found two thousand critical vulnerabilities and over 400 leaked secrets. Not because product people are dumb. Because the prototype never went through a review gate, never got a security scan, and never had an audit log. There were no true engineering principles being used or enforced. No one was sitting there asking critical engineering questions about how the new feature would impact users, or other systems more broadly. And if you’re in a shop with compliance or certifications hanging over you, forget it. An app can pass every functional test and still fail the compliance review because nothing is logged and nothing is auditable. Even with great infrastructure and CI, it’s going to take time and practice before anyone can just say “put this feature in production” and have it be safe. Maybe a year or two. There’s a lot of ground between here and there. And maybe honestly never.
That ground is where engineers go. And fwiw, I don’t know that we ever have to cede it. I think there’s always going to be a case for engineers to ask the right questions at different steps in the process. An engineer on a software team with domain knowledge for the product can ask questions no llm can think to ask.
This isn’t me just saying it, it happens to me and my team all the time. A session will be pumping out a change / feature / bug fix, whatever, and it says it’s going to do x, and we ask why it isn’t doing why, and it’ll honestly answer because it didn’t think to. We’ll regularly point out new directions for it to go and why, and the old direction would have been really bad to go in, or have just wasted a little time. But in either case, a competent, trained, domain knowing engineer did the correction. The LLM did the code.
The factory layer
Here’s the move. Engineers get out of the application layer and into what I call the factory layer. Somebody wants a feature? Have a system that builds that feature as deterministically as possible, so that every feature goes through the exact same steps before it gets anywhere near production. Think of it as advanced CI that does everything for you, where the only part not done for you is deciding what to build. The product owner owns the feature. You own the factory the feature goes through.
The industry has landed on a name for this in the last eight months: harness engineering. Mitchell Hashimoto, the Terraform guy, wrote in February that any time an agent makes a mistake, you take the time to engineer a solution so it never makes that mistake again. OpenAI put out a post a week later using the same term. Stripe has homegrown agents merging over a thousand PRs a week with no human-written code in them. Dropbox has one doing about one in twelve of their PRs. And every one of those shops still has engineers at the gates. They didn’t leave. They moved. That’s the factory layer.
The factory is a workflow, and the workflow has a lot of steps. Take a plan, make tickets. Take tickets, write acceptance criteria. Take tickets, build them. Make PRs. Run them through CI. And then don’t merge to production. A human gate. Humans review the plan to make sure it’s going to do what it’s supposed to before anything gets built, and humans decide what merges. Between those two gates sit all the checks: the ones that enforce compliance, the ones that make sure your system design principles and your architecture are being followed, the ones that make sure the thing knows how to talk to the other services it’s supposed to talk to. The shape everybody seems to be converging on is two or three human gates with deterministic checks between them. Spotify’s version runs format, compile and tests with no model involved at all before a model ever gets to judge anything. Stripe caps their agents at two CI runs, then it goes to a person. The frontier, by the way, is fewer human gates, not more. Mechanical changes auto-merge. Product changes get two sets of eyes. The gate matches the blast radius.
Deviation from the norm
Then there are the evals. Every output of the workflow gets checked against what the team expects, and when it doesn’t match, the evals tell you there’s a deviation from the norm. Your job at the factory layer is to shrink that deviation. You do it by understanding your agents well, and by putting deterministic steps before and after each one so the output has the least variability you didn’t ask for.
And yes for now the tooling here is young. Nobody has a mature, off-the-shelf eval suite for a coding pipeline, and I’ve looked. What people actually have is a layered thing. Deterministic checks where you can, lint, tests, schema checks, a diff size cap. A model-judged check where you can’t, like “does this PR actually meet each acceptance criterion, yes or no, with evidence.” And then the slow eval, which is your four delivery numbers over a month, PRs through the loop, clean-first-pass rate, rounds to clean, cost per merged PR. One paper out of the CI/CD research crowd this year put said that deployment is outpacing evaluation. So don’t wait for the tool. Count what you can and tighten what moves.
Every team’s factory is different, and some of it is standard
Some workflows are going to be standard. A story goes in one end and a feature comes out the other. An epic is a big story with some work to make the right tickets out of it, and the factory should be able to chew on many epics at once. There’s a bugs workflow. There’s dependency bumps and vulnerability fixes. And guess what, those last ones are already products. GitHub made Dependabot alerts assignable to Copilot, Claude or Codex in April, agent opens a draft PR with the fix and handles the test breakage. In September they let you hand 25 code-quality findings to Copilot in one shot. The engineering workflows are standard now. But, if you have specific needs, they aren’t hard to build from scratch, we did and they work better.
But every organization also has internal workflows that are their own, and those need to get built too, not just to speed up engineering but to speed up every part of the business. That’s the custom work. And every workflow that’s more deterministic than not means less variability and less time wasted fixing outputs by hand. I’ll say that one twice, because it’s the thing I keep relearning.
Fix the factory, not the PR
Here’s the example I like to use. A bug ticket comes in. The factory’s up and running. The bug gets worked, there’s a PR, and either it doesn’t merge because the fix is wrong, or worse, it merges and the bug isn’t fixed. Most engineers, me included for most of my life, would go “ok, I’ll just fix it myself.” Open the PR the agent made, poke at what went wrong, push a commit, get it merged. Done.
In factory thinking you don’t touch that PR. You take the output, you figure out what the factory got wrong, and you fix the factory. Then you put the bug back through. You learn what went wrong so the next bug doesn’t have the same problem.
Now, I’ll be honest, “never touch the PR” is a rule I aim at, and for me it’s an easy target to hit. As an engineering manager I haven’t coded seriously for more than a decade. Reading code is one thing, writing it is another. LLM’s have given me the ability to do that again. I know from experience though that for some engineers that transition is a hard one.
So, what do I tell my team to get them to make that transition? Well, it’s just an abstraction one layer up. What they’re working on isn’t code at the application layer anymore, it’s “code” (its not code its really workflow, diagrams, tech speak) one layer up, in the factory. They’ll need to review agent prompts sometimes, they’ll need to ensure that the deterministic code steps make sense, and do what they say they do. They need to be able to think agentically or what I like to call, think like a manager or a director of a software team.
Each engineer has the power to run their own small team, and the workflows themselves are those teams. Manage them, make sure they run smoothly, make sure their outputs are soundly engineered and met the specifications. That’s what a manager does, it’s what I did for more than a decade, and now all engineers should be doing it.
I mentioned in a previous post that engineers now are full stack, and not full stack like frontend, backend, infra, but full stack as in, they need to know how to create specifications, understand what the signals of the system are telling them, determine how to act on those signals, and know how to get a team of agents or a workflow to accomplish the needs of the system.
What this looks like at my house
I run one of these at home. Not a Stripe-sized one, obviously, it’s me and a server. A ticket with acceptance criteria goes in. An agent builds it in a sandbox where it can’t push. A second agent reviews it against each criterion. A fix loop runs with a hard cap on rounds. CI runs. The factory posts “would merge” and a human, me, merges. Over the last few months it put hundreds of PRs on more than a dozen of my own small projects through that loop and about 90% came back clean on the first review pass.
And I can tell you exactly where I spend my time, because it’s not on the tickets. The reviewer kept finding a new medium issue every round and PRs would loop, so I put in a hard cap and a “settle once, then human” rule. Dozens of runs were failing in the fixer’s test step and it turned out to be the environment, a port already in use, a formatter disagreeing with the linter, not the code. I fixed the environment. The search in one of my own tools regressed after an agent change and the eval caught it, so I tightened the eval. I haven’t opened a PR to fix a feature by hand in months. I’ve opened a lot of PRs to fix the factory.
That’s the job now. The factory builds the product. The product owner decides what goes in. You keep the factory honest. Yep, you’re going to need evals, and you’re going to need to know which deviations are fine and which ones aren’t so you can analyze and adjust. That’s the work. But that’s what being an engineer at the factory level is, versus being one at the app level. And I’d rather be at the factory level.
If you’re building one of these, or you think I’m wrong about the app layer, I’d like to hear it. I like talking about this stuff more than is probably healthy.