If there’s one thing AI coding agents are good at, it’s churning out a lot of code.<\/p><\/blockquote>\n
Their conclusion was blunt: “little evidence that firms increase software output or reduce employment.” Efficiency gained in one phase was “absorbed by downstream constraints in the production process.”<\/p>\n
So the question for anyone signing off on an AI pilot is simple. Are you tracking activity, or outcomes? Commits, prompts, and tokens consumed are activity metrics. Features in production, defects caught, orders shipped: those are outcomes.<\/p>\n This wasn’t a survey. Chen and Stratton pulled aggregated engineering analytics from Jellyfish covering work events and issue management data from 2021 through March 2026, which means the window spans the entire arc of commercial AI coding tool adoption, not a single hype cycle.<\/p>\n Adoption dates came from two sources: directly measured AI usage and analysis of GitHub activity. That let them pin down roughly when each firm switched on these tools, then run a difference-in-differences regression comparing each company against itself before and after, and against firms that hadn’t adopted yet.<\/p>\n The researchers deliberately separated two things most vendors blur together. Assistants autocomplete code a human is already writing. Agents write and submit code autonomously from a prompt, with the human stepping in afterward as a reviewer.<\/p>\n That distinction matters because the volume effects show up with agents. When a tool generates work that a person then has to validate, you’ve shifted the labour, not removed it. Any measurement that lumps both categories into “AI usage” will tell you nothing useful about where your throughput actually went.<\/p>\n Most coverage fixated on output. The employment finding is the sharper signal. If AI genuinely compressed the work, headcount would drift down or feature delivery would climb. Neither happened.<\/p>\n For an operations leader, that’s the number to carry into your next vendor conversation. Tool adoption changed what people spent their hours on without changing what the organisation delivered or what it cost to deliver it. Productivity you can’t see in finished output or staffing isn’t productivity. It’s rearranged effort.<\/p>\n Here is the mechanism, step by step. An agent submits a pull request. A human has to read it, understand intent that was never theirs, and decide whether it is safe to merge. Chen and Stratton found that after agents arrive, those pull requests are more likely to require revisions and reviewers leave more comments on each one.<\/p>\n So the time between submission and merge stretches by roughly half. Nothing about the review step changed. It just received more work, and work that was harder to verify.<\/p>\n Developers using these tools already know not to trust the accuracy of what comes out. That distrust is rational, and it is expensive. Every generated block still gets read line by line by someone who did not write it.<\/p>\n This is the same cost structure as an incoming inspection you cannot skip. You can accept material faster than ever, but if every lot still needs full inspection before it moves, your receiving dock is not your throughput. Your inspection bench is.<\/p>\n Throughput is set by the slowest step with the deepest queue, not by the fastest one. Lines of code are keystrokes. Merged, shipped features are output. The study measured both and found the first moved sharply while the second did not budge.<\/p>\n Treat review latency as your primary metric and the picture gets honest fast. Measure how long work sits waiting for a human decision, how often it comes back for rework, and how many comments each item attracts. Those three numbers tell you whether your AI spend is producing software or producing backlog.<\/p>\n Swap pull requests for defect flags and the mechanism is identical. A vision system starts catching ten times more candidate defects on the line. Every one of them still routes to the same quality engineer, who now spends her shift dispositioning false positives instead of fixing root causes. Detection got cheaper. Judgment did not.<\/p>\n Document automation follows the same curve. An AI drafts deviation write-ups and CAPA reports in minutes, then they queue behind an approval chain of three signatures that was already running two weeks behind. Automated data extraction from supplier certificates fills a review queue nobody staffed. Generation capacity is cheap to buy. Verification capacity is people, and people are what you were trying not to add.<\/p>\n Any one of these means your AI productivity gains are sitting as inventory, not output. The fix is rarely more AI at the front end. Count the queue, find where items wait longest, and either staff that step properly or change the decision rules so fewer items need a human at all.<\/p>\n Before you buy anything, walk the process end to end and put a stopwatch on every handoff. Not the work itself, the waiting between steps. In most quality and operations workflows, the generation step is already the fastest part. Queue time at approval, verification, and sign-off is where weeks disappear.<\/p>\n Once you can see where the time sits, the investment decision gets simple. If the delay lives in verification, funding faster generation buys you nothing except a bigger pile.<\/p>\n Pick one metric that represents finished work: deviations closed, batches released, audits completed, complaints resolved. Measure it for a month before the tool arrives. Record cycle time and volume separately so you can tell throughput from backlog burn.<\/p>\n Activity metrics will betray you. Chen and Stratton had 300 million work events and still found no significant change in resolved Issues and Epics, because commits and pull requests measure motion, not completion. Documents drafted, defects flagged, reports generated: all motion. If your pilot dashboard only shows activity, you have built a machine for proving success that cannot detect failure.<\/p>\n Redesign verification instead of staffing it harder. Tier reviews by risk so low-consequence items get a lighter path and high-consequence ones keep full human judgment. Use confidence thresholds to auto-route the clear cases and escalate only the ambiguous ones. Replace 100 percent human checking with statistically defensible sampling where your quality system allows it.<\/p>\n Credible ROI reads as cycle time on the outcome metric, not hours saved drafting. A deviation process that closed in eighteen days and now closes in six has relieved the constraint. Everything else is inventory.<\/p>\n![\"Bar](\"https:\/\/falcoxai.com\/main\/wp-content\/uploads\/2026\/10\/ai-coding-agents-create-code-inline-1.jpg\")
<\/figure>\nWhat the Harvard Study Actually Measured<\/h2>\n
Assistants vs. agents: two different adoption patterns<\/h3>\n
Why ‘no change in employment’ matters more than the headline<\/h3>\n
The Bottleneck Moved Downstream to Code Review<\/h2>\n
Trust is the hidden cost: humans still verify every output<\/h3>\n
Why queue length, not keystrokes, sets your throughput<\/h3>\n
![\"Bar](\"https:\/\/falcoxai.com\/main\/wp-content\/uploads\/2026\/10\/ai-coding-agents-create-code-inline-2.jpg\")
<\/figure>\nThe Same Trap in Quality and Operations AI Projects<\/h2>\n
Three signs your AI pilot is generating work-in-progress, not output<\/h3>\n
\n
How to Find and Fund the Real Constraint Before You Scale<\/h2>\n
Baseline the outcome metric before the pilot starts<\/h3>\n
Automate the review step, not just the production step<\/h3>\n
![\"Whiteboard](\"https:\/\/falcoxai.com\/main\/wp-content\/uploads\/2026\/10\/ai-coding-agents-create-code-inline-3.jpg\")
<\/figure>\n