Writing on cyber strategy, statecraft, operations and geopolitics in a personal capacity. Views are my own and do not represent any employer or client. I use AI tools in research and drafting; the analysis and judgement are mine.
In January 2026, the maintainers of cURL closed a bug bounty programme that had been running for nearly seven years. They received many vulnerability reports. They had too many, and an increasing share were machine-written submissions that still required a human being to work out whether there was anything useful in them. The proportion of reports that turned out to describe genuine vulnerabilities had fallen from more than 15 per cent to below 5 per cent.
Weeks later, AWS, Anthropic, GitHub, Google, Google DeepMind, Microsoft and OpenAI committed US$12.5 million through the Open Source Security Foundation and Alpha-Omega to help open-source maintainers deal with the growing burden of AI-generated vulnerability reports.
There is an irony in companies building the models that help fund the people absorbing their output. More usefully, it gives us an early look at where the economics of AI-assisted security may lead.
Most discussion still concentrates on capability. AI will write code faster, find vulnerabilities faster and generate exploits faster. Every few months another benchmark appears showing a model performing some security task that recently required a skilled researcher.
We are concentrating on the easy part.
Software insecurity has always contained an economic problem. Finding flaws is only useful if somebody has the time, authority and engineering capacity to decide which ones matter and to remedy them without breaking something else. As AI makes the first part cheaper, that old constraint becomes harder to hide.
This raises a larger question: who pays to secure the software civilisation we have built?
The hidden bargain beneath the digital economy
Banks settle payments, hospitals deliver care, aircraft fly, insurers process claims and governments run public services using software assembled from thousands of components. Much of it is commercial, but a remarkable amount rests on open-source code maintained by people most executives will never know. CISA has cited research that found open-source components in 96 per cent of the codebases studied, accounting for roughly three- quarters of the code within them.
That arrangement has worked extraordinarily well. Linux, OpenSSL, Kubernetes, PostgreSQL and thousands of smaller projects are part of the foundations of modern computing.
The funding has never matched the level of dependence.
Companies have built enormous commercial value on software maintained by foundations, small engineering teams and, in some cases, a handful of volunteers. Organisations consume the benefits without necessarily contributing anything close to the cost of sustaining the projects beneath them.
The XZ Utils backdoor attempt in 2024 showed how far that mismatch can reach. The operation appears to have involved a patient effort to gain influence over a small project before malicious code was inserted. The technical sophistication was impressive. The attacker’s understanding of the human system around the code was equally impressive.
Log4Shell revealed the same dependency from a different angle. Organisations around the world suddenly needed to know whether a logging library, obscure to almost everyone outside software engineering, was buried somewhere in the systems supporting important business services.
We usually put both cases under the heading of software supply chain risk. That describes the route through which the exposure appears. It says less about why the underlying fragility exists.
I increasingly think the better frame is capital allocation. The digital economy has become very good at extracting value from shared software, but it has been rather less disciplined about funding its upkeep in proportion to the reliance placed upon it.
For years, this mismatch was partly cushioned by another scarcity: finding vulnerabilities was difficult.
When searching gets cheap
Traditional vulnerability research takes skill, time and persistence. Even large organisations can inspect only a fraction of the software they use or produce. That constraint has limited attackers and defenders alike.
AI is beginning to loosen it.
In 2025, DARPA’s AI Cyber Challenge tested autonomous systems against synthetic vulnerabilities distributed across roughly 54 million lines of real software. The systems found 54 of 63 synthetic vulnerabilities and patched 43. They also found 18 genuine vulnerabilities in open-source projects and patched 11 of those. DARPA reported an average cost of about US$152 per competition task.
It would be silly to treat US$152 as the new market price of vulnerability research. The competition was engineered, the tasks were bounded and the environment was nothing like a messy enterprise estate. The result is still useful because it shows what happens when the cost of searching starts to fall.
Google’s Big Sleep has also reported finding previously unknown flaws in widely used software, including an SQLite vulnerability that Google says it identified before anticipated exploitation.
Anthropic’s Mythos work points in the same direction. Its discovery of a long-standing OpenBSD vulnerability is a useful reminder that some flaws persist not because the code is hidden, but because deep inspection has always been scarce. There are only so many skilled researchers, so many hours, and far more code than anyone can examine closely. AI changes the economics of that attention.
Good security researchers are unlikely to disappear. They are more likely to cover far more ground and once organisations can inspect far more code, they will inevitably find more things requiring attention.
Finding is only the beginning
A vulnerability does not become a lower business risk simply because a model has identified it.
Someone first has to establish whether the finding is real and whether the vulnerable code is reachable in the way the organisation uses it. Somebody owns the product. Somebody else may own the infrastructure beneath it. Development teams already have work committed. Changes need testing, operations need a safe deployment window, and security still needs evidence that the flaw has actually gone away.
NIST’s Secure Software Development Framework describes much of this lifecycle formally. Large organisations are familiar with the less tidy version. Getting a technically correct change into production can be harder than writing it.
AI will help here as well. Models can generate patches, write tests, review code and automate parts of deployment. In modern engineering environments with effective tests, clear ownership and disciplined release practices, that should reduce remediation costs. Any argument that assumes people will continue fixing everything manually is unlikely to survive very long.
But important software does not live exclusively in modern engineering environments.
Legacy systems have weak documentation and incomplete tests. Suppliers disappear or stop supporting products. Operational technology may have tightly controlled maintenance windows. A hospital cannot treat a production clinical platform as a test bench because a model has produced a plausible fix. Neither can a bank casually experiment with the payments stack.
The hard part is often more organisational than computational. Who owns the decision? Who approves the change? Who carries the consequence if it fails?
AI can write a patch. It cannot sign off on the change approval.
That is why the same model will deliver very different outcomes in different institutions. An organisation with sound architecture, good testing, clear ownership and enough engineering headroom can turn faster discovery into faster risk reduction. Another organisation may simply acquire a more accurate inventory of problems, but it still lacks the capacity to repair them.
For years, the security industry has invested heavily in finding more things. We may be approaching the point where finding them is no longer the impressive bit.
The bill is already arriving
The cURL episode is useful because it shows the problem before the technology has matured.
More reports did not translate automatically into more security. They consumed the attention of maintainers who still had to separate useful findings from machine-generated noise. The cost of producing a report had fallen; the cost of deciding whether it mattered had not fallen at the same rate.
That helps explain the March 2026 US$12.5 million commitment. The organisations funding it understand that AI-generated security work usually lands somewhere, often on people with limited time and competing responsibilities.
The funding is welcome. It also looks modest compared to the economic value of the software these programmes support.
More importantly, voluntary grants are a fragile way to fund infrastructure.
If maintainers are expected to absorb a growing volume of machine-generated security work because banks, technology companies and governments depend on their projects, then those beneficiaries will eventually have to pay more systematically. Otherwise the organisation running the model saves money, while the maintainer inherits the triage bill.
The risk has moved. It has not necessarily fallen.
The incentives inside the firm
The same tension exists inside a commercial software business.
Suppose a product team becomes capable of finding five times as many vulnerabilities in its own code. The security team may regard that as progress. The engineering director now has five times as many findings competing with features, reliability work and customer commitments.
Each genuine flaw creates work and, potentially, legal or regulatory exposure. A company that searches rigorously may expose far more problems than a competitor that does less assurance work.
Customers have difficulty pricing that difference. They can compare licence cost, features, integration and performance. They can only compare the vulnerabilities that both vendors have already found.
The incentive problem follows. The cost of searching appears immediately on the vendor’s balance sheet. Much of the benefit arrives later, and some of it takes the form of an incident that never happens.
Governments have begun to push that cost back onto producers.
The European Union’s Cyber Resilience Act places greater responsibility on vendors and introduces the concept of an open-source steward, recognising that organisations sustaining commercially important open-source software occupy a different position from individual developers publishing code.
In the United States, CISA has described shipping products with known exploitable vulnerabilities as a bad practice and has pressed vendors to adopt memory-safe roadmaps and provide free, timely security patches.
These are early policy answers to an economic problem. They will not converge neatly. Jurisdictions will choose different balances between producer responsibility, public funding, customer responsibility and the role of large platforms.
The result will matter beyond compliance because each model changes where money, responsibility and leverage sit.
Who carries the cost?
Making vendors carry more responsibility has an obvious attraction. The company selling a product bears more of the cost of securing it. Those costs will eventually appear in prices, support periods and decisions about whether some products or markets remain worthwhile.
Public funding makes a different choice. Taxpayers support shared digital infrastructure from which private firms derive value, much as governments already support roads, ports and other assets used by commerce.
Giving cloud providers more responsibility may improve security across millions of customers. It would also hand more influence to a small number of companies that already control much of the world’s compute and increasingly its most capable AI.
Leaving each customer to manage the problem alone favours organisations wealthy enough to employ large security and engineering teams. Smaller firms, public bodies and underfunded projects will struggle to keep pace.
There is no clever technical answer that makes these costs disappear. Policy decides who absorbs them, and that decision redistributes leverage along with money.
This is why I remain sceptical of AI security arguments framed mainly around productivity. Lower discovery costs can save money at one point in the system while creating more work somewhere else. AI may expose technical debt far faster than an organisation can remove it.
The invoice has merely changed address.
When software assurance becomes state capacity
The international version of the problem is harder to ignore.
The most capable AI systems remain concentrated among a small number of companies in a small number of countries. If those systems become materially better at finding and helping repair vulnerabilities, access to them begins to affect how quickly organisations and governments can understand their own software exposure.
No serious country needs to own every model it uses. For most, that would be economically absurd.
It does need to understand what happens if access changes.
Suppose the best tools for analysing sensitive government software or critical infrastructure are owned overseas. Can sensitive code be sent to them? What happens if export controls change? Can the provider withdraw the capability or alter the terms under pressure from its own government?
We already recognise this logic in semiconductors, satellite navigation, cloud infrastructure and energy. Dependence often looks efficient until the supplier and customer stop wanting the same thing.
Software assurance could develop the same characteristic.
For Australia, the useful question is not whether Canberra should build a national frontier model as a gesture of technological sovereignty. It is whether the country knows which defensive capabilities it cannot readily replace, where they reside and what alternatives exist if access narrows.
That is less glamorous than announcing sovereign AI. It is also much closer to strategy.
What boards should ask
At enterprise level, the immediate problem is measurement.
Boards should be cautious about treating the number of open vulnerabilities as a proxy for security performance. If AI improves discovery, an organisation that searches harder may report more weaknesses than one that remains largely ignorant of its own exposure.
A better question is whether a credible finding can be converted into reduced risk.
That depends on more than the security team. Critical systems need accountable owners. Technology needs to be supported. Test environments have to work. Suppliers need to patch products within useful timeframes. The organisation also needs enough engineering capacity to make changes without treating every urgent fix as an assault on the operating model.
The existing backlog matters just as much. An AI system that produces another 50,000 findings is not especially helpful if nobody can identify the handful that materially change exposure.
This is where AI may widen the gap between organisations. Strong engineering environments will use it to move faster. Weak ones may receive better evidence of why they are weak.
The hidden bargain is ending
For decades, engineers and maintainers made an unlikely economic arrangement function remarkably well.
Open-source projects supported software estates far larger than their funding suggested. Corporate engineering teams kept ageing platforms alive. Security teams prioritised more weaknesses than organisations had money or people to repair. The success of all that work allowed the rest of the economy to treat maintenance as plumbing.
AI will probably make much more of that hidden work visible.
If machines can inspect vastly more software, somebody still has to judge the findings, decide what matters, authorise the change and carry the cost of getting it wrong. Some of those tasks will also become cheaper. Some will not. The balance will move, and anyone pretending to know exactly where it settles is getting ahead of the evidence.
What seems harder to avoid is the distributional question.
When we can find far more of what is broken, who is expected to pay for fixing it?
That is where the interesting argument begins.
References
CISA, Open Source Software Security Roadmap (2024)
Andres Freund, Backdoor in XZ Utils disclosure (2024)
DARPA, AI Cyber Challenge Final Results (2025)
NIST, SP 800-218, Secure Software Development Framework (SSDF)
OpenSSF / Linux Foundation, US$12.5M Funding Announcement (2026)
The Register, cURL Ends Bug Bounty Programme (2026)
European Commission, Cyber Resilience Act: Open Source
Good piece John. You make the key point that fixing is organisational, not computational. I'd push it one step further. These are two separate cost curves, and AI is pulling them apart. Discovery falls fast because it is computational. Remediation stays flat because writing the patch was never the expensive part - the cost lives in ownership, verification, regression testing, and someone signing off on the risk. The gap between those curves is the attacker's margin. So a security economy optimised to find more, faster, may be industrialising the cheap half of the problem while the expensive half stays stubbornly human. Linus's Law told us more eyeballs make bugs shallow. It said nothing about who has to dig the patch.