Security Design Never Scaled

Take an ordinary service. It accepts connections from the internet and terminates TLS, so it parses attacker-controlled bytes with a general-purpose library. It proves its own identity to peers, so it holds a long-lived private key. It runs business logic against a database, so it holds a database credential. Written the ordinary way, that is one process under one uid, and all three jobs share an address space. There is nothing irresponsible about it. An enormous amount of software is built exactly this way.

Now write the same service as four processes under four uids, talking over local sockets that authenticate their peer. netd terminates TLS and holds no long-lived secret. authd holds the host key and exposes one operation, sign a well-formed challenge, and never returns key material. appd holds the database credential and accepts only a defined message schema, from netd and nobody else. logd takes append-only writes. None of this is novel. OpenSSH has been built this way since 2002, and Provos, Friedl and Honeyman[1] wrote up the reasoning the following year. Postfix, qmail, and modern browsers are the same idea. It also costs more, in four supervised processes instead of one, serialization on a path that used to be a function call, latency, new failure modes when a socket goes away, harder debugging, and a schema that now has to be maintained as a real interface because it has become a security boundary.

Now drop Heartbleed[2] into both.

Heartbleed was a buffer over-read in the TLS heartbeat extension, returning up to sixty-four kilobytes of adjacent process heap per request, repeatable as often as the attacker likes and invisible in the logs. Not code execution, but a read primitive, which is exactly why it is the useful example. What the attacker gets is precisely and only what happens to be in the address space.

In the monolith, the address space is everything. The host private key is on that heap, along with the database credential, the session state, and the decrypted traffic of every other user currently connected. Whether a given read returns the key is a matter of heap layout and patience, and in 2014 the answer turned out to be patience. Cloudflare put a server up as a challenge and researchers pulled the private key off it within a day. Recovery is not patching OpenSSL. It is rotating an identity, reissuing, notifying relying parties, and assuming the database was read.

In the separated design the same bug is in netd, whose heap holds TLS session state for the connections it is currently serving, and that is all it holds. The host key is in authd, under a different uid, in a different address space, and a read primitive cannot cross an address space. The database credential is in appd, likewise. What still leaks is the in-flight plaintext of the connections netd is handling, which is genuinely bad and which no amount of process separation fixes, because handling that plaintext is the job. The design did not make the bug harmless. It decided, before the bug existed, which secrets were allowed to be in the room when one arrived.

Give the attacker code execution instead of a read primitive. The monolith outcome is unchanged; they already had everything. In the separated design they can now speak the IPC protocol, driving appd through its schema and asking authd to sign well-formed challenges. Real capabilities, but bounded by an interface somebody designed rather than unbounded by an address space that happened to contain everything. Same architecture, two bug classes, two different residuals, and you can only see either one by reasoning about the shape.

Nothing we built to measure security tells these two systems apart on its own.

They link the same library, so the SBOM is identical. They are affected by the same disclosure, so the CVE is identical. The base score is computed on the vulnerable component rather than on your deployment, so that is identical too, and both land in the same remediation bucket under the same policy with the same deadline. The scanner output matches. The audit tests whether critical vulnerabilities were patched inside the window, and both pass or both fail together. A conventional externally scoped pentest sees the same exposed interface. A procurement questionnaire sees the same technologies. Every one of these default instruments agrees the two systems are in the same condition, and one of them loses its identity key while the other does not.

Heartbleed makes this almost too neat. NVD scored it 5.0 Medium under CVSS v2 and later rescored it 7.5 High under v3.1. The bug that forced a global TLS certificate reissue was rated medium, and the rescoring fixed nothing, because both numbers describe the component rather than your deployment of it, and the whole question was what your deployment kept in that address space. CVSS does have a place for the deployment, the environmental metrics, and almost nobody fills it in, because doing so takes the architectural reasoning nobody has budgeted. The instruments are not blind by nature. They are blind by default, because the input that would separate the two systems is the expensive thing.

The difference between those two services is not which controls they hold. Both can hold the same ones. It is how the parts are arranged, and arrangement is a fact about the whole system that no check on any single control can see. And the engineer who proposes the separated design is asking for budget to change a property that every instrument reports as already fine, while the engineer who ships the monolith is out of compliance with nothing. A compliance checklist tests whether the cached answers are present; it has no vocabulary for whether the arrangement those answers sit inside still makes sense.

This essay is about why that is the normal condition rather than an oversight. The design reasoning never scaled. We could not afford to apply it to every system, so we compressed it into controls, rules, patterns, certifications, and tests. The compression was rational and it worked. But it dropped the reasoning behind each control, the threat model that made it the right answer and the signal that would tell you when it had stopped being right. We learned to scale the answers. We never learned to scale the reasoning that produced them, so our machinery ended up measuring the pieces while attackers exploit what the pieces compose into. What is changing now is that the reasoning itself may be getting cheap enough to scale.

Say what secure design is, plainly. It is not the presence of any particular control but the shape of the whole, the arrangement that decides, before any particular bug, what a compromise can reach. It helps to hold that apart from two things the field keeps collapsing into it. One is reducing how many vulnerabilities get created, which memory-safe languages, safe-by-default frameworks, and hardened APIs all do, retiring whole classes of defect that once consumed a generation of researchers. Another is finding the vulnerabilities created anyway, which fuzzers, scanners, and now autonomous agents that top bug-bounty leaderboards all do, and which is getting cheaper by the month. Both attack the presence of vulnerabilities. Neither touches what a vulnerability reaches once it exists. That is design, the only one of the three that decides whether a given bug rotates an identity key or leaks a little session state. Presence never reaches zero, because no memory model rules out a logic flaw, an over-broad grant, or two sanctioned edges composing into a path, so consequence is the residual that design, and only design, owns.

The strongest version of this is not about bugs you know. Chrome shipped a sandboxed, multi-process design in 2008 that kept a compromised renderer away from the machine[3], and then spent roughly five years and 450,000 lines of change on Site Isolation, which puts every site in its own process so one site cannot reach another’s data[4]. It was expensive and, for years, unglamorous. Then Spectre arrived in 2018, a CPU flaw that lets code read memory in its own process with no software bug to patch and no clean software fix, and the work already done contained it. Because cross-site data no longer shared an address space, Chrome enabled Site Isolation by default and turned an unpatchable hardware class into a bounded one. The design did not anticipate Spectre. It anticipated that something would eventually read across whatever boundary was cheapest to cross, and it made that boundary a process.

Design in this sense is not confined to code. Google runs its services on a substrate that gives developers identity, isolation, and lifecycle management by default[5], so the cheap path is many small services, and small isolated services are trust boundaries whether or not anyone set out to draw one. iOS assumes an app can be killed at any moment and denies it unmediated access to another app’s data[6], so the sandbox and the disposable process are security decisions wearing power-management clothes.

None of these ask the developer to be a security architect. They bake the shape into the substrate so the contained thing is the default, which is the form of secure design that has actually scaled, and it scales the answer rather than the reasoning. Someone did the expensive design once, at the platform, and everyone above inherits the boundary for free, but only for the shapes the platform anticipated. The moment a system needs a boundary the substrate does not provide, you are back to reasoning it out by hand, at the cost the rest of this essay is about.

The move is not finished arriving. The newest place it is showing up is around AI agents, where the component processing hostile input is a language model that cannot be assumed immune to prompt injection. Google’s browsing agent does not try to build a planner that cannot be prompt-injected. It assumes the planner can be, and constrains what a manipulated one reaches, routing the decision through a separate critic that never sees the untrusted web content and confining the agent to a declared set of origins it may read and a narrower set it may act on[7]. That is Heartbleed’s lesson on unfamiliar hardware. Do not ask whether the complicated component can be made perfect. Decide what its failure is allowed to reach. Thirty years after privilege separation and fifteen after the browser sandbox, the same design move is being made against the least predictable component anyone has yet had to contain.

1Controls are cached answers

Security controls are cached answers to security-design questions, and we scaled security by distributing the answers and discarding the reasoning that produced them. Some began as a compensating control, a stand-in for design reasoning nobody could afford; others were always mechanisms or principles in their own right. What they come to share is the role they play in the pipeline, where each ends up standing in for the design question it once answered, present as an answer with the reasoning behind it gone.

This is not a new claim about what security is. Saltzer and Schroeder were writing about least privilege, complete mediation, and unwanted access paths in 1975[8]. NIST treats security as an emergent property of a system and systems security engineering as a subdiscipline of systems engineering, which is SP 800-160[9] in one sentence. Nobody forgot security design. It never scaled, and what we built in its place has been standing in for it ever since.

It never scaled because the reasoning is expensive. Understand what the system is for and who holds authority. Trace what becomes reachable when an identity, a component, or a trust boundary fails. Notice when two individually reasonable decisions compose into a dangerous path. Then decide which risks to eliminate, contain, accept, or transfer. That requires holding a whole system in one head, keeping it there while the system changes, and being in the room when the consequential decisions get made.

It also resisted the automation that arrived for everything around it, for reasons that matter later. Design reasoning is non-local. Whether an authority relationship is acceptable depends on parts of the system you are not looking at, so a scanner that recognizes a bad pattern inside a bounded window cannot recognize a bad architecture. It is counterfactual. The question is not what the system did but what becomes possible if a component behaves differently, so watching the system run does not answer it on its own. And it has no cheap oracle. A design is correct until an adversary demonstrates otherwise, and the absence of a demonstration is not proof that none exists. Incidents, red teams, and formal verification can all supply evidence, but none of them is the automatic crashed, therefore wrong signal that closes the write-it-and-run-it loop for the rest of software.

This is a different question from whether the code contains a vulnerability. Preventing defects and finding them are both worth doing and both getting cheaper, but neither decides what a defect that slips through is allowed to reach. The two are not substitutes for design, and no amount of the first two adds up to the third.

So we rationed. The highest-risk systems got real attention and everything else got something cheaper, expert threat reasoning compressed into portable substitutes (rules, patterns, controls, certifications, audit tests) that let thousands of people make mostly-good decisions without repeating decades of analysis. Every expert discipline does this. Building codes are compressed structural engineering; clinical guidelines compressed medicine. The difference is what those fields kept. A building code does not stamp drawings; a licensed engineer does, and the compression stayed subordinate to a judgment attached to a person accountable for departing from it. Security compressed and kept only the artifact.

The compressions worked, and the argument depends on that. MFA closed off the category of compromise that a stolen password alone used to enable. Hardware key storage prevents software extraction of the key. FIPS validation really does establish properties about a cryptographic boundary. A compression that produced nothing would have been abandoned decades ago. What went wrong is that they were useful enough to become the thing we buy, measure, mandate, and eventually mistake for the property they were supporting. And that dropped reasoning is exactly what tells you whether the control is still doing its job here.

Take keys belong in HSMs, a rule I have spent a lot of my career on both sides of. It drops the only question that matters, which is how many reachable extraction paths that boundary removes in your system, and it has no grammar for the one thing an engineer actually needs to say, that a design is better against this threat model, at this cost. There is usually an intermediate design the rule cannot express, a signing key in a separate process under its own user, exposing a narrow signing interface with a policy on what it will agree to sign. It is not an HSM; root still wins. But application compromise no longer discloses the key, dumping the environment no longer discloses it, and key use becomes separately auditable. Several attack paths vanish at a fraction of the canonical control’s cost, and the institution gives the design no credit because it is not the canonical control. Risk reduction is continuous; institutional credit is a step function. That gap is why a team that cannot afford the approved control rationally does nothing at all.

The cliff has a mirror, and it is worse because it looks like success. Encryption at rest is the clean example. It is a real control, usually implemented correctly, and its canonical threat model is loss or compromise of the storage medium, a stolen drive or a backup tape leaving the building in a van. Against that it works exactly as designed. But name the thing that actually takes the data, an over-privileged service account, an injection flaw, a compromised application host, and the database is asked politely for plaintext by something holding valid credentials, and it complies, because complying is its job. At-rest encryption is not defeated in those scenarios; it is simply not in the path. And it satisfies everything, the questionnaire that asks whether data is encrypted at rest, the auditor who tests whether the feature is enabled, the framework that awards the control, with nobody in the chain saying anything false. The cliff withholds credit from a design that removes real paths; the mirror grants full credit to one that removes none of the paths that matter. Both are what happens once the threat model has been stripped off, because the threat model was the only thing that would have told you which case you were standing in.

The same pattern appears with session cookies. Every control checklist demands HttpOnly, Secure, and SameSite. None of them asks whether the application put every user behind a single origin or gave each user, or each piece of untrusted content, its own. Custom user domains turn the same-origin policy into a cookie-isolation boundary. An XSS that would have stolen over-scoped cookies on a shared origin now executes in a different origin. The flags are identical; the residual is not.

This is the line between two kinds of compression. A process boundary, a sandbox, a per-user origin, or a platform default bounds what a compromise reaches by construction, and keeps working even when no one remembers why it is there. A rule like encrypt at rest, keys belong in HSMs, or set the cookie flags only asserts that a control is present, and can be satisfied in full while containing nothing. The first is design, cached into a substrate. The second is a compensating control that forgot what it was compensating for. This is why the objection that sandboxes and secure defaults obviously scaled is not a counterexample but the whole point. Those are answers cached into a shape, and caching the answer is exactly what scales, because the containment survives everyone forgetting the reasoning behind it. What never scaled is the reasoning that knows which shape a system needs, and the reasoning that notices when a cached answer has quietly stopped fitting the system it sits in.

2What compression cannot see

The loss shows up first as a boundary problem. A FIPS 140-3 validation[10] covers a defined cryptographic module in an approved mode and says little about the product, networking, and operating system around it, and isolation is no different, since a wall does no work until you cut the holes every real deployment needs, and each hole hands back a property the wall was providing. Each is correct inside a boundary drawn for its own convenience rather than for the risk. The boundary even runs the wrong way in time. A compliance certificate attests a specific version, so a security fix inside the validated boundary can produce code that is no longer the validated version until it is reassessed. CMVP provides paths for revalidating an updated module, including expedited ones, but updates are uncommon in the public record. Of 415 validated modules read from that record, 324 show no recorded update after initial validation. An absent entry does not by itself prove none happened, but the gap is large enough that choosing validated cryptography can leave you slower to absorb a fix in the component you were being most careful about.[11]

The same gap opens above the boundary. Put a key in the finest HSM money can buy and expose sign(anything) to a compromised caller, and the key is perfectly non-extractable while the authority it represents is wide open. What the device protects is key material, not the question of who may ask it to sign and what happens when that caller is compromised.

The deeper loss is compositional, and it is where the mismatch stops being philosophical. Consider a path:

internet-facing service → application identity → CI system → deployment credential → production administration → signing service

The application is supposed to have its identity. The identity may legitimately reach CI. CI may legitimately hold deployment authority. Operations may legitimately administer production. Production may legitimately invoke signing. Every edge is somebody’s intended behaviour. The composition is nobody’s. Six teams can each examine their own edge and find nothing absurd, because the defense is built by whoever was funded to build it and the funding follows reporting lines (endpoint, identity, AppSec, cloud, network), none of which is a taxonomy of attack paths. Where an organizational boundary happens to coincide with a trust boundary the defense is good. Where they diverge, a path crossing four teams is on none of the four backlogs. The attacker has the opposite cost function. They spend almost everything on identifying and exercising the single cheapest viable route, and are adequate nowhere except along it.

You cannot fix this by reshaping the organization until it produces the architecture you want. No org chart is isomorphic to an attack graph, which runs through vendors, SaaS, recovery paths, and temporary privileges you do not employ, and changes every time somebody grants a permission on a Tuesday afternoon.[12] Either the consequential paths stay unowned, or something other than a team has to hold them. This is why BloodHound[13] mattered. The permissions were never secret; the representation was new. Put identities, privileges, and sessions into a graph and ask what they compose into, and the unit stops being the permission and becomes reachability, much closer to the attacker’s question than to ours.

3Why the machinery keeps growing

Given a loss this shape, the industry’s response has been to build more machinery around it, and the machinery grows in a characteristic way.

For decades vendors effectively said trust us. Customers stopped accepting that and asked what is in it, and the SBOM is the answer the industry settled on. An accurate and current one could turn a scavenger hunt into a query when a catastrophic vulnerability lands in a widely deployed library. But presence is not what the operator needs to know. They want to know whether this vulnerability can hurt them in this deployment, which requires knowing whether the vulnerable code is even reachable, what privilege the process holds, and what a compromise of it actually gets you.

So institutions convert the thing they can require into the thing they wanted to know. Does the vulnerability affect this product? Add VEX. How do you substantiate VEX? Add reachability analysis. Under which configuration? Add configuration evidence. How do you know production still matches? Add runtime attestation. Can an attacker change it? Add authorization data. Who can become that administrator? Add identity relationships. What if the IdP is compromised? Add upstream trust relationships. Is any of it still true today? Add continuous reconciliation. None of those additions is silly, which is precisely why the process can continue indefinitely. Each one serializes another piece of the missing model; none recovers the reasoning that composes the pieces into claims about the whole system.

The regress also wins because it is demonstrable and good design is not. A product producing ten thousand findings a month is doing visible work and can show it continuously. The architecture that made those ten thousand events less consequential produces nothing anyone can screenshot for the quarterly review, no counter of attacks stopped, no alert queue, no dashboard turning greener. The largest buyers have started saying this out loud. In 2025 JPMorgan Chase’s CISO told suppliers that modern integration patterns have quietly dismantled the boundaries that used to contain a compromise[14]. It is the buyer’s side of the argument, from a buyer with unusual leverage, and it meets the same wall, because what it asks for is architectures whose properties a customer can cheaply check, which is the one thing the market cannot.

Follow the regress far enough and you get a plausible end state. Every product has a current SBOM, every vulnerability has a VEX statement, every configuration is attested, every identity relationship is inventoried, every control emits evidence, every dashboard is green, and a three-step route still runs from an internet-facing service to the authority that can end the company. Nobody in the chain has necessarily done anything wrong. We have documented the insecure system extremely well.

4What changed

The reasoning we compressed away is exactly the thing that is now getting cheap, which is what makes machine reasoning matter here rather than being one more tool in the pipeline. Of the three properties that kept design reasoning from being automated, two are exactly what a machine attacks, because it can hold a non-local model in view and explore counterfactuals at scale. The third, the absence of a cheap oracle, does not move at all. So it is worth being careful about which side of the ledger moves first.

Consider what an attacker does looking for a path. They gather information, form a hypothesis, follow an edge, find it blocked, back up, try another, combine two facts nobody had considered together, and rank several routes by cost and consequence. Almost all of that is reasoning over a representation. Now consider what the defender does after discovering a dangerous relationship. They change an API, migrate customers, rewrite authorization, reissue credentials, alter disaster recovery, accept latency, retrain operators, coordinate four teams, and support the old architecture for another year. Almost none of that is reasoning. Machine reasoning therefore reduces the attacker’s cost faster than it reduces the defender’s cost of changing the system, at least in the near term, and the gap holds even if model capability stopped improving today. This is an argument from the nature of the two jobs, not a measurement, but it is the near-term asymmetry the structure predicts.

There is a measurement pointing this way. Anthropic evaluated frontier models[15] building working exploits directly from disclosed patches. The counts are not the interesting number; the clock is. The first working Firefox exploit arrived in under an hour, and the release carrying the fix was still eighteen days away. Patch cadences were designed on the assumption that weaponizing a disclosed patch costs expert-weeks and that few people can do it. Tens of minutes against a release measured in weeks is a different regime. This does not mean every attacker suddenly wins, or that defenders get nothing from the same models. The near-term asymmetry is simpler. Information can move at machine speed while organizations still change at organizational speed.

It would be too glib to leave it there, because defenders are pointing the same machines at their own work, and not just at finding bugs. Google now reports fixing and critic agents proposing patches in iterative loops, test-writing agents checking that a fix holds before a human sees it, and a move toward shipping security releases twice a week, with Chrome 149 and 150 together closing 1,072 security bugs, more than the previous twenty-three milestones combined[16]. The reason that loop compresses is the same reason vulnerability discovery does. It has an oracle. A fuzzer reports that an input crashed, a patch is proposed, and the system reruns the crash to confirm it is gone and nothing else broke[17]. Find, triage, fix, test, and ship is becoming one machine-speed loop because every stage has checkable feedback. Deciding that an authority relationship should not exist, migrating the systems that depend on it, and accepting the residual risk acquires no oracle merely because the patch loop got faster. That is the real discontinuity. Not attacker against defender, but the loops that have a checkable answer against the architectural change that never will.

Within design reasoning itself there is a ladder worth walking, because the rungs are easy to conflate. Sensitive keys should be protected from extraction is retrieval. This application keeps the key in an environment variable, therefore application compromise discloses persistent signing authority is contextual analysis. A local constrained signer removes the highest-probability extraction paths at lower cost than a remote HSM, leaving host compromise as an accepted residual is design-tradeoff reasoning. The system only needs this long-lived authority because of the protocol we chose, and changing the protocol makes several attack paths disappear together is architecture. Everything below the last rung makes the existing machine faster; only the last rung changes what the machine is for.

It is worth being precise about why cheap search does not become cheap design. A great deal of security work has a cheap correctness witness. A fuzzer has one, because the target crashes or returns memory it should not have, and so do patch-derived exploits and much automated remediation, where rerunning the crash tells you whether the fix held. Anything with a defined space and a checkable answer rewards raw search, and machines are extraordinary at it, which is the whole of what the leaderboard-topping agents do. But the witness is a property of the question, not of the word vulnerability. Authorization logic, protocol design, confused deputies, and many abuse cases share the hard core that architecture has, and conversely, once someone states an architectural property precisely, that a compromised renderer must not read another origin’s data, much of the rest can be searched, tested, or even model-checked. The expensive act is not the search. It is choosing the property in the first place, deciding what authority should exist, what consequence is acceptable, and which tradeoff is justified, given what the system is for and what cannot be lost. That is a counterfactual measured against intent, and intent appears in no artifact and has no external oracle. You cannot try a trust boundary in production and watch it fail, because the failure is the loss you were preventing. Search settles what follows from a property. Judgment settles which property is worth holding, and no amount of search answers that.

That is why the top rung resists brute force. Design reasoning means holding a model of the system and interrogating it, so a system that only arrives at good answers by iterating against feedback will never reach it, while one whose competence comes from the model might. Whether today’s systems can is genuinely open. They reason impressively over bounded technical problems, while maintaining a reliable model of a large, evolving organization and making good architectural tradeoffs inside it remains much harder. The reason to care is not that a machine would replace the architect. It is that the architect’s reasoning would become something a buyer, an auditor, or an underwriter could inspect.

5What scaling security design would require

We already know what happens when the right representation makes an expensive reasoning problem cheap. Joern[18] constructs Code Property Graphs so properties of large codebases can be explored through queries rather than reconstructed by hand, CodeQL[19] does something similar, and BloodHound does it for identity relationships. Each is incomplete for this problem and each demonstrates the same trick. A better representation converts reasoning that was artisanal into reasoning that is merely a query. None of them answers a security question; all of them changed what it costs to ask one. Some reach design directly. Privtrans automatically partitioned programs for privilege separation in 2004[20], and later work made the cut quantitative, choosing where to draw the boundary that removes the most privilege for the least cost. The mechanism has been automatable for two decades, and what none of it supplies is the judgment for which boundary is worth its cost here.

Every one of them also begins too late, because it starts from something already built. Before the code there was a decision about what the system should permit. Before the IAM permission there was a decision that a principal should hold some authority. Before the private key there was a protocol that required persistent signing authority. A representation that starts earlier lays a design claim out as a sequence. The property somebody decided must remain true. The authority model it depends on, which is to say who can cause what. The design decision taken to make the property hold, with the alternatives considered and rejected. The assumptions that decision rests on, written down so they can be checked rather than silently inherited. The evidence that bears on those assumptions, drawn from the artifacts the system already emits. The attempted falsification that keeps looking for the case where the claim breaks, together with the conditions under which it would stop holding, which is the piece compression always drops and the only one that lets the representation invalidate itself. And last the residual risk knowingly accepted, which a rule never records because it has no way to say and we decided this was fine. None of these is a property of the code or a row in a control matrix. They are claims about the system that can be true or false independently of whether the usual scanners are green.

It is a design tool, not a documentation tool. Take the property compromise of the application must not disclose persistent signing authority. No product is named in that sentence, and the candidates become comparable against it. The environment-variable design yields app compromise → key disclosure → persistent signing authority. A separately permissioned local signer removes direct key disclosure but leaves host compromise. Moving the signer to another host removes another path. A remote KMS or HSM removes more extraction paths while leaving a compromised authorized caller able to exercise signing. A protocol redesign that eliminates the long-lived key removes the question entirely. Each option carries a different cost and a different residual, the reasoning the control cliff destroyed, made explicit enough to compare.

Once a decision is represented this way, reality can disagree with it. Say the team takes the local signer because a remote KMS is not justified by the residual host-compromise risk. The design now carries three assertions. The application cannot read key material, the signing interface accepts only transaction-shaped payloads, and the signer identity cannot be assumed from the workforce identity plane. Each is falsifiable by evidence somebody is already collecting. Code analysis finds a crash-dump endpoint that maps the signer’s memory. A runtime trace shows the interface accepting an arbitrary byte string from an integration harness that shipped by accident. The IAM graph finds a break-glass role granting assume rights to on-call, and on-call is a workforce group. Three contradictions, and nobody had to imagine the attack in advance. The design said what had to remain true and the implementation disagreed in three places.

That is where design and continuous assurance finally connect. Design says what must remain true, assurance asks whether reality still supports the claim, and continuous assurance without continuous design is a faster way to prove the wrong architecture is behaving exactly as specified. The danger is obvious enough to name before anyone builds it. The instant this becomes Design Graph Coverage: 94%, with a standard, a procurement mandate, and an audit test for currency, we have invented the next SBOM. The defense is in what the representation is for. It is not here is the graph, therefore we understand the system, but here is what we currently believe about the system; find the shortest path that proves us wrong. A representation optimized to be attacked is harder to game than one optimized to be certified, because its success is expressed as surviving attempts to contradict it rather than as a coverage number to hit.

There is a harder objection than institutionalization, and it is worth meeting directly. The reasoning may be getting cheap, but the representation might not. Building an accurate map of principals, authorities, assumptions, and residuals, and keeping it true as the system changes every Tuesday, could cost more than reasoning over it ever did. And an incomplete map is worse than none, because it reassures. The asymmetry is the one that has run through this essay. The attacker needs a single edge the map is missing; the defender is lulled by a graph that looks complete. Cheap reasoning over a wrong model produces confident, sophisticated, wrong answers.

It helps to see what the representation is made of, because the cost is not what it first appears. The organization already holds most of the facts the map needs, in its code, its identity graph, its configuration, its architecture documents, and its operators’ memory. The attacker has to reconstruct all of that from outside and against resistance, while the defender starts out owning it. What the defender has never had is those facts in a single representation a machine can reason over, rather than scattered across a dozen that do not compose or living nowhere but in someone’s head. So the task is not to build an omniscient model of the system from nothing. It is to make an advantage the defender already has cheap enough to compute, before an adversary maps the same ground faster than the defender ever instrumented it[21]. That does not make the map cheap to keep true, which is the harder half of the problem, but it does say where the map comes from.

This is the deeper reason to build the representation to be attacked, and it takes two kinds of attack, not one. One searches the model for a path that violates the property. That search is only ever as good as the model, because no search over a graph can find an edge the graph does not contain, so by itself it can interrogate the wrong world with great thoroughness. The other attacks the model itself, reconciling it against evidence gathered independently of it, the code, the runtime traces, the identity graph, the configuration, the recovery paths, and counting anything they contain that the model cannot explain as a contradiction. That second attack is the one that catches the forgotten recovery system wired to root, because the finding does not originate in the model at all. It originates in a witness the model has to answer to, which is exactly what the three contradictions above already were. Completeness is not the bar and never will be, because no map of a living system is ever complete. The bar is whether the model can be falsified from both directions and whether something is continuously trying, so a missing edge surfaces not as a gap the search politely steps over but as a place where reality refuses to match the map. The proposal does not promise a correct picture of the system. It promises one that fails loudly, which is the only kind worth having, and the only kind cheap reasoning makes affordable to keep attacking.

None of this makes controls obsolete. If machine reasoning reaches architecture, today’s artifacts move back into their proper position as premises. The SBOM becomes evidence about composition, the identity graph evidence about authority, runtime traces evidence about interaction. None of them has to pretend to be security, which makes the pipelines more valuable rather than less. And the questions change with them. Instead of do we have an HSM, which extraction and signing-abuse paths remain. Instead of do we have an SBOM, which component failures create reachable paths in this deployment. Instead of is this system compliant, which claims about its security properties the available evidence can falsify.

A design system worth having argues back. Propose an HSM and it asks what happens when the authorized caller is compromised. Propose network isolation and it finds the management plane crossing the boundary. Propose MFA and it inspects recovery. Propose doing nothing and it tells you which fact would make that irrational. That does not replace human judgment; it concentrates it where it belongs (materiality, business intent, proportionality, acceptance of residual risk) and hands the machine the part it may become extraordinarily good at, which is holding more relationships at once, searching more paths, and continuously trying to prove the current answer wrong. It also changes who can be answerable for the result. A residual risk that lived only in an expert’s head could not be signed, priced, or handed to anyone. A residual risk written as a property, a decision, and an invalidation condition can be attached to the engineer who accepted it, to the auditor who tests whether its assumptions still hold, and to the insurer who finally has something to price. The scarce thing was never the willingness to be accountable, nor even the facts. It was a representation of the architecture cheap enough for anyone to check.

We never stopped knowing how to do security design. We could not afford to apply the reasoning everywhere, so we compressed the results into standards, controls, patterns, certifications, checklists, findings, and artifacts. That was rational, and it let a small number of experts influence systems they could never personally review. The attacker never had to use the compression. They always got the real system (the identities, dependencies, trust relationships, configuration, recovery mechanisms, forgotten permissions, and ordinary decisions that compose into a path), and they searched it.

Cheap reasoning will make that asymmetry hurt before it helps, because searching a system gets cheaper before changing one does. But the rationing that produced all of this was never a law of computing. It was a price, and prices move.

Sources

Links are given where I verified them. Entries without a link are cited by identifier only.

1. Provos, N., Friedl, M. and Honeyman, P., Preventing Privilege Escalation, 12th USENIX Security Symposium, 2003. Privilege separation in OpenSSH, and the reasoning behind it.
2. CVE-2014-0160, the Heartbleed buffer over-read in OpenSSL’s TLS heartbeat extension, disclosed April 2014. NVD base score 5.0 Medium under CVSS v2, later 7.5 High under v3.1.
3. Barth, A., Jackson, C. and Reis, C., The Security Architecture of the Chromium Browser, 2009. Multi-process sandboxing that isolates the renderer from the operating system.
4. Reis, C., Moshchuk, A. and Oikonomou, N., Site Isolation: Process Separation for Web Sites within the Browser, USENIX Security 2019, a roughly five-year effort, and Google, Mitigating Spectre with Site Isolation, enabled by default in Chrome 67, July 2018.
5. Verma, A. et al., Large-scale cluster management at Google with Borg, EuroSys 2015. The platform whose isolation and lifecycle model became the basis for Kubernetes.
6. Apple, Apple Platform Security. The app sandbox, mediated access between apps, and the assumption that processes can be suspended or terminated at any time.
7. Google, Architecting security for agentic AI, 2026. A user-alignment critic kept away from untrusted web content, and Agent Origin Sets that separate the origins an agent may read from the ones it may act on.
8. Saltzer, J. H. and Schroeder, M. D., The Protection of Information in Computer Systems, Proceedings of the IEEE 63(9), 1975. Least privilege, complete mediation, separation of privilege, and unwanted access paths. Full text · doi:10.1109/PROC.1975.9939
9. Ross, R., Winstead, M. and McEvilley, M., Engineering Trustworthy Secure Systems, NIST SP 800-160 Vol. 1 Rev. 1, November 2022. Systems security engineering as a subdiscipline of systems engineering.
10. NIST FIPS 140-3, Security Requirements for Cryptographic Modules. The cryptographic boundary, approved modes, and the operational environment outside it.
11. Hurst, R., FIPS 140-3 Validation, in Practice, 2026. A near-census of 415 validated modules read from the public CMVP record, covering what the boundary excludes and how the certified state ages.
12. Conway, M. E., How Do Committees Invent?, Datamation 14(5), April 1968. Organizations are constrained to produce designs that copy their own communication structures.
13. SpecterOps, BloodHound OpenGraph and the BloodHound documentation. Attack-path analysis over identities and privileges.
14. Opet, P., An open letter to our suppliers, JPMorgan Chase, April 2025. A major buyer arguing that suppliers ship features faster than security and that modern integration patterns have dismantled the boundaries that once contained a compromise.
15. Anthropic, Measuring LLMs’ impact on N-day exploits, June 2026. Firefox and Windows kernel evaluations, and the collapse of the reverse-engineering bottleneck.
16. Google, Chrome is stronger with every update, 2026. AI applied across discovery, triage, patch generation, and testing; Chrome 149 and 150 together fixing 1,072 security bugs, more than the previous twenty-three milestones combined, alongside a move to more frequent security releases.
17. Google, From finding to fixing: reducing maintainer burden with automated patches, 2026. CodeMender treating an OSS-Fuzz crash as an oracle, generating a root-cause patch and verifying that the crash is resolved without regressing other tests.
18. Joern and the Code Property Graph specification, after Yamaguchi, F. et al., Modeling and Discovering Vulnerabilities with Code Property Graphs, IEEE S&P 2014.
19. GitHub, CodeQL. Extracting a queryable representation of code including syntax, control flow, and data flow.
20. Brumley, D. and Song, D., Privtrans: Automatically Partitioning Programs for Privilege Separation, 13th USENIX Security Symposium, 2004. Automatic privilege separation, later made quantitative by Liu et al., Program-mandering, ACM CCS 2019.
21. Fabian, D., The evolving role of the red team in the era of agentic security, Google, August 2026. Reconnaissance and lateral movement approaching machine time, and defenders’ home-field knowledge of their own environment as an advantage that has to be instrumented before an adversary maps it faster.