Two questions follow any system that acts on its own. How do you know it works? Where are the guardrails?
A demonstration usually answers the first question. Something runs, in front of you, under conditions somebody chose and the thing gets done. A mechanism usually answers the second question and the mechanism is usually a limit placed as low in the stack as you can manage with a rule that can’t be bypassed.
Neither answer concerns a real knowledge of the system as built and implemented. A singular demonstration is evidence about one occasion. A limit is a promise about outcomes. The questioner wants to know whether the beliefs the system rests on are true and whether anybody would find out if those beliefs were false.
A working system is a set of claims
Any machine that acts carries assertions, most of them silent. A sensor reports what its calibration says the sensor reports. A controller is stable at the latency the deployment has and not at the latency the bench had. A model fits the memory the model was mapped onto and a component labeled perception does the thing its label names.
Some assertions can be stated precisely enough to come back false. Other assertions cannot and the second kind are the dangerous ones, because from the inside both feel identical. Confidence is no property of a claim and only belongs to the person holding the claim.
A claim you cannot be wrong about is not a strong claim. An unfalsifiable claim is an empty one and a system assembled out of empty claims runs beautifully until the day the world disagrees.
What a limit tells you
Fuses are good. An overcurrent or a thermal event sits outside any model’s domain and hardware belongs there doing what hardware does well. I build in gateware for exactly that purpose.
Notice what a fuse reports, though. A fuse forecloses one class of outcome and stays silent about whether anything you believe is true. A clamp can fire all day in a system whose every hypothesis is sound. A clamp can also sit untouched for years in a system wrong in ways nobody thought to write into the clamp, which covers most of the ways a system can be wrong.
A second difficulty follows. A limit burned into transistors is no law of nature. Somebody chose the limit and silicon realizes it, which holds equally for a threshold in software. The two differ in how long they take to change and in nothing else that matters. Immutability is a virtue if you were right. If you were wrong, you have made the wrong thing permanent and no rule will tell you so because rules do not report.
Physical law deserves care here, since physical law gets invoked as though it were a safety property. Gravity does not stop you falling off a cliff. Gravity is why you fall. Laws of that kind constitute what is possible and say nothing whatever about what is desirable. Physical law is the ground on which the problem stands.
The device nobody can build
Consider the only general intelligences available for study.
We do not regulate human conduct with an implant that watches the inside of the skull and vetoes disfavored thoughts. Nobody seriously proposes such a device. Were somebody to try, the objection would arrive from two directions at once, since the implant would not work and it misunderstands where conduct lives.
Persons are governed at action and consequence. We test people, license them for things that matter, hold them answerable afterward and revise our confidence when the record demands revision. None of those instruments inspects a mechanism. Every one of them can come back and report that our assessment of a person was mistaken, which is precisely their value. A licensing exam nobody could fail would certify nothing.
The deeper trouble with the implant is that obligation presupposes the power to do otherwise. A promise that cannot be broken is not being kept. Compulsion does not produce fidelity but abolishes the category and whatever such a device delivered would not be the thing anyone wanted when they asked for trustworthiness. Covenant does real work as a word where force does not, because a covenant binds a party who could defect.
Anatomy makes the same point in another register. The brain has no supervisor anywhere in it. Inhibition is neurons. Homeostatic plasticity belongs to the synapse. Neuromodulation broadcasts through the same tissue that does the computing. Refractory periods are properties of the cell that fires. Regulation never sits above cognition as a separate plane, since the same elements do both jobs and the most robust controller anyone has ever met is arranged that way.
Biology at large agrees. A cell stays alive through homeostasis, redundancy, repair, programmed self removal and immune surveillance. Thousands of adjustable loops, almost no interlocks and the arrangement outlasts anything we build with a fuse in it.
Neuromorphic computing inherits the same arrangement from the biology it takes as model. A brain inspired part has no supervisor to interrogate and no place to put one. A brain inspired part also resists the instruments built for other silicon, which causes its own trouble.
Consider what the standard instruments assume. Throughput assumes dense arithmetic proceeding at a steady rate. Parameter count assumes that capability scales with stored weights. Benchmark accuracy assumes a fixed test set standing in for the world. Latency figures assume batching, since batching is how a graphics processor earns its throughput number.
Event driven silicon answers none of those assumptions. Work happens when something changes, so cost depends on the statistics of the input and a quiet scene costs almost nothing. Sparse computation offers no dense matrix multiplies to count. Batching, the very move that produces impressive throughput elsewhere, defeats the latency advantage entirely. Continuous adaptation cannot be expressed on a static test set, since the test set is frozen and adaptation is the whole point.
Measured with instruments built for a graphics processor, neuromorphic silicon looks like a poor graphics processor. Torrance’s rule applies without modification. Method must suit its object and a method built for another object returns an accurate verdict about the wrong question. Honest measures here are energy per event under realistic input statistics, latency from stimulus to decision without batching, behavior under distribution shift and adaptation without retraining. None of those measures appears on the scoreboard the industry currently keeps.
A heterogeneous compute ontology forces the difficulty into the open. Treating quantum, neuromorphic, graphics and general purpose silicon as peers means that whether something works ought to be asked per substrate, with instruments proper to each. A single scoreboard spanning all of them is no neutral arbiter. A single scoreboard is an argument for whichever substrate the scoreboard was built to describe.
The paradigm decides what counts as a question
Before a claim can be tested the claim has to be formable and formability is not neutral. The obvious, as a field understands the obvious, sets the outer edge of what anyone thinks to check.
T. F. Torrance argued in Theology in Reconstruction that empirical science required a premise and that the premise was theological before it was methodological. Today, many might say philosophical but the result is the same. We presuppose certain givens in the doing of science. Creation gives the world an order both real and contingent. Things did not have to be this way, so nature cannot be deduced from first principles the way Greek rationalism attempted and one has to go and look. Creation also makes the world creature instead of divinity, so one may look, which the enchantment of the medieval imagination made difficult and we waited at least through Renaissance and Reformation for modern science to form. Nature stopped being something to venerate or read allegorically and became something to interrogate on its own terms.
Torrance pressed further and his further point is where I keep returning, and it is the rule I mentioned earlier. Method must suit its object. We let a thing disclose itself according to its own nature instead of imposing a framework no one asked the thing to accept. Suiting method to object is a discipline and not a pleasantry. Study something with instruments built for something else and you will conclude, with evidence in hand, that the first thing is a poor version of the second.
Michael Polanyi described the same difficulty as a problem of attention. We attend from subsidiary particulars to a focal whole and the two modes cannot be held at once. Look focally at the individual notes and the melody disappears. A discipline trained to attend to stones keeps reporting, accurately, that there is not much in a stone.
Polanyi also had an account of how frameworks change and the account offers no comfort. A framework is held with commitment and supplies the criteria by which evidence gets judged, so evidence gathered inside a framework seldom tells against it. Moving to another framework is no deduction. A logical gap sits in the way and crossing that gap resembles conversion more than proof. Polanyi’s account explains why argument so rarely relocates anybody. People are relocated by a demonstration they can attend from the inside.
Georg Lukács made the resistance into a principle and did so without embarrassment. The opening essay of History and Class Consciousness argues that orthodox Marxism consists in no particular thesis of Marx. Suppose every individual claim Marx advanced were empirically disproven. Orthodoxy would survive intact, since orthodoxy names adherence to the dialectical method and not belief in any result the method produced.
The operation deserves care, since nothing about it is automatically illegitimate. Method does legitimately outlast particular results and a discipline abandoning its methods at every disappointing finding would learn nothing at all. Accountability is what separates a durable method from an insulated one. Relocating orthodoxy to method achieved insulation, namely that no result about the world could bear on the commitment. A framework surviving the refutation of every one of its claims has purchased permanence at the price of contact.
The popular version of the move needs no philosopher. Every implementation that disappointed was not the thing itself. The theory was never really tried. Whatever happened belongs to the fault of the world, or of the people, or of the circumstances. The methodology remains available and untouched.
Technology runs the same argument now. Disappointing results from ever-increasing model size get read as evidence that the models were not yet large enough. Data was insufficient. Compute was short. Nothing in the pattern of outcomes is permitted to count against the approach, since the approach is the thing being defended and every shortfall becomes a call for more of it. Lukács at least stated the principle out loud.
Michel Foucault would say the belief matters less than the machinery producing the belief. Pastoral government does not forbid. Pastoral government shepherds. The pastor attends to each and to all, takes an interest in your development, offers guidance where a sovereign would issue commands and works by shaping the field of what you are likely to do. Markets excel at pastoral work. An industry never has to prohibit a line of research. The industry funds adjacent things, hires for adjacent skills, benchmarks adjacent quantities and after a while the adjacent thing is simply what the work has become.
The machinery is being described in the present tense. On a recent Moonshots episode, Alexander Wissner-Gross argued that we are seeing the end of the endless frontier, meaning the research settlement Vannevar Bush proposed to Roosevelt in 1944 which organized the funding of American science for eighty years. His complaint is not that the settlement was wrong when it was made. The settlement arrived through the lens of the war effort and produced what he called an over institutionalization of which research directions would get funded and pursued and which were appropriate. He is blunt about the behavior the apparatus then selects. The apparatus rewards incrementalism and the reputation is earned.
There stands the shepherd with a budget. Nobody forbade the other directions. Other directions were not what a review panel recognized as appropriate and after eighty years of such recognition, the appropriate is what the field has become.
James Scott showed what such a settlement does to its object. In Seeing Like a State he describes German scientific forestry, which set out to make a forest measurable. Everything that was not merchantable timber counted as noise, so the undergrowth went and the deadfall and the mixed species and the birds and insects living among them. Norway spruce in tight straight rows remained, countable and predictable in yield. The first rotation was magnificent. The second rotation came in stunted and sick and the foresters who had produced the disaster even named the condition Waldsterben, forest death. Simplification had stripped out the soil ecology on which the first rotation had quietly depended. The map reshaped the territory into the map’s own image and then discovered that the image could not live.
Measurement is not the villain in Scott’s story. A measurement regime selects for what the regime can see and the field rearranges itself to be seen. Parameter counts, benchmark scores and floating point operations are legible. An architecture whose advantage lives in its arrangement is not legible, which is the difficulty a cathedral faces when the available instrument is a scale for weighing a certain kind of brick.
Herbert Marcuse, working downstream of the inheritance Lukács crystallized, named the political form of the same problem in One-Dimensional Man. Technological rationality does not defeat alternatives by argument. Technological rationality absorbs alternatives until the standing arrangement looks like rationality itself and the alternatives become unthinkable well before anyone refutes them. One dimensional thought has not lost an argument. One dimensional thought no longer holds the vocabulary to have one and so society plods on without the benefit it could have had by envisioning a different world.
The word intelligence sits about there now. AI has come to mean neural networks, neural networks have come to mean transformers and transformers have come to mean GPUs in data centers of ever-increasing size, as though data centers are the way, the truth and the life. Argument settled none of the sequence. Capital allocation settled it and a supply chain makes a powerful epistemology. A roadmap is no evidence about what intelligence is, though a roadmap decides very efficiently which questions get funded, which get published and which never get formed.
What it looks like when a settlement gives
The defense case turns on the same document. Alex Karp opens The Technological Republic on that November 1944 letter from Roosevelt to Vannevar Bush, treating the letter as the founding of the partnership between the American state and American technical talent. Wissner-Gross names the same letter as the thing now ending. Two men arguing about quite different subjects locate the boundary on the same page, which usually means the boundary is real.
Shyam Sankar supplies the mechanism in Mobilize and the mechanism is economic. One customer buys the entire defense market, a position economists call a monopsony and a single buyer does more than purchase. A single buyer defines what a product is. Sankar charges that the rules grown up around the position came to privilege the well connected over the innovative and that the Pentagon tried to central plan its way to success. Americans, he observes, turn out to be bad commies.
Consolidation followed and the consolidation was explicit. In 1993, with the Cold War over and procurement cut by more than half, Deputy Defense Secretary William Perry hosted a dinner for the heads of the leading defense contractors and told them the business would no longer support all of them. Military circles remember the evening as the Last Supper. Roughly fifty aerospace and defense companies became five. An irony sits inside that dinner, since Perry had been a founder himself, a man who left a vacuum tube company in 1964 because he could see a digital wave few others saw coming.
Scott would recognize what came next. Three of the survivors built the Army’s battlefield intelligence system, fielded in 2007. The program satisfied every procurement requirement while soldiers could not use the result. The system demanded hours of training, crashed often and remained unusable in remote locations. An officer who carried it in Afghanistan said you could not see anything as a whole. Optimizing for legibility buys exactly that outcome.
The settlement is coming apart now and nobody won an argument to make the change happen. Palantir sued the Army for the right to compete. Anduril grew out of the same network. A field that held five members holds a great many small builders today. The case against the primes had been available for years and moved nothing. The length of the feedback loop changed. In Ukraine and in the recent exchanges around Iran, equipment gets fielded and refuted in days instead of program cycles, in public, with consequences nobody can staff their way around. War is an unusually falsifiable environment. Something works or does not work, quickly, and the institution has to notice.
Procedural legibility was the primes’ advantage and the advantage held for exactly as long as the loop concealed what lived within it.
Who measures the measurers
A test is a claim about what matters and the claim is rarely stated as one, still less tested.
Every measurement carries assumptions. The quantity measured serves as proxy for a thing somebody valued. The threshold announced beforehand encodes a judgment about how much counts as enough. The population sampled encodes a judgment about who counts at all. Ask of any measurement who chose the instrument and what that person needed to be true and the answers are often more interesting than the reading.
Development work offers the clearest case at scale, since the measurement regime there is explicit, funded and global. The United Nations Sustainable Development Goals arrived in 2015 with seventeen goals and one hundred sixty nine targets, tracked by more than two hundred indicators. Universality was the announced virtue. Every country would be measured on one frame, so nobody could claim exemption and nobody would be left behind.
Universality of that kind carries a cost the framework cannot see from inside. The indicators encode a single model of a good life, assembled out of formal employment, formal education, formal financial inclusion and measured national output. A people organized around other goods does not register as different on the index but as behind, lacking the development the Sustainable Development Goals intend. Deficiency becomes an artifact of the instrument and the instrument then directs the money that arrives to repair the deficiency the instrument invented.
The category has a birthday. Truman’s inaugural address in 1949 named two thirds of the world underdeveloped areas and Arturo Escobar traces in Encountering Development the apparatus that followed, one that constituted the people it proposed to help as lacking. Reform has worked at the problem through data disaggregation, qualitative method and country led review. Reform of that kind acknowledges the difficulty while operating inside the architecture that produced the difficulty.
Escobar’s later work names the alternative. The pluriverse is a world in which many worlds fit, set against the assumption of a single world where everything else amounts to a deficient approximation of it. Design in Escobar’s account is never neutral, since what we build goes on to shape what we become. A measurement is a design.
The same structure operates in compute and will operate harder as the substrates multiply. A benchmark asserting universality across substrates does to neuromorphic silicon what a development index does to a people whose goods the index cannot express. Neither instrument reports hostility. Each instrument reports, accurately, that the thing in front of it falls short on the axis the instrument was built to read.
What a proof of concept proves
Enterprise technology keeps a standard for whether a thing works. The standard is a proof of concept at a client site, followed by a minimum viable product, followed by a production deployment somebody will name in a case study. Clearing those gates is what the industry means by proven.
Consider what clearing them establishes. One organization, holding one set of data, under one set of constraints, with one procurement process and one internal sponsor, agreed to run something and did not reject the result. The finding is a social fact at least as much as a technical one and the social part is not incidental. A proof of concept is the instrument by which a technology becomes legible to an institution’s purchasing apparatus.
Legibility of that kind decides who is permitted to be proven. Access to a client environment is granted by people whose interests the incumbent already serves, which returns the question to Foucault’s shepherd. Nobody forbids the alternative architecture. The alternative architecture simply never receives a pilot.
Approval and working also come apart and the defense case already showed how. The battlefield intelligence system fielded in 2007 satisfied every procurement requirement in force at the time. Institutional approval was complete. Soldiers could not use the result. Any standard conflating the two will go on certifying approval while claiming to report on function.
A pilot or proof-of-concept answers a narrow question well. Does this thing survive contact with one particular world, including that world’s data quality, its security review, its change control and its politics? Fitness for an enterprise environment is worth knowing and a pilot measures fitness honestly.
A pilot cannot measure whether an architecture generalizes, since a pilot has a sample size of one and the sample was chosen for accessibility.
The academy runs the same structure with different vocabulary. Peer review is the permission instrument for knowledge and the properties match. Cycles run in years. Access is granted by people invested in the arrangement under review. Incrementalism is rewarded, as Wissner-Gross said plainly of the funding apparatus, so a proposal that would unsettle its reviewers reads as a proposal that will not be funded. Grants take two years to award. Investigators in the biomedical sciences receive a first grant of their own in their early forties.
Corporate publishing adds a third gate with a different filter again. An engineer inside a large firm who wants to show a working result passes legal review, communications review, competitive review and partner review before anything reaches the public. Not one of those reviews asks whether the result is true. Each asks whether the result conflicts with a message already in the market, exposes a weakness a competitor could use, embarrasses a partner or moves a number that will draw questions on an earnings call.
A marketing message is a claim protected from testing. Once an organization commits to a message, evidence complicating the message becomes a liability instead of a finding and the machinery for handling liabilities is not the machinery for handling findings. The protection is rarely dishonest. The protection is procedural and the procedure carries an outcome the procedure never states, namely that the public record of what a technology can do gets assembled out of whatever was commercially safe to say.
Delay compounds the effect. By the time approval arrives the work has moved on, so the published version describes a system that no longer exists and the distance between what an organization holds and what an organization has said becomes permanent.
Three gates, three filters, and not one of them asks the question a reader actually wants answered.
Beneath the three gates sits something that is not a gate. The shape of the disciplines decides what can be formed at all and the shape answers to no object any discipline studies.
Lewis Ayres and Michel René Barnes made the case from inside patristic scholarship, which is close to the last place anyone would look for it. Modern readings of fourth century theology, they argue, have been distorted by three features of the contemporary academy. Disciplinary categories get constructed that the ancient writers never shared. What those writers held together gets separated into fields no longer on speaking terms. Credentialing penalizes the synthesis that would put the pieces back together.
Consider what the diagnosis means. A tradition with unbroken attention to its own sources, staffed by scholars of enormous learning, reading texts that were never lost, still misreads the fourth century because the twentieth century organized its universities a particular way. Ignorance explains none of it. The distortion is structural and a structure of that kind stays invisible to the people working inside it, which is the property that makes structures dangerous.
The doctoral requirement does its own work here. An original contribution to a densely worked field means finding ground nobody has turned, so the search selects for narrowness ahead of importance. Funding rewards the same motion, as Wissner-Gross observed of the American research apparatus and a career gets built on a strait instead of a sea. Newman argued in 1852 for a philosophical habit of mind able to hold things together. MacIntyre concluded in 1990 that the modern university had become fundamentally incoherent. I have written elsewhere about what the university was and what comes next and the argument there is that the integrative capacity specialization suppressed has become available again from another direction.
I publish in those venues and I value the work they carry. I have not made the venues the measure of whether a thing is true. I build and I set out the reasoning in Build It Anyway, an essay on Sankar and Karp and institutional decay. Institutions optimize for self preservation and accumulate the process meant to prevent failure until the process guarantees it. Seventeen layers of review separate the person who sees a problem from the person who can decide. The immune system attacks an internal critic more fiercely than an external competitor.
Karp and Sankar both argue for building and both have built. William McLean built the Sidewinder against instruction, disguising the work as a local fuze project. Hyman Rickover put the matter plainly when he said his job was not to work within the system but to get things done. The most radical act available inside a decaying institution is a working prototype delivered before anyone granted permission, since a working prototype removes theoretical impossibility from the argument.
Building is no escape from evidence. Building is a harder exposure, since an artifact carried into a new domain either works there or does not, in public, on a schedule nobody controls.
Computational development names the space I take that work to occupy. Code is a way of thinking about a domain and not merely a tool for implementing conclusions reached elsewhere. Implementation forces decisions theory leaves undetermined and surfaces dynamics theory alone cannot anticipate. Adjacent fields each leave the constructive question unasked. Work in information and communication technologies for development applies existing technology to goals set in advance without reconsidering the assumptions built into the technology. Development informatics criticizes the technology without building an alternative. Critical data studies exposes extraction without constructing a system that does not extract.
A different question becomes available once demonstrations accumulate. I have built close to seventy of them across rail, orbit, spectrum, genomics, wildfire, sleep, music and drone coordination, on neuromorphic, quantum, graphics and general purpose silicon. Any one of them proves very little alone. Taken together they test something no pilot can reach, namely whether the heterogeneous compute ontology carries when the domain changes underneath it.
Every demonstration is an opportunity for the ontology to fail. A new domain arrives with its own sensors, its own latencies, its own regulatory shape and its own definition of a good outcome. Either the ontology accommodates the new domain without special pleading or the ontology does not. Seventy chances to be refuted, taken in public, is a different kind of evidence from one pilot that cleared a procurement gate.
The word proof does two jobs and the jobs are opposed. Proof in the institutional sense names a permission granted by people entitled to grant it. Proof in the constructive sense names an artifact that survived a test the artifact could have failed. The old paradigm asks for the first and calls it the second. A newer one asks whether an invariant survived being carried somewhere the invariant had not been, which is the only question a general architecture can honestly be asked.
None of that argues for building over thinking and none of it is peculiar to me. The practices that let a demonstration count for anything are ordinary and available to anyone who wants them.
Making a claim falsifiable
Five practices carry most of the weight and not one of them is glamorous.
State the claim so that the claim can return false. “The perception module works” is not yet a claim. A specific quantity, measured a specific way, against a threshold announced beforehand, is a claim. The first formulation cannot fail. The second can fail in front of you, which is what makes stating it worthwhile.
Instrument the claim and not only the outcome. An outcome test reports that the run completed. An outcome test says nothing about why and goes on passing while the reason underneath quietly changes. The German foresters had superb outcome data for an entire rotation.
Vary one thing. Hold everything else and move a single parameter until the behavior breaks, then look hard at where the break occurred. A dose response curve turns a vague worry into a sensitivity with a number attached and names the variable you are actually fighting.
Test long enough that the test could fail. A short window on a noisy system is no lie about the system. A short window is a test too weak to be failed, which is worse, because a weak test hands back the same reassuring answer a strong test would have handed back if you were right.
Keep the hypotheses that did not survive. Refuted hypotheses are results. A design that did not do what the theory expected has told you something about the object no further theorizing would have surfaced and the next attempt should be built on that knowledge instead of around the memory of it.
One question sits underneath all five practices. What would have to be true for this to be wrong and would I see it?
Where the guardrails actually are
Guardrails live inside the loop and take the form of regulation.
A safe envelope computed continuously from the current state, with action bounded against the envelope at every step, moves as conditions move. A speed safe on a dry road becomes dangerous on the same road in rain. A fixed clamp is therefore wrong in both directions at once, too tight everywhere the clamp is not needed and still not tight enough where the clamp was aimed. Confidence that fades as a system approaches the edge of its evidence does more work than any wall and confidence does that work without forbidding anything.
Beneath the regulation sits what I have been calling a covenant and the word carries a specific meaning. A covenant states what must remain true and leaves the mechanism free to be whatever the substrate allows, gateware included. The usefulness has nothing to do with being soft.
Truth in the register I mean here is not correspondence. The modern habit takes a claim to be true when the claim matches a state of affairs, which is a serviceable account for a proposition and a poor one for a bridge. An older conception runs differently. The Hebrew scriptures work with a family of words around emet and emunah in which truth means reliability, steadfastness, the quality of a thing that holds when weight comes onto it. The root behind them stands behind amen, a word asking that something be made firm or recognizing that it always is. A true road does not give way. A true friend does not vanish. A true beam carries the load placed on it.
Faithfulness in that sense is what a covenant asks for and the sense is the one engineering has always quietly used. Nobody asks whether a bridge corresponds. Everybody asks whether the bridge holds. The question is temporal and it is loaded, since holding is something a thing does over time and under conditions instead of something a statement possesses at an instant.
Physics enters here the way the opening described. Laws constitute what is possible and never prescribe what is desirable, so faithfulness to physics is no obedience to a rule but a refusal to lie to oneself about the terrain. A design pretending that grip is available where grip is absent has broken no law, since no law of that kind can be broken. The design has made a claim the world will decline to honor and the world will decline at a moment of its own choosing.
A covenant asks what must go on holding. The mechanism stays free because holding can be secured many ways and invariant preservation is the standard because the only evidence that a thing held is that the thing went on holding. You check that what had to remain true did remain true, across changes to the mechanism underneath. The standard sits higher than proving a clamp fired and unlike a clamp, the standard is one you can fail.
Invariants belong to a domain and not to the universe. A covenant is the pluriversal form of a guardrail. A covenant asserts no single governance for every world. A covenant states what must remain true in the world actually in play and asks faithfulness to that world. Faithfulness requires one thing of a vehicle at the limit of grip, another of a spacecraft holding attitude on a shrinking power budget and something else again of a clinical instrument. New designs and new behaviors have to act in line with the stated goods of the domain in which they operate, which is why the word carrying obligation does work the word carrying force cannot.
What can really be said
Return to the two questions.
How do you know it works? Strictly, you do not. Nothing in silicon or process will change the situation. What you can hold is a set of claims that have so far survived tests capable of refuting them, carried into domains that could have broken them, with more tests still coming. Knowledge of that kind is provisional by construction and it is the only kind on offer for anything that acts in the world.
Where are the guardrails? In the same place. A practice able to report that you are wrong is the guardrail. A limit that forecloses without reporting is a fuse, which is a fine thing to own and a poor thing to mistake for knowledge.
What remains is a discipline instead of a guarantee. Guarantees are what an immutable rule promises and cannot deliver, since a rule that cannot be broken has removed the possibility of learning that the rule was wrong in the first place. The discipline is smaller and harder. State the claim so the claim can fail. Carry it somewhere it has not been. Watch whether it holds. Keep the record when it does not, since that record is the only thing that improves the next design.
A system built that way makes no promise of never being wrong. The promise is smaller and worth a great deal more, namely that being wrong will show, early and in public, while there is still time to build differently.