Verification

Has it worked outside the lab?

A technology passes its pilot. The results are good, the report is positive, the certification is in order. It is procured and deployed.

Then it meets conditions nobody tested. Operators under time pressure instead of an engaged project team. Integration with systems that were never in the pilot. Degraded power, poor connectivity, interference, cold, dust, load. An adversary who behaves differently from the test scenario. Within months, the capability that worked in the pilot is being worked around.

Most of the people involved did their jobs properly. The pilot showed what it was designed to show. The problem is that it was taken to show something else: that the capability works where it is needed.

This is the problem we mean by verification. Capabilities are routinely deployed on the strength of evidence gathered somewhere else, under other conditions, often by someone with an interest in the result. Until a capability has been shown to work where it is needed, its deployment rests on an assumption.

Our position is simple to state. Validated is not deployed. Proof is a property of a capability under stated conditions, and it is only as strong as the conditions in which it was gathered.

A successful pilot proves the pilot

Pilots are designed to succeed. That is not cynicism; it is how they are set up. The scope is limited, the team is motivated, the supplier is closely involved, the environment is prepared, and problems are fixed as they arise.

All of this makes a pilot a good test of whether something can work. It makes it a poor test of whether it will work once the supplier steps back, the original team moves on, and the capability has to run inside ordinary operations.

The gap between the two is well known. In innovation policy it is often called the “valley of death”: the stretch between a promising result and a deployed capability, where many initiatives stop. The usual response is more funding for more pilots. That produces more promising results, and the same valley.

What is missing is rarely another pilot. It is a way of gathering evidence under the conditions the capability will actually face.

Verification and validation are different questions

Systems engineering has long separated two questions that are easy to blur.

Verification asks whether the system was built right: does it do what its specification says? Validation asks whether the right system was built: does it do what its users actually need, in the setting where they need it? Standards such as IEEE 1012 treat the two as distinct activities for good reason.

A capability can pass verification and fail validation. It meets every requirement on paper and still does not work for the people and the conditions it was bought for, because the requirements themselves were written for a cleaner world.

We use “verification” in the broad sense that people outside engineering use it: establishing, with evidence, that a capability works where it is meant to. That includes both questions. Answering only the first is how systems end up certified and unused.

Readiness levels measure maturity, not fit

Technology Readiness Levels are a useful common language. They describe how far a technology has travelled, from basic principles to a system proven in its operational environment. European research programmes use them to describe where a project starts and where it should end.

They are also easy to over-read. A readiness level says how mature a technology is along a general path. It does not say whether it fits a particular institution’s conditions, systems and people. A system demonstrated in “an operational environment” has been demonstrated in some operational environment, not necessarily in yours.

We therefore treat a readiness level as the start of a conversation about evidence, not the end of one. The question that follows is always the same: demonstrated where, under which conditions, and how close are those to ours?

The environment is part of the test

Where a capability is tested shapes what the test can show. A laboratory can establish performance under controlled conditions. It cannot establish behaviour under the conditions that make real operations hard.

Different environments expose different kinds of failure. An isolated or hardened setting tests what happens when support, connectivity and supply are cut off. A live maritime setting tests exposure to weather, corrosion, distance and the difficulty of physical access. A setting that crosses an administrative or national boundary tests whether authority, procedures and handovers work when they have to. An ordinary office or a cloud test environment tests none of these.

This is why we treat the test environment as an instrument in its own right. Choosing it is a design decision: it determines which failure conditions can be exercised, and therefore which claims the result can support.

Every result has conditions

A test result that does not state its conditions invites being generalised beyond them.

For any verification result, we want to know at least: what was tested, in which configuration, in which environment, by whom, under which conditions, for how long, what was measured, what happened, and what was deliberately left out. The last item is the one most often missing and the one that matters most to the next buyer.

A narrow result that states its limits is useful. It can be relied on, compared and built on. A broad claim without conditions can only be believed or not.

This is the same discipline that applies to resilience claims. A result inside one envelope does not prove anything outside it. See Resilience.

Negative results are evidence

Tests that fail tell us at least as much as tests that pass. They show where a capability’s limits actually are, which conditions it cannot handle, and which assumptions in its design did not hold.

They are also the results most likely to disappear. Suppliers have little reason to publish them. Buyers may not want to admit to a failed procurement. Programmes are judged by success stories.

We treat documented failure as part of the evidence base. A verification record that contains only passes is either describing a capability that was never pushed to its limits, or leaving something out.

The buyer must be able to judge

Most verification evidence reaches an institution through someone with an interest in the outcome: a supplier’s test report, a certification obtained by the vendor, a reference from another customer. None of these is worthless. None is independent.

An institution that cannot read such evidence critically has no way to tell a strong claim from a well-presented one. It cannot ask which conditions were tested, whether they match its own, or what was left out. It buys the label.

This is where verification depends on judgement. The buyer needs people able to evaluate evidence produced by others and to ask for the evidence that would actually settle the question. That capability has to exist inside the institution, not only with its advisers. See Capability transfer.

From pilot to infrastructure

The end point of verification is not a report. It is a capability that the institution operates, under its own conditions, with evidence that it works there.

Many initiatives end at the pilot because the funding does. The result is a demonstration that nobody runs. We treat a pilot as one step in a longer sequence: test under controlled conditions, then under the conditions that matter, then in operation by the people who will run it, with the evidence recorded at each step.

That is the Operations discipline in The sequence: implemented, tested under real conditions, and converted into something that functions independently and durably after the initiative that created it has ended.

How we work on it together

Our work on verification with a partner begins with the conditions the capability will face, not with the capability itself.

Name the operational conditions. Agree where and how the capability will actually be used, including the conditions nobody likes to plan for.

Read the existing evidence. Establish what has been demonstrated, where, under which conditions, and what was left out.

Design the test around the gap. Choose an environment and scenarios that exercise the conditions the existing evidence does not cover.

Record the result with its conditions. Including what failed, and what was not tested.

Leave the ability to judge. The institution should end with people who can assess the next supplier claim, test report or certification on its own.

When we look at a capability together with a partner, these are the questions we typically work through:

Question	What it reveals
Where, and under which conditions, will this actually be used?	What the evidence needs to cover.
What has been demonstrated so far, and where?	The real extent of existing evidence.
Who produced that evidence, and with what interest?	How much independent weight it carries.
Which conditions have never been tested?	The gap a new test must close.
What environment can exercise those conditions?	Whether the test can support the claim.
What failed, and what was left out?	The limits of the result.
Who will operate it, and have they tested it themselves?	Whether the pilot can become infrastructure.

Where we are careful

We hold ourselves to a few commitments about what we claim.

We do not generalise a result beyond the conditions in which it was obtained.

We do not treat a readiness level, a certification or a supplier report as proof that a capability works in a particular setting.

We do not present test environments as results. An environment makes stronger evidence possible; it is not evidence in itself.

We do not leave failures out of the record.

And we do not describe a capability as proven without naming where, under which conditions and by whom.

Changes to this position

24 September 2026 — First published.

Material changes are listed here with their date. Corrections of wording or links are not.