Healthcare organizations spend significant effort deciding what an artificial intelligence system should do. They should give equal attention to a less glamorous question: When should the workflow stop?

A stop state is a defined condition that prevents an AI-assisted step from silently advancing to the next clinical or operational action. It returns the work to a named person, preserves the reason, and makes the next permitted action explicit. This is a practical foundation for healthcare AI governance because it translates a policy such as “a human reviews the output” into an observable workflow.

Without that design, human oversight in healthcare AI can become ceremonial. A reviewer may see an output but lack the source, authority, or workflow controls needed to challenge it. A warning may appear, yet the downstream order, message, claim, or task may already have moved forward. The organization technically kept a person in the loop, but the person did not control the loop.

The better design question is not simply whether a human is present. It is whether the system can stop safely, assign responsibility, preserve evidence, and resume only after an authorized decision.

Begin with the decision boundary

Before configuring a model, define the action boundary around its output. Is the system drafting text, classifying information, proposing a next step, prioritizing a queue, or initiating an external transaction? Who may accept, change, or reject that output? What must remain impossible until that person acts?

This distinction matters because an AI output is not automatically the same thing as a clinical decision or completed work. A draft note is not a signed note. A suggested code is not a final coding decision. A predicted priority is not an authorized order. A prepared authorization packet is not a payer transmission.

The federal HTI-1 Decision Support Interventions criterion provides a useful transparency anchor for certain certified health IT. Its source attributes include the intervention’s intended use, intended patient population, intended user, intended decision-making role, and cautioned out-of-scope uses. The criterion also addresses access to plain-language source information and risk-management practices for predictive interventions supplied as part of certified health IT. These requirements do not prescribe one universal user-interface pattern, but they reinforce a critical design principle: the system’s purpose and limits need to be visible where decisions occur.

An organization can turn that principle into five explicit stop states.

1. Stop when required source evidence is missing

An output should not advance when the reviewer cannot inspect the evidence required for that use. The missing item might be a source document, a current laboratory result, the version of a clinical reference, the input features used by a predictive model, or the provenance of imported data.

The workflow should say what is missing and what can happen next. “Unable to proceed: source document unavailable” is more useful than a generic error. The task should remain assigned, or route to a named queue, rather than disappear behind a warning banner.

The required evidence will vary by use case. A documentation draft and a patient-specific recommendation do not need identical controls. The organization should define the minimum review packet for each workflow and test what happens when every required element is absent, stale, or inaccessible.

2. Stop for an out-of-scope population, setting, or use

Every AI-assisted workflow needs a visible operating boundary. That boundary may involve age, specialty, care setting, geography, data availability, device type, or the action the user is attempting to take.

HTI-1’s source-attribute framework expressly includes cautioned out-of-scope uses for predictive decision support interventions, including tasks, situations, or populations where use is cautioned and known limitations or inappropriate settings. A workflow should do more than place those details in documentation that few users open. When the system detects an out-of-scope condition, it should stop the affected step and explain the boundary in plain language.

The stop should not imply that the patient or case is invalid. It means only that this intervention is not approved or configured for that situation. The next action may be a standard non-AI workflow, escalation to a specialist queue, or closure with a recorded reason.

3. Stop when the result does not meet the organization’s review threshold

Not every model exposes a meaningful confidence score, and a numeric score should not be mistaken for truth. Organizations still need a documented rule for outputs that are incomplete, internally inconsistent, ambiguous, or otherwise insufficient for the intended decision.

That rule should be specific to the use. For example, a document-classification workflow might stop when two categories remain plausible. A summarization workflow might stop when required sections cannot be grounded in the available record. A predictive intervention might stop when inputs fall outside its validated or locally accepted conditions.

The important point is that the threshold belongs to the governed workflow, not to a universal percentage copied from a model dashboard. Reviewers need to know why the work stopped, what evidence they can inspect, and whether they may correct the output, request more information, or abandon the AI-assisted path.

4. Stop at an authorization boundary

Some actions require an identified person with an appropriate role to approve, sign, attest, transmit, or finalize them. The AI system may prepare the work, but it should not cross that boundary by implication.

Design the authorization step as a state transition, not merely a button. Record who is authorized, what they are approving, which version they reviewed, when they acted, and whether they accepted, modified, or rejected the proposed output. If the output changes after review, the prior authorization should not silently carry forward.

This is where “human in the loop” becomes operational. The person has actual control over whether the workflow continues, and the record distinguishes the system’s contribution from the human decision.

5. Stop when an external handoff fails or remains unconfirmed

Healthcare workflows cross organizational and technical boundaries: an order moves to a laboratory, a referral packet moves to another organization, a message moves to a patient, or an authorization request moves to a payer. Preparing or sending data is not the same as confirmed receipt or completed work.

The workflow should represent the handoff state explicitly. If transmission fails, acknowledgment does not arrive, or a response cannot be matched to the original request, the work should move to a failed, held, or exception state with an owner and follow-up path. It should not display a generic “complete” state simply because the local system finished its portion.

This stop state is especially valuable because it exposes a common source of context loss: each system knows what it attempted, but no one owns the gap between them.

Represent the stop as workflow data

Standards can help make stop states portable and inspectable, although using a standard does not by itself complete the governance work.

In HL7 FHIR R4, the Task resource can represent an activity’s status, business status, reason for the current status, owner, inputs, outputs, and relevant history. An implementation might use those elements to record that work is on hold, why it stopped, who owns the next step, and what evidence is attached. Exact profiles and codes should be defined for the organization’s use case rather than improvised across teams.

The FHIR Provenance resource can record information about the activity that created, revised, deleted, or signed a resource, including the agents and entities involved. That can help distinguish an AI-generated draft, a human edit, and a signed result. Provenance and workflow state serve related but different purposes: one helps explain how information came to be; the other helps coordinate what must happen next.

Consider a synthetic example. An AI-assisted workflow prepares a follow-up recommendation from a patient record. One required source document is unavailable. The system places the task on hold, records “required source unavailable” as the reason, assigns the task to the appropriate review queue, and preserves references to the inputs that were available. A reviewer obtains the missing document, edits the draft, and authorizes the next action. The workflow then records the human decision and the source used before it resumes.

The model did not need to resolve every exception. The workflow needed to make the exception visible, owned, and recoverable.

Test recovery, not just accuracy

Model evaluation often focuses on the quality of outputs. Production readiness also depends on what happens when the system should not proceed.

A predeployment test should intentionally trigger each stop state and answer six questions:

1. Did the workflow stop before the protected action?
2. Did it explain the reason in language the intended reviewer understands?
3. Did it preserve the relevant sources, inputs, output, and version?
4. Did it assign the next step to a named role or queue?
5. Could an authorized person correct, reject, or resume the work?
6. Did the audit record distinguish the AI output from the human decision?

NIST’s voluntary AI Risk Management Framework is designed to help organizations manage AI risks across design, development, deployment, and use. A stop-state test is one concrete way to operationalize that broader risk-management work within a healthcare workflow. It does not replace clinical, legal, privacy, security, or regulatory review, but it gives those reviewers something observable to evaluate.

Healthcare AI governance becomes stronger when a limit is represented as more than policy text. A well-designed stop state protects the decision boundary, tells people what happened, preserves the evidence, and gives the work a safe path forward. Before asking whether an AI system can complete a task, make sure the surrounding workflow knows when it must not.

Sources

• Assistant Secretary for Technology Policy/Office of the National Coordinator for Health Information Technology, Decision Support Interventions certification test method.
• Assistant Secretary for Technology Policy/Office of the National Coordinator for Health Information Technology, HTI-1 Decision Support Interventions fact sheet.
• HL7 International, FHIR R4 Task.
• HL7 International, FHIR R4 Provenance.
• National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework 1.0.