Human + AI: The System Is Both

The system is both

What gets deployed is not a model sitting alone on a server. It is a joint system: a person, an interface, a model, tools, and the rules about who may act.

AI is a category of machine-based systems. [nist-ai-glossary] [oecd-legal-0449] It is not defined as learning. Search, rules, and planners sit in the same category. See What is AI? and AI Is More Than Machine Learning.

This page is the design of that joint system. Who does which part of the work. When the machine should stop. What the screen shows. Who still owns the call.

It is not Ethics, Bias & Fairness. That lesson is principles, fairness, and accountability. Link it when the question is “is this fair.” This page is “who acts.”

It is not AI Safety. That lesson is risk and alignment. Do not recopy it here.

It is not a redo of Evaluating AI or How to Check an AI Claim. You may say this, and then stop: whether the joint system is good is an evaluation of the composed system (human + model + interface + tools), not of the model alone.

Three allocations of work

Automation, assistance, and human-only control are allocations of work. They are not moral slogans. The same task can move among them.

• Automate. The machine does the action under a rule. A person set the rule. A person may never see the case.
• Assist. The machine proposes. A person owns the call.
• Human-only. A person does the whole task. The machine stays out.

NIST’s Appendix C on human-AI interaction says configurations can span from fully autonomous to fully manual: the system can decide, defer to a person, or sit as an extra opinion. Some systems need no human oversight. Others specifically do. [nist-airc-human-ai] [nist-ai-rmf-2023]

Keep a person in the loop when the cost of a silent error is high, when the task is underspecified, or when accountability requires a named person.

A draft is not a signed note

A doctor finishes a visit. A model drafts the note and a medication line. The interface can show uncertainty (“the dose was not in the transcript”) or hide it. The doctor still owns the call: the diagnosis, the prescription, the chart.

Skipping the review is the failure mode. Fluency is not a check. Probability, Uncertainty, and Confidence already taught that fluent language is not confidence. This page adds the allocation: the person still has to act, or escalate.

The deployed system is the doctor, the interface, the model, the record tools, and the rule about who may sign. It is not the model alone.

Automation bias is over-trust. Appropriate reliance is the goal.

Automation bias is over-trusting a fluent suggestion. A clean paragraph looks finished. A ranked route looks decided. People skip the check.

Appropriate reliance is using the suggestion when it is actually reliable, and not using it when it is not. Maximum trust is the wrong target. Trust should track reliability, not fluency. [lee-see-2004] [parasuraman-riley-1997]

A fluent draft is not a checked draft.

Cognitive offloading is a design consequence

If the machine always drafts the note, the person can lose fluency in writing it. They can forget which fields they used to check by hand. That is cognitive offloading: moving the work out of the person’s head, and sometimes losing the skill.

This is not a sermon about character. It is a design consequence of the allocation you chose. If the leftover human tasks are the hard ones, and practice is gone, the joint system is weaker when the machine fails. Bainbridge called that pattern an irony of automation. [bainbridge-1983]

What the interface shows is a design choice

The screen is part of the system. It can show a suggestion, uncertainty, a source, and who is accountable. It can hide all of those.

Hidden uncertainty is a design choice. So is a button that says “Accept” and nothing else. Probability, Uncertainty, and Confidence is the lesson on scores versus calibrated probabilities. Here the point is allocation: what you hide changes whether a person can actually review.

Escalation, feedback, and rubber stamps

Escalation is when the machine should stop and a person should take the decision. Write the rule before the busy hour. High cost of a silent error. Underspecified task. A case that falls outside the intended use.

Suggestion, then review, then act or escalate. That is the loop. A rubber stamp is a false loop: the human appears to review, always clicks yes, and the labels look like human judgments. Those labels can become training data. Data: What Machines Actually Learn From already taught that feedback is a sample, not the world. Hollow review makes hollow labels.

If the system calls tools, permissions are part of the joint system. Who may send the message, write the chart, or move the dispatch. See AI Agents & Tool Use for the agent. This page does not write the security lesson.

Responsibility does not vanish

A model produced the draft. The organization and the person still own the decision. Responsibility does not evaporate because the text was fluent.

Fairness, bias, and governance belong on Ethics, Bias & Fairness. This page is who acts.

What to take with you

• The system is both: user, interface, model, tools, and the rules about who may act.
• Automate, assist, and human-only are allocations of work. The same task can move.
• Keep a person in the loop when a silent error is costly, the task is underspecified, or accountability requires a person.
• Automation bias is over-trust. Appropriate reliance tracks reliability, not fluency.
• Cognitive offloading is real. Skill fade is a design consequence.
• Escalation is when the machine stops and a person takes the decision.
• The organization and the person still own the call.