Start with friction, not the tool
The strongest AI implementations rarely begin with “which model should we use?”. They begin with a careful look at the work. Where do people repeat the same step? Where does information disappear between systems? Where does a decision need better evidence without surrendering human judgement?
Automation earns its place when it removes genuine friction. Choosing the technology first often produces an impressive demonstration that changes very little in everyday work.
Five control questions
Does the problem happen often? Can the result be checked? Is a mistake reversible? Does a person retain control of important decisions? Is the time saved greater than the cost of maintenance? These questions quickly separate fashionable ideas from valuable ones.
High risk and weak oversight call for more restraint. Repetition, clear quality criteria and an easy route back to the previous state create a much better starting point.
Five questions before automating
An original framework for assessing the idea before choosing a model or vendor.
- 01Frequency
Does the problem recur often enough?
- 02Verifiability
Can a person or test check the output?
- 03Reversibility
Can a mistake be undone quickly?
- 04Control
Who owns the important decision?
- 05Economics
Does the gain exceed full maintenance cost?
A small loop and fast learning
Instead of automating an entire workflow, choose one closed loop: input, decision, output and review. Measure time, quality and exceptions. Expand only when that loop proves dependable.
Being modern does not mean putting AI everywhere. It means placing it deliberately where it creates a clear advantage and remains understandable to the people using it.
A process map exposes the real source of cost
Mateusz’s framework begins with observing the work rather than selecting a tool. A team records process steps, waiting time, manual data transfer, recurring questions and decisions that repeatedly require the same evidence. Only then does it identify friction that is frequent, costly and measurable. AI becomes a candidate for solving a defined problem instead of a compulsory addition to every product or organization.
Not every delay requires automation. Removing an unnecessary step, standardizing a form or assigning clear ownership may be better. Mateusz proposes three questions before building: must this step exist, can the rule be simplified, and does the outcome require generation or merely reliable retrieval? This order prevents teams from investing in a sophisticated system that only executes a poorly designed process more quickly.
Match autonomy to the cost of error
The level of automation should follow the reversibility of an action and how easily its result can be checked. Low-risk classification, a draft summary or a proposed response may run automatically when errors are visible and cheap to correct. Publishing, sending a message, changing permissions, making a purchase or deciding something that affects another person needs a stronger gate. In Mateusz’s model, AI can prepare the recommendation and artifacts while the consequence remains paused for conscious approval.
Good control does not require confirmation for every minor action. Excessive warnings teach people to ignore them and slow the workflow without materially improving safety. Mateusz divides actions into three levels: automatic and reversible, automatic with later audit, and approval required before execution. This is his proposed operating model. It preserves momentum where mistakes are inexpensive and concentrates human attention on outcomes that are public, consequential or difficult to reverse.
A map of a strong first use case
Four qualitative signals help select a workflow that can be tested safely and judged honestly.
A recurring problem produces enough observations to evaluate the change.
The output can be checked against a rule, example or human judgement.
A poor result does not trigger an irreversible consequence.
A named person owns quality and the decision to scale.
A pilot that can be judged honestly
The first test should cover one closed loop with a clear input, defined output, acceptance criteria and a named reviewer. Before it begins, Mateusz recommends recording a baseline: completion time, correction rate, exceptions and the cost of the current process. Without that comparison, an impressive demonstration can be mistaken for progress even when it merely moves work from initial execution into later error checking.
Pilot reporting should include the full cost of one accepted result: model use, tools, retries, review time, manual recovery and integration maintenance. An average can hide expensive exceptions, so the worst common cases deserve separate attention. Success is not a larger volume of generated material. It is a shorter route to an outcome that meets the required standard while preserving control, traceability and the ability to reproduce what happened.
Automation as a living system
A workflow does not stop changing after launch. Data, models, prices, external interfaces and team practices all drift. Mateusz’s operating model therefore requires a process owner, a change log, stable control examples and a clear route back to manual work. The system should show when it was last checked and which parts remain assumptions. That prevents gradual degradation from hiding behind the belief that automation which worked once will continue to work indefinitely.
A stopping condition matters as much as a launch target. If interventions increase, quality declines or maintenance exceeds the value saved, the scope should be reduced or disabled. That is mature product management, not a technological failure. The strongest automation remains understandable, measurable and replaceable. It helps people make better decisions without removing their knowledge of the process or their ability to take control when reality no longer matches the original design.
Instrument the process before model execution
Describe automation first as a sequence of observable events. Job acceptance, data validation, model calls, tool use, human handoff and the final outcome share a trace identifier and timestamps. The record measures the route to an accepted result, not model response speed alone. It also reveals whether work disappeared or moved into review or repair that the demonstration omitted.
This telemetry can run on the manual workflow before AI enters it, so the baseline uses the same definitions of start, finish and exception as the deployed system. Instrumentation should not copy complete documents into logs by default. Event metadata, status, error category and component versions answer many operational questions. Collection scope, retention and read access belong in the measurement design, not in a response improvised after an incident.
Data contracts separate missing information from model judgement
Every input needs a versioned contract covering required and optional fields, formats, origin, collection time and empty values. A missing case number must not be replaced with a guess, and an unknown code should not enter whichever category sounds similar. A validator accepts a conforming record, routes an incomplete one for enrichment or rejects it with a reason. Raw material remains auditable; only an explicitly labelled representation advances.
The output needs the same precision. Besides content, it carries state, references, detected gaps, the permitted next action, and whether the result is proposed, validated, approved or executed. Valid and invalid examples become integration contract tests. When a schema changes, a consumer does not infer a new field. It understands the version or stops the record safely. Fluent model language can no longer conceal disagreement between systems.
The queue is process memory, and an exception is a first-class result
Background automation needs a queue that records job ownership, priority, deadline, attempt count and the last confirmed state. An idempotency key prevents the same external effect from occurring twice when a service response is lost or an operator repeats a request. Retries should apply only to failures classified as transient and must have a limit. Once that limit is reached, the job does not disappear. It enters a visible exception queue with the context required for a person to complete it.
Exceptions should be classified by the response they require, not by a low-level error string. Missing evidence, conflicting data, an unavailable dependency, a policy block, an ambiguous decision and an invalid format belong to different owners and next steps. Human handoff preserves existing artifacts, completed actions and pending effects. The operator can see what may be retried and what must not be repeated. The unusual path becomes part of the designed service instead of a private investigation for whoever happens to be on call.
Observability joins business outcome, system health and decision trace
One dashboard should not collapse three different questions. The outcome layer shows which jobs reached a result accepted by the intended recipient and what corrections or exceptions were required. The operational layer covers latency, failed dependencies, queues, retries and execution cost. The decision layer connects a particular result to the model, instruction, retrieved data, tool and rule versions that produced it. Together they distinguish infrastructure failure from a quality regression or a change in the mix of incoming work.
An alert matters only when it names an action and an owner. Rising queue age might restrict new intake; absent provenance should block publication; divergence on a control case can stop a release. A trace identifier leads from the signal to a chronology without exposing unnecessary content. Logs should be redacted and sampled according to data risk. Observability exists to support diagnosis, not to create a second information repository with weaker safeguards than the systems it describes.
An automation version is a bundle, not a model name
Workflow behaviour depends simultaneously on the model, instructions, input template, retrieved sources, data schema, tool permissions and business rules. Those elements should form one identifiable release bundle. Changing any component opens a new version even when the model name remains unchanged. Each result records the bundle that produced it. Without this link, a team can observe a difference but cannot determine whether it came from the model, a new document, a revised prompt or wider access to an operational system.
A release candidate first runs against stable regression cases and a sample of current work in shadow mode. Limited traffic can then reach the new bundle while the previous one remains a comparator and rollback route. Reversal planning also covers jobs already queued: the team decides whether they finish on their original version or migrate safely. Results produced under different contracts should not be pooled without marking a break in comparability. Release history should explain a decision rather than merely preserve a number.
Unit economics counts the cost of an accepted outcome
The model call is only one cost component. A complete unit includes data retrieval and preparation, inference, external tools, storage, observability, reviewer time, corrections, exception handling and an allocated share of maintenance for every accepted result. The denominator needs discipline too: a generated draft is not an outcome when its recipient rejects it or rewrites it from the beginning. Comparison with manual work must apply the same definition of quality and completion, otherwise the saving exists only in a spreadsheet.
An average can hide a class of work that repeatedly requires expensive recovery. The calculation should therefore be segmented by input type, exception path and level of required control, while examining both typical and difficult runs. Value may appear as shorter waiting time, increased team capacity, more consistent output or reduced exposure to error; each needs separate evidence. A scaling decision should use a range that changes with volume and prices instead of one point estimate built on perfect utilization.
The shutdown criterion belongs in the operating design
Before launch, the owner records conditions that require reduced scope or suspension. They may include an inability to verify sources, increasing queue age, repeated side effects, an unhandled data-contract change, quality below the agreed threshold, cost above value or a case that an operator cannot safely take over. Every condition needs a signal source, a decision owner and a default response. Otherwise an alarm remains an optional suggestion that can be deferred whenever current demand creates pressure to continue.
A controlled shutdown stops new intake, preserves the state of work in progress, routes viable jobs to the manual process and removes permissions the automation no longer needs. The team communicates the confirmed state without promising completion that the system has not established, and archives the release bundle and event trace for analysis. Re-enablement requires an understood cause, a verified correction, successful regression checks and an explicit owner decision. The capacity to stop is not the opposite of automation; it shows that the process still has an accountable operator.
Frequently asked questions
Are the five questions an official AI evaluation standard?
No. They are Mateusz Więcek’s editorial framework for structuring product experiments. They are not a certification, formal standard or scientifically validated safety score.
Where should the first pilot begin?
Choose one frequent, reversible fragment with a clear input, output and reviewer. Record current time, quality, exceptions and cost first, then compare the full cost of an accepted outcome.
When should a person approve an action?
The framework proposes prior approval for public, financial, hard-to-reverse actions or changes to another person’s access. Inexpensive local changes can rely on preview, history and undo.
When should automation be stopped?
Stop or reduce scope when corrections and exceptions rise, quality falls, maintenance exceeds value, or safe manual fallback fails. Stopping is a legitimate outcome of a mature pilot.