The approval form for AI projects

1. Start with the task, not the model

The first question on the form should be: what decision or activity will the AI perform? Write the task as an observable operation, with an input, an expected output and a limit on its scope. “Help customer service” is too broad. “Classify messages as duplicate invoice, cancellation, technical question or other” makes the result verifiable. “Draft a response based on approved articles” is also more testable than “respond to customers”.

Do not combine different tasks in the same approval. Classification, field extraction, text generation and action execution have different failure modes. If the flow combines several stages, evaluate each one separately and then test the complete chain. This separation helps reveal whether the problem lies in interpreting the request, retrieving data, choosing a tool or producing the final response.

• Define who provides the input and in which format.
• State what counts as a correct, incomplete or prohibited output.
• Record what the AI may suggest and what it may not execute.
• Choose someone responsible for the decision to release, pause or remove the workflow.

Sources and references: [1][2]

2. Build a sample that represents real work

The sample should contain common cases, difficult cases and inputs that tend to confuse people or systems. Use permitted historical examples, data created specifically for testing and situations constructed by specialists. It is not enough to select only the easiest examples or cases that confirm the initial idea. The sample distribution should approximate expected use without concealing important exceptions.

Set aside part of the sample for the final decision. If every case is used to adjust instructions, rules or tools, the form loses strength as an independent test. As a hypothetical illustration, a team could reserve 120 requests for the final evaluation, distributed across frequent, ambiguous and adversarial situations. That number is only a planning example, not a quality standard.

• Include language, spelling, format and length variations when they are plausible.
• Add requests with multiple intentions and incomplete context.
• Test missing, contradictory data and fields with ambiguous names.
• Identify which examples contain sensitive information and control access to them.

Sources and references: [1]

3. Record critical errors before calculating averages

An average can conceal a failure that makes a project unviable. For that reason, the form should separate errors by severity. A slightly longer text may be acceptable in a draft. Inventing a contract condition, sending private information or calling a tool with the wrong identifier may require an immediate block.

Describe each error in observable terms. Instead of “bad response”, record “assigns the customer a deadline that does not appear in the source” or “classifies cancellation as a general question”. For each category, indicate the consequence, tolerable frequency and expected response. Tolerance may be zero for some classes, especially when the output triggers an external action or exposes data.

• Distinguish content errors, format errors, routing errors and action errors.
• Mark failures that require blocking, automatic correction, referral or simple logging.
• Test attempts to alter the instructions through the received content.
• For agents, check the selected tool, the arguments sent and the execution limits.

Sources and references: [3]

4. Compare with a non-AI reference and use human evaluation

The right question is not only whether the AI works. It is whether it improves, speeds up or simplifies the process compared with a concrete alternative. The reference may be an existing rule, a traditional classifier, structured search or the current manual procedure. Document that alternative’s operating cost, time, referral rate and types of errors without assuming that it is perfect.

Human evaluation should use a short rubric, examples of quality levels and approval criteria. For subjective tasks, ask evaluators to rate or compare responses without relying on an overall impression. An automated evaluator may help expand testing, but its agreement with human judgments must be verified in the specific context. Isolated scores do not replace analysis of failed cases.

• Define the criteria before reviewing the results of the new implementation.
• Use side-by-side comparison when the question is which output better meets the objective.
• Record disagreements between evaluators and refine the rubric when necessary.
• Require explicit approval for each critical error, even when the overall average is good.

Sources and references: [1]

5. Make the release decision with clear conditions

The form must end with a decision that someone can audit: approve, approve with limited scope, return to development or reject. “Approved” should not mean that the AI is reliable in every situation. It may mean that it will be used only for drafts, with restricted data, without automatic execution and with an interruption mechanism.

A good decision rule combines an aggregate result with category-specific safeguards. As a hypothetical example, a team might require at least 90 percent correct classifications, no leakage cases in the test set and mandatory referral for ambiguous inputs. These limits are illustrative. Each organization should define them based on the task’s impact, the cost of error and its ability to recover.

The architecture’s complexity should also be justified by the form. If a simple call or fixed workflow solves the problem, do not add an autonomous agent merely to increase flexibility. Systems with more stages, tools or handoffs provide more points to evaluate and more ways to fail. The recommendation is to increase complexity only when testing shows a measurable benefit for the task.

• Limit the users, data, tools and hours covered by the first release.
• Specify the conditions that automatically suspend the workflow.
• Record the version of the instructions, reference data and components used.
• Define who may alter the system and when a new evaluation is mandatory.

Sources and references: [2][3]

6. Monitor behavior after going live

Approval is a snapshot; monitoring follows the film. Record relevant inputs, outputs, routing decisions, tool calls, refusals, referrals and error signals, while respecting internal access and retention rules. Without these records, it will be difficult to distinguish a change in the model from a change in the user profile, available documents or connected system.

Track metrics tied to the task, not only latency or volume. Observe the proportion of referred cases, correction frequency, critical errors, category distribution and complaints related to the output. Create a queue of new cases to expand the evaluation sample. Every significant change to instructions, tools, sources or the model should go through the tests again to protect against regressions.

For a workflow that uses tools, monitor invalid arguments, unexpected calls and attempts to act outside the scope. External or user-submitted content should not directly control a privileged instruction. Structured fields, validations and confirmations for sensitive operations reduce paths for improper commands to propagate, although they do not eliminate the need for continuous testing.

• Define thresholds that trigger investigation and suspension.
• Periodically review a sample of real cases and edge cases.
• Compare the current version with the last approved version.
• Maintain a simple path back to the procedure without AI.

Sources and references: [1][3]