AI in compliance: traceable evidence and approval

Start with the evidence object, not the chat

The first element of the system should be structured evidence, not an isolated textual response. Each record can contain a source identifier, file name, version, collection date, owner, passage location, confidentiality classification, and an extracted assertion. The AI-generated response should point to these records instead of replacing the original documents.

This design changes the question from “what did the AI conclude?” to “which data supports this assertion, and who can challenge it?”. It also makes it possible to distinguish a lack of evidence, conflicts between documents, and outdated evidence. The application can store a state such as pending, confirmed, rejected, or superseded, with a reason and date for each change. This is software design for investigation and control, not a decision about compliance.

• Define an evidence schema before choosing the model or interface.
• Separate the original text, the extraction performed by the AI, and the team’s decision into different entities.
• Use stable identifiers for related sources, versions, passages, and assertions.

Sources and references: [2]

Build traceability during retrieval

Documents should enter through an ingestion flow that preserves the original file and produces verifiable metadata. The system can then divide the content into searchable passages and retrieve semantically related sections. Semantic search is useful when the question and the document use different words, but the result still needs to carry the file identifier, the passage found, and its attributes so that the origin remains visible.

Attribute filters help limit searches by period, region, responsible area, language, or document status. This layer matters because a relevant passage may be unsuitable for the current question. The recommendation is to apply filters before generating the synthesis and reject responses when there is not enough supporting source material. The application should also record the original query, any query reformulation, the retrieved results, and the version of the index used.

External text should be treated as untrusted data, even when it appears to be an instruction. The model should not receive this content in a privileged context or decide on its own which tools may be called. Structured outputs, enumerated fields, and schema validation reduce the paths by which imported text can alter the system flow, although they do not eliminate every risk. [1]

• Store a hash or another technical file identifier to detect silent replacements.
• Show the user the passage used, not only the document name.
• Define an explicit policy for documents that have been removed, replaced, or are still being processed.

Sources and references: [1][2]

Make approval a product step

Approval should not be an informal comment in a chat box. Create a case queue with enough essential context: the proposed assertion, source passages, conflicting documents, uncertainty level, change history, and requested action. The responsible person should be able to confirm, reject, request additional information, or route the case to another function, always with a recorded reason.

Permissions should limit both what the AI can access and what each role can change. One person may validate the classification of a source, while another may approve publication of a synthesis. The system should prevent an old approval from being reused automatically after the source, prompt, index, or extraction rule changes. In that situation, the item returns for review or receives an outdated status.

For actions that alter records, send communications, or update external systems, use explicit confirmation and record the request, parameters, result, and user responsible for authorization. The AI can prepare the action, but the software needs to control the state transition. This boundary makes the flow more predictable without assuming that technical approval, by itself, resolves an organizational issue.

• Use clear states: new, under analysis, awaiting information, approved, rejected, and obsolete.
• Record who approved the item, when, with which version of the material, and for what reason.
• Separate permission to view evidence from permission to change decisions.

Sources and references: [1]

Test the entire chain, not just the response

A test that checks only whether the response appears correct leaves out more dangerous failures. Evaluate whether the system retrieved the right source, preserved the passage, applied the correct filters, identified a conflict, rejected an instruction inserted into the document, and sent the case to the appropriate queue. Each step can have its own criteria because a plausible final result may have been produced from the wrong source.

Build a set of cases with current documents, older versions, incomplete sources, contradictory texts, different formats, and malicious inputs. For each case, define the expected result in observable terms, such as the source identifier, extracted fields, routing decision, and blocking reason. The evaluation should be repeated whenever there is a change to the model, prompt, document segmentation, filters, or interface.

Automated metrics can verify fields and calls, but they should not be treated as substitutes for well-defined criteria. Compare results between versions, record regressions, and keep failure examples together with the reason for failure. The recommended practice is to evaluate early and continuously, combining flow-specific tests with judgments about relevance and completeness. [3]

• Test source selection, extraction, synthesis, and approval as separate points.
• Include cases in which the correct answer is “there is not enough evidence”.
• Record inputs, outputs, retrieved sources, and decisions so that a failure can be reproduced.

Sources and references: [3]

Use a hypothetical pilot to choose the architecture

Consider a hypothetical company that receives internal policies, supplier reports, and training records in varied formats. The first pilot does not need to answer open-ended questions about the entire collection. It can locate evidence for one category, present three related passages, identify version conflicts, and create a case for approval. This scope makes it possible to observe where the team loses time and which fields are actually necessary.

In this scenario, the initial architecture can use ingestion with mandatory metadata, hybrid or semantic search, fixed-format extraction, evidence storage, and an approval queue. The synthesis is produced only after the passages pass basic scope rules. If a source lacks a date, owner, or version, the system can classify it as incomplete and request specific handling instead of filling gaps through inference.

The choice between a simple flow and an agent with tools should follow the process’s actual needs. If the steps are predictable, a workflow with explicit transitions is usually easier to test and explain. More autonomy makes sense only when there is a task that genuinely requires dynamic choice and when the tests can detect improper calls. The pilot’s purpose is to learn which controls the work requires, not to present AI as the final arbiter.

• Choose one evidence type and one approval queue for the pilot.
• Define in advance which changes require a new analysis of the case.
• Stop generation when the source is missing, conflicting, or outside the scope.

Sources and references: [1][3]