Buyer guide

Evaluating AI governance evidence: what to ask for and how to read it.

A policy tells you what a vendor intends. Evidence tells you what happened. Ask for the evidence record itself, check who produced it and whether it can be changed, and trace one decision path end to end before you believe the summary.

Written for Compliance, risk, audit and security leads evaluating AI tools or AI governance products in regulated work, and anyone who has to sign off on an agent that acts on company systems.

The method

  1. Separate policy from evidence.

    A policy document, a framework mapping or a certification badge describes intent or a point-in-time assessment. Evidence is the record of a specific action, decision or check. Ask for both and weigh them differently.

  2. Ask for the record, not the screenshot.

    Request an export of the evidence for a period, in a form you can open without the vendor present. A screenshot shows that a screen exists.

  3. Check the chain.

    For each record ask who or what produced it, when, whether it can be edited or deleted after the fact, and how you would know if it had been. Records chained or signed so that a change is detectable are stronger than records that are merely stored.

  4. Map to the framework you already report against.

    If you report under ISO 42001, the EU AI Act, NIST AI RMF, HIPAA, FFIEC or CMMC, ask which controls the evidence covers and which it does not. A mapping that covers everything is a mapping to be suspicious of.

  5. Trace one decision path.

    Pick one agent action that matters to you. Ask to see the policy that governs it, the decision the system made, the evidence written for that decision, and what the calling system did with the answer. If any link is described rather than shown, write that down.

  6. Check access and action authority.

    Identify the sources a workflow can read and the changes it can make. Test required approval in the system that performs the action. An instruction in a chat is not a substitute for that permission check.

  7. Test instructions in external content.

    Prompt injection means instructions hidden in a document, message or website try to redirect AI. Test with synthetic examples and limit accessible data and tools. A detected attempt is not necessarily a blocked action.

  8. Distinguish training from storage.

    Not used for model training does not mean not stored. Ask where inputs, outputs, business records and logs are retained, who can access them and when they are deleted. Confirm the specific product and configuration.

  9. Test the stop procedure.

    Name the person who can revoke access, then test that revocation reaches the connected systems. Record what stops, what may continue and how the workflow recovers. Local hosting alone does not establish these controls.

  10. Ask what happens when logging fails.

    Some systems write the audit record first and stop the action if the write fails. Others write it alongside and carry on if it fails, so an audit outage never blocks a customer's work. Both are defensible. What matters is that the vendor can tell you which one you are getting, path by path.

  11. Settle who can read and who can export.

    Evidence a regulator cannot receive, or that only the vendor can produce, is weaker than evidence you hold yourself. Ask about retention, deletion, and what survives if you leave.

The checklist

What to have in place, or to have asked for, before you decide. Each line is something you can point at.

  • Policy documents and evidence records requested and kept separate.
  • An evidence export for a period, opened without vendor help.
  • For each record: who produced it, when, and whether it can be altered.
  • Tamper detection: how a changed record would be noticed.
  • Framework coverage stated control by control, gaps included.
  • One decision path traced from policy to decision to record to effect.
  • For each audit path: blocking or best-effort, stated by the vendor.
  • Who can read the evidence, who can export it, and in what form.
  • Retention period, deletion process, and what you keep on exit.
  • What is built today and what is planned, in the vendor's own words.
  • Data and action permissions reviewed against the actual connected systems.
  • Hidden-instruction tests distinguish detection from blocked effects.
  • Training use and storage/retention answered as separate questions.
  • A named containment owner and a tested way to stop access.

Questions to ask

Ask these of any vendor, including us, and of any internal team proposing the work. A vague answer to any of them is an answer.

  1. Can you export the evidence for last month in a form we can open ourselves?
  2. Who or what produces each record, and can it be edited or deleted afterwards?
  3. How would we detect a changed record?
  4. Which controls in the framework we report against does this evidence cover, and which does it not?
  5. For one action we choose, show us the policy, the decision, the record and what the calling system did with it.
  6. When the audit write fails, does the action stop or continue, and is that the same on every path?
  7. Does the decision engine carry the outcome out, or does the calling system have to honour it?
  8. What is retained, for how long, and what do we take with us if we leave?
  9. Which of these is running today, and which is planned?
  10. Can content from an external source redirect the workflow, and what contains the effect?
  11. Where is approval enforced before a business system changes?
  12. Who can stop access, and when was that procedure last tested?

Related products, and where each one stands

What to do next

Apply for early access See the governed AI route

← All guides