Buyer guide

Evaluating AI governance evidence: what to ask for and how to read it.

A policy tells you what a vendor intends. Evidence tells you what happened. Ask for the evidence record itself, check who produced it and whether it can be changed, and trace one decision path end to end before you believe the summary.

Written for Compliance, risk, audit and security leads evaluating AI tools or AI governance products in regulated work, and anyone who has to sign off on an agent that acts on company systems.

The method

  1. Separate policy from evidence.

    A policy document, a framework mapping or a certification badge describes intent or a point-in-time assessment. Evidence is the record of a specific action, decision or check. Ask for both and weigh them differently.

  2. Ask for the record, not the screenshot.

    Request an export of the evidence for a period, in a form you can open without the vendor present. A screenshot shows that a screen exists.

  3. Check the chain.

    For each record ask who or what produced it, when, whether it can be edited or deleted after the fact, and how you would know if it had been. Records chained or signed so that a change is detectable are stronger than records that are merely stored.

  4. Map to the framework you already report against.

    If you report under ISO 42001, the EU AI Act, NIST AI RMF, HIPAA, FFIEC or CMMC, ask which controls the evidence covers and which it does not. A mapping that covers everything is a mapping to be suspicious of.

  5. Trace one decision path.

    Pick one agent action that matters to you. Ask to see the policy that governs it, the decision the system made, the evidence written for that decision, and what the calling system did with the answer. If any link is described rather than shown, write that down.

  6. Ask what happens when logging fails.

    Some systems write the audit record first and stop the action if the write fails. Others write it alongside and carry on if it fails, so an audit outage never blocks a customer's work. Both are defensible. What matters is that the vendor can tell you which one you are getting, path by path.

  7. Settle who can read and who can export.

    Evidence a regulator cannot receive, or that only the vendor can produce, is weaker than evidence you hold yourself. Ask about retention, deletion, and what survives if you leave.

The checklist

What to have in place, or to have asked for, before you decide. Each line is something you can point at.

  • Policy documents and evidence records requested and kept separate.
  • An evidence export for a period, opened without vendor help.
  • For each record: who produced it, when, and whether it can be altered.
  • Tamper detection: how a changed record would be noticed.
  • Framework coverage stated control by control, gaps included.
  • One decision path traced from policy to decision to record to effect.
  • For each audit path: blocking or best-effort, stated by the vendor.
  • Who can read the evidence, who can export it, and in what form.
  • Retention period, deletion process, and what you keep on exit.
  • What is built today and what is planned, in the vendor's own words.

Questions to ask

Ask these of any vendor, including us, and of any internal team proposing the work. A vague answer to any of them is an answer.

  1. Can you export the evidence for last month in a form we can open ourselves?
  2. Who or what produces each record, and can it be edited or deleted afterwards?
  3. How would we detect a changed record?
  4. Which controls in the framework we report against does this evidence cover, and which does it not?
  5. For one action we choose, show us the policy, the decision, the record and what the calling system did with it.
  6. When the audit write fails, does the action stop or continue, and is that the same on every path?
  7. Does the decision engine carry the outcome out, or does the calling system have to honour it?
  8. What is retained, for how long, and what do we take with us if we leave?
  9. Which of these is running today, and which is planned?

Related products, and where each one stands

What to do next

Apply for early access See the governed AI route

← All guides