AI Vendor Evidence Gap Notes #9
A completed questionnaire can still leave the buyer without usable evidence.
AI vendor reviews often ask:
“Do you use customer data for model training?”
“Do you support data residency?”
“Are audit logs available?”
“Do you maintain a subprocessor list?”
The vendor answers.
The questionnaire is marked complete.
The review may still be incomplete.
The question is not only:
Did the vendor answer?
The better question is:
What source supports the answer, what does that source cover, and what remains unresolved?
A questionnaire collects statements.
An evidence review tests whether those statements support the buyer’s intended use.
Claim
The review file says:
“Vendor questionnaire completed.”
Or:
“Vendor confirmed required controls are in place.”
That establishes that the vendor responded.
It does not establish that the answers are evidence-complete.
Why it sounds sufficient
Questionnaires create structure.
They force buyers to ask about security, privacy, retention, subprocessors, model training, audit logs, data residency, and access controls.
That is useful.
The problem starts when the answer becomes the evidence.
A questionnaire may contain:
No customer data is used for training.
SOC 2 Type II available.
Audit logs supported.
Data stored in the United States.
See Trust Center.
Each answer may be accurate.
But none of them tells the buyer, by itself:
which source supports it;
which product and plan it covers;
which data types are included;
which exceptions apply;
whether the evidence is current;
whether the commitment is contractual.
The questionnaire records what the vendor said.
It does not automatically record what the buyer can rely on.
What it actually proves
A completed questionnaire may establish that the vendor responded to a defined set of questions and that relevant claims have been collected.
It can also identify areas that need follow-up.
It does not establish the strength or scope of the evidence behind each answer.
Consider:
Question: Do you use customer data for model training?
Vendor answer: No.
The review file still needs:
Source: Where is that commitment written?
Scope: Which product, plan, endpoint, or feature does it apply to?
Data covered: Prompts, outputs, files, metadata, logs, or all customer content?
Exceptions: Support, abuse monitoring, opt-in evaluation, or other workflows?
Evidence date: When was the source checked?
Contract boundary: Is this public documentation or part of the customer agreement?
Gap: What remains unresolved?
That is the difference between an answer and an evidence record.
A public evidence example
The CodeYourCompliance public evidence profile for Cohere shows how a vendor review can move beyond a questionnaire answer.
The profile separates public evidence surfaces rather than treating the company name as one source.
A response such as:
“SOC 2 available.”
or:
“Customer data is not used for training.”
still needs to be mapped to the specific source that supports it.
The reviewer then needs to check the product and plan scope, evidence date, data types covered, known exceptions, and whether a stronger contractual or customer-specific source is required.
The profile helps locate evidence surfaces.
It does not make the review conclusion.
That distinction matters.
Questionnaire answer → evidence source → scope → evidence gap
That is evidence conversion.
What it does not prove
A completed questionnaire does not automatically establish:
Source strength. “Yes,” “No,” or “See Trust Center” does not show whether the source is contractual, audited, product documentation, marketing material, or unsupported.
Product scope. A company-level answer may not apply to the exact product, plan, feature, endpoint, or integration.
Data-path scope. A training answer does not establish retention. A residency answer does not establish processing location. A subprocessor answer does not establish routing.
Freshness and exceptions. Supporting documents may change, and binary answers may hide support access, safety review, fallback providers, beta features, or configurable behaviour.
Contract applicability. Public documentation may not create the same commitment as a DPA, order form, or product addendum.
None of these gaps means the vendor gave a false answer.
They mean the answer still needs evidence mapping.
Weak-answer pattern
This is the answer-without-source pattern.
The buyer asks:
“Are audit logs available?”
The vendor answers:
“Yes.”
The spreadsheet becomes green.
But the review file does not identify which events are logged, which plan is covered, how long logs are retained, whether customers can export them, or whether AI-specific activity is visible.
The questionnaire looks complete.
The evidence record is not.
Evidence request
Do not ask the vendor to complete another questionnaire.
Ask the vendor to support the existing answers.
For each material AI, security, privacy, retention, logging, subprocessor, data-residency, and human-access claim, identify the supporting source, applicable product and plan, relevant data types, known exceptions, evidence date, and contractual basis where applicable.
For higher-impact claims, ask:
Does the source apply to the exact product and feature?
Which data types are explicitly covered?
Which exceptions exist?
Has the supporting evidence changed?
Is stronger private or customer-specific evidence required?
The goal is not more questions.
The goal is claim-to-evidence mapping.
Review note
The vendor completed the requested questionnaire, but several material responses have not yet been mapped to product-specific supporting evidence. Source strength, product and plan scope, data-type coverage, exceptions, evidence freshness, and contractual applicability remain incomplete for key AI data-handling claims.
The questionnaire is complete.
The evidence mapping is not.
Usage boundary
Until material answers are mapped to evidence:
Limit use to low-sensitivity internal or test data. Do not expand to customer-confidential data, regulated data, source code, employee records, autonomous actions, or externally relied-upon outputs where unsupported questionnaire answers would materially change the appropriate usage boundary.
That is not approval.
That is review preparation.
The review unit remains:
vendor + product + plan + use case + data type + region + contract terms + evidence date.
A questionnaire can collect claims.
It cannot determine whether those claims are supported.
The useful sequence is:
vendor answer → evidence source → scope → evidence gap → buyer question → usage boundary
A completed questionnaire is an input.
It is not an evidence-complete review.
Boundary
This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.
The goal is not to approve or reject a vendor.
The goal is to make the evidence gap visible before real data use.
Reply if you want the sanitized sample evidence gap memo.
#AIVendorRiskAssessment #ThirdPartyRisk #AIGovernance