AI Vendor Evidence Gap Notes #3
SOC 2 is useful evidence.
It is not automatically evidence that the AI data path is covered.
A SOC 2 report may support the existence of a tested control environment.
It may still not establish whether the specific AI product, feature, model route, support workflow, retention path, region, and contract boundary match the buyer’s intended use.
That is the evidence gap.
The question is not only:
Does the vendor have SOC 2?
The better question is:
What does the SOC 2 report prove for this AI workflow, and what still needs to be mapped before real data use?
Claim
The vendor says:
“We have a SOC 2 Type II report.”
Or:
“Our platform is SOC 2 compliant.”
Or:
“Our security controls are independently audited.”
These statements may be useful.
They do not, by themselves, establish that the buyer’s actual AI data path is covered.
Why it sounds sufficient
SOC 2 sounds formal.
It usually involves an independent auditor, a defined system, a reporting period, management assertions, control descriptions, testing procedures, and test results.
For a conventional SaaS review, the report may answer important baseline questions about access management, change management, incident response, monitoring, vendor management, and security operations.
That matters.
But an AI vendor review usually asks a narrower question.
The buyer is not simply purchasing “the platform.”
The buyer may be sending prompts, files, meeting audio, transcripts, summaries, source code, customer records, metadata, embeddings, logs, support tickets, or evaluation data into a specific workflow.
That workflow may involve:
a hosted application;
one or more model providers;
retrieval or embedding systems;
transcription services;
support and troubleshooting tools;
safety or abuse-review systems;
logging and monitoring platforms;
regional infrastructure;
customer-selected integrations.
A SOC 2 report may cover some or all of those elements.
It may also apply only to a more narrowly defined system.
The buyer cannot determine that from the SOC 2 label alone.
What it actually proves
A SOC 2 Type II report may establish that specified controls operated within a defined system and reporting period, subject to the scope, criteria, testing, exceptions, and limitations described in the report.
It may support findings such as:
access-control procedures were documented and tested;
privileged access was reviewed;
system changes followed controlled processes;
relevant systems were monitored;
incident-response procedures existed;
selected vendors were subject to review;
security policies and operational controls were maintained.
These are meaningful evidence points.
They establish something about the audited control environment.
They do not automatically establish the scope of the buyer’s AI workflow.
A public evidence example
The CodeYourCompliance public evidence profile for Harvey records several public assurance surfaces, including Harvey’s Trust Center, security material, DPA references, subprocessor disclosures, and dated SOC 2 and ISO updates.
The profile is an evidence index.
It is not verification that a particular Harvey product, feature, plan, integration, or customer workflow falls within the scope of a specific audit report.
Harvey’s public Trust Center lists SOC 2 Type II among its compliance evidence and states that its current SOC 2 documentation is available through the portal.
Harvey’s security page also says that the company undergoes annual SOC 2 Type II and ISO 27001 audits. The same page describes a wider product environment involving uploaded documents, queries, responses, model providers, regional processing, retention controls, and subprocessors.
Those are useful public evidence points.
They show that independent assurance exists and that the service has multiple data-handling surfaces.
They do not allow a public reviewer to determine the complete audit boundary.
The public Trust Center does not expose the full SOC 2 system description, control tests, exceptions, complementary user-entity controls, reporting period details, or the treatment of every AI feature and external provider without access to the underlying report.
Harvey’s Security Addendum provides an important next step. It states that customers may request the SOC 2 Type II report and, where applicable, relevant penetration-test summaries and data-flow diagrams.
That distinction matters.
The public SOC 2 statement establishes the existence of assurance evidence.
The private report establishes the audited system boundary.
The product documentation and data-flow materials help determine whether that boundary maps to the buyer’s actual AI workflow.
None of these sources should be stretched beyond what it proves.
What it does not prove
A SOC 2 statement does not automatically establish:
Product scope. The report may cover a defined service environment without identifying every AI product, feature, add-in, agent, integration, or beta capability.
Plan and configuration scope. The buyer’s product tier, regional deployment, optional feature, or customer configuration may differ from the environment described in the report.
Model-provider scope. The report may not establish which external model providers process data, what those providers receive, or whether their controls are included or treated as subservice organizations.
Data-type coverage. Prompts, outputs, uploaded files, transcripts, embeddings, metadata, logs, support artifacts, and derived data may follow different paths.
Operational workflows. Support access, abuse monitoring, safety review, troubleshooting, evaluation, and incident response may introduce systems or personnel not obvious from the SOC 2 badge.
Retention and deletion. The report does not automatically establish deletion across active storage, caches, logs, backups, support copies, and downstream providers.
Regional routing. A company-level audit does not prove where each data type is processed or whether routing changes by region, model, fallback path, or integration.
Current applicability. A Type II report covers a historical period. The buyer still needs to consider the evidence date, subsequent changes, bridge coverage, new features, and material changes to the service.
Contract scope. Audit evidence does not replace the DPA, order form, security addendum, regional commitment, or product-specific contractual term.
None of these gaps show that the vendor’s controls are deficient.
They show that the SOC 2 claim is narrower than the decision the buyer is trying to make.
Weak-answer pattern
This is the certification-scope blur.
The buyer asks:
Can we use this AI vendor for this workflow?
The vendor answers:
We have SOC 2 Type II.
That is not a bad answer.
It is an incomplete review record.
The missing question is:
Which parts of this AI workflow fall within the audited system boundary?
Without that mapping, SOC 2 remains valuable source material.
It is not evidence-complete for the intended use case.
Evidence request
Do not ask only:
Can you provide your SOC 2 report?
Ask for scope mapping.
A stronger request is:
Please confirm whether the current SOC 2 report covers the exact product, plan, features, region, and AI workflow under review. Identify how prompts, outputs, uploaded files, transcripts, embeddings, metadata, logs, support artifacts, model-provider routes, integrations, and subprocessors relate to the audited system boundary.
Then ask:
What legal entity, system, products, and services are included in the report?
What reporting period does it cover?
Which AI features and customer plans are included?
Which model, transcription, embedding, retrieval, or infrastructure providers are involved?
Are those providers included in the scope, carved out, or addressed through vendor-management controls?
Are support, troubleshooting, safety, abuse-review, and human-access workflows covered?
What complementary user-entity controls must the customer operate?
Have material product or provider changes occurred since the reporting period?
Which data-flow document, DPA, security addendum, or order-form term confirms the operational boundary?
That is how SOC 2 becomes reviewable.
Review note
A weak review note says:
SOC 2 received. Risk resolved.
A stronger review note says:
The SOC 2 report may support the existence and operation of controls within the vendor’s stated system boundary and reporting period. The available evidence does not yet establish whether the specific AI product, feature, plan, model route, data types, support workflow, retention path, region, and customer configuration are included. Product-specific scope mapping is required before relying on the report for the intended AI workflow.
That note does not reject the vendor.
It identifies where the assurance evidence stops.
Usage boundary
Until the SOC 2 scope is mapped to the intended workflow:
Limit use to low-sensitivity internal data. Do not introduce customer-confidential data, regulated data, source code, privileged business material, or externally relied-upon outputs where uncertainty about product scope, model routing, support access, retention, subprocessors, regional handling, or contractual coverage would materially change the appropriate usage boundary.
That is not approval.
That is review preparation.
The review unit is:
vendor + product + plan + use case + data type + region + contract terms + evidence date.
SOC 2 can support that review.
It cannot perform the scope mapping by itself.
Boundary
This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.
The goal is not to approve or reject a vendor.
The goal is to make the evidence gap visible before real data use.
Reply if you want the sanitized sample evidence gap memo.
#AIVendorRisk #AIVendorAssessment #SOC2 #ThirdPartyRisk #AIgovernance