AI Vendor Evidence Gap Notes #6
AI vendors often describe retention as a setting:
“Zero data retention is available.”
“Enterprise customers can configure retention.”
“Prompts are deleted after 30 days.”
These statements sound operational.
They suggest that the buyer controls the data lifecycle.
But a retention setting is not automatically a retention control.
The review question is not only:
What retention period does the vendor advertise?
The better question is:
What data lifecycle does the available evidence establish for this product, plan, feature, configuration, region, and use case?
A setting may control one data object in one part of the service.
It may not cover files, metadata, logs, embeddings, support copies, caches, backups, or exception paths.
That is the evidence gap.
Claim
The vendor says:
“Enterprise customers can configure prompt retention.”
Or:
“Zero data retention is enabled by default.”
Or:
“Customer content is deleted after 30 days.”
These statements may establish that a retention feature or policy exists for a defined category of data.
They do not establish the complete lifecycle of the buyer’s data.
Why it sounds sufficient
Retention is often reduced to one number:
zero days;
thirty days;
customer-configurable;
deleted on request.
That makes the answer easy to enter into a questionnaire and mark complete.
But an AI workflow rarely creates only one data object.
A single interaction may create:
prompts and outputs;
uploaded files;
embeddings or indexes;
metadata and usage records;
security or moderation events;
logs and diagnostic records;
support artifacts;
cached copies;
backups.
The vendor may use “retention” to describe visible conversation history.
The buyer may assume that the same setting applies to every object created by the workflow.
It may not.
A prompt can disappear from the user interface while related metadata remains in an operational log.
A file can be removed from active storage while a backup copy remains until rotation.
A zero-retention inference endpoint can coexist with another feature that stores conversations by default.
The setting may be real.
The assumption that it controls the entire lifecycle is the problem.
What it actually proves
A current, product-specific retention statement may establish that:
a retention feature exists;
a defined data type follows a stated retention period;
administrators can shorten or disable certain storage;
a product or plan provides additional lifecycle controls;
the vendor has documented a customer-facing retention option.
These are useful evidence points.
They establish something about a defined setting or data category.
They do not automatically establish the lifecycle of every object, storage layer, feature, or exception path involved in the buyer’s workflow.
A public evidence example
The CodeYourCompliance public evidence profile for Fireworks AI records zero data retention as a public evidence signal and points to the vendor’s data-handling documentation.
The profile is an evidence index.
It is not implementation verification for a particular customer account, endpoint, configuration, or request.
Fireworks AI’s data-retention documentation shows why scope matters.
For open-model inference, the documentation states that prompt and generation data is not logged or stored persistently by default without explicit user opt-in. It also states that service metadata, such as token counts, is logged.
The same documentation explains that certain advanced features allow customers to opt into prompt and generation logging.
Separately, the Fireworks Responses API follows a different lifecycle. When store=True, which is the documented default, complete conversation data is retained for 30 days. This may include user prompts, model responses, and tools called by the model.
Storage can be disabled by setting store=False.
These statements are not necessarily contradictory.
They describe different services, endpoints, data types, features, and configurations.
For one path, prompt and generation data may not be stored persistently.
For another path, conversation data may be stored by default for a defined period.
Metadata may still be logged.
Advanced features may introduce opt-in content storage.
That is exactly why “zero retention” cannot remain a vendor-level label.
The buyer must determine which lifecycle applies to the exact workflow being reviewed.
What it does not prove
A configurable retention statement does not automatically establish:
Data-type coverage. The setting may cover prompts and outputs without covering files, embeddings, metadata, logs, moderation records, support artifacts, or backups.
Storage-layer coverage. Deletion from the primary application does not establish deletion from caches, indexes, vector stores, observability platforms, support systems, or recovery environments.
Deletion timing. “Deleted” may mean removed immediately, queued for deletion, removed from active systems later, or retained in backups until scheduled rotation.
Exception paths. Data may follow different retention rules for security investigations, abuse monitoring, customer support, legal obligations, incident response, or customer-enabled features.
Configuration scope. The lifecycle may depend on an endpoint parameter, administrator setting, feature toggle, workspace default, or customer opt-in.
Administrative enforcement. A setting does not show whether changes are logged, whether access is restricted, whether the customer can verify enforcement, or whether new features inherit the same policy.
Product and plan scope. API products, hosted applications, enterprise plans, individual plans, integrations, beta features, and model routes may follow different retention rules.
Contract scope. Public documentation does not necessarily provide the same commitment as a DPA, order form, service agreement, product addendum, or customer-specific term.
None of these gaps proves that the vendor retains data improperly.
They show that a retention statement may be narrower than the lifecycle the buyer needs to understand.
Weak-answer pattern
This is the setting-without-lifecycle pattern.
The vendor provides a retention period or configuration option.
The buyer records:
“Zero retention available.”
Or:
“Prompts deleted after 30 days.”
But the review file does not identify:
which data types are covered;
which endpoint or feature the setting controls;
whether it applies by default;
which storage layers are included;
whether metadata or logs remain;
which exceptions apply;
how deletion is verified;
which contractual source supports the claim.
The vendor has described a setting.
The buyer has not yet established the control boundary.
Evidence request
Do not ask only:
“What is your prompt-retention period?”
Ask for the lifecycle.
A stronger request is:
Please provide a product- and plan-specific retention matrix for the AI workflow under review. For prompts, outputs, uploaded files, embeddings, metadata, logs, moderation events, support artifacts, caches, indexes, and backups, identify the default and configurable retention period, storage layer, deletion timing, exception paths, human-access boundary, customer controls, audit evidence, and supporting contractual source.
Then ask:
Does the setting apply by default or require configuration?
Which endpoints, features, and data types are covered?
Does a setting change apply to existing data or only newly created data?
Which metadata, security records, or operational logs remain after content deletion?
How are cached, indexed, backed-up, or support-system copies handled?
Can the vendor override the setting for support, safety, abuse, incident, or legal purposes?
Are configuration changes recorded in audit logs?
Do new features and integrations inherit the same retention policy?
Which DPA, order-form term, or product addendum confirms the lifecycle?
The vendor does not need to disclose sensitive architecture.
It does need to establish an operationally usable boundary.
Review note
A weak review note says:
“Prompt retention is configurable.”
A stronger review note says:
The vendor identifies a configurable retention feature, but the available evidence does not establish the complete data lifecycle for the intended workflow. Coverage of files, embeddings, metadata, logs, support artifacts, caches, indexes, backups, configuration changes, and exception paths remains unclear. The response is not evidence-complete for data requiring verified deletion, short-lived processing, or customer-controlled retention.
That note does not say the vendor’s statement is false.
It prevents a setting from being treated as proof of the entire lifecycle.
Usage boundary
Until the complete retention lifecycle is evidenced:
Limit use to low-sensitivity internal or test data. Do not introduce customer-confidential data, regulated data, source code, employee records, privileged business material, or other data requiring verified deletion where uncertainty about storage layers, logs, metadata, backups, exception paths, or configuration would materially change the appropriate usage boundary.
That is not approval.
That is review preparation.
The review unit is:
vendor + product + plan + use case + data type + region + contract terms + evidence date.
A retention setting can support that review.
It cannot remain a single number inside it.
A setting is part of the evidence.
It is not the whole control.
Boundary
This material is for evidence structuring and review preparation. It does not provide legal, regulatory, audit, procurement, certification, or implementation advice.
The goal is not to approve or reject a vendor.
The goal is to make the evidence gap visible before real data use.
Reply if you want the sanitized sample evidence gap memo.
#AIVendorAssessment #AIGovernance #ThirdPartyRisk #Privacy #VendorRisk