Ask eight things, in this order: where our data physically sits and whether it shares infrastructure with other customers; whether any of it trains a model, yours or a third party's; what the automation can reach, acting as whom, and who approves changes to that access; what gets logged, for how long, and who can read or alter the log; which subprocessors touch the data; what happens in the first hours of an incident and when we are told; what we are contractually allowed to inspect; and what evidence backs each of the previous seven answers. Everything else in a standard security questionnaire is a variation on one of those.
The answers themselves are not the useful part. Any competent vendor will answer all eight well in a sales call. The real test is whether they can produce the artifact behind each answer — a tenancy diagram, the contract clause, an export of the actual IAM policy, a sample audit log with real fields in it, the current subprocessor list, a past incident write-up, signed audit rights — without escalation drama and without a month of delay. A vendor whose controls are real has these sitting in a folder. A vendor whose controls live in one engineer's head will send a polished questionnaire response and then go quiet when you ask for the export. Our answers are on our security page, and the supporting documentation sits in our trust center. Hold us to the same standard you would hold anyone else.
Data residency and tenancy
Residency is two questions, not one. First: in which countries does the data rest, and in which does it transit or get processed? Support tooling, backups, log aggregation and disaster-recovery replicas frequently land in a different region from the primary workload, and that is where residency commitments quietly break. Second: is the environment single-tenant, logically separated within shared infrastructure, or genuinely shared? Most managed automation runs as logical separation, which is fine for most workloads — but say so plainly rather than letting "dedicated" do ambiguous work in a slide.
Push for specifics on where the data goes at rest, in backup, in logs, and in any model or OCR service in the chain. If the process touches regulated records, ask what happens when the vendor needs a support engineer in another time zone to debug a failed run at night. The honest answer is usually that someone reaches into an environment containing your data. That is acceptable if it is scoped, logged and time-limited. It is not acceptable if nobody has thought about it.
Training on your data
Ask it as a chain, not a single question, because the vendor is rarely the only party involved. Does the vendor train on your data? Does the vendor's model provider train on data sent through its API? Does any embedded third-party component — document extraction, transcription, translation, a vector database, an analytics SDK — retain inputs for product improvement? Each link needs its own answer, and each needs to be in writing, either in the contract or in a referenced policy that cannot change without notice.
Watch for two soft spots. One is "we do not train on customer data" that covers the vendor but not their upstream provider. The other is retention for abuse monitoring, which is legitimate but means your documents persist somewhere for a period. Ask what the retention window is, whether zero-retention processing is available on your tier, and who can access buffered content during that window. If the answer only exists verbally, it is not an answer.
Access model and least privilege
Automation is a set of credentials with a job. The critical question is what those credentials can do beyond the job. Ask for the specific service accounts the automation will use, the exact scopes or roles attached, and whether they are read-only anywhere they could be. Then ask the harder question: who can change those scopes, and does a change generate an alert to your side or only to the vendor's?
Insist on named human accounts for anyone on the vendor side who can reach production, with SSO and MFA enforced through your identity provider where possible — not shared logins, not a generic integration account that three people use. Ask how vendor staff access is granted, reviewed and revoked when someone leaves their team, and how quickly you are told. Ask whether standing production access exists at all or whether it is requested per incident and expires. Standing, permanent, broad access held by a rotating cast of contractors is the single most common weakness we see, and it is also the easiest one to fix before a project starts.
Logging and auditability
Every automated action needs to be reconstructable after the fact by someone who was not there. Ask to see a real log line, not a description of logging. It should identify what ran, on which record, with which inputs, under which identity, at what time, what it decided, and what a human did or did not approve. If the log records that a run succeeded but not what it changed, you cannot investigate anything and you cannot answer an auditor.
Then ask where the log lives and whether the vendor can edit it. Logs that only exist inside the vendor's platform, editable by the vendor, are evidence of good intent and nothing more. Ask whether logs can be streamed to your own SIEM or log store, in what format, and at what latency. Ask retention: how long online, how long archived, and whether that matches your record-keeping obligations rather than the vendor's default. For approval steps, ask whether the system captures the approver's identity and the state they actually saw, which matters far more than a timestamp when someone later disputes a decision.
Subprocessors
Ask for the current list, in writing, with each party's role and the data it sees. Then ask the process question that actually protects you: how are you notified before a new subprocessor is added, how much notice do you get, and do you have a right to object — and what happens if you do object? A subprocessor list with no change-notification mechanism is a snapshot of a moving target.
Pay particular attention to anything in the AI layer. Model providers, document-parsing services and speech vendors are subprocessors even when they feel like infrastructure, and they are the ones most likely to change mid-contract as the vendor tunes cost and quality. If the vendor cannot tell you today which model provider processes your documents, they cannot tell you next quarter either.
Incident response
Skip the policy document and ask for the timeline. Who decides that something is an incident? Within what window are you notified, and does that window start at detection or at confirmation? Who calls you, and do they have your after-hours contact today rather than after the paperwork? What is communicated in the first update when the facts are still thin?
Then ask about the automation-specific failure that most policies ignore: what happens when the system does the wrong thing correctly. A misconfigured rule that posts hundreds of bad journal entries or emails the wrong customers is not a breach, so it may not trigger the incident process at all — yet it is the failure mode you are far more likely to experience. Ask how a run is stopped mid-flight, how work already performed is identified and reversed, and who has the authority to hit the switch without waiting for a meeting. A vendor who has built production automation before will answer this immediately, because they have lived it.
Right to audit and what evidence to ask for
Right-to-audit clauses are commonly granted and rarely exercised, which is fine — the point is to establish that inspection is possible, on reasonable notice, without renegotiating. What you should actually exercise is the lighter-weight version: an annual review of reports, configuration exports and access lists, plus the ability to inspect after an incident. Ask what the vendor will accept in a signed agreement, and treat a flat refusal as informative.
Compliance reports are necessary but frequently over-read. A report covers a defined scope over a defined period, and the scope is often narrower than the product you are buying. Read the scope section and the exceptions, not the logo.
| What you asked for | What actually demonstrates it |
|---|---|
| "We're SOC 2 certified" | The full report with scope, period and exceptions — and confirmation the system you're buying is in scope |
| "Data stays in your region" | An architecture diagram covering backups, logs and support access, plus the contractual commitment |
| "Least privilege access" | An export of the actual roles and scopes on the service accounts, plus the access review record |
| "Everything is logged" | A sample log entry with real fields, and a described path to export into your own tooling |
| "We never train on your data" | The clause, extended to the model provider and any parsing subprocessor in the chain |
| "We have an incident process" | A redacted postmortem from a real event and a named notification window |
When the answer is "don't automate this yet"
We will say this on a scoping call and we would rather say it early. Hold off when the underlying process has no owner who can define what a correct outcome looks like — automation makes an undefined process fail faster, not better. Hold off when the only way to complete the work is to grant a service account broad standing access to a system of record because nobody can scope it down; fix the entitlement model first, because you will want that anyway. Hold off when the source data is so inconsistent that every run requires human interpretation, which is a data quality problem wearing an automation costume.
And hold off when the process has a regulatory consequence you cannot yet evidence. If an examiner would ask who approved a decision and your candidate design cannot answer that in a log, automating it moves you backward regardless of how much manual effort it saves. The cost drivers here are mostly one-time: access redesign, data cleanup, defining the approval and exception paths. They are usually small relative to the work of unwinding an automation that produced unauditable output for months.
Running the review without stalling the project
Send the eight questions before the demo, not after the contract is drafted. Ask for evidence on the same timeline as the commercial discussion so security is not a gate at the end. Decide up front which answers are firm requirements and which are preferences, because a vendor who is strong on access control and thin on residency may still be the right choice for a process that never touches regulated data — and the reverse is also true. Then scope the first automation narrowly enough that the security model is actually testable in production rather than theoretical, which is how we structure our engagements for exactly this reason.
If you want to run these questions against a process you are considering — or against a proposal already on your desk — talk to us. We will go through the eight with you, tell you where the gaps are, and say plainly if the honest recommendation is to wait.
Questions we get asked about this
No, but it is a reasonable starting filter. A report covers a specific scope over a specific period, and the product you are actually buying is sometimes outside that scope. Read the scope section, the exceptions and the period, and pair it with the access, logging and residency evidence that applies to your particular deployment.
Require it when a regulation, contract or internal policy says so; otherwise treat it as a preference you can trade. The more important question is whether the vendor can actually honor it end to end, including backups, log aggregation, support access and any model or document-parsing service in the chain. A residency commitment that covers only the primary database is not a residency commitment.
It depends on the provider, the tier and the contract, so ask each link in the chain separately. Major enterprise APIs generally commit to not training on submitted data, but some retain inputs briefly for abuse monitoring, and embedded third-party components may have their own terms. Get the commitment in writing for the vendor, the model provider and any parsing or transcription service they use.
Usually yes in some form, though rarely as an unlimited on-site inspection right. What is commonly achievable is inspection on reasonable notice, annual review of reports and access lists, and a clear right to inspect after an incident affecting your data. A vendor who refuses every version of it is telling you something worth hearing.
You should, in your identity provider, with the vendor granted scoped access to them. That way you can see the entitlements, review them on your own cycle and revoke them in minutes without depending on the vendor's offboarding process. It also means the access survives a change of vendor, which matters more than most buyers expect.
Standing broad access. A service account gets provisioned with wide permissions during the build because scoping it down is fiddly, and nobody goes back to tighten it after go-live. The second most common is logging that records that a run happened but not what it changed, which makes both investigation and audit response far harder than they need to be.
Test the integration surface and any custom interface you expose, and ask the vendor for the results of their own testing on the platform. For most internal process automation, the higher-value exercise is a permissions review and a failure drill: confirm what the service accounts can reach, then practice stopping a bad run and reversing the work it did.
Less time than people fear if the questions go out before the demo, and considerably more if they go out after the contract is drafted. The delay almost always comes from evidence that has to be created rather than retrieved. Asking early tells you quickly which kind of vendor you are dealing with.
Bring us the process you were reading this for
Confidential assessment led by senior engineers. No obligation.