When an organisation sends a standard vendor security questionnaire to an AI development partner, they are assessing the right things — and missing the most important ones. SOC 2 Type II verifies that the vendor's organisational security controls operated effectively over time. It says nothing about whether their AI system sanitises inputs before they reach a language model. ISO 27001 certifies their information security management system. It says nothing about whether training data contains your customers' personal information in a way that the model can reproduce under the right query.
Traditional assessments were built for traditional software. AI introduces a category of risk — model-level vulnerabilities, training data exposure, autonomous decision-making without audit trail — that did not exist when those frameworks were designed. The organisations that discover this gap after a prompt injection attack, a model memorisation incident, or an EU AI Act compliance investigation discover it at the worst possible moment.
#1
AI attack vector: prompt injection — yet most traditional vendor questionnaires contain no question about it
OWASP Top 10 for LLMs, 2025
82.6%
of phishing emails now use AI technology in some form — the attack surface your vendors face has changed
Tech Advisors / ATL Systems, 2026
Aug 2026
EU AI Act full enforcement for high-risk AI — Annex IV documentation required; most vendors unprepared
EU AI Act enforcement timeline
$1.9M
average saving per breach for organisations with extensive AI security tools vs those without
IBM Cost of Data Breach, 2025
What traditional assessments cover — and what they miss
Traditional security assessment vs AI-specific security assessment
✓ What SOC 2 + ISO 27001 covers
- Organisational access controls and user provisioning
- Encryption in transit and at rest for stored data
- Incident response plan existence and testing
- Vulnerability management and patch cadence
- Physical and network security controls
- Third-party subprocessor disclosure requirements
- Business continuity and disaster recovery
✕ What AI-specific risk adds — uncovered by SOC 2
- Prompt injection resilience — does the system validate inputs before they reach the model?
- Training data provenance — where did the model's training data come from, and does it contain your customer PII?
- Model memorisation — can the AI reproduce sensitive training data when queried in specific ways?
- Model drift monitoring — how is degrading model performance detected and escalated?
- AI supply chain — which foundation model, which version, which third-party components?
- IP ownership of AI-generated code — who owns what was built using AI tools?
- EU AI Act Annex IV documentation — for high-risk AI, does technical documentation exist?
"Generic vendor security assessment checklists verify encryption standards and SOC 2 compliance but remain silent on critical AI-specific concerns: model lineage, training data provenance, and autonomous decision-making safeguards."
Protecto — AI Vendor Risk Assessment, July 2026
Domain 1 of 6
Data handling & training provenance
Data
"Where does our data go during AI development — specifically, is it used to train or fine-tune any model, including the foundation model you are building on?"
Foundation model providers such as OpenAI, Anthropic, and Google have different default data usage policies. Some use API inputs for model improvement by default unless explicitly opted out. A development agency using these APIs with your data may inadvertently contribute your proprietary information or customer PII to third-party model training without your knowledge or consent — creating data protection liability under GDPR and potential IP loss.
✓ Passing answer
Names the specific foundation model and version, confirms data processing terms with that provider, states explicitly whether your data is used for training and how to opt out if so, and can produce the DPA (Data Processing Agreement) from the foundation model provider.
✕ Walk-away answer
"We use industry-standard cloud providers" without naming which model, which version, or what their data usage terms say about your inputs. Vagueness here is not discretion — it is either ignorance of the supply chain or concealment of an unfavourable answer.
Data
"Can the trained or fine-tuned model reproduce any of our training data in its outputs — and how have you tested for this?"
Model memorisation is a documented vulnerability where AI models can reproduce verbatim or near-verbatim content from their training data when queried in specific ways. If your model is fine-tuned on customer records, proprietary documents, or personally identifiable information, an attacker — or even a well-intentioned user — may be able to extract that information through targeted queries. This is not theoretical: research has demonstrated extraction of training data from major commercial models.
✓ Passing answer
Describes specific memorisation testing performed (canary injection, extraction attack simulation), the results, and the mitigation measures applied — differential privacy, data minimisation, output filtering. Can point to documentation or test results.
✕ Walk-away answer
"Our model is secure" or "we use encrypted training data" — encryption during storage does not prevent a trained model from reproducing training content in inference outputs. An agency that does not understand the distinction has not done memorisation risk assessment.
Data
"Show me your full AI supply chain — which foundation model, which version, which third-party components, and which subprocessors have access to our data during development and inference."
AI vendors rarely build everything in-house. A typical AI system relies on a foundation model from one provider, a vector database from another, an embedding service from a third, and cloud infrastructure from a fourth. Each dependency in this chain introduces a risk: if any subprocessor experiences a breach, that breach potentially exposes your data. GDPR Article 28 requires documented subprocessor disclosure; the EU AI Act adds supply chain transparency requirements for high-risk systems.
✓ Passing answer
Produces a written AI supply chain map naming each component, provider, and version. Confirms DPAs are in place with each subprocessor. Can describe what data each component has access to and for what purpose.
✕ Walk-away answer
Cannot name the specific foundation model version, or says "various providers" without enumeration. Your vendor's vendors are your risk too — a supplier who cannot name their own supply chain cannot manage it.
Domain 2 of 6
AI-specific attack surface — prompt injection, extraction, poisoning
Attack Surface
"How does the system defend against prompt injection — both direct (user input) and indirect (content retrieved from external sources)?"
Prompt injection is ranked #1 in OWASP's Top 10 for Large Language Models (2025). Direct prompt injection occurs when a user sends instructions that override the system's intended behaviour. Indirect prompt injection — often harder to defend — occurs when malicious instructions are embedded in documents, web pages, or database entries that the AI retrieves during operation, causing it to act on attacker instructions without the user or operator being aware. RAG systems and agentic AI are particularly vulnerable to indirect injection.
✓ Passing answer
Describes input sanitisation and validation before prompts reach the model; separate trust levels for system instructions vs user inputs; retrieval content scanning in RAG systems; rate limiting; and evidence of penetration testing that specifically included prompt injection scenarios with documented results.
✕ Walk-away answer
"Our model handles that" or no specific articulation of input validation architecture. The model itself does not defend against prompt injection — defensive architecture must be implemented at the application layer. An agency without this answer has not built it.
Attack Surface
"How do you protect the model against extraction attacks — systematic querying designed to reconstruct the model's behaviour or training data?"
Model extraction attacks use systematic querying to reverse-engineer a model's behaviour, effectively stealing the intellectual property embedded in a fine-tuned model through its outputs. For organisations that have invested in proprietary fine-tuning on their data, an unprotected inference API is a potential IP leak. Rate limiting on the API is the primary defence — combined with output monitoring for patterns consistent with systematic extraction attempts.
✓ Passing answer
API rate limiting with named thresholds; IP allowlisting or authentication mechanisms; monitoring for query patterns consistent with extraction (high volume, systematic input variation); and documented response procedure for detected extraction attempts.
✕ Walk-away answer
No rate limiting on the AI inference API, or API access limited only by API key with no query volume constraints. An API without rate limiting is an extraction attack waiting to occur.
Attack Surface
"What is your process when a security vulnerability specific to the AI system is discovered — including model-level vulnerabilities, not just infrastructure?"
AI systems introduce vulnerability classes that infrastructure security tools do not detect: jailbreaks that bypass safety controls, novel prompt injection techniques, and model behaviour changes after foundation model updates. A development partner's vulnerability response process must extend to AI-layer vulnerabilities, not just the standard CVE-based infrastructure patching cycle.
✓ Passing answer
Describes a specific AI vulnerability disclosure process distinct from infrastructure vulnerabilities; responsible disclosure channel; AI-specific monitoring (output quality, safety filter bypass rates, unusual query patterns); and named procedure for communicating AI-layer security issues to clients.
✕ Walk-away answer
"We follow standard software vulnerability management" — redirecting an AI-specific vulnerability question to a standard CVE patch process indicates the agency has not mapped AI-specific risk types to their incident management framework.
Skip the questionnaire
Find AI agencies already assessed on security depth
TechRadiant evaluates AI development agencies on security certifications, data handling practices, compliance depth, and technical delivery track record. The agencies in our verified index have already been assessed on the criteria this checklist describes — so you start the conversation with confidence, not a questionnaire.
Domain 3 of 6
Compliance & regulatory readiness — EU AI Act, GDPR, HIPAA, ISO 42001
Compliance
"If this AI system qualifies as high-risk under the EU AI Act, can you produce Annex IV technical documentation — and have you conducted an Article 27 Fundamental Rights Impact Assessment?"
The EU AI Act became fully effective for high-risk AI systems in August 2026. High-risk AI includes systems used in medical diagnosis, credit scoring, employment screening, educational assessment, biometric identification, and critical infrastructure. Annex IV requires technical documentation covering the system's intended purpose, architecture, training data description, performance metrics, and cybersecurity measures. Article 27 requires a Fundamental Rights Impact Assessment. A development partner building high-risk AI for EU-market products must be able to support both, or you carry the compliance gap.
✓ Passing answer
Can confirm whether the system qualifies as high-risk; describes what Annex IV documentation they maintain; demonstrates NIST AI RMF alignment; and can describe their FRIA process. Does not need to have completed all documentation at proposal stage, but must demonstrate they understand what is required.
✕ Walk-away answer
"What is the EU AI Act?" or a vague reference to "GDPR compliance" without demonstrating awareness of the AI Act's additional requirements. For any EU-market product, this answer carries regulatory exposure that lands on you, not the agency.
Compliance
"For a healthcare AI application, does your BAA explicitly name the AI feature — and how do you handle AI-specific HIPAA obligations including minimum necessary access and decision audit trails?"
A standard HIPAA Business Associate Agreement may not explicitly name AI features that access PHI, creating a documentation gap that regulators can interpret as an uncontrolled use. AI introduces HIPAA-specific challenges: minimum necessary access must be applied at the prompt level (what PHI does the AI actually need to answer the query?), AI-generated clinical suggestions may constitute a covered function requiring additional documentation, and the audit trail requirement must extend to AI decisions, not just data access events.
✓ Passing answer
Confirms BAA explicitly names AI features; describes minimum necessary access enforcement at the prompt or query level; provides decision audit logging that attributes AI outputs to specific model versions, inputs, and timestamps — the record required if a regulator asks to explain a specific AI-influenced clinical decision.
✕ Walk-away answer
"We have a standard HIPAA BAA in place" without confirming it explicitly names the AI feature. A generic BAA that was not updated when AI capabilities were added may not cover PHI access through the AI inference layer — creating uncontrolled BAA exposure.
Compliance
"Have you adopted ISO 42001 or NIST AI RMF, and how does your development process align with AI-specific governance requirements beyond SOC 2?"
ISO 42001 (AI Management System, published 2023) and NIST AI RMF 1.0 (2023) are the two primary AI-specific governance frameworks addressing risk categories that ISO 27001 and SOC 2 do not cover: algorithmic accountability, AI bias monitoring, training data governance, and model performance oversight. An agency operating under these frameworks has structured processes for the AI-specific risks that traditional security frameworks miss. Neither is yet mandated in most jurisdictions, but both are increasingly requested by enterprise procurement and required by financial services regulators (including the Fannie Mae LL-2026-04 vendor accountability standard for mortgage AI).
✓ Passing answer
Can describe specific ISO 42001 or NIST AI RMF controls they apply; names the AI-specific risks these address (algorithmic bias, training data governance, model performance monitoring); and can produce evidence of implementation, not just awareness.
✕ Walk-away answer
"We follow best practices for AI development" without naming which framework, which controls, or how compliance is evidenced. Governance without documentation is aspiration, not control.
Certifications decoded — what each one actually tells you in 2026
| Certification / standard | What it verifies | AI-specific coverage | 2026 status |
|---|---|---|---|
| SOC 2 Type II | Organisational security controls operated effectively over a defined period (not just exist at a point in time) | None — does not cover model security, prompt injection, or training data | Baseline required — insufficient alone for AI |
| ISO 27001:2022 | Information security management system — policies, processes, and controls for data protection | None — predates AI risk categories | Baseline required — insufficient alone for AI |
| ISO 42001:2023 | AI management system — training data governance, algorithmic accountability, model performance oversight, bias monitoring | Specifically designed for AI — covers the gap SOC 2 leaves | Emerging requirement — increasingly required by enterprise procurement |
| NIST AI RMF 1.0 | Framework for AI risk identification, assessment, and management across the AI lifecycle | AI-specific — covers trustworthiness, bias, transparency, explainability | Increasingly required by US federal contractors and financial services |
| EU AI Act Annex IV | Technical documentation for high-risk AI systems — architecture, training data, performance metrics, cybersecurity measures | AI-specific — mandatory for high-risk AI systems effective August 2026 | Legally required in EU — most vendors not yet compliant |
| OWASP LLM Top 10 compliance | Security posture against the 10 most critical LLM application vulnerabilities including prompt injection, insecure output handling, training data poisoning | AI-specific — the most practical security framework for LLM applications | No certification body — ask for evidence of controls, not a certificate |
Domain 4 of 6
Model governance & monitoring post-deployment
Governance
"How do you monitor for model drift — and what metric threshold triggers an investigation or retraining cycle?"
Model drift is the degradation in AI performance as real-world input distributions diverge from training data. It is one of the most common causes of quiet failures — the system produces outputs that look plausible but are increasingly inaccurate, with no visible error to trigger investigation. Without monitoring, model drift may go undetected for weeks or months while the system accumulates incorrect decisions at scale.
✓ Passing answer
Names specific metrics monitored (accuracy on a holdout set, output confidence scores, user correction rate, hallucination rate on a test suite); defines numerical thresholds that trigger escalation; and specifies who owns drift monitoring post-launch and with what frequency.
✕ Walk-away answer
"We'll monitor it" or "our model handles edge cases well" — neither defines a monitoring process nor a threshold for action. A model without defined drift monitoring is a model you will discover has failed when a user reports a problem, not before.
Governance
"If a regulator asks me to explain a specific AI-generated decision that affected a customer or employee, what audit trail does the system produce?"
The EU AI Act's Article 26 obligation on deployers, the Fannie Mae LL-2026-04 vendor accountability standard for mortgage AI, and HIPAA's audit trail requirements all converge on the same demand: when an AI system makes a decision that affects a real person, you must be able to produce a contemporaneous record of what model, what version, what inputs, and what logic produced that decision. Without per-decision logging, regulatory accountability is structurally impossible regardless of how well the model otherwise performs.
✓ Passing answer
Produces audit log schema showing: decision timestamp, model name and version, input summary, output produced, confidence score or uncertainty metric, and any human review that occurred. Deployer (you) has read access to logs — the audit trail is not locked inside the vendor's systems.
✕ Walk-away answer
"We log all API requests" — API request logs capture what was sent; decision audit trails capture what the model did with it and why. These are different data structures. A vendor who conflates them has not built decision accountability into their system.
Domain 5 of 6
IP ownership & data rights in AI development
IP Rights
"Who owns the AI-generated code produced during this engagement — and what are the IP implications of the coding tools your engineers use?"
GitHub Copilot, Cursor, and equivalent AI coding tools are used in the vast majority of development workflows in 2026. The IP status of AI-generated code is an unresolved legal area. Some foundation model providers' terms of service include licence provisions over generated outputs; some have indemnification programmes that protect customers; others do not. Without explicit contractual language transferring full ownership of AI-generated code to the client, the IP status of what you paid to build may be ambiguous.
✓ Passing answer
Names the AI coding tools used, confirms the IP position under their respective terms (ideally cites a specific indemnification or IP transfer provision), and confirms the development contract explicitly transfers all IP — including AI-generated portions — to the client on delivery.
✕ Walk-away answer
"You own everything we build" without addressing AI-generated code specifically, or uncertainty about what coding tools are used. In 2026, every professional development team uses AI coding tools. Not knowing their IP implications is not a minor gap.
IP Rights
"If the engagement ends early, what do we receive — and can we operate the AI system independently without ongoing dependency on your infrastructure or proprietary components?"
AI systems can create hidden dependencies: proprietary prompt libraries, custom evaluation frameworks, vendor-managed fine-tuning infrastructure, or model weights hosted exclusively on the agency's infrastructure. Without explicit exit terms, an AI system you paid to build may be inoperable the moment the agency relationship ends — because the components required to run it are owned or managed by the agency, not you.
✓ Passing answer
Provides a specific exit artifact list: model weights or API configuration, prompt library, evaluation test suite, deployment infrastructure documentation, and onboarding documentation sufficient for an internal team or new agency to operate the system independently. Confirms no proprietary component is required for basic operation post-handoff.
✕ Walk-away answer
"We handle the infrastructure" without describing what you receive at exit. A system whose operation depends on a vendor's proprietary infrastructure is a vendor dependency, not a deliverable.
Domain 6 of 6
AI incident response — failures that don't look like traditional breaches
Incident Response
"What is the procedure for taking the AI system offline immediately — and what service continuity fallback exists while it is offline?"
AI systems can fail in ways that require immediate shutdown: a successful prompt injection attack that causes the system to act against its intended purpose; a model update from the foundation model provider that breaks expected behaviour; drift that has degraded output quality below acceptable thresholds. The ability to take the system offline instantly — a kill switch — with a defined fallback to manual processes or a prior system version, is a basic AI operational requirement that many production AI systems lack.
✓ Passing answer
Describes a specific kill-switch mechanism (named control, named authority to execute it, response time); a fallback workflow that maintains service continuity during AI outage; and a version rollback capability that can restore prior model behaviour if a new model version causes unexpected degradation.
✕ Walk-away answer
"We'll figure it out if it happens" or no articulation of a kill switch. An AI system without a tested shutdown procedure is a system whose failure mode is uncontrolled.
Incident Response
"If the AI system produces widespread harmful or false outputs — affecting many users simultaneously before detection — what is the notification and communication procedure?"
AI incidents are categorically different from traditional data breaches in their scope and speed. A prompt injection that causes a customer-facing AI to provide dangerous medical advice, or model drift that causes an AI underwriting system to systematically misclassify applications, can affect thousands of users before the pattern is detected. The communication strategy for AI-specific failures requires different procedures from targeted breach notification — potentially including proactive outreach to all affected users, not just those who reported a problem.
✓ Passing answer
Distinguishes AI-specific incident categories (model failure, safety bypass, data leakage through output) from standard security incidents; defines severity tiers and notification timelines for each; names the client contact who will be notified; and describes the audit log review process to determine affected scope.
✕ Walk-away answer
"We follow our standard incident response plan" without demonstrating that the plan addresses AI-specific failure modes. Standard IR plans were not designed for widespread simultaneous incorrect AI outputs — they were designed for data breaches affecting specific records.
Scoring the responses — Satisfied / Partial / Missing
Score each question in one of three states before evaluating the overall vendor. The three-state model — from DeepInspect's AI vendor due diligence framework — is more useful than a binary pass/fail because it separates vendors who have not yet implemented a control from vendors who are unaware of the risk the control addresses. Those are different risk profiles requiring different responses.
Scoring rubric — apply per question
Satisfied
The vendor provided a specific, verifiable answer with named controls, documented processes, and evidence available on request. No follow-up required — move to the next question.
Partial
The vendor demonstrated awareness of the risk but could not evidence a specific control. Requires a follow-up document request before the evaluation closes. Partial answers on data handling or IP ownership require written responses, not verbal assurance. Partial answers on prompt injection or kill-switch procedures require a live technical demonstration.
Missing
The vendor could not answer, gave a vague response that doesn't address the risk, or demonstrated they do not understand why the question is being asked. Missing answers are high-risk indicators. On critical questions (training data provenance, prompt injection, IP ownership, HIPAA BAA scope), a single Missing answer warrants a walk-away assessment regardless of all other scores. The procurement decision must weigh the Missing answers against business need — but the risk team's job is to surface them, not to normalise them.
Reassess annually, not once. AI systems change faster than traditional software — foundation model providers release updates, new attack techniques emerge, regulatory requirements evolve. A vendor who passes this assessment in July 2026 may not pass it in July 2027 without documented progress on any Partial answers. Build annual reassessment into the vendor management programme as a contractual obligation, not a discretionary review. For the broader agency evaluation framework covering all domains — not just security — see our complete red flags and green flags guide and our AI agency briefing guide.
