Third-Party Risk

The Limits of Vendor Security Questionnaires

XR
Xcigence Research
Cyber Risk Intelligence Team
8 min read

The security questionnaire is the most widely used instrument in third-party risk management and the least interrogated. It is self-attested, point-in-time, unverified and easy to answer aspirationally — and yet it still has two jobs that nothing else can do. Knowing which two is the whole skill.

Where questionnaires came from

Questionnaires solved a real problem in a world with no alternatives. If you needed to know whether a supplier encrypted data at rest and had no way to observe them, asking was the only option — and putting the answer in writing created a record and a degree of accountability. ISO/IEC 27036 formalized this kind of supplier information exchange as part of managing supplier relationships.[4]

What has changed is not that asking became useless. It is that asking is no longer the only way to know, while questionnaire volume has grown to the point where the instrument collapses under its own weight — vendors receiving hundreds a year, buyers receiving answers nobody has capacity to read.

The four structural limits

These are not implementation problems to be fixed with a better template. They are properties of the instrument itself.

  • Self-attestation — The answers come from the party with an interest in the outcome, usually from someone who cannot personally verify them. "Do you enforce MFA?" is answered from policy, not from a coverage report.
  • Point-in-time — A questionnaire describes the day it was completed. Vendor posture changes weekly — acquisitions, staff departures, new internet-facing systems, newly disclosed vulnerabilities. The document ages into fiction while remaining on file as evidence.
  • Unverifiable at scale — Verifying a single answer properly requires evidence review or audit. Across 400 vendors, verification does not happen, so the whole population is effectively trusted on assertion.
  • Aspiration bias — Questions written as controls invite answers describing intent. Policies exist; enforcement coverage, exception volume and tested recovery times are what actually matter — and questionnaires rarely distinguish them.
The uncomfortable test
Ask any vendor for their MFA coverage percentage across administrative accounts, with the exception list, rather than whether they "require MFA". The gap between the two answers is the gap between a questionnaire and reality.

What external evidence sees instead

Externally observable evidence is the mirror image of a questionnaire: it is unverifiable-by-assertion proof, continuously refreshed, requiring no cooperation — but blind to everything not visible from outside.

PropertyQuestionnaireExternal evidence
SourceThe vendorDirect observation
VerificationNone, unless separately auditedInherent — the finding is the evidence
FreshnessAnnual at bestContinuous
Cooperation neededYes — and often slowNo
Scales to 400 vendorsNoYes
Sees internal controlsClaims toNo
Sees actual exposureNoYes
Gaming riskHighLow — findings are observed, not declared
Questionnaires and external evidence compared.

Notice that neither column dominates. Evidence cannot tell you whether a vendor tests its backups; a questionnaire cannot tell you that the vendor has an unpatched, known-exploited vulnerability on an internet-facing server right now — which is precisely the class of finding most strongly associated with real incidents.[3][5]

What questionnaires are still for

Two jobs, and they are genuinely important.

  • Seeing inside — Internal segmentation, backup testing cadence, joiner-mover-leaver process, privileged access management, subprocessor governance, incident response rehearsal. None of these are externally observable. If you need to know them, you must ask — and then verify through evidence or audit.
  • Creating obligation — A written answer is a contractual and legal artefact. It establishes what the vendor represented, which matters for liability, and it forces an internal conversation on the vendor's side. NIST's supply chain guidance treats this documentation role as part of a broader programme, not as the assessment itself.[1]

Third-party attestations occupy a middle position worth understanding: a SOC 2 Type II report is independently examined over a period, which is stronger than self-attestation — but only within the scope and system boundary the vendor defined.[2] Always read the scope section before the opinion.

A better operating model

The sequence matters more than the instruments. High-functioning programmes invert the traditional order:

  • 1. Score the whole population externally — Every vendor gets a measured baseline immediately, with no vendor effort at all. For the majority of vendors, this is the entire assessment.
  • 2. Tier by consequence — Rank by what the vendor could cost you, not by spend — the logic set out in building a third-party programme.
  • 3. Question only what evidence cannot answer — For critical vendors, replace the 300-question template with a short, pointed set aimed at internal controls — and at anything the external evidence flagged.
  • 4. Demand artefacts, not assertions — Coverage reports, dated recovery test results, penetration test summaries with remediation status.
  • 5. Monitor continuously thereafter — The questionnaire's expiry problem is solved by monitoring, not by asking again next year.

Twelve questions worth asking

When you do question a critical vendor, ask for numbers and dates rather than yes/no. What percentage of administrative accounts enforce phishing-resistant MFA, and what is on the exception list? When did you last restore production data from backup, and how long did it take? What is your median time to patch internet-facing critical vulnerabilities, measured over the last quarter? Which subprocessors can access our data, and how are they assessed? What was your last security incident, and what changed as a result? Who can access our data, and how is that access reviewed?

Twelve questions of that character, sent to a vendor you have already measured externally, will tell you more than any standard template — because you already know where to press. That combination of continuous external measurement under our patented method[6] and targeted human enquiry is the model Xcigence is built around; see third-party cyber risk scoring.

Part of our guide to
Third-Party Cyber Risk Scoring

Scoring vendors, suppliers, and downstream providers on one methodology.

References

  1. [1]NIST. SP 800-161r1 — Cybersecurity Supply Chain Risk Management Practices
  2. [2]AICPA. SOC 2 — Trust Services Criteria and report scope
  3. [3]Verizon. Data Breach Investigations Report (DBIR)
  4. [4]ISO. ISO/IEC 27036 — Information security for supplier relationships
  5. [5]CISA. Known Exploited Vulnerabilities (KEV) Catalog
  6. [6]USPTO. Granted patent US 11,418,531 (application US 16/822,691) — Global Dossier record

See your own cyber risk score

Xcigence computes a standardized 300–850 cyber risk score for your organization and every vendor in your ecosystem — no agents, no questionnaires.

We use cookies to improve your experience, analyze traffic, and support our marketing. See our Privacy Policy.