SOC 2 and Penetration Tests: What Each Tells an AI Buyer

Diane runs operations at a promo distributor. On Monday her owner forwarded her a security questionnaire from their biggest client, with a note that said "can you just do this." By Wednesday the AI vendor they were evaluating had sent two PDFs. One said SOC 2 Type 2. The other said Penetration Test Summary.

Mark, the controller, looked at both covers and said, "Great, we're secure." Diane asked him secure against what. Mark thought about it, said "against audits," and left for lunch. The IT contractor, who bills in fifteen minute increments, then spent two of them noticing that the pen test covered the vendor's marketing website. The SOC 2 period had ended almost a year earlier. Neither document was wrong. They answered questions nobody in the building had asked yet.

A SOC 2 report tells you whether a vendor's controls were designed sensibly and, for a Type 2, whether they operated over a review period. A penetration test tells you what a tester could break inside an agreed scope during a fixed window. Read both against the system you are actually buying.

What a SOC 2 report covers

SOC 2 is an attestation report issued by an independent CPA firm. It evaluates a service organization's controls against the Trust Services Criteria published by the AICPA. The five categories are security, availability, processing integrity, confidentiality, and privacy. Security is always included, and the report states which other categories are in scope. See the AICPA SOC resources for the official description.

SOC 2 is not a certification. The auditor issues an opinion, and that opinion can carry qualifications. The useful parts of the report are the system description, the auditor's opinion, the list of controls, any exceptions, the complementary user entity controls, and the treatment of subservice organizations such as hosting or model providers.

A Type 1 report looks at whether controls were suitably designed as of one date. A Type 2 report also tests whether those controls operated effectively across a review period, and it lists the tests and results.

What a penetration test covers

A penetration test is a time boxed attempt to find and exploit weaknesses within an agreed scope and set of rules. Its value depends on what was in scope, when the work happened, who did it, and what access the testers had. NIST describes planning, execution, and reporting for this kind of work in its technical guide to security testing.

A good report lists findings by severity, explains how each was found, and records remediation and retest status. Many vendors share a summary or an attestation letter in place of the full report. A summary can be reasonable, but it should still tell you the scope, the dates, and whether serious findings were fixed and retested.

Why one does not replace the other

The two documents look at different things. A SOC 2 report asks whether processes like access reviews, change management, incident response, and vendor oversight ran consistently. A penetration test asks whether someone skilled could break specific parts of the system on specific days.

A clean pen test says nothing about whether a former employee lost access on their last day. A clean SOC 2 report does not prove that a particular API endpoint resists a particular attack. Each one fills a gap the other leaves open, so buyers should ask for both and read each for what it is.

What AI workflows add to the review

An AI agent often reads email and attachments, calls a model, and writes to an ERP or CRM. Those are typical operations workflows, and they raise questions a standard review may not cover. Ask whether the system description in the SOC 2 report includes the agent runtime and the stores that hold your business data. Ask how model providers and other subservice organizations are handled, since a report can carve them out of scope.

For the pen test, ask whether testers covered authenticated application functions, APIs, tenant isolation, and the connectors that touch your systems. Ask whether anyone tested what happens when a malicious instruction arrives inside a purchase order or email. That kind of attack is different from a classic web vulnerability, and our prompt injection checklist covers it in more detail.

Review checklist

Use this list when a vendor sends security evidence. Any line you cannot check becomes an open question for the vendor.

• The SOC 2 report names the service and environment you will use.
• You know whether it is a Type 1 or a Type 2, and the date or period it covers.
• You have checked for a bridge letter covering the time since the period ended.
• You have read the auditor's opinion and every listed exception.
• Each complementary user entity control has an owner on your side.
• You know which subservice organizations are carved out or included.
• The pen test scope includes the application, APIs, and tenant boundaries you rely on.
• You know the pen test dates and what changed in the product since.
• Serious findings show remediation and retest evidence.
• Someone has asked about testing against instructions hidden in emails and attachments.
• The evidence, the open questions, and the next review date are recorded in one place.

Trelium builds AI agents that automate order entry, invoice matching, and similar work for distributors and promo and apparel companies. If you are evaluating Trelium, ask us every question on this list, and ask for the current status and scope of any report we mention.

FAQ

1Is SOC 2 a certification?

No. SOC 2 is an attestation report in which an independent CPA firm gives an opinion on a service organization's controls. There is no official SOC 2 certificate or badge, so ask for the report itself and read its scope, period, opinion, and exceptions.

2What is the difference between SOC 2 Type 1 and Type 2?

A Type 1 report evaluates whether controls were suitably designed as of a single date. A Type 2 report also tests whether those controls operated effectively over a review period and lists the results. A Type 2 gives stronger evidence that controls work in practice over time.

3Can a penetration test replace a SOC 2 report?

No. A penetration test checks whether specific parts of a system can be exploited during a fixed window, within an agreed scope. A SOC 2 report evaluates whether organizational controls such as access management and change management are designed and operating, so the two cover different ground.

4How recent should a vendor's penetration test be?

There is no universal rule that fits every buyer. Ask for the test dates, what has changed in the product since then, and whether the vendor retests after major releases or infrastructure changes. A test of an older version of the system tells you less about what you would use today.

5What should I ask if a vendor's SOC 2 audit is still in progress?

Ask which stage it is in, such as a readiness assessment, an open review period, or fieldwork. Ask when a report is expected and what interim evidence, like policies or a controls summary, the vendor can share now. Record the status as in progress and plan a follow up review once the report exists.

6Do SOC 2 reports and pen tests cover AI risks like prompt injection?

Not necessarily. Neither format requires testing for instructions hidden in documents or for an agent misusing its tool permissions. Ask the vendor directly what testing covered prompt injection, tool authorization, access to unrelated records, and approval boundaries.