Zod-Validated LLM Output: The Repair Loop That Makes It Safe to Ship
Most tutorials still teach the deprecated generateObject path. Here is the Zod 4 + AI SDK 7 repair loop I run, with measured failures and the provider matrix.
On this page
Structured outputs guarantee shape, not truth
Start with the sentence every vendor writes in its own docs, in different words: structured output constrains what tokens the model can emit, so the response matches your schema. It says nothing about whether the values are right. A schema-valid response can carry a wrong total, an invented SKU, or a confident enum value nobody would have picked. OpenAI, Anthropic and Google all say this. The guarantee is shape. Truth is still your job.
There are three layers of failure, and only the first two are what most tutorials cover:
Syntax failures. The output does not parse: a trailing comma, an unclosed brace, prose wrapped around the JSON, a markdown fence. Constrained decoding eliminates most of these, but only when you actually use a strict mode. Prompt-enforced JSON - "return JSON only" in the prompt - does not.
Schema failures. The output parses but does not match the contract: a key named invoiceNo instead of invoice_number, a string where a number was required, an enum value paraphrased instead of exact, a nested object on the wrong discriminated-union branch. Strict modes prevent most of these at decode time. They cannot prevent the next layer.
Semantic failures. Valid, well-typed, and wrong. The best clean tool-use rate in an October 2026 benchmark across 19 models on 1,000 single-turn tasks was about 86 percent. The other 14 percent were wrong, skipped, or unnecessary calls and fabricated results - none of them malformed JSON. An agent that never produces malformed calls can still be wrong 15 to 20 percent of the time. No decoder fixes that. Only validation of business rules the schema cannot express, and a human in the loop for the expensive ones, does.
The numbers behind the first two layers are now measured, not guessed. An October 2026 study (Verma, arXiv 2610.33949) ran 3,997 generation attempts across nine OpenAI and Anthropic model configurations with the schema stated in the prompt and Pydantic validation after. Markdown-fenced JSON was the most common failure: of 681 fenced failures, only 48.0 percent passed validation after the fence was removed. The rest hid further violations, mostly field-length violations. Fence-stripping feels like a fix. Half the time it is a bandage over a second failure.
And truncation is a failure mode of its own. A separate October 2026 study of harness trimmers found 3,827 of 11,189 tool results were truncated into invalid JSON by the harness itself - langchain-core keeps the first 500 characters, smolagents deletes the middle past 20,000. A truncated response is not a wrong response. It is an incomplete one, and retrying it without raising the token budget reproduces it exactly.
The failure battery I ran
I wanted my own numbers before writing a word of advice, so I ran a battery of eight realistic model outputs against a real triage schema - the same shape I use in production for document extraction, where every AI-generated section is validated against its own Zod schema before it is assembled. The schema:
import { z } from "zod";
const triage = z.object({
summary: z.string().min(10).max(200),
category: z.enum(["billing", "bug", "feature", "account"]),
priority: z.enum(["low", "medium", "high", "urgent"]),
confidence: z.number().min(0).max(1),
needsHuman: z.boolean(),
tags: z.array(z.string()).max(5),
});The battery, run with Zod 4.4.3 from this site's own dependencies (the article pins zod@^4.6.7, the current release as of November 17, 2026):
| Model output | Raw safeParse | After salvage |
|---|---|---|
| Clean JSON object | Pass | Pass |
| Prose + fenced code block | Fail: not valid JSON | Pass |
Hallucinated enum value (category: "refund") | Fail: invalid option | Fail |
| Truncated mid-object (token cap cut) | Fail: not valid JSON | Fail: no object found |
Number as string (confidence: "0.92") | Fail: expected number | Fail |
Missing required field (no priority) | Fail: invalid option | Fail |
Extra unknown key (internalNote) | Pass (key stripped) | Pass |
Number out of range (confidence: 1.4) | Fail: too big | Fail |
Two of eight pass as returned. Three of eight pass after a deterministic salvage step - strip the prose and fences, extract the first {...}. The other five need a model retry with the validation error fed back, and one of those five (truncation) needs a bigger token budget as well, because the same cap will cut the same output again.
Two details in that table deserve a second look. First, the extra-key case passes, and that is a choice, not an accident: a plain z.object strips unknown keys silently. If a model-added field vanishing bothers you, z.strictObject rejects it instead, and z.looseObject passes it through. Pick one per feature and write down why. Second, the string-number case is fixable without a retry: z.coerce.number() accepts "0.92" and returns 0.92. I use coerce only where the source genuinely cannot be trusted to type its numbers - query strings, form fields, model output - and never as a blanket habit, because it also accepts things you would rather reject.
The check itself is effectively free. I measured 20,000 safeParse calls on this schema at 25.1 milliseconds total, about 1.3 microseconds per check. The model call costs seconds and cents. Skipping validation to save time is not a trade-off; it is a rounding error dressed as one.
Run the battery yourself. Every scenario below is the real schema, running in your browser:
Zod 4 is the contract
If you learned Zod on v3, half of what you remember is now the deprecated way, and the v4 migration guide is the document to read before you write a tutorial. Zod 4.0 shipped in July 2025; the current release is 4.6.7 (October 12, 2026). The parts that matter for LLM output:
The schema converts to JSON Schema natively. z.toJSONSchema(schema) is built into v4 and replaces the zod-to-json-schema package. This is the bridge the whole post stands on: one schema produces the TypeScript type, the runtime validator, and the JSON Schema the provider's structured-output mode consumes. When I print the triage schema through it, the output is a draft 2020-12 JSON Schema with additionalProperties: false on the object - which is exactly what OpenAI's strict mode demands. Print yours once and read it. The provider sees the JSON Schema, not your Zod code.
Unknown keys are a three-way choice. z.object strips them. z.strictObject rejects them with an error. z.looseObject passes them through. The method forms .strict() and .passthrough() still work but are deprecated. For model output I default to plain z.object (strip) on extraction features and z.strictObject on anything where a surprise key means a client bug.
Salvage lives per field. .catch(value) substitutes a fallback when a field exists but fails validation - z.enum(["light", "dark"]).catch("light"). .default(value) fills a missing field. .prefault(value) is new in v4 and parses the fallback through the schema, which is what .default() used to do in v3. Every automatic repair is a data decision: log it, because a field you silently defaulted is a field you chose not to know about.
There is no partial parse. .parse() throws, .safeParse() returns success or error, all or nothing. .partial() makes top-level fields optional; .deepPartial() was removed in v4. Salvage is per-field with .catch and .default, or at the streaming layer where the SDK yields only fully validated elements. Do not write a partial-parse helper that guesses at the missing half of an object. That is how corrupt rows get written with a confident face.
The error API changed. Issues live on .issues, not .errors. ZodError.format() and .flatten() are deprecated in favour of top-level z.treeifyError() and z.flattenError(). The error customisation is one unified error parameter; invalid_type_error, required_error and errorMap are gone, and a schema-level error now wins over the error map passed to .parse() - the opposite precedence to v3. String formats are top-level functions now: z.email(), z.url(), z.uuid(); the method forms still work but are deprecated. And z.coerce fields now take unknown input, with a missing key on a coerce field erroring unless you add .default() explicitly.
One more v4 note, because it decides what you pin: Zod 4 requires strict: true in tsconfig and TypeScript 5.5 or later. If your project cannot meet that yet, stay on v3 deliberately and say so in your code, rather than fighting the compiler.
The provider matrix is not one schema
Here is the part that costs teams the most time: the three providers do not accept the same JSON Schema subset, and the differences are not cosmetic. A schema that is valid for OpenAI can be rejected by Anthropic, and a constraint Gemini silently ignores is a constraint your Zod schema must still enforce locally.
| OpenAI | Google Gemini | Anthropic | |
|---|---|---|---|
| How you ask | response_format: { type: "json_schema", json_schema: { name, strict: true, schema } } | responseFormat with mimeType: "application/json" and the schema (Interactions API); legacy responseSchema + responseMimeType on generateContent | output_config.format: { type: "json_schema", schema } (GA January 29, 2026) |
| What it guarantees | 100% schema adherence under strict: true - constrained decoding means a non-conforming token cannot be emitted | Shape when both the mime type and the schema are set; the mime type alone is only a strong hint | Grammar-constrained sampling - the model cannot emit a character that breaks the schema |
| Hard limits | 5,000 object properties, 10 nesting levels, 1,000 enum values; every property must be in required; additionalProperties: false on every object | No published property or nesting ceiling; very large or deeply nested schemas may be rejected with a 400 | About 20 strict tools, 24 optional parameters, 16 union-typed parameters per request; no recursive schemas; no external $ref |
| What it strips or rejects | Rejects allOf, oneOf, not, dependentRequired, if/then/else with a 400 | Ignores minLength, maxLength, minProperties, maxProperties, uniqueItems - your Zod schema must enforce them | minimum, maximum, multipleOf, minLength, maxLength not supported; minItems only 0 or 1; enum casing not guaranteed to match exactly |
| When it still fails you | Safety refusal (detectable refusal field); truncation at the token cap (finish_reason: "length") | Refusals and truncation return HTTP 200 with schema-violating output; Google's own docs say to validate again on your side | Refusal (stop_reason: "refusal"); token-cap cut-off (stop_reason: "max_tokens"); forced tool choice errors with manual extended thinking |
Three warnings from that table, because each one has bitten someone I know. First, Gemini's legacy pair is on the way out: Google's API reference marks responseSchema deprecated and points to responseFormat and the Interactions API, while the Vertex docs still present the legacy pair without a deprecation flag. Teach the new path, show the legacy one as widely deployed, and do not build examples on gemini-2.5-* model strings - 2.5 Pro and 2.5 Flash are deprecated, and from December 1, 2026 API requests auto-route them to 3.8 Flash and 3.5 Flash-Lite. Second, Anthropic's SDK helpers strip the unsupported keywords into field descriptions and validate client-side - which is exactly why your local Zod check is not redundant even under strict mode. Third, OpenAI's launch-era limits (100 properties, 5 nesting levels, 500 enums) still circulate in old posts; the current figures are 5,000, 10 and 1,000.
The practical consequence is simple: one schema, three wire formats, and your Zod schema is the only layer that sees all of them. Generate the JSON Schema from Zod, send the provider its subset, and validate the response against the original Zod schema when it comes back. The provider's guarantee and your validation are two different checks, and you need both.
The current AI SDK path
Most tutorials you will find still teach generateObject. That function was deprecated in AI SDK 6, and the current docs teach generateText with the output option instead. The SDK's own guide page lagged the migration guide for months, which is why the stale path is everywhere. If you are writing new code in November 2026, the shape is AI SDK 7 with Zod 4:
import { generateText, Output, NoObjectGeneratedError } from "ai";
import { z } from "zod";
const { output } = await generateText({
model,
output: Output.object({ schema: triage }),
prompt: "Triage this ticket. Return JSON only.",
});Output.object is the common case. Output.array takes an element schema plus minItems and maxItems, and with streamText its elementStream yields each element only once it is complete and validated. Output.choice classifies from a fixed set of strings - the successor to the old output: "enum". Output.json checks only that the response is valid JSON, with no schema validation at all. Structured output counts as a step in the SDK's multi-step model, so account for it in stopWhen when you combine it with tool calls.
When the response cannot be parsed or validated, generateText rejects with NoObjectGeneratedError. It preserves the raw text, the cause (for example a JSON parse error), the response metadata, and the token usage - everything the repair loop needs to decide what to do next. There is no partial object on failure. It throws; it does not hand back a half-broken object for you to guess at.
One thing the SDK does not do for you: maxRetries is a transport-level retry for failed requests, not a validation retry. A validation failure throws immediately, and your code decides whether to retry. That is the correct design - the retry needs the validation error as context, and only your code has it.
The repair loop
Here is the loop, in the shape I run. It has four stages, and the order matters: validate first, salvage second, retry with the error third, dead-letter last.
import { generateText, Output, NoObjectGeneratedError, type LanguageModel } from "ai";
import { z } from "zod";
const MAX_ATTEMPTS = 3;
/** Deterministic salvage: strip prose and fences, extract the first {...}. */
function salvage(raw: string): string | null {
let s = raw.trim();
const fence = s.match(/```(?:json)?\s*([\s\S]*?)```/);
if (fence) s = fence[1].trim();
const start = s.indexOf("{");
const end = s.lastIndexOf("}");
if (start === -1 || end === -1 || end <= start) return null;
return s.slice(start, end + 1);
}
function safeJsonParse(raw: string): unknown {
try {
return JSON.parse(raw);
} catch {
return raw;
}
}
/** Providers surface truncation differently; read whichever reason is present. */
function isTruncated(error: NoObjectGeneratedError): boolean {
const cause = error.cause as { finishReason?: string; stopReason?: string } | undefined;
const reason = cause?.finishReason ?? cause?.stopReason;
return reason === "length" || reason === "max_tokens";
}
function isRefusal(error: NoObjectGeneratedError): boolean {
const cause = error.cause as { stopReason?: string; refusal?: unknown } | undefined;
return cause?.stopReason === "refusal" || cause?.refusal !== undefined;
}
/** Write the raw text, the schema version, and the model id to a review queue. */
async function deadLetter(entry: { raw: string; reason: string; ticketText: string }) {
// Your queue of choice: a database table, a BullMQ dead-letter queue,
// a Slack channel. The point is that it exists and a human reads it.
console.error("dead-lettered extraction", entry);
}
async function extractTriage(ticketText: string, model: LanguageModel) {
let lastIssues = "";
let lastRaw = "";
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
const prompt =
attempt === 1
? `Triage this ticket. Return JSON only.\n\n${ticketText}`
: `The previous output failed validation:\n${lastIssues}\nReturn corrected JSON only.\n\n${ticketText}`;
try {
const { output } = await generateText({
model,
output: Output.object({ schema: triage }),
prompt,
});
return output; // typed, validated, done
} catch (error) {
if (!NoObjectGeneratedError.isInstance(error)) throw error;
lastRaw = error.text ?? "";
// Truncation is not a retryable shape error. Raise the budget.
if (isTruncated(error)) {
throw new Error("output truncated: raise max_output_tokens", { cause: error });
}
// A refusal is not a validation failure. Route it to review.
if (isRefusal(error)) {
await deadLetter({ raw: lastRaw, reason: "refusal", ticketText });
throw new Error("model refused: routed to review", { cause: error });
}
// Stage 2: deterministic salvage before spending another model call.
const salvaged = salvage(lastRaw);
if (salvaged) {
const check = triage.safeParse(JSON.parse(salvaged));
if (check.success) return check.data;
}
// Stage 3: the next attempt carries the validation issues, not a
// generic "try again". Field path plus message, nothing more.
const failed = triage.safeParse(safeJsonParse(lastRaw));
const issueLines: string[] = [];
if (failed.error) {
for (const issue of failed.error.issues) {
issueLines.push(`${issue.path.join(".")}: ${issue.message}`);
}
}
lastIssues = issueLines.join("; ");
}
}
// Stage 4: whatever survives three attempts never touches the database.
await deadLetter({ raw: lastRaw, reason: "validation failed after 3 attempts", ticketText });
throw new Error("triage failed validation after 3 attempts");
}The rules that keep this loop honest:
Retry only when it helps. A wrong enum, a missing field, a string where a number belongs - the model can fix all of these when told exactly what was wrong. A refusal cannot be fixed by asking again. A truncation cannot be fixed without a bigger token budget. The loop checks the failure kind before it spends the next call, because a retry that cannot succeed is not persistence, it is a billing event.
Cap it at three. The first attempt is the model doing its best. The second is the model with the error in front of it. The third is the last chance for a flaky output. Past three, you are no longer repairing; you are hoping, and hoping at model-call prices. Whatever survives goes to the dead-letter queue with the raw text, the schema version, and the model id, so a human can see exactly what failed.
Feed the error back, not a summary of it. The retry prompt carries the Zod issues verbatim - field path plus message. "Try again" teaches the model nothing. category: Invalid option: expected one of "billing"|"bug"|"feature"|"account" teaches it the exact constraint it violated. Cap the feedback at a handful of issues; a wall of twenty errors is noise the model will skim.
Log every automatic repair. A .catch() fallback, a salvage rescue, a second-attempt success - each one is a data decision your code made on the reader's behalf. Log the attempt count and the failure kind next to the record. When a user asks why a ticket was triaged "billing" instead of "account", the log is the answer.
Cost it before you ship it. Validation costs about 1.3 microseconds per check - I measured it. The retry is the real cost: three attempts worst-case triples the extraction cost of that record. At low volume that is invisible; at a million extractions a month it is a line item. Budget the worst case, not the happy path, and alert on the retry rate the same way you would alert on an error rate. A climbing retry rate is a schema problem or a model problem, and either way it is cheaper to find in your dashboard than in your invoice.
I ran this loop's logic against stub model outputs before publishing: a fenced first attempt rescued by salvage, a wrong-enum first attempt fixed on the second attempt with the error fed back, and a permanently invalid output dead-lettered after three attempts. The SDK call shape above is verified against the AI SDK docs I read while writing this post; the loop logic is verified by execution.
The production checklist I run before shipping
Six checks, in the order I run them, on the real feature - not on the demo. The first three are about the schema, the last three about the loop.
- The schema is pinned and the wire schema is printed.
zod@^4.6.7in package.json, andz.toJSONSchema()logged once and read by a human. The provider sees the JSON Schema, not your Zod code. If the conversion dropped a constraint, the wire is where you find out. - The unknown-key policy is deliberate. Strip, pass through, or reject - chosen per feature, written down. The default (strip) is fine for extraction and wrong for anything where a surprise key means a client bug.
- The salvage step is tested on all eight failure shapes. Fence-stripping rescues the most common failure and only about half of fenced outputs are clean underneath. Run the battery from this post against your schema before you trust the salvage path.
- Retries are bounded at three, carry the validation error, and never retry a refusal. The retry prompt includes the Zod issues, not a generic "try again". Refusals go to human review. Truncations get a bigger token budget, not another identical attempt.
- Truncation is detected from the finish reason, not from a parse error.
finish_reason: "length",stop_reason: "max_tokens", or anincompletestatus means the output was cut, not wrong. A parse error on a truncated response is a symptom. The fix ismax_output_tokens. - The dead-letter queue exists, and you have replayed one failure from it. Whatever survives three attempts lands in a review queue with the raw text, the schema version, and the model id. A dead-letter queue you have never replayed from is a dead-letter queue you have not tested.
My verdict
Use generateText with Output.object, a Zod 4 schema as the single contract, and the repair loop above - validate, salvage, retry with the error, dead-letter at three attempts. That is the setup I would defend for any feature whose output lands somewhere it cannot be undone: a database, another system, a user acting on it.
The conditions that would flip it: if a human reviews every output before anything depends on it, drop the loop and keep the schema - the schema is the part that is hard to retrofit, the loop is easy to add later. If you run one provider and one model at high volume, put that provider's strict mode first and Zod second, and spend the saved effort on the business-rule checks no schema can express. And if the feature is a prototype or a read-only view, do not build any of this yet - log the raw outputs, and build the loop when the first bad JSON costs you a user or an hour.
What I would not do, in any of those cases, is the thing I did on call fifty-one: trust the prompt. "Return JSON only" is a request, not a contract. A schema is a contract. The difference between them is the afternoon your support agent spends untangling a row that should never have been written.
Not sure which posture fits your feature? Answer four questions and I will place you:
If your agent features are growing past one extraction call and you want the loop, the harness, and the cost controls built as one system instead of bolted on one feature at a time, that is the layer I build for teams. Tell me about your project and I will tell you honestly whether you need the loop, the harness, or nothing at all.
Related reading: Zod as a Contract, Not a Comment, the primer this post builds on - one schema as type, validator, and JSON Schema. TanStack AI vs Vercel AI SDK if you are still choosing the SDK the loop runs on.
FAQ
Does structured output mean I can skip Zod validation?
No. Structured output guarantees shape, not truth: a schema-valid response can still carry a wrong total, an invented enum value, or a factually wrong field. Providers also strip JSON Schema keywords they do not support, and refusals and token-cap truncation bypass the guarantee entirely. Keep Zod validation after every provider call.
Is generateObject still OK to use?
It was deprecated in AI SDK 6 and the current docs teach generateText with Output.object instead. Old tutorials still show generateObject, and the SDK's own guide page lagged the migration guide for months. New code should use generateText with Output.object, Output.array, Output.choice or Output.json.
What does Zod 4 change for LLM output?
z.toJSONSchema is built in, replacing the zod-to-json-schema package. z.strictObject and z.looseObject replace the deprecated .strict() and .passthrough(). .catch and .prefault give you per-field salvage, and the error API is now .issues with z.treeifyError. There is still no partial-parse API: safeParse is all or nothing.
How many retries should the repair loop allow?
Three attempts, then dead-letter. Each retry carries the Zod issues (field path plus message) back to the model. Never retry a refusal blindly - route it to human review - and never retry a truncation without raising the token budget first, because the same cap will cut the same output again.
Which provider's structured output should I pick?
Pick the provider by the model you need, not by the structured-output feature: all three guarantee shape under their strict modes and all three strip or reject part of the JSON Schema subset. OpenAI is the most complete, Gemini ignores string-length bounds, Anthropic ignores numeric bounds and caps minItems at 0 or 1. Validate locally no matter which one you use.
How much does validation cost?
Almost nothing. I measured 20,000 safeParse calls on a real triage schema at 25.1 ms total, about 1.3 microseconds per check. The model call costs seconds and cents; the check is free by comparison. The real cost of the loop is retries: three attempts worst-case triples the extraction cost, so budget it.
The tools I would actually use
| Tool | What it is | What I use it for |
|---|---|---|
| Zod 4.6.7 | TypeScript schema validation, free and open source | The contract: type, validator, and JSON Schema from one object |
| Vercel AI SDK 7 | Provider-agnostic AI SDK, free and open source | generateText with Output.object, structured output across providers |
| OpenAI structured outputs | response_format with json_schema, strict: true | The most complete strict mode when the model choice is OpenAI |
| Google Gemini structured output | responseFormat / Interactions API | Structured output when the model choice is Gemini |
| Anthropic structured outputs | output_config.format with json_schema | Structured output when the model choice is Claude |
Comments
No comments yet. Be the first.
Leave a comment