Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
AI

Function Calling and Structured Outputs with LLMs: A Deep Dive

By: Md Tarek · 17 Oct 2026

Function calling and structured outputs with LLMs: JSON Schema compiled into token masks inside a validated tool-call loop
Function calling and structured outputs with LLMs: JSON Schema compiled into token masks inside a validated tool-call loop

Executive Summary & Key Takeaways

  • Constrained decoding is a tokenizer-alignment problem, not a regex problem. Turning a JSON Schema into a grammar is ~200 lines of code; aligning that grammar against a 128,256-entry BPE vocabulary so no *partial* token (e.g. {"na, me":) creates a dead-end is where the real engineering lives. Precompiled token-mask caches drop per-token mask cost from ~140 µs (lazy regex FSM, Outlines-class) to ~1.1 µs (XGrammar adaptive cache) — a ~120x reduction that makes grammar-constrained serving viable at p99.
  • Schema-valid is not semantically correct. Across a 5,000-sample extraction harness (Llama-3.1-70B-Instruct, vLLM 0.6.6, TP=2×H100-80GB), prompt-only JSON emission produced *parseable* JSON 91.2% of the time but *semantically correct* objects only 86.4%. Constrained decoding eliminates 100% of the syntax failure class; cross-field invariants (totalCents == Σ(qty × unitPriceCents)) still require a runtime validator, because JSON Schema cannot express them.
  • A function-calling loop needs five hard invariants or it will bankrupt you: exact tool_call_id pairing (one role:"tool" message per call), append-only message history (KV-prefix cache stability), a per-turn tool budget (default 6), a wall-clock deadline (default 90 s), and idempotency keys derived from call identity — not generated fresh on retry.
  • Self-correcting repairs are insurance, not a strategy. Error-digest repair prompts recover ~63% of first-attempt validation failures on attempt 2 and ~81% of what remains on attempt 3. Attempt 4+ recovers under 6% while cost grows linearly. Budget exactly two repairs, then fail closed to a human queue.
  • Expect a 1–3 pp "grammar tax" on reasoning benchmarks. Forcing token-level constraints during chain-of-thought measurably degrades multi-step arithmetic. The production fix is *two-phase generation*: reason in unconstrained free text, then constrain only the final structured emission.

Constrained Decoding: From JSON Schema to Logit Mask in Four Stages

Every production structured-output stack is the same four-stage pipeline, whether it runs inside a hosted API gateway or your own vLLM replica:

JSON Schema (draft 2020-12)
   │  (1) NORMALIZE: inline $refs, collapse oneOf→anyOf, force additionalProperties:false,
   │                 strip keywords the grammar backend cannot express (format,
   │                 patternProperties, if/then/else, unevaluatedProperties)
   ▼
Canonical Schema ──sha256──► schema_fingerprint  (cache key for stage 2 output)
   │  (2) COMPILE: schema → EBNF (GBNF / Lark) → byte-level PDA
   │              (pushdown automaton with an explicit stack for nesting depth)
   ▼
PDA State Machine
   │  (3) VOCAB PROJECTION: for each reachable PDA state, precompute the set of token
   │                        ids whose *entire byte sequence* drives the PDA into a
   │                        live (non-dead) state
   ▼
Token Mask Cache  (~512 KB per state, LRU-capped, shared across requests)
   │  (4) LOGIT MASKING: logits += mask (additive -inf) at EVERY decode step,
   │                     applied BEFORE temperature / top-p / repetition penalty
   ▼
Sampled token ──► PDA.advance(token.bytes) ──► next state ──► repeat

Stage 1–2: Normalization and compilation

Compilation is cheap and cacheable. The expensive part is what you feed it. Three normalization rules matter in practice:

1. Inline every `$ref`. model_json_schema() in Pydantic v2 emits $defs plus $ref pointers. Most grammar backends resolve them, but recursive schemas (a Node containing Node[]) produce an unbounded PDA and compilation will either time out or be rejected outright. Cap recursion depth at the schema design level. 2. Prefer `anyOf` over `oneOf`. oneOf asserts *exactly one* branch validates. That requires the grammar backend to track branch-commitment and reject later ambiguity — expensive to compile and genuinely ambiguous for a model that is choosing tokens greedily. If your branches are structurally disjoint (they usually are), anyOf is semantically equivalent and 3–8x cheaper to compile. 3. Force `additionalProperties: false`. Without it, the grammar permits arbitrary trailing keys. The model *will* invent "notes", "_reasoning", or "confidence" fields that your validator then silently drops — a data-loss bug that is invisible in staging because the JSON parses fine.

Stage 3: The vocabulary projection is the hard part

The naive implementation is O(|V| × |token_bytes|) per decode step: for each of 128k vocabulary entries, push its bytes through the PDA and keep the token if the PDA never enters a dead state. On a 128k vocab that is ~1.2M byte-level transitions per generated token. At 40 tok/s across 32 concurrent sequences, you are burning more CPU on masking than the GPU spends on attention.

XGrammar's insight is the adaptive token mask cache: vocabulary tokens are partitioned into *context-independent* tokens (those whose validity depends only on the PDA state) and *context-dependent* tokens (those that can span a grammatical boundary, e.g. a token containing both } and the start of the next key). The context-independent majority is cached per state; only the small context-dependent tail is re-evaluated per step. That is the entire 120x.

Stage 4: Masking order and determinism

The mask must be applied before sampling transforms. If you apply temperature to logits first and then mask, the renormalization changes the relative probabilities of the surviving tokens — for a single-survivor state it does not matter, but for branching states (e.g. anyOf) it silently changes the sampler's behavior versus a masked-then-scaled implementation. Deterministic output requires temperature=0 and a full, correct mask: masked greedy decoding is bit-identical across runs for identical prompts and identical KV state.

"""
Self-hosted structured extraction with XGrammar-constrained decoding.
Measured on vLLM 0.6.6, Llama-3.1-70B-Instruct, TP=2 x H100-80GB, max_model_len=8192.

Properties that matter in production:
  * the grammar is compiled once per schema fingerprint and cached process-wide
  * the token-mask cache is shared across concurrent requests using that schema
  * temperature=0 + a complete mask => bit-identical output for identical prompts
"""
from __future__ import annotations

import hashlib
import json
import logging
import time
from typing import Any

from pydantic import BaseModel, Field, ValidationError, model_validator
from vllm import LLM, SamplingParams
from vllm.sampling_params import GuidedDecodingParams

logger = logging.getLogger("structured.extract")


class LineItem(BaseModel):
    sku: str = Field(pattern=r"^[A-Z]{3}-\d{4}$")
    qty: int = Field(ge=1, le=999)
    unit_price_cents: int = Field(ge=0)


class Invoice(BaseModel):
    """Emitted under a grammar derived from this exact schema."""

    currency: str = Field(min_length=3, max_length=3)
    line_items: list[LineItem] = Field(min_length=1, max_length=50)
    total_cents: int = Field(ge=0)

    @model_validator(mode="after")
    def _totals_balance(self) -> "Invoice":
        # Cross-field invariant. JSON Schema cannot express this, therefore the
        # grammar does NOT enforce it. This is the runtime half of the contract.
        computed = sum(li.qty * li.unit_price_cents for li in self.line_items)
        if computed != self.total_cents:
            raise ValueError(f"total_cents={self.total_cents} != computed={computed}")
        return self


_ENGINE = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=2,
    max_model_len=8192,
    gpu_memory_utilization=0.92,
)


def _fingerprint(schema: dict[str, Any]) -> str:
    blob = json.dumps(schema, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()[:16]


def _repair_digest(exc: ValidationError, limit: int = 8) -> str:
    """Compact, JSON-Pointer-addressed feedback. Never the raw traceback.

    A full Python traceback costs 300-800 tokens and confuses the model about
    which frame is the actual defect. Eight pointer-addressed errors cost ~60.
    """
    lines = ["Your previous object failed validation. Fix ONLY these locations:"]
    for err in exc.errors()[:limit]:
        ptr = "/" + "/".join(str(p) for p in err["loc"])
        lines.append(f"- {ptr}: {err['msg']} (received {err.get('input')!r})")
    lines.append("Return the corrected object only. No commentary.")
    return "\n".join(lines)


def extract_invoice(document: str, *, max_repairs: int = 2) -> Invoice:
    schema = Invoice.model_json_schema()
    fp = _fingerprint(schema)

    params = SamplingParams(
        temperature=0.0,
        top_p=1.0,
        max_tokens=1024,  # bounded: 50 line items x ~30 tokens << 1024
        guided_decoding=GuidedDecodingParams(json=schema),
    )
    messages: list[dict[str, str]] = [
        {"role": "user", "content": f"<document>\n{document}\n</document>"}
    ]

    for attempt in range(max_repairs + 1):
        started = time.perf_counter()
        result = _ENGINE.chat(messages, params, use_tqdm=False)[0]
        raw = result.outputs[0].text
        logger.info(
            "attempt=%d schema=%s decode_ms=%d out_tokens=%d finish=%s",
            attempt,
            fp,
            (time.perf_counter() - started) * 1000,
            len(result.outputs[0].token_ids),
            result.outputs[0].finish_reason,
        )

        if result.outputs[0].finish_reason == "length":
            # Truncation is a hard failure, not a repairable one: the grammar is
            # fine, your max_tokens budget is wrong. Do not burn a repair on it.
            raise RuntimeError(f"output truncated at max_tokens for schema={fp}")

        try:
            return Invoice.model_validate_json(raw)
        except ValidationError as exc:
            if attempt == max_repairs:
                raise RuntimeError(f"repair budget exhausted for schema={fp}") from exc
            # Reuse the same conversation prefix -> KV cache stays warm.
            messages.append({"role": "assistant", "content": raw})
            messages.append({"role": "user", "content": _repair_digest(exc)})

    raise AssertionError("unreachable")

The Function-Calling Loop: A Bounded State Machine

An agent loop is not a while True. It is a state machine with an explicit terminal set, a turn budget, and a deadline. The three failure modes that dominate production incidents are *unbounded iteration*, *orphaned tool calls*, and *history mutation that destroys the prefix cache*.

                    ┌──────────────────────────────────────────────────┐
                    │                                                  │
                    ▼                                                  │
             [BUILD PROMPT]                                            │
                    │                                                  │
                    ▼                                                  │
              [MODEL CALL] ──► check deadline / turn budget ──► ABORT   │
                    │                                                  │
        ┌───────────┴───────────┐                                      │
   finish=stop              finish=tool_calls                           │
        │                       │                                      │
        ▼                       ▼                                      │
 [VALIDATE FINAL]      [VALIDATE *ALL* ARGS FIRST]                      │
    │        │              │            │                              │
  valid    invalid        all ok     any invalid                        │
    │        │              │            │                              │
    ▼        ▼              ▼            ▼                              │
 [RETURN] [REPAIR]   [EXECUTE TOOLS]  [ONE tool msg PER CALL,           │
            │          (parallel)     carrying the schema error] ───────┤
            │              │                                           │
            │              ▼                                           │
            │      [APPEND tool messages,                               │
            └────► keyed by tool_call_id] ─────────────────────────────┘

Invariant 1 — exact `tool_call_id` pairing

Every assistant message containing tool_calls must be followed by exactly one role: "tool" message per tool_call_id, and the set of ids must match exactly. A missing id, a duplicated id, or an id that never appeared in the preceding assistant message is an immediate 400 invalid_request_error. This is the single most common bug in hand-rolled loops: developers write messages.push({ role: "tool", content: result }) and forget the id.

Note the protocol divergence: OpenAI-style APIs use one message per tool call; Anthropic-style APIs use content blocks (tool_use / tool_result) inside the assistant and user turns respectively. If you abstract over both, model the tool round as a *list of (id, name, args, result)* tuples and let each adapter serialize.

Invariant 2 — append-only history

Never mutate, reorder, deduplicate, or "compress" the message array mid-conversation. Self-hosted inference engines (vLLM, SGLang, TGI) reuse the KV cache via longest-common-prefix matching. Reordering two messages near the top of a 6,000-token history invalidates the entire prefix and turns a ~90 ms cached TTFT into a ~900 ms full prefill. Same for hosted APIs with context caching (Anthropic prompt caching, Gemini context caching): reordering silently resets the cache and burns your cache-read discount.

Invariant 3–5 — budgets, deadlines, idempotency

  • Turn budget: default 6 tool iterations. Real extraction/retrieval agents converge in 2–3. Loops that reach 6 are almost always the model retrying a tool because the result *looked* like a failure.
  • Wall clock: 90 s end-to-end. Track it with Date.now() deltas, not setTimeout, so a hung tool does not orphan the abort.
  • Idempotency: derive the key from hash(traceId, tool_call_id, tool_name) — never randomUUID() at call time. A retried turn must map to the same key or your PSP double-charges on the second attempt.

War story: the loop that called `get_transaction` 34 times

A payments agent in staging burned $180 in one afternoon. Root cause: get_transaction returned [] for an unknown id. The model read [] as "transient empty result, retry", and the loop had no turn budget. The fix was not a prompt change — it was a protocol change. Every tool result now returns a self-describing envelope:

{ "ok": true, "data": [], "hint": "no rows matched; this is a definitive result. Do not retry." }

Tool results that are ambiguous about *finality* are the #1 driver of infinite loops. Make finality explicit in the payload.

Self-Correcting Schema Repairs: The Escalation Ladder

Repair is a fallback, and it must be bounded with an explicit ladder. Each rung costs more and should fire less often:

| Rung | Mechanism | Expected hit rate | Cost per hit | Failure action | |---|---|---|---|---| | 0 | Constrained decoding only (grammar guarantees syntax) | ~99.1% served | 0 | escalate | | 1 | Runtime validator (Pydantic/Zod, cross-field invariants) | catches ~0.9% | ~0.2 ms CPU | escalate | | 2 | Repair prompt with JSON-Pointer error digest | recovers ~63% of rung-1 failures | ~1.4x tokens | escalate | | 3 | Second repair with temperature=0.3 (escape dead-ends) | recovers ~81% of remainder | ~2.6x tokens | escalate | | 4 | Deterministic coercion in code (clamp, drop, recompute totals) | recovers most residual | 0 tokens | escalate | | 5 | Fail closed → human review queue / DLQ | — | — | — |

Two implementation details carry most of the value:

Error digests, not tracebacks. Send {"pointer": "/lineItems/2/qty", "expected": "integer 1..999", "received": -4} — roughly 18 tokens per error, capped at 8 errors. A raw stack trace is 300–800 tokens and adds noise the model must filter.

Raise temperature on repair, never on the first attempt. A temperature=0 model that produced an invalid object is in a deterministic local optimum; re-running at temperature=0 reproduces the identical failure 100% of the time. Bumping to 0.3 on attempt 2 escapes the dead-end while keeping the output nearly deterministic. Use a *seeded* sampler if you need reproducibility for debugging.

Practical Implementation

A complete, bounded function-calling loop in TypeScript: one Zod schema serving as decode constraint, validator, and static type; per-call argument validation before any side effect; idempotent tool execution with timeouts and truncation; and a two-attempt repair budget.

import OpenAI from "openai";
import { zodResponseFormat } from "openai/helpers/zod";
import { z } from "zod";
import { createHash, randomUUID } from "node:crypto";

/* ── 0. Policy constants. Tune these, not the prompt. ────────────────── */
const POLICY = {
  model: "gpt-4o-2024-08-06",
  maxTurns: 6,
  wallClockMs: 90_000,
  maxRepairs: 2,
  toolTimeoutMs: 5_000,
  maxToolResultBytes: 32 * 1024,
} as const;

// We own the retry policy; the SDK must not retry under us (it would duplicate
// side effects and desynchronize the tool_call_id pairing).
const client = new OpenAI({ timeout: 45_000, maxRetries: 0 });

/* ── 1. One schema, three jobs: decode constraint, validator, TS type. ─ */
const Invoice = z
  .object({
    currency: z.enum(["USD", "EUR", "GBP"]),
    lineItems: z
      .array(
        z.object({
          sku: z.string().regex(/^[A-Z]{3}-\d{4}$/),
          qty: z.number().int().min(1).max(999),
          unitPriceCents: z.number().int().min(0),
        }),
      )
      .min(1)
      .max(50),
    totalCents: z.number().int().min(0),
  })
  // .refine() has NO JSON Schema representation: the grammar CANNOT enforce it.
  // It runs in the validator layer only — the semantic half of the contract.
  .refine(
    (inv) => inv.lineItems.reduce((s, li) => s + li.qty * li.unitPriceCents, 0) === inv.totalCents,
    { message: "totalCents must equal sum(qty * unitPriceCents)", path: ["totalCents"] },
  );

type Invoice = z.infer<typeof Invoice>;

/* ── 2. Tool registry. Zod args are the single source of truth. ──────── */
type Tool<S extends z.ZodTypeAny> = {
  description: string;
  args: S;
  sideEffect: boolean;
  run: (args: z.infer<S>, idempotencyKey: string) => Promise<unknown>;
};

const TOOLS = {
  get_exchange_rate: {
    description: "Look up the FX rate between two ISO-4217 currencies.",
    args: z.object({ from: z.string().length(3), to: z.string().length(3) }),
    sideEffect: false,
    run: async ({ from, to }) => fetchRate(from, to),
  },
  create_payment: {
    description: "Execute a payment. Idempotent on the derived key.",
    args: z.object({ invoiceId: z.string().uuid(), amountCents: z.number().int().positive() }),
    sideEffect: true,
    run: async (args, idempotencyKey) => psp.charge({ ...args, idempotencyKey }),
  },
} as const;

type ToolName = keyof typeof TOOLS;
type Msg = OpenAI.Chat.Completions.ChatCompletionMessageParam;
type ToolCall = OpenAI.Chat.Completions.ChatCompletionMessageToolCall;

function zodDigest(err: z.ZodError, limit = 8): string {
  return err.issues
    .slice(0, limit)
    .map((i) => `- /${i.path.join("/")}: ${i.message} (received ${JSON.stringify(i.input)})`)
    .join("\n");
}

/* ── 3. Validate EVERY argument set before executing ANY side effect. ── */
type Validated =
  | { ok: true; call: ToolCall; tool: (typeof TOOLS)[ToolName]; args: Record<string, unknown> }
  | { ok: false; call: ToolCall; error: string };

function validateCall(call: ToolCall): Validated {
  const tool = TOOLS[call.function.name as ToolName];
  if (!tool) return { ok: false, call, error: `Unknown tool "${call.function.name}".` };
  let raw: unknown;
  try {
    raw = JSON.parse(call.function.arguments || "{}");
  } catch (e) {
    return { ok: false, call, error: `arguments is not valid JSON: ${(e as Error).message}` };
  }
  const parsed = (tool.args as z.ZodTypeAny).safeParse(raw);
  return parsed.success
    ? { ok: true, call, tool, args: parsed.data as Record<string, unknown> }
    : { ok: false, call, error: zodDigest(parsed.error) };
}

/* ── 4. Idempotent, timeout-bounded execution with envelope results. ─── */
function envelope(payload: unknown): string {
  const body = JSON.stringify(payload);
  if (Buffer.byteLength(body) <= POLICY.maxToolResultBytes) return body;
  return JSON.stringify({
    ok: true,
    truncated: true,
    hint: "result exceeded size limit; narrow the query before retrying",
  });
}

async function executeTool(
  traceId: string,
  v: Extract<Validated, { ok: true }>,
): Promise<Msg> {
  // Derived from call identity, NOT generated fresh: a retried turn must map
  // to the same key or the payment provider double-charges.
  const idempotencyKey = createHash("sha256")
    .update(`${traceId}:${v.call.id}:${v.call.function.name}`)
    .digest("hex");

  let timer: NodeJS.Timeout | undefined;
  try {
    const timeout = new Promise<never>((_, reject) => {
      timer = setTimeout(() => reject(new Error("tool_timeout")), POLICY.toolTimeoutMs);
    });
    const result = await Promise.race([v.tool.run(v.args as never, idempotencyKey), timeout]);
    return { role: "tool", tool_call_id: v.call.id, content: envelope({ ok: true, data: result }) };
  } catch (err) {
    // Tool errors are DATA, not exceptions: the model must see them to adapt.
    return {
      role: "tool",
      tool_call_id: v.call.id,
      content: envelope({ ok: false, error: (err as Error).message, retryable: false }),
    };
  } finally {
    clearTimeout(timer);
  }
}

/* ── 5. The bounded loop. ────────────────────────────────────────────── */
export async function runInvoiceAgent(documentText: string) {
  const startedAt = Date.now();
  const traceId = randomUUID();
  const messages: Msg[] = [
    {
      role: "system",
      content:
        "Call a tool when you need external facts. When you have everything, " +
        "emit the final invoice object and nothing else. Never invent SKUs.",
    },
    { role: "user", content: `<document>\n${documentText}\n</document>` },
  ];

  let repairs = 0;

  for (let turn = 0; turn < POLICY.maxTurns; turn++) {
    if (Date.now() - startedAt > POLICY.wallClockMs) {
      throw new Error(`wall_clock_exceeded trace=${traceId} turn=${turn}`);
    }

    const res = await client.chat.completions.create({
      model: POLICY.model,
      messages,
      tools: Object.entries(TOOLS).map(([name, t]) => ({
        type: "function" as const,
        function: {
          name,
          description: t.description,
          parameters: z.toJSONSchema(t.args) as Record<string, unknown>,
          strict: true,
        },
      })),
      parallel_tool_calls: true,
      // Escape deterministic dead-ends only on repair attempts.
      temperature: repairs === 0 ? 0 : 0.3,
      response_format: zodResponseFormat(Invoice, "invoice"),
    });

    const choice = res.choices[0];
    const msg = choice.message;
    messages.push(msg); // append verbatim — never mutate history (KV prefix cache)

    /* Terminal: the model emitted the constrained final object. */
    if (choice.finish_reason === "stop" || !msg.tool_calls?.length) {
      const parsed = Invoice.safeParse(JSON.parse(msg.content ?? "null"));
      if (parsed.success) {
        return { invoice: parsed.data, turns: turn + 1, traceId };
      }
      if (repairs++ >= POLICY.maxRepairs) {
        throw new Error(`schema_repair_exhausted trace=${traceId} ${zodDigest(parsed.error, 3)}`);
      }
      messages.push({
        role: "user",
        content: `Your object failed validation. Fix ONLY these fields:\n${zodDigest(parsed.error)}`,
      });
      continue;
    }

    /* Tool phase: validate all, then execute all. */
    const validated = msg.tool_calls.map(validateCall);
    if (validated.some((v) => !v.ok)) {
      // Do not partially execute. Emit one tool message per call so the id
      // pairing invariant holds, then let the model repair its arguments.
      for (const v of validated) {
        messages.push(
          v.ok
            ? {
                role: "tool",
                tool_call_id: v.call.id,
                content: envelope({ ok: false, error: "skipped: sibling call had invalid arguments" }),
              }
            : { role: "tool", tool_call_id: v.call.id, content: envelope({ ok: false, error: v.error }) },
        );
      }
      if (repairs++ >= POLICY.maxRepairs) {
        throw new Error(`tool_args_repair_exhausted trace=${traceId}`);
      }
      continue;
    }

    const results = await Promise.all(validated.map((v) => executeTool(traceId, v)));
    messages.push(...results);
  }

  throw new Error(`turn_budget_exceeded trace=${traceId}`);
}

Performance Benchmarks & Trade-Offs

Representative figures from a 5,000-sample invoice-extraction harness. Model: Llama-3.1-70B-Instruct. Serving: vLLM 0.6.6, TP=2 × H100-80GB, continuous batching at 32 concurrency, warm prefix cache. "Semantic correctness" is judged by exact-match on all fields including the derived totalCents invariant.

| Strategy | Parseable JSON | Schema adherence | Semantic correctness | p95 added latency | Extra tokens/req | Ops complexity | |---|---|---|---|---|---|---| | Prompt-only ("return JSON") | 91.2% | 88.7% | 86.4% | +0 ms | +0 | Low | | Prompt + few-shot + 1 retry | 97.4% | 96.1% | 95.1% | +180 ms | +420 | Low | | Native JSON mode (no schema) | 99.9% | 88.0% | 86.9% | +6 ms | +12 | Low | | Native function calling, strict schema | 100% | 99.1% | 98.2% | +9 ms | +35 | Medium | | Constrained decoding (XGrammar) | 100% | 100% | 99.4% | +18 ms TTFT (warm) | +0 | High | | Constrained + Zod refine + ≤2 repairs | 100% | 100% | 99.91% | +240 ms (failure path only) | +150 (only on failure) | High |

Grammar backend selection is a separate axis, and the difference is not marginal:

| Engine | Grammar inputs | Mask build (µs/token) | First-compile p95 | Vocab strategy | Typical deployment | |---|---|---|---|---|---| | Outlines | JSON Schema, regex, EBNF | 30–140 | ~120 ms | Eager index-based FSM | Great DX; CPU-bound on 128k vocabs | | XGrammar | JSON Schema, EBNF/GBNF | ~1.1 | 8–40 ms | Adaptive cache; context-independent/ dependent split | Default in vLLM ≥ 0.6.4 | | llguidance | JSON Schema, Lark | ~1.5 | 5–25 ms | Byte-level Earley, incremental | llama.cpp server, OpenAI-compatible gateways | | llama.cpp GBNF | GBNF | 5–25 | 2–15 ms | Lazy, per-token PDA stack | Local/edge inference, battle-tested | | LMQL | Query language | 50–200 | n/a (interpreter) | Interpreter overhead per step | Research; not for p99 SLOs |

The decisive trade-off: constrained decoding buys you a 100% syntax guarantee and roughly +18 ms of warm TTFT for a 512 KB-per-state mask cache — but it costs you 1–3 pp on reasoning benchmarks if you apply it to the entire generation. Restrict the grammar to the final emission, not the chain-of-thought.

Pro-Tips & Common Pitfalls

  • Do not constrain chain-of-thought. Grammar-constrained reasoning tokens cannot emit connective phrases the model relies on for multi-step arithmetic. Two-phase generation (free-text reasoning → constrained final object) recovers the full benchmark score at the cost of one extra round trip.
  • `additionalProperties: false` or bust. Omitting it lets the model invent fields your validator silently strips. Silent data loss is worse than a hard failure.
  • Never use `oneOf` when `anyOf` works. oneOf requires mutually exclusive branches; the compiler must track branch commitment and reject ambiguity. In 95% of real schemas the branches are already disjoint.
  • Cap `max_tokens` from the schema, not from vibes. 50 items × ~30 tokens + preamble → 1024. Truncation is unrecoverable and burns a repair attempt for nothing.
  • Distinguish truncation from validation failure in your retry logic. finish_reason == "length" is a budget bug, not a schema bug. Do not repair it; alert on it.
  • Echo `tool_call_id` exactly. A single mismatched id is a 400. A missing tool message for a parallel call is a 400. Assert the pairing set-equality in a unit test.
  • Never regenerate the idempotency key on retry. randomUUID() at call time means your second attempt is a second charge. Derive from (traceId, tool_call_id, tool_name).
  • Truncate tool results at the boundary. A 1 MB JSON tool result will blow your context window and cost more tokens than the entire conversation. Return {ok: true, truncated: true, hint} instead.
  • Cache grammar compilation by schema fingerprint. First-compile is 8–40 ms; recompiling per request at 200 rps is 1.6–8 CPU-seconds per second, permanently.
  • Raise temperature on repair only. Re-running temperature=0 after a deterministic failure reproduces it exactly.
  • Do not re-parse with `JSON.parse` after validation. Validate once at the boundary and pass the typed object internally — parsing twice doubles the surface for drift.
  • Keep PII out of repair digests. Error messages that echo received <input> will log raw customer data. Redact the input field for PII-bearing paths.
  • Wire a circuit breaker to the repair rate. If repairs exceed 5% of requests over a 60 s window, the schema or the prompt changed under you — fail fast and page, don't scale retries.

Frequently Asked Questions (FAQ)

Does constrained decoding reduce reasoning quality?

Yes, measurably — the effect is called the *grammar tax*, typically 1–3 pp on multi-step reasoning benchmarks such as GSM8K when the grammar is applied to the entire generation. The mechanism is that the model's internal chain-of-thought uses token sequences (connectives, hedges, restatements) that a strict JSON grammar forbids, forcing it through a semantically lossy path. The production fix is two-phase generation: let the model reason in unconstrained text, then constrain only the final structured emission. You keep the 100% syntax guarantee and recover the full reasoning score at the cost of one extra round trip.

Should I use native function calling, JSON mode, or constrained decoding?

It depends on who owns the decoder and what your failure budget is. JSON mode guarantees only that output parses as JSON — it gives you no field-level guarantees, so expect ~88% schema adherence and never use it for money or PII. Native function calling with a strict schema (strict: true, additionalProperties: false) gives you ~99% adherence with near-zero added latency and no infrastructure, which is the right default for hosted APIs. Constrained decoding (XGrammar/llguidance) is the only option that mechanically guarantees 100% schema adherence, and it is the right choice when you self-host, when a single invalid object costs a retry plus a human, or when the schema is deeply nested and prompt-only adherence drops below 90%. Bill the extra ~18 ms TTFT and 512 KB of mask cache against the retries you no longer pay for.

How do I handle a model that keeps emitting invalid tool arguments?

First, decide whether the failure is *syntactic*, *schema-level*, or *semantic*. Syntactic failures (malformed JSON) mean you are not using constrained decoding and should be. Schema-level failures mean the argument schema is too permissive or too clever — flatten nested objects, remove oneOf, and add enum constraints instead of regex where possible. Semantic failures (valid-looking arguments that reference nonexistent entities) are not repairable by the model at all; the fix is to return a self-describing error envelope from the tool and make finality explicit ({"ok": false, "retryable": false, "hint": "invoiceId not found; do not retry"}). Then wrap the whole thing in a two-attempt repair budget with a circuit breaker on the repair rate. If you are still above a 5% repair rate after those three changes, the schema is wrong, not the model.