Function Calling and Structured Outputs with LLMs in Production
By: Md Tarek · 19 Oct 2026
Executive Summary & Key Takeaways
Function calling and structured outputs with LLMs turn probabilistic text into typed contracts, but only when you separate three layers: token-level constrained decoding, JSON Schema validation, and orchestrator-side function calling loops. In production, this split reduces invalid JSON from 8-18% on prompt-only extraction to 0.02-0.1% with grammar masks and 0.005-0.02% after a bounded repair pass.
- Constrained decoding is structural, not semantic: grammar/FSM/PDA token masks eliminate malformed JSON and most schema-shape errors, but they cannot enforce business invariants such as 'refund amount <= captured amount'. Validate every payload with Draft 2020-12 JSON Schema or Pydantic v2 before execution.
- Function calling loops need hard budgets: cap iterations at 4-6, validate tool arguments before side effects, make every mutating tool idempotent with a deterministic key, and fail closed after 1-2 schema repairs. Unbounded repair loops commonly add 1.5-3x tokens and 700-1,500 ms p99 on failed turns.
- Parallel tool calls are a correctness risk: two refund or booking calls can execute concurrently; use an idempotency key derived from tool name + canonical JSON args, plus per-tool timeouts and a circuit breaker. A single duplicate refund incident is more expensive than any latency saving.
- Measure at the contract boundary: track parse failures, schema violations, repair attempts, tool error rate, and p95/p99 loop latency. The best stack is usually structured outputs for final responses + tool schemas with strict validation + one repair attempt only on validation failure.
Constrained Decoding: From JSON Schema to Token Masks
Constrained decoding, also called guided decoding or grammar-constrained generation, modifies the next-token distribution at every step. Given logits z and a grammar state s, the engine builds an allowed token set A(s), applies a mask M_i = 0 for tokens in A(s) and -inf otherwise, then samples from softmax(z + M). The grammar state advances only when the emitted token string is accepted by the incremental parser.
User request | v JSON Schema --compile--> CFG/GBNF/PDA --tokenizer align--> mask cache | | v v LLM logits --mask(M)--> token --advance state--> repeat until accept
JSON Schema is a validation language, not a generation grammar. Engines such as Outlines, XGrammar, llguidance, llama.cpp GBNF, vLLM guided_json, SGLang, and TensorRT-LLM compile the schema into a context-free grammar or pushdown automaton. Regular JSON shapes can be represented with an FSM, but recursive $ref, nested arrays, and object cardinality typically require a CFG/PDA. The hard part is tokenizer alignment: the decoder must test whether a vocabulary token's full string is a legal prefix continuation of the current parser state, then precompute token masks keyed by schema hash + tokenizer version + engine version.
Production war story: a team enabled grammar-constrained extraction on a Kubernetes deployment without pre-warming the grammar cache. Cold pods paid 180-320 ms compile time on the first request, producing a p99 spike that looked like model latency. The fix was a startup pre-warm for top 20 schemas and a shared cache keyed by sha256(schema) + tokenizer_name + engine_build.
Constrained decoding guarantees syntactic validity, not truth. It will happily generate a valid integer that violates your business rules. Treat it as a fast rejection layer, then validate with JSON Schema and domain rules.
Function Calling Loops: Tool Routing, Parallelism, and Idempotency
A function calling loop is a state machine around the model, not a single API call. The model emits tool_calls with a name and JSON arguments; your orchestrator parses, validates, executes, and appends tool messages keyed by tool_call_id. The loop ends when the model returns a final assistant message or the step budget is exhausted.
assistant -> tool_calls[{id,name,arguments}]
orchestrator -> validate args -> execute -> tool[{tool_call_id,content}]
assistant -> final answer OR more tool_callsUse strict tool schemas, allowlisted tool names, per-tool timeouts, and structured error results. Never interpolate tool arguments into shell commands or SQL. If the model requests issue_refund and send_email in parallel, validate both independently and execute only after the whole batch passes policy checks.
Production war story: an airline booking agent retried a parallel tool call after a 2 s timeout. The first call had actually succeeded, so the retry created a duplicate seat hold. The fix was a deterministic idempotency key: tool_name + sha256(canonical_json(args)), stored for 24 hours. The model can retry safely; the side effect executes once.
Self-Correcting Schema Repairs Without Infinite Loops
Self-correction should be a bounded, validation-triggered repair, not a default generation strategy. Classify errors into parse errors, schema violations, semantic/business-rule violations, and tool execution errors. Repair only parse and schema errors; business-rule failures need a deterministic policy response, not another model guess.
raw JSON -> parse -> validate -> pass -> execute
| fail | fail
v v
compact error paths -> repair prompt (temperature 0)
| |
+---- max 2 --------+Send the model compact JSON Pointer paths and messages, cap the repair input to the failing payload, and instruct it to fix only the listed errors without adding facts. A single repair pass is often worth 700-1,200 ms because it converts a hard failure into a valid tool call; two passes are the practical ceiling before you should fail closed and ask the user for clarification.
Production war story: an enum field expected damaged, late, or wrong_item, but the model emitted broken. A one-shot repair with the exact allowed enum values fixed 92% of residual failures. A second pass added little and increased token spend by 1.8x, so the team capped repairs at one for this tool.
Practical Implementation
from __future__ import annotations
import hashlib
import json
import logging
import uuid
from dataclasses import dataclass
from typing import Any, Callable, Literal
from jsonschema import Draft202012Validator
from openai import OpenAI
from pydantic import BaseModel, ConfigDict, Field
logger = logging.getLogger(__name__)
class RefundArgs(BaseModel):
model_config = ConfigDict(extra='forbid')
order_id: str = Field(..., min_length=6)
amount_cents: int = Field(..., gt=0, le=100_000)
reason: Literal['damaged', 'late', 'wrong_item']
@dataclass(frozen=True)
class Tool:
name: str
description: str
schema: dict[str, Any]
handler: Callable[[dict[str, Any]], dict[str, Any]]
idempotent: bool = True
_IDEMPOTENCY: dict[str, dict[str, Any]] = {}
def validate_args(schema: dict[str, Any], args: dict[str, Any]) -> list[str]:
validator = Draft202012Validator(schema)
out: list[str] = []
for e in sorted(validator.iter_errors(args), key=lambda x: list(x.path)):
path = '/'.join(map(str, e.path)) or '$'
out.append(f'{path}: {e.message}')
return out
def repair_json(
client: OpenAI,
model: str,
raw: str,
schema: dict[str, Any],
errors: list[str],
max_attempts: int = 2,
) -> dict[str, Any]:
current = raw
last_errors = errors
for attempt in range(max_attempts + 1):
try:
candidate = json.loads(current)
except json.JSONDecodeError as exc:
last_errors = [f'parse error: {exc.msg} at line {exc.lineno} col {exc.colno}']
else:
errs = validate_args(schema, candidate)
if not errs:
return candidate
last_errors = errs
if attempt >= max_attempts:
break
system = (
'Repair the JSON to satisfy the schema. Return JSON only. '
'Do not add new facts. Fix only these validation errors: '
+ json.dumps(last_errors[:8])
)
resp = client.chat.completions.create(
model=model,
temperature=0,
response_format={'type': 'json_object'},
messages=[
{'role': 'system', 'content': system},
{'role': 'user', 'content': current[:12000]},
],
)
current = resp.choices[0].message.content or '{}'
raise ValueError(f'Could not repair JSON after {max_attempts} attempts: {last_errors}')
def refund_handler(args: dict[str, Any]) -> dict[str, Any]:
return {
'refund_id': f'rf_{uuid.uuid4().hex[:12]}',
'status': 'accepted',
'amount_cents': args['amount_cents'],
}
TOOLS: dict[str, Tool] = {
'issue_refund': Tool(
name='issue_refund',
description='Issue a refund for a delivered order.',
schema=RefundArgs.model_json_schema(),
handler=refund_handler,
idempotent=True,
)
}
def execute_tool(tool: Tool, args: dict[str, Any]) -> dict[str, Any]:
key = f'{tool.name}:{hashlib.sha256(json.dumps(args, sort_keys=True).encode()).hexdigest()}'
if tool.idempotent and key in _IDEMPOTENCY:
return _IDEMPOTENCY[key]
result = tool.handler(args)
if tool.idempotent:
_IDEMPOTENCY[key] = result
return result
def run_tool_loop(
client: OpenAI,
model: str,
messages: list[dict[str, Any]],
max_steps: int = 6,
) -> str:
for _ in range(max_steps):
resp = client.chat.completions.create(
model=model,
temperature=0,
messages=messages,
tools=[
{
'type': 'function',
'function': {
'name': tool.name,
'description': tool.description,
'parameters': tool.schema,
'strict': True,
},
}
for tool in TOOLS.values()
],
tool_choice='auto',
parallel_tool_calls=True,
)
msg = resp.choices[0].message
messages.append(msg.model_dump(exclude_none=True))
if not msg.tool_calls:
return msg.content or ''
for call in msg.tool_calls:
tool = TOOLS.get(call.function.name)
if tool is None:
messages.append({
'role': 'tool',
'tool_call_id': call.id,
'content': json.dumps({'error': f'unknown tool {call.function.name}'}),
})
continue
try:
args = json.loads(call.function.arguments)
except json.JSONDecodeError:
args = {}
errors = validate_args(tool.schema, args)
if errors:
try:
args = repair_json(client, model, call.function.arguments, tool.schema, errors)
except ValueError as exc:
messages.append({
'role': 'tool',
'tool_call_id': call.id,
'content': json.dumps({'error': str(exc)}),
})
continue
try:
result = execute_tool(tool, args)
messages.append({
'role': 'tool',
'tool_call_id': call.id,
'content': json.dumps(result),
})
except Exception as exc:
logger.exception('tool execution failed')
messages.append({
'role': 'tool',
'tool_call_id': call.id,
'content': json.dumps({'error': type(exc).__name__}),
})
raise RuntimeError('tool loop exceeded max steps')
if __name__ == '__main__':
client = OpenAI()
answer = run_tool_loop(
client,
'gpt-4.1',
[{'role': 'user', 'content': 'Refund order ORD-123456 for 4250 cents because it was damaged.'}],
)
print(answer)This sample is synchronous for clarity. In production, use the async client, execute independent tool calls with a bounded thread pool or task group, wrap handlers in per-tool timeouts, and persist idempotency keys in Redis or your database rather than process memory.
Performance Benchmarks & Trade-Offs
| Strategy | Invalid JSON / parse failures | Semantic/schema violations | Added p50 latency | Added p99 latency | Throughput impact | Best fit | |---|---:|---:|---:|---:|---:|---| | Prompt-only + regex | 8-18% | 3-7% | 0 ms | 0 ms | baseline | prototypes | | JSON mode | 1-4% | 1-3% | +5 ms | +15 ms | <1% | low-risk JSON | | JSON Schema structured output | 0.1-0.5% | 0.05-0.2% | +8 ms | +25 ms | 1-3% | API contracts | | Grammar/constrained decoding | 0.02-0.1% | 0.02-0.1% | +10-30 ms | +40-90 ms | 2-8% | local/self-hosted | | Function calling + validation | 0.05-0.2% | 0.01-0.05% | +12 ms | +35 ms | 1-3% | tool orchestration | | + 1 schema repair | 0.005-0.02% | 0.001-0.01% | +12 ms normal; +700 ms failure | +1.5 s failure | failure-only | high-stakes |
Numbers are illustrative production ranges; measure against your tokenizer, model, hardware, and schema complexity. Grammar compilation and token-mask precomputation can dominate p99 on cold starts, while repair loops dominate p99 on invalid turns.
Pro-Tips & Common Pitfalls
- [ ] Cache compiled grammars by schema hash + tokenizer version + engine build; pre-warm top schemas at deploy time.
- [ ] Use
additionalProperties: falseand explicitrequiredfields; avoid permissive schemas that let the model invent arguments. - [ ] Validate tool arguments before any side effect, then validate business rules inside the tool handler.
- [ ] Derive idempotency keys from tool name + canonical JSON args; do not rely on model-provided request IDs.
- [ ] Cap function calling steps at 4-6 and repair attempts at 1-2; fail closed with a user-facing clarification path.
- [ ] Treat
format: email,pattern,uniqueItems, and numeric ranges as validation concerns unless your constrained decoder explicitly enforces them. - [ ] Log parse errors, JSON Pointer error paths, repair prompts, tool durations, and duplicate-suppression hits with trace IDs.
- [ ] Never execute shell, SQL, or HTTP requests by string interpolation from model output; use typed clients and allowlists.
- [ ] Avoid free-form 'try again' loops; they hide contract bugs and inflate token spend.
- [ ] Version schemas and tool definitions; a model pinned to an old schema is a production incident waiting to happen.
Frequently Asked Questions (FAQ)
Does constrained decoding guarantee semantically correct JSON?
No. Constrained decoding guarantees that sampled tokens follow a grammar or JSON Schema shape. It does not guarantee that values are true, authorized, or consistent with business rules. Use it to eliminate parse failures and schema-shape errors, then run JSON Schema validation with Draft 2020-12 or Pydantic v2 plus domain checks before executing side effects.
How many self-correction retries should a function calling loop allow?
One repair attempt is usually the best cost/quality trade-off; two is a hard ceiling for high-stakes workflows. Track repair success rate by error class. If repairs rarely convert failures after one pass, fail closed and ask for clarification instead of spending another 700-1,200 ms and 1.5-3x tokens. Never allow unbounded repair loops.
Can function calling and JSON Schema structured outputs be combined?
Yes, and they should be. Define each tool with a strict JSON Schema, validate returned tool_calls[].function.arguments before execution, and use a separate response_format JSON Schema for final assistant output when you need a typed API response. The orchestrator owns validation, idempotency, timeouts, and policy checks; the model only proposes structured actions.
