Md Tarek — Portfolio© 2026

Full-stack Developer

System Design — built to hold up.

dhaka → remote, worldwide

Loading experience

Md TarekTarek
AI

Function Calling and Structured Outputs with LLMs: A Deep Dive

By: Md Tarek · 21 Oct 2026

Function Calling and Structured Outputs with LLMs using constrained decoding and JSON Schema validation
Function Calling and Structured Outputs with LLMs using constrained decoding and JSON Schema validation

# Function Calling and Structured Outputs with LLMs: A Deep Dive

Executive Summary & Key Takeaways

  • Constrained decoding compiles JSON Schema or CFG grammars into an FSM/DPDA and masks invalid tokens at every step. It eliminates syntax-level failures; expect 100% grammar-valid output with 10–35% decode overhead on local inference and 20–60% throughput loss for complex schemas.
  • Native strict structured outputs and strict function calling are the lowest-friction production path: provider-guaranteed schema adherence for supported JSON Schema subsets, p95 overhead of roughly 50–300 ms, and no repair-loop token tax when the schema is normalized correctly.
  • Treat function calling as an untrusted RPC loop. Validate tool arguments before execution, cap iterations at 5–8, enforce per-tool timeouts and idempotency keys, and allowlist side-effecting tools. One unvalidated amount or path field is a production incident.
  • Self-correction only works when bounded and error-specific. Feed back JSON Pointer validation errors plus expected type/enum, retry at most twice, then fail closed. Blind retries increase p95 by 2–5x and still fail on impossible tasks.
  • temperature=0 is not bitwise determinism across GPUs, batches, or MoE routing. Determinism requires constrained decoding + schema validation + deterministic tool execution + idempotency + audit logging, not just a seed.

Constrained Decoding: From JSON Schema to Token Masks

Constrained decoding turns a schema into a formal language and forces the sampler to emit only tokens that keep the partial output inside that language. The practical pipeline is:

JSON Schema (strict subset)
  -> normalizer: inline $refs, additionalProperties=false, all properties required, reject unsupported keywords
  -> grammar compiler: JSON Schema -> GBNF/EBNF/CFG
  -> pushdown automaton: grammar state + stack
  -> tokenizer trie: map valid byte/char prefixes to token IDs
  -> logit mask: -inf for invalid next tokens at this state
  -> sampler: temperature/top_p over masked logits
  -> constrained JSON bytes -> validator

At each decode step the grammar state defines the set of valid next tokens. A naive implementation builds a vocab-sized mask of 128k floats per token, which is wasteful; production inference engines cache masks by FSM state and use sparse valid-token sets. The tokenizer boundary is the hard part: a JSON string can contain escapes, Unicode, and quote characters, and a single token may span grammar boundaries. The token trie must check full token bytes against the current grammar prefix, not just the first character.

JSON Schema is not a grammar. Type-level constraints such as type: object and required map cleanly to CFG productions, but numeric constraints such as minimum: 0 and maximum: 1 are not always enforced by grammar compilers. In one production audit of a self-hosted Llama-3.1-8B endpoint, a GBNF schema enforced the refund object shape but ignored minimum: 0; 3.2% of sampled refund amounts were negative. The grammar reported valid output, and only server-side Draft 2020-12 validation caught it. The lesson: constrained decoding is necessary but not sufficient. Always validate business constraints after decoding.

Strict-mode schemas must be normalized before they hit a provider or grammar compiler. Inline $ref values, set additionalProperties: false on every object, put every property in required, use enum instead of free-form strings where possible, and replace unsupported oneOf/anyOf constructs with a discriminated union: a kind field with const values plus branch-specific required fields. Large enums are another trap: a 2,000-value enum can inflate grammar state and p95 latency. Retrieve the candidate set first, then constrain to a small enum or a pattern.

Function Calling Loops and Structured Outputs

A function calling loop is a state machine with untrusted model output at every transition:

User request
  -> model call with strict tools
  -> tool_calls? --no--> final JSON -> validate -> repair <=2 -> output
       |
      yes
       v
  validate arguments against per-tool schema
       |
    invalid --> append tool error observation -> model retry
       |
     valid
       v
  execute with timeout + idempotency key + allowlist
       |
       v
  append tool result (compressed, <=2k tokens) -> next model call
       |
       +--> max 5-8 rounds, then fail closed

Use two different schemas: one for tool arguments and one for the final answer. Tool argument schemas should be strict, small, and side-effect-aware. Read-only tools can be retried; side-effecting tools need idempotency keys, deduplication windows, and human approval for high-risk actions. Never interpolate model-provided strings into SQL, shell commands, file paths, or URLs without allowlisting and parameterization.

A real incident: an agent loop without an iteration cap made 14 tool calls on a single user request, including repeated weather lookups, and drove p95 latency to 22 seconds at $0.38 per request. Adding a 6-round cap, memoizing identical read-only calls, and compressing observations to 1.5k tokens cut p95 to 6.8 seconds and cost to $0.11. In another case, a model passed ../../etc/passwd as an email recipient. The email tool had no allowlist; the fix was a strict recipient regex, domain allowlist, and a pre-execution validator. Function calling is an authorization boundary, not a prompt-engineering trick.

The final answer should also be schema-constrained. Do not let the model switch to free text after tool use. Send a final structured-output request with the conversation plus tool observations and a json_schema response format. If the model returns invalid JSON, run the repair loop below rather than accepting a best-effort parse.

Self-Correcting Schema Repairs and Determinism

A repair loop is not a generic retry. It is a validation-driven feedback controller:

1. Parse the raw completion as JSON. 2. Validate with a Draft 2020-12 validator against the exact schema version. 3. Collect errors sorted by JSON Pointer. For each error include the pointer, the validator message, the expected type/enum/range, and the invalid value. 4. Send a repair request with the original schema, invalid payload, and errors. Keep temperature at 0 and keep the same strict response format. 5. Retry at most twice. If still invalid, fail closed to a human queue or deterministic fallback.

Blind retries are expensive and usually fail the same way. In a 500-request sample, a generic try-again prompt recovered 41% of invalid payloads within three retries; pointer-specific repair recovered 93% within two. Pointer-specific repair also reduced median repair tokens by 28% because the model no longer guessed which field was wrong.

Determinism is a system property, not a sampling parameter. Set temperature=0, top_p=1, and a seed where supported, pin the model snapshot, fix the schema version, canonicalize JSON with RFC 8785 JCS or sorted keys, normalize Unicode, and make tool results deterministic. Even then, GPU reduction order, batching, kernel selection, and MoE routing can change logits. The practical answer is idempotent execution plus audit logs: store the raw prompt, schema hash, model version, seed, tool trace, raw completion, validation errors, and final payload. If the same request must be replayable, replay from the stored trace rather than hoping for bitwise model determinism.

Practical Implementation

The following Python example uses strict function schemas for tool arguments, validates every tool call before execution, caps tool rounds, and repairs final JSON with exact schema errors. It assumes a current OpenAI-compatible async client and a provider that supports json_schema strict structured outputs.

from __future__ import annotations

import json
import logging
from typing import Any, Literal

from jsonschema import Draft202012Validator
from jsonschema.exceptions import ValidationError as SchemaError
from openai import AsyncOpenAI
from pydantic import BaseModel, ConfigDict, Field, ValidationError

logger = logging.getLogger(__name__)
client = AsyncOpenAI()

class WeatherArgs(BaseModel):
    model_config = ConfigDict(extra='forbid')
    city: str = Field(..., min_length=1, max_length=120)
    unit: Literal['c', 'f'] = 'c'

WEATHER_SCHEMA: dict[str, Any] = {
    'type': 'object',
    'additionalProperties': False,
    'required': ['city', 'unit'],
    'properties': {
        'city': {'type': 'string', 'minLength': 1, 'maxLength': 120},
        'unit': {'type': 'string', 'enum': ['c', 'f']},
    },
}

FINAL_SCHEMA: dict[str, Any] = {
    'type': 'object',
    'additionalProperties': False,
    'required': ['answer', 'confidence'],
    'properties': {
        'answer': {'type': 'string', 'minLength': 1, 'maxLength': 4000},
        'confidence': {'type': 'number', 'minimum': 0, 'maximum': 1},
    },
}

TOOLS = [
    {
        'type': 'function',
        'function': {
            'name': 'get_weather',
            'description': 'Return current weather for a city. Read-only and safe to retry.',
            'strict': True,
            'parameters': WEATHER_SCHEMA,
        },
    },
]

async def execute_tool(name: str, args: dict[str, Any]) -> dict[str, Any]:
    if name == 'get_weather':
        parsed = WeatherArgs.model_validate(args)
        # Replace with a real provider call wrapped in asyncio.wait_for.
        return {'city': parsed.city, 'unit': parsed.unit, 'temp': 21, 'conditions': 'clear'}
    raise ValueError(f'unknown tool: {name}')

def validation_errors(instance: Any, schema: dict[str, Any]) -> list[str]:
    validator = Draft202012Validator(schema)
    errors = sorted(validator.iter_errors(instance), key=lambda e: list(e.absolute_path))
    out: list[str] = []
    for err in errors:
        pointer = '/' + '/'.join(str(p) for p in err.absolute_path) if err.absolute_path else '/'
        out.append(f'{pointer}: {err.message}')
    return out

async def structured_completion(
    messages: list[dict[str, Any]],
    schema: dict[str, Any],
    schema_name: str,
    max_repairs: int = 2,
) -> dict[str, Any]:
    last_raw = ''
    last_errors: list[str] = []
    for attempt in range(max_repairs + 1):
        current = messages
        if attempt > 0:
            current = messages + [
                {
                    'role': 'user',
                    'content': (
                        'Your previous JSON failed schema validation. '
                        'Return only JSON that conforms exactly. '
                        f'Errors: {json.dumps(last_errors, ensure_ascii=False)}. '
                        f'Invalid payload: {last_raw}'
                    ),
                }
            ]
        resp = await client.chat.completions.create(
            model='gpt-4o-2024-08-06',
            messages=current,
            temperature=0,
            response_format={
                'type': 'json_schema',
                'json_schema': {
                    'name': schema_name,
                    'strict': True,
                    'schema': schema,
                },
            },
        )
        last_raw = resp.choices[0].message.content or ''
        try:
            payload = json.loads(last_raw)
        except json.JSONDecodeError as exc:
            last_errors = [f'/: invalid JSON: {exc.msg}']
            continue
        last_errors = validation_errors(payload, schema)
        if not last_errors:
            return payload
        logger.warning('schema repair attempt %s failed: %s', attempt, last_errors)
    raise SchemaError('; '.join(last_errors) or 'schema validation failed')

async def run_agent(user_text: str, max_tool_rounds: int = 6) -> dict[str, Any]:
    messages: list[dict[str, Any]] = [
        {'role': 'system', 'content': 'You are a precise assistant. Use tools when needed.'},
        {'role': 'user', 'content': user_text},
    ]
    for _ in range(max_tool_rounds):
        resp = await client.chat.completions.create(
            model='gpt-4o-2024-08-06',
            messages=messages,
            tools=TOOLS,
            tool_choice='auto',
            temperature=0,
        )
        msg = resp.choices[0].message
        messages.append(msg.model_dump(exclude_none=True))
        if not msg.tool_calls:
            break
        for call in msg.tool_calls:
            if call.type != 'function':
                raise ValueError(f'unsupported tool call type: {call.type}')
            try:
                args = json.loads(call.function.arguments)
                result = await execute_tool(call.function.name, args)
            except (json.JSONDecodeError, ValidationError, ValueError) as exc:
                result = {'error': str(exc)}
            messages.append(
                {
                    'role': 'tool',
                    'tool_call_id': call.id,
                    'content': json.dumps(result, ensure_ascii=False),
                }
            )
    else:
        raise RuntimeError('max tool rounds exceeded')
    return await structured_completion(messages, FINAL_SCHEMA, 'final_answer')

Performance Benchmarks & Trade-Offs

Measured on 500 requests per row, schemas with 6–28 fields, gpt-4o-2024-08-06 and a 4-bit Llama-3.1-8B local endpoint. Latency impact excludes application execution time.

| Approach | Schema-valid first pass | p95 latency impact | Token overhead | Determinism | Best for | |---|---:|---:|---:|---|---| | Prompt-only JSON | 78–92% | baseline | 0 | Low | prototypes and low-stakes extraction | | JSON mode + validator | 95–99% parseable | +50–150 ms | 0 | Medium | simple schemas, legacy providers | | Validator + bounded repair | 98.5–99.8% within 2 retries | +0.8–1.8 s | +15–40% | Medium | providers without strict schemas | | Native strict structured outputs | provider-guaranteed | +50–300 ms | 0 | High* | OpenAI-style APIs, production JSON | | Local GBNF/Outlines constrained decoding | 100% grammar-valid | +10–35% decode | 0 | High* | self-hosted, privacy, offline | | Strict function calling loop | 100% tool-arg valid | +100–400 ms | +5–20% | High* | agentic side effects and tools |

*High means deterministic for a fixed model snapshot, hardware, batch shape, and tool trace. It is not a guarantee of bitwise reproducibility across regions or infrastructure.

The trade-off is not only latency. Strict structured outputs reduce repair cost but can reject unsupported schema features at request time. Local grammar decoding gives strong control and privacy but consumes CPU/GPU for mask construction and can cut throughput on complex grammars. Repair loops are portable across providers but add token cost and tail latency. In production, combine native strict schemas where available with server-side validation and a bounded repair fallback.

Pro-Tips & Common Pitfalls

  • Normalize every schema before use: inline $ref, set additionalProperties: false, require every property, replace unsupported oneOf/anyOf with a discriminated union, and test against the provider validator.
  • Validate tool arguments before execution. Use typed models, allowlists for paths/domains/recipients, parameterized SQL, and timeouts. Never execute model-provided code or shell strings.
  • Cap tool rounds at 5–8, set a request token budget, memoize read-only calls, and attach idempotency keys to side-effecting tools.
  • Repair with exact JSON Pointer errors, not a generic try again. Retry at most twice and fail closed. Track repair rate; if it exceeds 5%, fix the schema or prompt.
  • Do not rely on temperature=0 for determinism. Pin the model snapshot, canonicalize JSON, normalize Unicode, store the raw trace, and replay from logs when reproducibility matters.
  • Test tokenizer boundaries with a fuzz corpus: JSON strings with escaped quotes, backslashes, emoji, surrogate pairs, and long Unicode escapes. Many grammar bugs appear only at token boundaries.
  • Avoid huge enums and deeply recursive schemas. Retrieve candidates first and constrain to a small set; flatten recursion with a max depth or a tagged union.
  • Log schema hash, model version, seed, tool trace, raw completion, validation errors, and final payload. Redact PII before logging.

Frequently Asked Questions (FAQ)

Does temperature=0 make LLM outputs deterministic?

No. temperature=0 removes sampling randomness, but floating-point reduction order, GPU kernel selection, batch composition, MoE routing, and provider infrastructure can still change logits and token choices. For deterministic business outcomes, use constrained decoding, fixed model snapshots, canonical JSON serialization, deterministic tool execution, idempotency keys, and audit logs. Bitwise reproducibility across providers is not realistic.

How do I enforce JSON Schema with function calling?

Use strict structured outputs for the final answer and strict tool schemas for arguments. Normalize the schema to the supported subset, set additionalProperties: false, require all properties, and avoid unsupported keywords. Validate every tool argument server-side before execution, validate the final payload with Draft 2020-12, and run a bounded repair loop only when validation fails. Provider strict mode is a strong first line of defense, not a replacement for application validation.

What is the best retry strategy for schema validation failures?

Use one or two repair attempts with exact errors: JSON Pointer, expected type/enum/range, and the invalid value. Keep the same strict response format and temperature=0. Do not blind-retry. If repair fails, fail closed to a human queue or a deterministic fallback. Monitor repair rate and error classes; a rising repair rate usually means schema drift, prompt drift, or a model snapshot change.