We tested every model we serve with a real strict json_schema request instead of trusting the catalogue tags. Twelve of them can call a function and still cannot hand you parseable JSON. Here is how to tell, and what to do about the ones that fail.
If you are parsing a model’s output, “supports structured output” is a claim worth verifying rather than reading. We verified it across 46 models and the suppliers’ own tags were wrong often enough to matter.
What we tested
For every model, a real request: response_format with a json_schema and strict: true, asking for
an object with three required keys. Then the only test that counts.
JSON.parse succeeds AND the required keys are present.
That bar is deliberately unforgiving, and specifically it fails JSON wrapped in markdown code fences. Fenced output looks correct in a terminal and throws in your code, and parsing the content is the entire reason you asked for JSON in the first place.
What we found
Of the 46 models we serve, 36 do tool calling and 28 do structured output. So twelve models can call a function and cannot reliably return parseable JSON, which is not the pairing most people assume when they see a “supports tools” badge.
The instructive failure was llama-3.3-70b. Its supplier’s catalogue tags it for both
structured-output and json. It ignores response_format outright: four consecutive strict: true
requests came back as conversational prose. Asked for json_object mode instead, it returned JSON
wrapped in fences, which JSON.parse rejects.
Our registry therefore declares only ["tools"] for that model. If you built against the supplier’s
catalogue you would ship code that parses its output and passes your tests, because it does sometimes
comply.
How to build against this
Prefer a hard failure to a soft one. Ask us for a capability a model does not declare and you get a
400 naming the parameter, before any paid work happens. That is deliberate. A 400 is a five-minute fix;
silently dropping the parameter gives you a 200, a paragraph of prose, and a bug you chase for a day.
We shipped the silent version by accident once and that is precisely how it went: callers sending
tools got prose, no tool call and no error, and it read exactly like a model deciding not to use the
tool.
Validate every parse anyway, even on a model that passed. A model that complies 99% of the time will hand you prose under an unusual prompt or during a bad hour upstream. Wrap the parse, and on failure retry once with a stricter instruction before you surface an error.
Keep the schema flat and small. Deeply nested schemas with many optional fields failed far more often than flat ones with three or four required keys, on every model we tested. If you need a complex object, two simple calls are usually more reliable than one complex one.
Strip fences defensively if you must use a model that does not declare support. Three lines that
remove a leading json and a trailing recover most of those failures. It is a patch rather than a
fix, and it is worth having if that model is otherwise right for your task.
Do not describe the schema in the prompt as well. If the model honours response_format, restating
the shape in prose is input tokens you pay for on every request for no extra compliance.
Check it yourself before you commit
GET /v1/models is public, needs no key, and every entry carries the capabilities we measured rather
than the ones the supplier advertises:
curl https://api.dawnstackai.com/v1/modelsFilter on structured-output there, or use the JSON output filter on the
models page. An unmeasured capability counts as absent in our registry, so the worst case is
a clear error rather than a plausible wrong answer.