Structured output: JSON by a schema

Code in:JavaC#PythonJavaScript

In short: Structured output is a model answer as JSON that follows a schema with types and allowed values, which the program checks and, on an error, sends back to the model to fix.

Problem

The code that expects JSON

The support inbox gets hundreds of emails a day. A program sorts them into queues: deliveries, returns, payments, general. It asks a model for the email’s topic and expects JSON with the field intent. Here are three answers of the model for the same email:

List<String> answers = List.of(
        """
        Sure! Here is the data:
        ```json
        {"intent": "delivery", "order_id": "1042", "urgent": true}
        ```""",
        """
        {"intent": "shipping", "order_id": "1042", "urgent": "yes", "summary": "The courier did not come."}""",
        """
        {"intent": "delivery", "order_id": "1042", "urgent": true, "summary": "The courier did not come, the customer leaves tomorrow."}""");
for (String answer : answers) {
    Map<String, Object> json = parseJson(answer);
    if (json == null) {
        System.out.println("not JSON: " + answer.lines().findFirst().get());
        continue;
    }
    String queue = QUEUES.get(json.get("intent"));
    System.out.println(queue == null
            ? "no queue for \"" + json.get("intent") + "\""
            : "queue: " + queue);
}

Output

not JSON: Sure! Here is the data:
no queue for "shipping"
queue: Deliveries

Only the third one works. The first is polite, but it is text around JSON and the program cannot read it. The second is JSON, but the topic is a word the program does not know and there is no such queue. In the lesson about the prompt you saw that examples help. But the program has to work on the night the model decides to be polite.

The idea

A schema, a check, a second attempt

When the answer goes into code and not to a person, it has a contract. The contract is a schema: which fields there are, the type of each and which values are allowed.

1. The prompt states the schema. The fields, the types and the allowed values are in the instruction.
2. The program reads the JSON. If it is not JSON, that is an error, not a crash.
3. The program checks it against the schema: is every field there, is the type right, is the value allowed, are there no extra fields.
4. The errors go back to the model as a new message, and it tries again.
5. There are at most three attempts. Then the email goes to a person.

An answer by a schema that the program checks is structured output.

The model’s output is untrusted input like everything that comes from outside: the program does not trust that it keeps the contract, it checks.

See it

Reading and checking

The email’s schema: four fields, intent with four allowed values:

static final List<Field> TICKET = List.of(
        new Field("intent", "string", List.of("delivery", "return", "payment", "other")),
        new Field("order_id", "string", List.of()),
        new Field("urgent", "boolean", List.of()),
        new Field("summary", "string", List.of()));

Reading returns the object, or null if the text is not exactly one JSON object:

static Map<String, Object> parseJson(String text) {
    return FlatJson.parse(text);
}

Java has no JSON in its standard library. A project uses Jackson or Gson. Here parseJson calls the small reader FlatJson from “More detail”: it understands a flat object with text, numbers, true, false and null, as much as the lesson needs.

The type of a value in JSON’s own words:

static String jsonType(Object value) {
    return switch (value) {
        case null -> "null";
        case String s -> "string";
        case Boolean b -> "boolean";
        default -> "number";
    };
}

The check does not throw an exception. It returns a list of errors, because the program has something to do with them:

static List<String> validate(Map<String, Object> json, List<Field> schema) {
    List<String> errors = new ArrayList<>();
    for (Field f : schema) {
        if (!json.containsKey(f.name())) {
            errors.add("missing \"" + f.name() + "\"");
            continue;
        }
        Object value = json.get(f.name());
        String type = jsonType(value);
        if (!type.equals(f.type())) {
            errors.add("\"%s\" is %s, must be %s"
                    .formatted(f.name(), type, f.type()));
        } else if (!f.allowed().isEmpty() && !f.allowed().contains(value)) {
            errors.add("\"%s\": \"%s\" is not one of %s".formatted(
                    f.name(), value, String.join(", ", f.allowed())));
        }
    }
    for (String key : json.keySet()) {
        if (schema.stream().noneMatch(f -> f.name().equals(key))) {
            errors.add("extra field \"" + key + "\"");
        }
    }
    return errors;
}

Four answers of the model:

List<String> answers = List.of(
        """
        {"intent": "delivery", "order_id": "1042", "urgent": true, "summary": "The courier did not come, the customer leaves tomorrow."}""",
        """
        {"intent": "shipping", "order_id": "1042", "urgent": "yes", "summary": "The courier did not come."}""",
        """
        {"intent": "return", "urgent": false, "summary": "Wants to return a jacket.", "mood": "calm"}""",
        """
        Sure! Here is the data:
        ```json
        {"intent": "delivery", "order_id": "1042", "urgent": true}
        ```""");
for (String answer : answers) {
    Map<String, Object> json = parseJson(answer);
    List<String> errors = json == null ? List.of("not JSON") : validate(json, TICKET);
    System.out.println(errors.isEmpty() ? "ok" : String.join("\n", errors));
    System.out.println("---");
}

Output

ok
---
"intent": "shipping" is not one of delivery, return, payment, other
"urgent" is string, must be boolean
---
missing "order_id"
extra field "mood"
---
not JSON
---

Try it

Predict the output

What does the check print for this answer?

List<String> answers = List.of(
        """
        {"intent": "refund", "order_id": 1042, "urgent": false, "summary": "Wants the money back."}""");
for (String answer : answers) {
    Map<String, Object> json = parseJson(answer);
    List<String> errors = json == null ? List.of("not JSON") : validate(json, TICKET);
    System.out.println(errors.isEmpty() ? "ok" : String.join("\n", errors));
    System.out.println("---");
}

Answer

ok
---

"intent": "refund" is not one of delivery, return, payment, other
"order_id" is number, must be string
---

"intent": "refund" is not one of delivery, return, payment, other
---

Check

Try it

Fill in the blank

A line from the check: the type is right, now the value must be one of the allowed ones. What is missing?

} else if (!f.allowed().isEmpty() && !f.allowed().(value)) {

Check

The idea

The errors back to the model

What do you do with an answer that does not pass? If you ask the question once more, the model often repeats the same error. It is better to tell it what is wrong.

The model’s answer goes into the history with the role assistant, followed by a new user message with the errors: “The answer does not match the schema: intent… Return only JSON that matches the schema.” The model sees its attempt and the exact errors and fixes them. This is a second attempt with feedback.

Every attempt sends the whole history. The second costs more than the first, and the third more than the second: from the lesson about context you know the API has no memory and everything starts again. That is why the attempts are few, and after them the email goes to a person.

See it

Three emails, three fates

static Ticket triage(FakeClient client, String email) {
    List<Message> messages = new ArrayList<>();
    messages.add(new Message("user", email));
    for (int attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
        Reply reply = client.create(SYSTEM, messages);
        Map<String, Object> json = parseJson(reply.text());
        List<String> errors = json == null
                ? List.of("not JSON")
                : validate(json, TICKET);
        String verdict = errors.isEmpty()
                ? "ok"
                : String.join("; ", errors);
        System.out.printf("  attempt %d (~%d tokens): %s%n",
                attempt, reply.inputTokens(), verdict);
        if (errors.isEmpty()) {
            return toTicket(json);
        }
        messages.add(new Message("assistant", reply.text()));
        messages.add(new Message("user",
                "The answer does not match the schema: " + verdict + ". Return only JSON that matches the schema."));
    }
    return null;
}

Step by step, in words

1. The email goes to the model with an instruction: return only JSON with four fields.
2. The answer is valid JSON and is read. That does not mean it is right.
3. Every field is checked against the schema: is it there, is the type right, is the value one of the allowed ones.
4. Two errors: intent and urgent. They are a list of strings, not an exception: the program decides what to do.
5. The errors go back to the model as a new message. Attempt 2 sends the whole history: ~108 tokens instead of ~53.
6. The second answer passes every check. Only now does it become a Ticket.
7. The email goes to the delivery queue, marked urgent. At most 3 attempts; after that a person takes over.

The instruction and how many attempts the program has:

static final String SYSTEM = "Sort the customer email. Return only JSON with the fields "
        + "intent (delivery, return, payment or other), order_id (text), "
        + "urgent (true or false) and summary (one sentence).";
static final int MAX_ATTEMPTS = 3;

One email: ask, read, check, and on an error a new attempt with the errors. When everything passes, the map becomes record Ticket, and from then on the code works with fields, not with text:

static Ticket triage(FakeClient client, String email) {
    List<Message> messages = new ArrayList<>();
    messages.add(new Message("user", email));
    for (int attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
        Reply reply = client.create(SYSTEM, messages);
        Map<String, Object> json = parseJson(reply.text());
        List<String> errors = json == null
                ? List.of("not JSON")
                : validate(json, TICKET);
        String verdict = errors.isEmpty()
                ? "ok"
                : String.join("; ", errors);
        System.out.printf("  attempt %d (~%d tokens): %s%n",
                attempt, reply.inputTokens(), verdict);
        if (errors.isEmpty()) {
            return toTicket(json);
        }
        messages.add(new Message("assistant", reply.text()));
        messages.add(new Message("user",
                "The answer does not match the schema: " + verdict + ". Return only JSON that matches the schema."));
    }
    return null;
}

static Ticket toTicket(Map<String, Object> json) {
    return new Ticket((String) json.get("intent"), (String) json.get("order_id"),
            (Boolean) json.get("urgent"), (String) json.get("summary"));
}

FakeClient client = new FakeClient();
String[] inbox = {
        "The courier did not come yesterday, order 1042. It is urgent, I leave tomorrow!",
        "I want to return order 1187, the jacket is too big.",
        "hi, something is wrong",
};
for (String email : inbox) {
    System.out.println("email: " + email);
    Ticket ticket = triage(client, email);
    if (ticket == null) {
        System.out.println("  → to a person");
    } else {
        String urgent = ticket.urgent() ? ", urgent" : "";
        System.out.println("  → " + ticket.intent() + ", " + ticket.orderId() + urgent);
    }
}

Output

email: The courier did not come yesterday, order 1042. It is urgent, I leave tomorrow!
  attempt 1 (~53 tokens): "intent": "shipping" is not one of delivery, return, payment, other; "urgent" is string, must be boolean
  attempt 2 (~108 tokens): ok
  → delivery, 1042, urgent
email: I want to return order 1187, the jacket is too big.
  attempt 1 (~49 tokens): ok
  → return, 1187
email: hi, something is wrong
  attempt 1 (~40 tokens): not JSON
  attempt 2 (~71 tokens): not JSON
  attempt 3 (~102 tokens): not JSON
  → to a person

• The email about 1042: the first answer has two errors, the second passes. The second attempt costs ~108 tokens instead of ~53.
• The email about 1187: passes the first time.
• “hi, something is wrong”: the model finds no topic and answers with a sentence all three times. The program does not invent a category; it gives the email to a person.

The model here is a fake client, as in the lesson about the prompt: the answers are recorded and the same every time. Its rule is in “More detail”.

Try it

Find the bug

A colleague writes a shorter check. The answer {"intent": "return", "urgent": false, "summary": "…"} passes as “ok”, and a minute later the program crashes somewhere far away, when it looks up the order. Which line has the bug?

Pick the line with the bug.

1static List<String> validate(Map<String, Object> json, List<Field> schema) {2 List<String> errors = new ArrayList<>();3 for (Field f : schema) {4 if (!json.containsKey(f.name())) {5 continue;6 }7 String type = jsonType(json.get(f.name()));8 if (!type.equals(f.type())) {9 errors.add(f.name() + " is " + type);10 }11 }12 return errors;13}

Check

The idea

The schema in the API itself

Newer APIs accept the schema as part of the request. Then the model generates only text that matches it: every next token is chosen among those that keep the JSON by the schema. The polite “Sure!” and "urgent": "yes" become impossible.

The check in the code stays, though:

• A refusal or a cut-off answer. If the model refuses (stop_reason is refusal) or reaches max_tokens, the text may not match the schema.
• The shop’s rules. The schema says “text”, but it does not know whether order 1187 exists and whether it belongs to this customer. That is a check in the program.
• Another model or provider. The code that checks does not depend on who generated the JSON.

Run it at your place. This program is not part of the lesson’s check: you need an account with a model provider, a key and money for every call. The key is in LLM_API_KEY, never in the code. Providers ship official SDKs for Java, C#, Python and JavaScript with the same shape (messages, roles, tokens, tools); the exact class and method names are in the provider’s documentation. The request as JSON: the schema is in the field output_config, and the model name is a placeholder:

{
  "model": "<model name>",
  "max_tokens": 1024,
  "system": "Sort the customer email. Return only JSON with the fields intent (delivery, return, payment or other), order_id (text), urgent (true or false) and summary (one sentence).",
  "messages": [
    {
      "role": "user",
      "content": "The courier did not come yesterday, order 1042. It is urgent, I leave tomorrow!"
    }
  ],
  "output_config": {
    "format": {
      "type": "json_schema",
      "schema": {
        "type": "object",
        "properties": {
          "intent": {
            "type": "string",
            "enum": ["delivery", "return", "payment", "other"]
          },
          "order_id": {"type": "string"},
          "urgent": {"type": "boolean"},
          "summary": {"type": "string"}
        },
        "required": ["intent", "order_id", "urgent", "summary"],
        "additionalProperties": false
      }
    }
  }
}

An example answer. The JSON by the schema is text in the text block, and the program still reads and checks it:

{
  "content": [
    {
      "type": "text",
      "text": "{\"intent\": \"delivery\", \"order_id\": \"1042\", \"urgent\": true, \"summary\": \"The courier did not come, the customer leaves tomorrow.\"}"
    }
  ],
  "stop_reason": "end_turn",
  "usage": {"input_tokens": 53, "output_tokens": 21}
}

Without an SDK, the request goes out with the language’s standard HTTP client. The address is a placeholder:

// requestJson is the request JSON above.
// The exact key header is in the provider’s documentation.
HttpClient http = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder(URI.create("https://<provider>/messages"))
        .header("Authorization", "Bearer " + System.getenv("LLM_API_KEY"))
        .header("Content-Type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString(requestJson))
        .build();
HttpResponse<String> response =
        http.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.body());

Every object in the schema has additionalProperties: false: that way an extra field like mood cannot appear at all.

The idea

When it hurts

• JSON for everything. The answer to a customer is text. A schema belongs where the answer goes into code: a queue, a database, another service.
• A deep schema. Five levels of nested objects and twenty fields give more errors and more expensive attempts. Two simple requests are better than one complicated one.
• Patches instead of a check. “Cut from the first { to the last }” works until the first answer with two {. If the API supports a schema, use it. If not, check and try again.
• The right shape is not the right answer. "order_id": "1043" passes the schema, but the model may have got a digit wrong: models invent plausibly. The number is checked against the database before it is used.
• Attempts have a price. Every attempt is more tokens and more seconds for the customer. How much exactly, and how to keep a budget, is in the lesson about cost and speed.
• The schema and the prompt change together. A new category in the schema without a new category in the prompt means a model that never says it. The schema’s version is in the code, next to the prompt.

More precisely, if you are curious

The words of JSON Schema. The schema in the API is JSON Schema: type (object, string, number, boolean, array, null), properties with the fields, required with the mandatory ones, enum with the allowed values and additionalProperties: false against extra ones. Our check does a subset of that, enough to show the mechanism. A project uses a ready library for JSON Schema.

From text to a type. After the check the map becomes Ticket, and the rest of the code checks nothing more: the compiler or the fields say what is there. That is the idea “read and check at the entrance, and work with types inside”. The check is in one place, not scattered across the program.

Tools use the same. In the next lesson the model asks the program to run a function. The function’s arguments also arrive as JSON by a schema (input_schema) and are also checked.

The fake client inside. Its answers are recorded by the order number in the email: for 1042 the first one is wrong, and after the correction it is right; for 1187 it is right at once; for anything else it is a sentence. The tokens are words × 1.3.

// Stands in for the real model: recorded answers,
// with no network and the same every time.
class FakeClient {
    static final String WRONG_1042 = """
            {"intent": "shipping", "order_id": "1042", "urgent": "yes", "summary": "The courier did not come."}""";
    static final String GOOD_1042 = """
            {"intent": "delivery", "order_id": "1042", "urgent": true, "summary": "The courier did not come, the customer leaves tomorrow."}""";
    static final String GOOD_1187 = """
            {"intent": "return", "order_id": "1187", "urgent": false, "summary": "Wants to return the jacket, it is too big."}""";
    static final String PROSE = "I cannot tell what the customer wants.";

    Reply create(String system, List<Message> messages) {
        String email = messages.get(0).content();
        boolean corrected = messages.size() > 1;
        String text;
        if (email.contains("1042")) {
            text = corrected ? GOOD_1042 : WRONG_1042;
        } else if (email.contains("1187")) {
            text = GOOD_1187;
        } else {
            text = PROSE;
        }
        int input = tokens(system);
        for (Message m : messages) {
            input += tokens(m.content());
        }
        return new Reply(text, "end_turn", input, tokens(text));
    }

    // Roughly: words × 1.3, rounded up.
    static int tokens(String text) {
        String t = text.strip();
        int words = t.isEmpty() ? 0 : t.split("\\s+").length;
        return (words * 13 + 9) / 10;
    }
}

The FlatJson reader. Recursive descent in one pass: object reads {, key–value pairs and }, and value recognises text, true, false, null and a number. It does not read nested objects or arrays, nor \n inside text. Every unexpected place throws an exception, and parse turns it into null.

// Java has no JSON in its standard library. This reader understands
// a flat object: text, numbers, true, false and null. In a project: Jackson.
class FlatJson {
    private final String s;
    private int i;

    private FlatJson(String s) {
        this.s = s;
    }

    // The object, or null if the text is not exactly one JSON object.
    static Map<String, Object> parse(String text) {
        FlatJson p = new FlatJson(text.strip());
        try {
            Map<String, Object> obj = p.object();
            return p.i == p.s.length() ? obj : null;
        } catch (RuntimeException e) {
            return null;
        }
    }

    private Map<String, Object> object() {
        Map<String, Object> obj = new LinkedHashMap<>();
        expect('{');
        skip();
        if (peek() == '}') {
            i++;
            return obj;
        }
        while (true) {
            skip();
            String key = string();
            skip();
            expect(':');
            skip();
            obj.put(key, value());
            skip();
            if (peek() == '}') {
                i++;
                return obj;
            }
            expect(',');
        }
    }

    private Object value() {
        if (peek() == '"') {
            return string();
        }
        for (String word : new String[] {"true", "false", "null"}) {
            if (s.startsWith(word, i)) {
                i += word.length();
                return word.equals("null") ? null : Boolean.valueOf(word);
            }
        }
        int start = i;
        while (i < s.length() && "+-.0123456789eE".indexOf(s.charAt(i)) >= 0) {
            i++;
        }
        return Double.parseDouble(s.substring(start, i));
    }

    private String string() {
        expect('"');
        StringBuilder out = new StringBuilder();
        while (peek() != '"') {
            char c = s.charAt(i++);
            out.append(c == '\\' ? s.charAt(i++) : c);
        }
        i++;
        return out.toString();
    }

    private void skip() {
        while (i < s.length() && Character.isWhitespace(s.charAt(i))) {
            i++;
        }
    }

    private char peek() {
        return s.charAt(i);
    }

    private void expect(char c) {
        if (peek() != c) {
            throw new IllegalStateException("expected " + c);
        }
        i++;
    }
}

Exercise

The number is four digits

Solve it in your editor.

Add a rule to the check: order_id is exactly 4 digits. If not, the error is "order_id": "10420" is not 4 digits. Run the check for the three answers that differ only in the number: 1042, 10420 and #1042.

Output

ok
---
"order_id": "10420" is not 4 digits
---
"order_id": "#1042" is not 4 digits
---

More tasks

• An empty summary. summary cannot be empty text or only spaces. Add the rule and a test with "summary": " ".
• A limited number of tokens. Stop the attempts also when the email’s total tokens pass 100, even before three attempts. How many attempts does the email “hi, something is wrong” make then?

Test

Test yourself

1. The model returned valid JSON. Does that mean the program can use it?

Yes: if the JSON can be read, the data is rightYes, if the prompt says “return only JSON”No: it must be checked against the schema, because fields can be missing or have the wrong type and value

2. Why are the errors sent to the model on the second attempt, instead of asking the question once more?

That way the model sees exactly what it got wrong and fixes thatBecause the API does not allow the same question twiceBecause it is cheaper

3. The email makes no sense and the model returns a sentence instead of JSON three times. What does the program do?

It keeps asking until it gets JSONIt puts the email into the “General” queue with empty fieldsIt stops after the third attempt and gives the email to a person

4. The API now guarantees JSON by the schema. Does the check in the code stay?

Yes: the answer can be a refusal or cut off, and the schema does not know the shop’s rulesOnly for Java, because it has no JSON in the standard libraryNo: the schema in the API makes it unnecessary

Check

Start the lesson over

Related lessons

• Builds on: The context window · The limits of the model · Calling a model through an API · The prompt is code
• Continues in: Tool use: the model asks, the program acts · The agent loop: plan, action, observation · Prompt injection: the email that gives orders · Guardrails: checks before and after the model
• Tasks on this topic: Apps with LLMs