Warning: This is where the exam gets sneaky. Wrong answers sound like good engineering. Right answers require knowing which technique applies to which specific problem.

4.1 Explicit Criteria

Concept

Specific categorical criteria obliterate vague confidence-based instructions.

# WRONG: Vague
"Be conservative. Only report high-confidence findings."

# RIGHT: Specific categorical criteria
"""Flag comments only when:
- Claimed behaviour contradicts actual code behaviour
- Security vulnerabilities (SQL injection, XSS, auth bypass)
- Bugs that would cause runtime errors

Skip:
- Minor style preferences
- Local patterns that differ from team convention but are functional
"""

The False Positive Trust Problem

High false positive rates in one category destroy trust in ALL categories.

Fix: Temporarily disable high false-positive categories while improving prompts for those categories. Restores trust while you iterate.

Severity Calibration

Define explicit severity criteria with concrete code examples for each level — not prose descriptions.

severity_examples = {
    "critical": "# Unvalidated user input in SQL query\n"
                "cursor.execute(f'SELECT * FROM users WHERE id = {user_input}')",
    "major": "# Missing null check before method call\n"
             "user.profile.get_preferences()  # user.profile may be None",
    "minor": "# Variable name could be more descriptive\n"
             "x = calculate_total(items)"
}

4.2 Few-Shot Prompting

Core Principle

Few-shot examples are the most effective technique for consistency. Not more instructions. Not confidence thresholds.

When to Deploy

• Detailed instructions alone produce inconsistent formatting
• Model makes inconsistent judgement calls on ambiguous cases
• Extraction tasks produce empty/null fields for information that exists in the document

How to Construct

• 2-4 targeted examples for ambiguous scenarios
• Each example shows reasoning for why one action was chosen over plausible alternatives
• This teaches generalisation to novel patterns, not just pattern-matching

<example>
Input: "The contract expires on the 15th"
Output: {"expiry_date": "unclear - no month/year specified", "confidence": "low"}
Reasoning: Only a day number is given. Do not fabricate month or year.
</example>

<example>
Input: "Agreement valid through December 2025"
Output: {"expiry_date": "2025-12-31", "confidence": "high"}
Reasoning: Month and year specified. Use last day of month as default.
</example>

Hallucination Reduction

Few-shot examples showing correct handling of varied document structures (inline citations vs bibliographies, narrative vs structured tables) dramatically improve extraction quality.

4.3 Structured Output with tool_use

Reliability Hierarchy

What tool_use Does NOT Prevent

• Semantic errors: Line items that do not sum to stated total
• Field placement errors: Values in wrong fields
• Fabrication: Model invents values for required fields when source lacks information

tool_choice Settings

Schema Design for Preventing Fabrication

{
  "type": "object",
  "properties": {
    "company_name": {"type": "string"},
    "revenue": {
      "type": ["number", "null"],
      "description": "Annual revenue in USD. null if not stated in document."
    },
    "fiscal_year": {
      "type": ["string", "null"],
      "description": "Fiscal year for the revenue figure. null if not stated."
    },
    "industry": {
      "type": "string",
      "enum": ["technology", "healthcare", "finance", "manufacturing", "other"]
    },
    "industry_detail": {
      "type": ["string", "null"],
      "description": "If industry is 'other', specify here."
    },
    "data_quality": {
      "type": "string",
      "enum": ["complete", "partial", "unclear"]
    }
  },
  "required": ["company_name", "data_quality"]
}

Key patterns:

• Optional/nullable fields when source may not contain information → prevents fabrication
• "unclear" enum value for ambiguous cases
• "other" + freeform detail string for extensible categorisation
• Format normalisation rules in prompts alongside strict schemas

4.4 Validation-Retry Loops

Retry-with-Error-Feedback

# Validation-retry loop
for attempt in range(3):
    result = extract(document)
    errors = validate(result)

    if not errors:
        break

    # Send back: original doc + failed extraction + specific error
    messages.append({
        "role": "user",
        "content": f"Extraction had errors: {errors}\n"
                   f"Original document: {document}\n"
                   f"Your extraction: {json.dumps(result)}\n"
                   f"Please fix the specific errors and re-extract."
    })

Retry Effectiveness Boundary

The exam presents both scenarios. You must identify which is fixable.

detected_pattern Fields

Add to structured findings to track which code construct triggered the finding. Enables analysis of dismissal patterns → improves prompts over time.

Self-Correction Flows

{
  "stated_total": 1250.00,
  "calculated_total": 1237.50,
  "conflict_detected": true,
  "line_items": [...]
}

Extract calculated_total alongside stated_total to flag discrepancies.

4.5 Batch Processing

Message Batches API

Matching Rule

Exam trap: A manager proposes using batch for everything. Correct answer: keep blocking workflows synchronous.

Batch Failure Handling

1. Identify failed documents by custom_id
2. Resubmit only failures with modifications (e.g., chunking oversized documents)
3. Refine prompts on a sample set BEFORE batch processing to maximise first-pass success

4.6 Multi-Instance Review

Self-Review Limitation

A model reviewing its own output in the same session retains reasoning context. It is less likely to question its own decisions. An independent instance without prior context catches more subtle issues.

Multi-Pass Architecture

Pass 1: Per-file local analysis → consistent depth per file
Pass 2: Cross-file integration  → catches data flow issues

Prevents attention dilution and contradictory findings.

Confidence-Based Routing

• Model self-reports confidence per finding
• Route low-confidence findings to human review
• Calibrate confidence thresholds using labelled validation sets

Domain 4 Practice Exam

Build Exercise

Build a Structured Data Extraction Pipeline

1. Define an extraction tool with a JSON schema containing required and optional fields, an enum with an "other" + detail string pattern, and nullable fields for information that may not exist in source documents. Process documents where some fields are absent and verify the model returns null rather than fabricating values.
2. Implement a validation-retry loop: when Pydantic or JSON schema validation fails, send a follow-up request including the document, the failed extraction, and the specific validation error. Track which errors are resolvable via retry (format mismatches) versus which are not (information absent from source).
3. Add few-shot examples demonstrating extraction from documents with varied formats (e.g., inline citations vs bibliographies, narrative descriptions vs structured tables) and verify improved handling of structural variety.
4. Design a batch processing strategy: submit a batch of 100 documents using the Message Batches API, handle failures by custom_id, resubmit failed documents with modifications (e.g., chunking oversized documents), and calculate total processing time relative to SLA constraints.
5. Implement a human review routing strategy: have the model output field-level confidence scores, route low-confidence extractions to human review, and analyse accuracy by document type and field to verify consistent performance.

Domains reinforced: Domain 4 (Prompt Engineering & Structured Output), Domain 5 (Context Management & Reliability)

→ Try related coding exercises

Quick Reference Card