Reference vs. candidate · sealed holdout

Prove the cheaper workflow keeps its quality.

Kourum compares one reference workflow with one candidate through a vendor-diverse panel, deterministic checks, and a Final Exam that revision logic cannot read.

Define the task

Plain text or JSON. Uploaded documents are never parsed on the server.

Task definitionDescribe the inputs, required output, constraints, and acceptance boundary.

Add context or reference filesUp to 5 files, 1 MB each. PDF/DOCX extraction remains disabled until its library TBD is resolved.

Choose the pair

Choose from Kourum's verified, priced model catalog. Same-vendor pairs are allowed.

Reference workflow

1 · Vendor

2 · AI Model

Verified pricing and provider support are required.

3 · Reasoning effort

The only orchestration-level consistency lever; sampling fields are omitted.

ConfiguredOpenAI · GPT-5.6 SolHigh

Workflow instructionsThe candidate instructions may be revised; the reference remains fixed.

Candidate workflow

1 · Vendor

2 · AI Model

Verified pricing and provider support are required.

3 · Thinking

Minimal disables thinking; Thinking enables it for this model.

ConfiguredAnthropic · Haiku 4.5Minimal

Workflow instructionsThe candidate instructions may be revised; the reference remains fixed.

Add evaluation cases

Cases are split once into Study, Practice, and the sealed Final Exam at creation.

Case 1

PromptRequired

Reference answerOptional

Task familyDefaults to General

Human-approved canonical example

Set the quality gate

Deterministic checks run before rubric aggregation; any failure blocks a pass.

Deterministic checks

Sandbox check definitions

[
  {
    "name": "candidate-output-present",
    "type": "contract",
    "config": {
      "runtime": "node24",
      "entrypoint": "check.mjs",
      "timeoutMs": 5000,
      "files": [
        {
          "path": "check.mjs",
          "content": "import fs from 'node:fs';\nconst rows = JSON.parse(fs.readFileSync('inputs/manifest.json', 'utf8'));\nif (!rows.length) process.exit(1);\nfor (const row of rows) {\n  if (row.role === 'candidate' && !fs.readFileSync(`inputs/${row.path}`, 'utf8').trim()) process.exit(1);\n}\n"
        }
      ]
    }
  }
]

Node 24 or Python 3.13 only. Each check runs in a fresh, network-denied sandbox.

Optional human anchor calibration

Upload anchor JSON

Human anchors JSONOptional array of human-approved dimension scores keyed to dataset case IDs.

Funded runs need NEXT_PUBLIC_TURNSTILE_SITE_KEY before launch.

Start with a validated Study roundThe Final Exam remains unread until you explicitly proceed.