Code Execution & DSA Engine: Test Case Generation & Verification

Part 2 of 5 in the Code Execution & DSA Engine series.

Judge0 & Code Execution covered how a submission gets run once its test cases already exist. This page covers where those cases come from, and why Ediky trusts them: tools/qgen/ in the ediky_tools repo, which turns a drafted question into a fuzz-tested, cross-language-verified file before it ever reaches src/content/dsa/. The failure mode this exists to prevent already happened once — a reference solution that didn't pass its own committed test case — which is why verify_solution.py is now a mandatory gate, not an optional check.

Four layers, four different bugs

No single check catches every way a question can be wrong, so the pipeline stacks four independent gates, each closing a gap the others structurally can't:

Layer	Catches	Never catches
lint-content.mjs	Structural issues — invalid JSON, non-kebab slug, banned <bits/stdc++.h>	Whether the logic is correct
fuzz.py	Primary solution vs. an independent brute-force disagreeing on tiny random inputs	A bug both solutions share
run_cases.py cross-check	One language's solution disagreeing with another on the real cases	Drift between the file and its own committed expected values
verify_solution.py	The committed expected string not being reproducible by the committed solution (a hand-edit, a post-generation tweak, a bad merge)	Nothing above it — this is the last gate

Fuzz testing: brute force as the oracle

fuzz.py compiles brute.cpp — an intentionally naive, "obviously correct by inspection" solution — alongside the primary reference language, then runs thousands of tiny random inputs through both. The generator must support a second CLI argument (python3 generator.py <idx> fuzz) that forces a deliberately small input regardless of what <idx> alone would produce, small enough that even an O(n²) or O(2ⁿ) brute force finishes instantly.

# Simplified real shape of the fuzz loop (tools/qgen/fuzz.py)
for i in range(1, num_fuzz_cases + 1):
    stdin_text = run(["python3", "generator.py", str(i), "fuzz"])
    if len(stdin_text.encode()) > FUZZ_MAX_STDIN_BYTES:
        oversized += 1  # generator's fuzz-mode branch is probably broken
        continue
    primary_out = run_lang(primary_runner, stdin_text)
    brute_out = run_lang(brute_binary, stdin_text)
    if primary_out.strip() != brute_out.strip():
        mismatches += 1  # a real correctness bug, in one solution or the other

The FUZZ_MAX_STDIN_BYTES guard is not a tuning knob — "fuzz" mode is contractually supposed to emit n in roughly [1, 8]. If a generator's fuzz-mode branch is broken and silently falls through to normal sizing, that would otherwise burn thousands of iterations of a multi-second brute force. fuzz.py is advisory: it never blocks the rest of the pipeline (even a mismatch just writes fuzz_report.txt and prints a summary) because it runs on random tiny inputs, not the shipped cases — a human still decides whether a flagged disagreement is a real bug or a legitimately ambiguous multi-valid-output case.

Cross-language verification, with an honest exception for Python

Every compiled reference language (C++, Java, and any others attached to the draft) must pass every case, visible and hidden — no exceptions. Python is different: langs.py marks it tle_lenient = True, because Python is often genuinely too slow on large hidden inputs even when the algorithm is correct — PyPy3 would pass the identical logic. Treating every Python TLE as a hard failure would mean either wasting time micro-optimizing CPython, or never shipping a Python solution at all.

diagram100%

rendering diagram…

drag to pan · ctrl / ⌘ + scroll to zoom

The two-tier check: Tier 1 runs the lenient language against visible cases only, with a capped timeout — a wrong answer here is always a hard failure regardless of language. Tier 2 tolerates TLEs on hidden cases, but only once a strict language (C++, Java) has already proven the test data correct end to end. A wrong answer anywhere, visible or hidden, is never tolerated by either tier — only timeouts are.

The last gate: is the shipped file self-consistent?

fuzz.py and the cross-language check both catch logic bugs. Neither re-checks that the expected strings actually committed to the .md file are reproducible by the committed solution — they can drift apart from a hand-edited value, a solution tweaked after cases were already generated, or a case copied between files during a merge. verify_solution.py closes that gap by treating the committed solution as the oracle for the committed cases, re-running every shipped case (visible and hidden) through it and failing if any expected isn't reproduced exactly. It deliberately reuses the same compile/run recipes as generation (langs.py, run_cases.py) and the same output comparison the live judge uses (normalizeOutput from Judge0 & Code Execution, ported into Python) — so a case that passes this gate structurally cannot fail on the real judge for a whitespace reason, and vice versa.

Adversarial, not just random

Generators are written to deliberately include edge patterns — empty inputs, max-N, degenerate structures like all-equal or already-sorted arrays — rather than relying on random noise to stumble into them. Random fuzzing is what catches an unanticipated bug; adversarial generation is what guarantees the known edge cases are always exercised, every time a question is regenerated.

Self-assessment

Word count for this page: 861 words.

---

← Previous: Judge0 & Code Execution→ Next: DSA Question Design & Structure