Appearance
Code Execution & DSA Engine: ediky_tools Pipeline & Empirical Analysis
Part 5 of 5 in the Code Execution & DSA Engine series.
Multi-Language Support & SQL Runner closed the loop on how a submission is judged. This closing page covers the orchestrator that ties the verification layers from Test Case Generation into one command, and then an honest correction: where empirical complexity analysis actually happens in this system, which is not where a first guess would place it.
qgen.py: automating steps 4 through 8, not 1 through 10
tools/qgen/qgen.py is described in its own docstring as automation for "I already have the draft, just run the pipeline." Of the ten-step authoring model, steps 1–3 (decide the question, draft it, save under drafts/<slug>.md) and 9–10 (smoke-test on the real judge, commit) stay manual on purpose — they need a human or an AI assistant genuinely in the loop. Everything between is one command:
text
python3 tools/qgen/qgen.py drafts/two-sum.md --preset standard rendering diagram…
drag to pan · ctrl / ⌘ + scroll to zoom
build_tests.sh internally runs the fuzz and cross-language layers from Test Case Generation and ends with its own step 5/5 internal consistency gate; if that specific gate fails, qgen.py prints an explicit warning not to rerun the whole script — by that point generator.py/brute.cpp have already been stripped from the draft, so the fix is to edit the <!--SOLUTION--> section directly and re-run just verify_solution.py until it passes.
Presets: three tiers, auto-detected
presets.json sizes the pipeline to the problem rather than using one budget for everything. A draft's topic: frontmatter is matched against each preset's keyword list — simple catches topics like string, array, sorting, two pointers, hashing, implementation; anything unmatched falls back to standard:
simple | standard | complex | |
|---|---|---|---|
| generated cases | 60 | 100 | 120 |
| solution timeout | 5s | 10s | 12s |
| fuzz iterations | 1,500 | 5,000 | 10,000 |
| max stdin / expected | 250 / 100 KB | 200 / 100 KB | 400 / 200 KB |
Every value is overridable per-run (--cases, --fuzz, --timeout, …), and qgen.py resolves its own bash at runtime specifically to dodge the WSL launcher stub Windows ships at System32\bash.exe — a Python subprocess call to "bash" can hit that stub even with Git Bash correctly on PATH, because Windows' native process search checks System32 before honoring PATH order.
Why the pipeline resolves a live website checkout, not its own copy
ediky_tools deliberately keeps no local copy of lint-content.mjs or src/content/. Every path resolves through tools/site_config.py, which points at the real ediky-website checkout (siblings by default, overridable via EDIKY_WEBSITE_ROOT). This exists because of an incident, not a design preference: an earlier version assumed the pipelines lived inside the website repo and computed paths relative to themselves. Split into a separate repo, they kept running — against a local, nearly-empty src/content/ and a locally drifting copy of the lint script. The duplicate-title check therefore only ever knew about the tools repo's own handful of files, never the website's live set, and a 75-question batch shipped with 34 accidental duplicate questions. The fix wasn't a better duplicate-detection algorithm; it was making sure every check runs against something that cannot drift.
Exit codes are the pipeline's own status protocol
qgen.py fails loudly with the stage that broke rather than a generic error:
| Exit code | Meaning |
|---|---|
| 0 | Completed (or reached however far --no-lint/--no-move asked it to) |
| 1 | A pipeline stage failed — build, trim, or the step 5/5 consistency gate |
| 2 | Lint failed |
| 3 | Bad arguments, missing draft, or a malformed presets.json |
The two most common real failures are a generator emitting out-of-constraint inputs (caught by verification) and oversized cases (caught by audit_sizes.py) — both fixed by editing the draft, never by loosening a gate.
Empirical complexity: a runtime feature, not an authoring-time one
It would be reasonable to assume qgen also estimates a problem's time complexity while generating cases. It doesn't — and the actual mechanism is arguably more interesting. src/lib/complexityEstimate.js, on the website side, fits a log-log linear regression over (inputSizeBytes, cpuTimeMs) pairs harvested from a single already-accepted submission's own hidden test cases — a real student's real solve, measured empirically at grading time, not a synthetic authoring-time benchmark.
javascript
// Simplified real shape of estimateComplexity (complexityEstimate.js)
function estimateComplexity(cases) {
const pts = cases.filter((c) => c.timeMs >= NOISE_FLOOR_MS); // 8ms floor
if (pts.length < MIN_POINTS) return { status: "insufficient_data" };
const xs = pts.map((c) => Math.log(c.sizeBytes));
const ys = pts.map((c) => Math.log(c.timeMs));
const slope = leastSquaresSlope(xs, ys); // ln(time) vs ln(size)
return { status: "ok", label: bucketForSlope(slope), slope, r2: rSquared(xs, ys, slope) };
}The slope of that fit is bucketed into a human label — under 0.3 reads as O(1)/O(log n), up to 1.3 as O(n) to O(n log n), up to 2.5 as O(n²), and so on — with O(n) and O(n log n) deliberately merged because timing alone can't reliably separate them. Below 4 surviving points, or less than 3x spread between the smallest and largest input size, the estimator reports insufficient_data rather than guessing — a poor fit is the honest signal that a given submission's case sizes didn't vary enough to say anything, and r2 is persisted alongside the label so a low-confidence estimate is visibly low-confidence, not silently wrong. This single function is shared by both judgeRunner.js (the persisted per-submission estimate) and the UI's ComplexityPanel.jsx chart, so the fitted line drawn on screen and the label stored in the database can never disagree.
What this series covered, and what it plugs into
Five pages: the client-side judging loop and its Judge0 status-to-verdict mapping (Judge0 & Code Execution); a four-layer verification stack that catches four genuinely different bug classes before a question ships (Test Case Generation & Verification); the frontmatter-plus-section-marker format both qgen and the website content pipeline read (DSA Question Design & Structure); a single language registry driving ten runtimes plus a SQL runner that deliberately isn't Judge0 at all (Multi-Language Support & SQL Runner); and this page's orchestrator plus a correction about where complexity analysis really lives.
Every page leans on what came before it in this documentation set: Task 1's Firestore/SQLite split is what makes a submission's queue-then-grade flow and a durable SQL grade job possible at all; Task 2's progressive-hints and split-pane patterns are the UI surface this execution engine feeds results into. The architecture, the interface, and the judge are three layers of the same product — this series was the third.
Self-assessment
Word count for this page: 921 words (slightly over the 800–900 target to cover the orchestrator, presets, the site_config incident, exit codes, and the complexity-analysis correction without cutting any of them short).
← Previous: Multi-Language Support & SQL Runner↩ Series start: Judge0 & Code Execution