Question Bank & Content Pipeline

This is the part of the project most worth walking a new developer through, because it's a real scaling problem with a real constraint, not a hypothetical.

The constraint

Cloudflare Pages has a hard limit of 20,000 files per deployment. An early version of the content system used import.meta.glob("../content/**/*.md"), which made Vite emit one JS chunk per markdown file. At a few hundred questions that's manageable; at the 10,000–20,000 questions the bank is scaling toward, that alone would exceed the file limit — before counting any other build output — and it also meant every question's full metadata shipped inside the main JS bundle whether a user ever opened it or not.

The design

Markdown never enters the Vite module graph. Content is split into two concerns, each served differently at runtime:

diagram100%

rendering diagram…

drag to pan · ctrl / ⌘ + scroll to zoom

1. Metadata (list/search/filter data) — sharded JSON indexes

tools/build-content.mjs walks each content section (dsa, cp, cs, aptitude, dsa-technical, sql-practice) at build time, parses frontmatter, and writes one JSON file per section to public/content-index/<section>.json. A list page (/problems, /cs, etc.) fetches only its own section's shard — kilobytes, not megabytes — once per page load.

2. Bodies (statement/solution/tests) — static files, fetched on demand

Full question bodies are copied verbatim to public/content/<section>/<slug>.md. Because these live under public/, Vite copies them straight through without bundling — no per-file JS chunk, no ?raw inlining. A question's body is fetched only when its detail page opens.

3. Heavy hidden tests — split into sidecars

If a question's full test-case JSON exceeds 16KB, build-content.mjs splits the hidden cases into a <slug>.tests.json sidecar, fetched only when the user hits Submit — not when they open the problem. Opening a problem costs ~10KB; submitting triggers the (potentially multi-MB) hidden-case fetch, exactly once, exactly when needed.

4. CP is the exception — served from a CDN

The 600-problem CP set ("God's Will") is marked source: "cdn" in the manifest: its bodies are fetched from jsDelivr against the content repo rather than shipped inside the Pages deployment at all. That keeps the single largest section entirely outside the 20k file budget. See CP System.

Runtime caching on top of that

The client-side loader (src/lib/questionBank.js) adds two more layers so the split-fetch design doesn't trade file-count problems for latency problems:

• Index shards cache in sessionStorage with stale-while-revalidate: a page reload renders the list instantly from cache while a background fetch checks for changes, rather than blocking on the network every time.
• Parsed question bodies sit in a small in-memory LRU of promises — navigating list → problem → back → same problem never refetches or reparses, and two concurrent requests for the same slug (e.g. two OA questions resolving in parallel) share a single fetch instead of duplicating it.

The measurable payoff of the metadata/body split plus lazy routes: the main bundle went from ~1,153KB to ~855KB gzipped — details in Performance.

The authoring format

Questions are structured markdown: YAML frontmatter (id, title, topic, difficulty, companies, evaluationMethod) plus HTML-comment section markers the parser splits on:

<!--STATEMENT-->   problem statement (GFM + LaTeX)
<!--HINTS-->       progressive hints
<!--SOLUTION-->    editorial + reference code per language
<!--TESTS-->       visible + hidden cases (JSON)

CS-fundamentals concept questions add <!--RUBRIC--> / <!--EXPECTED--> for AI grading; SQL questions add <!--SETUP--> for the per-question schema/seed script. src/lib/contentParse.js is the single parser all sections share.

The gate before commit

Nothing reaches src/content/ by hand-writing tests. The authoring pipeline in the tools repo (qgen.py → generators → verify_solution.py fuzz cross-check → lint-content.mjs) generates, fuzz-tests, and cross-language-verifies (C++17 / Python / Java) every problem before it's committed — verify_solution.py is a mandatory gate specifically added after a bug where a reference solution didn't pass its own generated tests. The lint step also bans <bits/stdc++.h> in reference solutions after a production compile-timeout incident. Full walkthrough in qgen.

Why this matters

It demonstrates recognizing a scaling failure mode before it happens (the bank is at ~1,400 questions today, nowhere near the 20k limit) and designing the data-loading strategy around a platform constraint rather than working around it after a deploy starts failing. It's the "ship metadata eagerly, ship payload lazily" pattern that shows up in most content-heavy production apps — worth being able to generalize past this project.