OTF Web Blog
Inside the on-demand dev server

Inside the on-demand dev server

How OTF Web boots in milliseconds by compiling only the route you visit — and how a module graph keeps hot reloads precise.

A development server's first obligation is to stay out of the way. When dev takes fifteen seconds to start, every restart becomes a context switch. On our documentation site, the OTF Web dev server now starts in roughly 40 milliseconds — achieved not by performing the same work faster, but by performing less work, later.

The old shape: compile everything, then serve

The initial implementation bundled the entire application up front: every route, every component, and every MDX page passed through the compiler before the first request was answered. Two factors made this slow:

  1. One process per file. The compiler ran as a subprocess invoked once per module, so startup paid the binary-startup cost dozens of times, serially.

  2. Eager compilation. Although routes were code-split, all of them were compiled at boot — the cost of 34 routes was incurred to view one.

Step one: one compiler process, not hundreds

The first change was to stop spawning processes. The compiler gained a long-lived serve mode that reads compile requests on stdin and writes results on stdout, remaining resident across requests:

TEXT
<id_len> <source_len> <component> <ssg>\n  <id bytes><source bytes>
→ OK <len>\n<code>     or     ERR <len>\n<message>

A single process now compiles every module. That change alone reduced the initial bundle from ~16.7s to ~3.1s, because the binary-startup cost is paid once.

Step two: compile only what is requested

The remaining 3.1s was spent on routes that had not been visited. The dev server therefore stopped bundling up front and began serving the way the browser asks for work — module by module:

  • /@fw.js — the runtime, bundled once and shared by every chunk through an import map (so the router and signal registry are a single instance).

  • /bundle.js — the app entry: mountApp plus a route table whose loaders point at /__route/<id>.js URLs. The route modules are not in this bundle.

  • /__route/<id>.js — one route, compiled the first time you navigate to it, then cached.

Startup compiles nothing. First paint pays for the entry and the runtime; each route pays its own cost on first visit and is subsequently served from memory in well under a millisecond.

What makes hot reload precise: a module graph

On-demand serving raises a question: when a file changes, what should be invalidated? Invalidating too much recompiles the world; invalidating too little serves stale code.

The answer is a module graph, in which nodes are modules, edges are imports, and each node carries a content fingerprint. When a file changes, the graph answers "what transitively imports this?" and the dev server drops exactly those cached chunks. Editing a component used by a single post recompiles that post's chunk; unrelated routes stay warm.

TEXT
edit  app/components/CounterDemo.jsx
  → affected = { …/docs/page.mdx, … }   // only importers
  → drop those route chunks, reload

The same graph will drive incremental builds and the client/server split as the toolchain matures. It is the underlying structure that makes the rest possible.

Direction

The dev server remains an intermediate step. The long-term home for the graph and for build orchestration is the Rust toolchain, where both can share data structures with the bundler rather than serializing across a process boundary. The principles, however, are settled: start instantly, compile on demand, invalidate precisely.

If you have used the dev server and found any operation slow, we would like to hear about it.