Guide14 min read5 sections

Terrain travel and residency

docs/TERRAIN_STREAMING.mdSource updated 2026-10-28
Markdown
On this page 5 sections

The bounded terrain streaming workflow is delivered; broader scale, transition polish and full performance qualification remain open. World format 24 adds explicit, versioned ground-chunk bounds and independent per-traveler residency. Format 25 adds a compact, always-resident overview of unloaded ground. Feature implementation is 70/70 (100%); qualification and polish are tracked separately in ENGINE_PROGRESS.

#Integrated repair — October 28

GPU staging now consumes the already filtered resident place exactly once. A second residency filter was rejecting a valid catalog before graphics preparation with “Terrain streaming requires independent static ground.” Native checks cover a genuinely distinct delayed chunk, continued movement during fetch/preparation, rejection, commit and checkpoint recovery. The complete native run passes (1,446 passes, zero failures; 259 environment-dependent skips). Earlier browser soak records below predate this repair; they are not a new integrated soak result.

#Try the larger landscape

Run npm run dev:editor, open its private URL, and choose New world → Misty uplands → Preview → Save → Publish. Open the invitation in two browser profiles. The landscape spans 128 × 128 metres; walk or sprint through it with WASD and Shift. Each traveler loads ground around their own position. Preparing the landscape ahead… appears while chunks download and decode in the background. Movement and rendering continue on resident ground through downloads, decoding and GPU catalog preparation. Only the final frame drain and publication hold movement; held keys resume afterward. At a slow-loading boundary, movement stops safely before missing collision.

In Terrain → Travel & loading, enable or disable travel-based loading and set the ground view distance. New uplands and newly saved streaming settings include distant ground. Old format-24 documents retain their exact generated models until edited; publishing an updated release enables the overview for its travelers. Save and use durable Undo/Redo as with sculpting. Publish a new invitation to test a changed release; existing invitations retain their original content.

#Edit a shared landscape

Choose Apply after saving Misty uplands, then open the development room in two browser profiles. Leave one traveler near the arrival and walk the other east across the hills. In the editor, sculpt or paint with the Terrain tool, choose Preview → Save → Apply, and wait for both travelers to resume. Saved Undo → Apply and Redo → Apply use the same coordinated handoff. Each traveler retains only their own nearby detailed chunks, while both receive the updated distant overview. Walk away and return, or reload the traveler, to revisit the saved edit. Published invitations remain immutable; use the development room for live changes.

The optional extended acceptance journey is:

Terminal
DW_GPU_PROFILE=hardware DW_TERRAIN_SOAK=1 npm run test:browser:terrain-streaming -- --grep 'shared uplands edits'

It requires a qualified physical WebGPU browser and measures at least ten minutes of repeated travel and 100 terrain residency publications, with recurring shared undo/redo. Raw resource samples attach to the Playwright result. This is a terrain resource-lifetime check, separate from the full desktop/mobile frame-time, JavaScript-heap and 100-place-crossing gates.

Add DW_TERRAIN_MEMORY=1 to the extended command for a retained-memory census of the editor and both players. This mode uses 1920 × 1080 player viewports and samples each page and its direct workers after garbage collection. It reports JavaScript heap, embedder heap and backing storage (ArrayBuffers plus external strings) separately from GPU resources, rejecting missing or replaced isolates. It does not measure process RSS, server memory or GPU execution time. Debugger attachment and forced garbage collection add overhead, so its frame observations cannot qualify the plan's separate no-DevTools-overhead frame-time target. The helper was checked against allocation/reclamation of a 16 MiB worker ArrayBuffer before use.

#Ownership and safety

  • terrain-stream.1 declares at most 16 independent static ground chunks per place. Generated terrain retains the existing total-world limit of 16 chunks. Ground bounds must match admitted geometry. Gameplay-bound, animated, scaled, rotated, hidden or parented objects cannot be streamed as independent ground.
  • terrain-overview.1 compacts certified, border-locked derivatives into one static model with one part per chunk and one shared paint atlas. The compiler admits only matching chunk bounds and exact oriented perimeter positions, normals and UVs. It selects a retained derivative with at most 0.12 metres of simplifier-reported error, or keeps the full mesh when none qualifies. The overview has no collision. Resident detail suppresses the corresponding overview part in the same presentation publication; editor and fully resident portal views suppress all overview parts. Existing model, document and undo budgets remain unchanged.
  • Content preparation fetches only selected chunk models plus ordinary place dependencies, including the overview. Closed bounds include both sides of a seam; a larger retention radius reduces repeated loading at the fringe. Global authored entities and world/content hashes remain stable.
  • Old collision and content leases remain owned while the candidate and graphics prepare. Downloads and presentation asset decoding run while the live traveler continues moving. After graphics staging, a fresh native candidate captures the latest player/input/entity state and takes over the prepared content lease exactly once. The GPU validates a fresh presentation baseline before consuming either catalog. Background ground updates retain existing surface, lettering and portal resources; portal projection rebinds to the accepted canonical store without replacing its stream. A late coverage check rejects ground selections that became unsafe while the traveler moved. Publication swaps both before releasing old ownership. Movement waits before reaching missing ground; rejected or canceled preparation preserves the prior scene.
  • Exact checkpoints include the current ground selection. Streamed recovery reinstalls that selection, native physics and global entity state. Lightweight saves in format-24 worlds support positions beyond the legacy 100-unit limit.
  • Sculpted source stays in the authoring document and publication. Unloading is a residency operation, not deletion. Reloading uses the same immutable artifacts.
  • Authoring reuses admission only for exact, privately tracked, deeply frozen document objects. Fresh external documents still undergo checksum, world and source validation; a matching supplied hash or caller-applied freeze cannot bypass it. Newly sealed documents are canonicalized once so export identity agrees with a fresh read. Reapplying the validated active document returns without recompiling both worlds or interrupting travelers.
  • Large world-replacement undo commands use replacement-textures.1 to store shared images once. Expansion verifies closed references and reserves the existing world byte budget before copying images. Normal image/model admission still verifies the payloads. Neither artwork resolution nor the 8 MiB durable-undo budget was reduced or raised to fit the larger landscape.

#Verification and remaining work

The full headless run passed 932 tests with 68 skips and no failures. Fourteen final focused terrain/template checks passed, including independently traveling native simulations, exact checkpoint recovery, movement holds, cancellation, saved edits, deduplicated replacement undo across restart, identity remapping, bounded image expansion and rejection of a residency choice after a traveler reverses direction. Existing LOD variants were verified to retain every boundary edge, neighboring normals and full collision geometry.

The final browser journey passed on physical Intel Arc B390 WebGPU (about 1.6 minutes) and SwiftShader (about 2.5 minutes). Both travelers cross ground boundaries, one using main-thread graphics and one using Worker graphics. It also covers editor settings, reload, saved undo/redo, publication and position retention on reload. The editor is closed after publication during the two-player portion. An initial software run with the editor also rendering exceeded the 180-second aggregate deadline during cleanup; its trace is retained. No deadline was raised. Reviewed player view.

The production build passes, including engine 474901fb9029eaf688fb298caf059399f011c0280af19e9a4ab652e637fbb1e4. A detached packaged CLI passed template/check/export/import/check/compile for format 24 and all 16 chunks. A temporary-filesystem quota interrupted the first export; retrying its output in workspace scratch succeeded. Logs and temporary artifacts are in .data/terrain-streaming/. The isolated browser command is npm run test:browser:terrain-streaming and is included in CI.

The background-loading follow-up passes 22 distinct focused terrain/travel checks (21 in the first run, with the new cancellation case and two handoff cases checked afterward). Both physical-GPU and software-WebGPU browser journeys also deliberately hold each traveler's first new ground download until they move at least two more metres, then verifies both main/Worker graphics crossings and reload retention. The production build passes with engine 20022b34b2be55fcf632af55067c1b875c0f22a0e68f75be6ad01c2c7b4447e6. Its first browser attempt was interrupted after finding an incorrect test URL matcher; the corrected hardware and software tests passed without changing deadlines. Logs are retained with the background- prefix in the same scratch directory. The earlier full headless and packaged-CLI evidence above predates this player handoff change.

The GPU-staging follow-up passes 41 distinct focused checks across native terrain/travel, streamed simulation, graphics ownership and portal rebinding. Physical-GPU and software-WebGPU browser checks prove both travelers move and both main/Worker renderers submit frames while their prepared catalog acknowledgements are deliberately held. Three native catalog journeys also pass on both profiles, covering rejected final mesh references, cancellation, pixel-identical retained scenes, fresh-scene publication and disposal. Production build passes with engine e33f45abda0865d12cd6ee22b0dda1a528adc2f2ddb5ca95b4a208e707c3862d. Results and screenshots use the graphics- prefix in the same scratch directory. The native catalog journeys now have a dedicated CI matrix entry. These checks do not replace the outstanding sustained-performance/device matrix qualification.

Terrain LOD now uses explicit presentation format 17 for declared ground. Runtime model preparation certifies oriented boundary edges; any derivative with missing, additional or reversed boundary edges falls back to the original ground mesh. Ground uses conservative box depth and projected pixel error instead of the generic small-object diameter guard. Source/collision geometry and residency identities remain unchanged. In the four-tile High-quality native fixture, visible triangles fall from 2,048 to 1,152 (43.75%), with RGB RMSE 0.0221 / 255, zero large-error pixels, unchanged full-detail shadows, exact damaged-LOD fallback and exact recreation pixels. Physical-GPU main and Worker results match. Reviewed ground fixture, selection policy.

The LOD headless run passed 937 checks with 68 skips; four engine-pinning setup checks hit the system temporary-filesystem quota and one voxel persistence check reported a storage failure. Retrying the two affected files with workspace TMPDIR passed all 11 checks, including those five (942 distinct passing checks overall). Four native LOD browser checks pass on hardware and software, and the multiplayer editor/publication/crossing journey passes on both profiles. The production engine build is 9fe338f4c3ab64c8beec2c974caaaae401b4d1537c029f6e0d619d2a2d47da89. Logs/results use terrain-lod- in the scratch directory; native LOD checks have a dedicated CI entry.

The distant-ground follow-up (world format 25) preserves full-format-24 compatibility and upgrades new uplands/edited streaming settings explicitly. The uplands overview uses 2,176 triangles and 1,616 vertices, versus 8,192 triangles and 4,624 vertices in all detailed chunks, with one shared atlas and no collider. Its complete portable document is approximately 11.15 MB, within the existing 16 MiB limit. A native fixture with four detailed and twelve overview sections keeps all sixteen sections visible. On the physical GPU, both main and Worker paths measure RGB RMSE 0.066 / 255, zero pixels with channel error above 32, identical recreation pixels, and model GPU allocation 775,960 → 535,960 bytes versus its fully loaded catalog. These are fixture-specific measurements, not a general frame-rate or total-memory guarantee.

Full headless validation passes 943 checks, with 68 skips; an additional targeted assertion rejects a stale overview after a shared-border sculpt. The production build passes with engine f214f9c7feed84673c9dff797734127866e71e81b717578cfd1ee09c94e7b933. The detached packaged CLI passes template/check/export/import/check/compile with unchanged exported world hash. Both main/Worker overview tests and the two-traveler save/undo/redo/publication/crossing/reload journey pass on physical and software WebGPU (hardware journey 1.9 minutes; software journey 2.8 minutes under the unchanged three-minute deadline). Logs and result files use the overview- prefix in .data/terrain-streaming/. Reviewed player view. Visual review retains the final art work: sparse procedural vegetation and overlapping travelers at the initial spawn are still visible in the arrival capture; ground continuity does not constitute art approval.

Shared editing now has a native and two-traveler browser journey covering sculpt/paint, saved undo/redo, independently resident catalogs, detailed/overview exclusivity, actual detail eviction and reload, position retention and reconnect. The final software journey passes in 5.1 minutes; initial room opening takes 4,987 ms under the unchanged 20-second check. Sculpt/paint, Undo and Redo Apply take 24,160 / 24,345 / 23,170 ms in that software run. These are single observations, not general latency guarantees. An initial software attempt missed the initial-room deadline; redundant document admission and active-document recompilation were removed without weakening validation or raising the deadline.

The current full headless run passes 946 checks, with 68 skips and no failures; type/boundary checks and production build pass. Engine build: 3d64161f26290c78eccc08e0de0d3e5b5d9a679c7ca10510c84a86ed6ba6585a. New admission checks reject forged hashes and invalid frozen copies, preserve input ownership and deep immutability, and verify serialization identity after reopening. An initial optimization omitted canonicalization when sealing; five focused failures exposed legacy ordered-JSON content identities differing after restart. Canonicalizing once before admission fixes those failures, and all 39 checks in the affected authoring/development suites pass. The additional active-document no-op check is included in the full run. The fresh standalone authoring bundle checks the earlier format-25 document and exports a byte-identical world, retaining compiled world hash 020a13968340769a8ed79bfa09661fb9587666ce459aa416956e67713bbcda42. Logs and reports use shared- in .data/terrain-streaming/.

The final physical Intel Arc B390 journey passes in 14.0 minutes, including 120 terrain residency handoffs over 626,827 ms of sustained travel and three additional shared Undo/Redo cycles. Both main and Worker travelers receive the revisions; the moving traveler reconnects to the latest revision. Across all twelve round-trip samples, the stationary Worker traveler holds 157 buffers, 5 textures and 3,114,112 model GPU bytes; the moving main-thread traveler holds 165 buffers, 5 textures and 3,154,112 model GPU bytes. Content reservations are 3,009,709 bytes for the stationary traveler and 3,096,844–3,181,564 bytes for the moving traveler, returning to the same range after edits. Every sample has one prepared place, no graphics recovery and at most two pending frames. Initial room opening takes 2,922 ms; the nine shared Apply operations take 16,126–16,162 ms. These are observations from this workload, not general performance guarantees. The 1440 × 900 journey does not measure JavaScript heap/array buffers or qualify the separate 1080p frame-time and 100-place-crossing gates.

An earlier hardware soak stopped at 60 handoffs because its resource sample captured the valid old/candidate lease overlap after arrival. The test now requires a fresh post-arrival content frame with one prepared place and no queued/fetching/pending content before sampling, with a ten-second deadline; it still fails a lease that does not settle. No resource leak was established by that first sample, and its raw report is retained. The final resource and timing attachments are copied under shared-soak-final- in the scratch directory. Visual review confirms continuous ground and shared revision feedback; the sparse procedural vegetation still requires the final art pass.

The next hardware measurement separates background preparation from the final publication hold. Before the atlas-reuse correction, fifteen terrain handoffs take 1,264–2,822 ms overall (median 2,050 ms), with a final reported hold of 64–104 ms (median 89 ms). The hold starts after the synchronous native rebase, so it does not count that worker stall. These local event timings are not a frame-time qualification. Format-7 LOD admission was dropping the catalog's explicit texture-reuse policy when reading the base artifact; it now forwards that policy, avoiding repeated validation of identical terrain atlases. Cache lookup still compares complete texture data, standalone admission retains its normal validation path, and an altered atlas is rejected. The regression first failed with two redundant atlas digests, then passed with none after a validated cache warm-up. All 16 focused asset checks and the integration run (947 passed, 68 skipped) pass; production engine build is b2d12db28f611b7ab4650fd9b4dd5c4bb3e9ed7d107c9f6a081bcf54a1eb0afa. Raw measurements and logs use handoff- in the same scratch directory. The final hardware journey passes in 2.4 minutes with main/Worker graphics, shared sculpt/paint, Undo/Redo, eviction/revisit and reconnect. Initial room opening measures 1,923 ms, and Apply measures 11,119 / 10,116 / 10,120 ms versus 16,169 / 16,129 / 16,158 ms in the baseline. Sixteen final terrain handoffs have median total time 1,493 ms and median reported hold 92 ms (85–102 ms range); the baseline contains fifteen handoffs because travel samples allow slight position overshoot. The same scripted workload thus shows less preparation time, while the final hold remains. These single before/after runs do not prove a universal speedup. Visual review retains continuous ground and expected shared-revision feedback. The earlier software journey and long hardware resource soak predate this admission-policy fix; neither was repeated for this scoped correction.

The retained-memory follow-up passes in 13.2 minutes, including 143 terrain handoffs over 632,175 ms, three repeated shared Undo/Redo cycles and final reconnect. Both player renderers run at 1920 × 1080 on the physical Arc B390. Thirteen censuses (initial plus twelve rounds) account for six distinct, unchanged isolates: the editor page, Worker-player page/simulation/graphics workers, and main-player page/simulation worker. No missing worker is represented as zero memory. The sampler's isolated native-browser check observes a 16 MiB worker ArrayBuffer being allocated and reclaimed; type/boundary checks pass. Production runtime code is unchanged from the preceding validated build, so the full suite/build were not repeated.

After two warm-up rounds, retained memory ranges are:

Browser group JavaScript heap, MB Backing storage, MB
Editor 74.20–74.41 7.472456–7.472462
Worker player, three isolates 39.81–41.55 41.816–41.865
Main player, two isolates 38.06–41.66 35.218–35.330

MB here means 1,000,000 bytes. The editor ends 204,684 heap bytes above its third-round sample; the Worker-player group ends 155,372 bytes above, and the main-player group ends 2,054,832 bytes below. These observations show limited retained client memory in this workload, not a proof of zero growth or the full 100-place-crossing memory gate. They exclude server/process-native memory. GPU buffer/texture counts remain 157/5 and 165/5, with unchanged model allocations and no recovery. Rolling 600-frame p95 intervals are 16.7–16.8 ms; individual sampled render CPU times are 1.7–3.4 ms. These are not whole-run percentiles or GPU execution times and do not qualify the no-DevTools-overhead desktop target. Reports, complete isolate samples and helper-check output use memory- in .data/terrain-streaming/.

Remaining qualification and extended scope: sustained frame-time and broader memory qualification beyond this terrain workload. Detail changes may still be visible, and the handoff retains a short final frame-drain/publication hold. A visible portal destination retains its full ground; vegetation/species models remain ordinary place dependencies. Full-detail vertices and textures remain resident for selected chunks; the overview is a separate compact artifact, while selected chunk detail variants still share full-resolution vertices. Larger-than-16-chunk worlds and multilevel distant residency remain open. LOD draw savings depend on camera, resolution and terrain error, and do not establish frame-rate or total-memory improvements. Desktop/mobile frame-time, JavaScript-heap/array-buffer and 100-place-crossing gates remain distinct from terrain handoffs. The final character/environment/arrival art pass also remains required.

End of Terrain streaming