A persistent room keeps its identity, engine release and PostgreSQL ledger when its process restarts. A new owner receives a higher fenced lease epoch. This applies to both cooperative and trusted static-world profiles. It does not implement live host-to-host simulation transfer or a multi-region room directory.
#Graceful restart
Send SIGTERM or SIGINT to the room process, or stop its service normally. The bootstrap forwards the signal to the retained runtime. The runtime:
- Stops new HTTP admissions, socket upgrades, inputs and transactions; readiness becomes HTTP 503 with
lifecycle: "draining". - Announces the reconnect to existing travelers. Both persistent profiles pause simulation while disconnected. The player status reads Keeping your place · Reconnecting… during an announced drain.
- Finishes work already admitted to the room queue. Trusted inputs already admitted continue at the normal 30 Hz clock until their action boundaries and input queue settle.
- Freezes motion and saves each connected traveler's final accepted position through the ordinary fenced ledger transaction. Saved messages are sent only after commit. Disconnecting during this drain does not discard that traveler's pending final save.
- Stops renewal, retires the lease and closes sockets with recoverable code 1012. A replacement process can acquire the room immediately. Clients reconnect with the same credential, consume a new baseline and retry unconfirmed operations with their original IDs.
The room allows ten seconds for the drain and bounds socket closing to another 250 ms. The runtime has a fifteen-second hard exit deadline. Independently, its bootstrap allows eighteen seconds after forwarding a stop signal, then kills the owned child if it has not exited. The Compose service's twenty-second stop grace exceeds those limits. A deadline or storage failure produces a nonzero exit; it does not claim the final movement was saved. Unacknowledged work may have committed and must be retried by its original transaction ID. Previously acknowledged ledger changes remain durable.
Cooperative presence follows accepted interaction/checkpoint positions unless a newer pose arrived while the transaction was committing. This prevents shutdown from replacing the latest committed position with stale presence, while preserving movement received during a slow commit.
A graceful restart captures positions and the existing durable ledger. It does not preserve an in-flight jump's velocity or arbitrary dynamic physics/script state. An abrupt kill can still lose motion since the four-second autosave; committed inventory, rewards and travel are retained. These are distinct recovery paths.
#Fatal storage or ownership failure
The runtime stops publication and closes connections if durable work, simulation or lease renewal fails. The production process observes this failure and exits with status 1. It no longer remains alive indefinitely serving 503. restart: unless-stopped in the supplied Compose configuration lets the deployment supervisor restart an unexpectedly failed process. A deliberate service stop stays stopped. Local npm run dev:persistent / dev:trusted may need to be started again; the launcher is not an independent production supervisor.
The room requests a 15-second lease and attempts renewal every second, leaving more headroom for busy physics and browser workloads. The host tracks a conservative monotonic expiry derived from the start of its last successful lease request. After an event-loop stall, callbacks check that bound before admitting work, publishing motion or returning readiness. A late renewal response cannot revive a host already marked failed. Database lease checks remain the authority for every durable commit.
A new process waits for any prior lease to expire if retirement was impossible, restores the pinned world/engine and admits the original traveler. An ungraceful crash can therefore require up to the 15-second lease duration plus retry and startup time before recovery. An engine mismatch or incompatible database still fails closed; recovery never substitutes new content for a retained release.
Retries of the same transaction on one connection share its in-flight queue entry. After it finishes, a retry is processed normally and can recover the saved receipt. Reusing an in-flight operation ID for a different command or input boundary closes the connection. This prevents duplicate retries from consuming the pending-work budget; it does not bound storage latency or replace the durable ledger's idempotency checks.
#Runtime stalls
The bootstrap monitors its retained runtime from a separate process. Node loads a small IPC responder before executing the verified entry point; replies run on that runtime's main event loop. This works with an older retained entry that has no heartbeat API. It does not modify retained engine files, select a different engine, or add a database migration. Deploying the new bootstrap is sufficient for this monitoring; other runtime features still require their normal explicit engine upgrade.
The parent checks once a second with one unpredictable challenge outstanding and a five-second reply window. Replies must match that challenge and arrive within its window. A main loop that remains blocked is killed with SIGKILL, and the bootstrap exits 1 after observing the child's exit. Scheduling and observation add overhead to that reply window. This bounds a stalled child independently of its own timers and makes the existing Docker restart policy applicable. It does not claim that an unhealthy Docker healthcheck by itself restarts a container.
Only the spawned child is signaled: the watchdog never selects a process by listening port. If the observer itself resumes after a scheduling gap longer than three seconds, it requests a fresh challenge and gives a full reply window. A delayed observation alone is insufficient evidence of a dead child. If the bootstrap disappears, a responsive child exits when its IPC channel disconnects; a blocked orphan still requires the operating system/container supervisor to terminate it.
This is event-loop liveness supervision, not a security sandbox or proof that every asynchronous operation is making progress. It does not monitor the bootstrap itself, impose a total process-memory ceiling, or implement an internal restart loop. Supervised room children use a 256 MiB V8 old-generation heap to leave room for young-generation memory, native physics and compiled code beneath the default 512 MiB RSS admission budget. This makes collection happen earlier; it does not replace RSS admission checks or qualify maximum room capacity. A forced kill uses the abrupt recovery path described above: PostgreSQL fencing and idempotent retries protect durable work, while movement can rewind to its last autosave. Production deployment still needs its restart policy, storage, readiness checks and operational monitoring.
Trusted clients keep at most sixty unacknowledged input frames. When that prediction window fills, they now pause prediction and reconnect with the same traveler credential instead of treating the delayed server as a fatal simulation error. The old connection is fenced locally before closing, so delayed callbacks and a stalled close handshake cannot replace the new baseline. The next authoritative stream resets prediction. Pending durable commands retain their original transaction IDs and input boundaries for exact retry. A seven-second transport timeout also covers an unfinished connection handshake. This client behavior is packaged with the engine and needs a new room or an explicit engine upgrade for an existing pinned room; deploying only the bootstrap does not replace its old client.
#Readiness and deployment
GET /health returns 200 only for a locally live owner, with ready, lifecycle, pending, leaseRemainingMs, authority profile, player count, durable revision and epoch. Draining or failed rooms return 503. This is not full operational telemetry: tick/commit latency distributions, memory pressure, load policy, routing and production recovery objectives remain open.
Build the candidate first. During a content upgrade, wait for the old process to exit and then create a fresh upgrade preview: its final movement saves can change revisions and invalidate a preview made before draining. Use the persistent-room upgrade workflow. Do not manually edit leases or migration records in a deployed database.
No database migration is added; the current runtime uses schema 5. Existing rooms still run their pinned engine until explicitly upgraded. Use a new room ID to test new runtime features without changing an existing room. The bootstrap watchdog also supervises older pinned runtimes as described above.
#Verification
npm run check
npm test
node --import tsx --test tests/host-watchdog.test.ts
npm run test:control
npm run test:browser:lifecycle
npm run test:browser:lifecycle:cooperative
npm run test:browser:trustedThe isolated PostgreSQL tests cover admitted transactions during drain, final trusted/cooperative poses, deadline fencing, original-traveler recovery, real database outage and process exit, and an event-loop stall beyond lease expiry with no late motion publication. The stall check was rerun with the 15-second lease; broad deployment and load qualification of the new window remains to be done. Existing transaction/crash/backup/upgrade tests remain in the full control suite.
The lifecycle browser journey uses two travelers, the packaged retained engine/bootstrap and 75 ms transport delay each way. It verifies graceful exit 0, failure exit 1, both clients reconnecting without page reload, frozen simulation while unavailable, preserved identity/inventory and final position, and higher lease epochs. Its Linux-only fault controller also suspends precisely the bootstrap's retained child with SIGSTOP, waits for the external watchdog to kill it, and starts a replacement. Both travelers must receive its new epoch and keep their identities and committed lantern. The harness delays close notifications after previously received messages so simulated latency preserves WebSocket delivery ordering. Docker execution itself, automatic Coolify restart timing and broad device/load behavior are not qualified by this native-process test.
Native watchdog tests exercise permanent main-loop blocks during startup and play, a blocked shutdown handler, replayed replies, independent child isolation, unchanged legacy entry bytes, normal exit propagation, observer suspension and a responsive child's exit when its parent disappears.
An additional PostgreSQL/socket case withholds authoritative motion for sixty inputs and prevents the old socket's close request from completing. Prediction stops at the bound, the same traveler reconnects on a new stream, the committed lantern remains, exactly one participant is active, and fresh input is acknowledged without a fatal client error.
The recorded final runs and their source hashes are in the original lifecycle evidence and the external watchdog and prediction recovery evidence.
#Admission under measured pressure
Host health adds bounded runtime/storage measurements and pauses new arrivals under pressure. This does not replace lease fencing, fatal storage handling, graceful draining or the external watchdog. Saved returns remain eligible subject to capacity.