Measured room health and admission

Persistent rooms report bounded operational measurements alongside their fenced lease. Regional routing uses those measurements to keep new arrivals away from a room under pressure. Existing travelers, saved returns and admitted durable work continue under the normal lease, capacity and durability rules. A soft pressure signal does not kill a room or move its inventory elsewhere.

#Measurements

GET /health on a standalone persistent host includes metrics.version: 1. This is a public numeric diagnostic: it contains no player IDs, names, credentials, inventory, SQL text or exception stacks. Routed children expose the same measurements only over their private Unix sockets. The regional gateway's public /health remains a coarse coordinator/database readiness check; it is not a claim that every child has spare capacity.

Duration fields contain count, mean, maximum and p95. Measurements use sixty fixed one-second buckets with fixed histogram boundaries. The p95 is a conservative histogram upper bound capped by the observed maximum. A bucket expires at a second boundary; a sixty-second window covers the current second and fifty-nine previous seconds. There are no unbounded raw sample lists or per-actor metric labels.

Reconnect recognition uses a sixty-second horizon and retains at most 1,024 actor IDs internally. Expired entries are pruned on the next successful hello; process exit discards the map. It counts repeat successful hellos, not failed transport attempts. A process restart resets observations; a return after the observation window is counted as a join. Actor IDs never appear in the metric response. This is a bounded operational signal, not an account/session analytics system.

The transaction timer stops before post-commit delivery callbacks. A lost receipt after successful COMMIT remains a successful database commit. Rollbacks contribute to the failure counter and never to successful commit latency. Ordinary pool reads are not commit samples. Lease renewals use transactions and do contribute to these measurements. All transactions sharing that database object contribute; production children each own their database pool.

#Admission policy

New admission pauses when any of these conditions hold:

• The maximum maintenance-timer delay in the most recent ten buckets exceeds 250 ms.
• Trusted tick p95 exceeds 25 ms, with at least thirty tick observations in those buckets.
• There are at least 48 pending durable jobs, or the oldest admitted job exceeds 1,000 ms.
• A successful transaction in those buckets exceeded 1,000 ms end to end.
• RSS exceeds DW_ROOM_RSS_LIMIT_MIB, default 512 MiB, configurable as an integer from 64 to 8,192.

Completed duration spikes age out after roughly nine to ten seconds; current queue and memory pressure clears when the measurements recover. The sixty-second diagnostic window retains the spike longer than the admission window. Reasons are event-loop, simulation, durable-queue, storage, and memory.

These are conservative initial admission policies, not measured production SLOs or a certified capacity envelope. RSS is a soft admission budget, not an OS/container memory limit. There is no automatic process kill for pressure; the existing watchdog, lease fencing and storage-failure behavior remain separate. Existing connections can still consume resources while admission is paused. Operators must set container/host resource limits independently.

A new regional arrival skips a responsive but pressured room and can select another compatible instance. A direct invitation to a pressured room gets a retryable unavailable response. A valid saved return and its reconnect ticket can still enter their original room, subject to its player cap. Already admitted work finishes without being reclassified halfway through a transaction.

Old retained engines without metrics.version: 1 are unmeasured and excluded from new matching. Their saved travelers can still return. Creating a new instance additionally requires the retained runtime's persistentABI.health: 1 declaration, before allocating persistent room state. Deploy a current candidate or explicitly upgrade a stopped room to enable measured new admission. No database schema migration or automatic room upgrade occurs.

#Regional operator inspection

Set a separate 32–256 character DW_OPERATIONS_TOKEN on the regional gateway. This is an operator secret, distinct from the room passphrase. It is optional; without it, the inspection endpoint is disabled. compose.rooms.yaml passes this variable and the RSS budget to the deployment.

From a trusted terminal, use your regional HTTPS address:

curl --fail --silent --show-error \
  -H "Authorization: Bearer $DW_OPERATIONS_TOKEN" \
  https://rooms.example.com/api/operations/health

The response contains region, capture time, active-process count, configured process cap, and one row per currently running room: immutable engine/release identity, lease epoch and remaining time, player count/cap, new-admission status and numeric measurements. A failed private probe is explicitly unavailable with metrics: null; missing data is not a zero-valued healthy sample. No actors, return tokens or passphrases are included.

Inspection does not start or reserve rooms. Concurrent inspections share one bounded probe pass; each private probe has a 750 ms deadline. Normal gateway connection, request and per-address rate limits also apply. Save the JSON with incident records if needed. This endpoint does not replace historical monitoring, alerts, log aggregation or a complete downloadable support bundle.

#Verification and remaining work

npm run check
npm test
npm run test:control
npm run test:browser:rooms
npm run test:browser:lifecycle:cooperative
npm run test:browser:trusted

Native acceptance uses a deferred PostgreSQL trigger that delays actual COMMIT, a held room row lock that stalls durable work, and real room-process event-loop stalls. It checks separate queue and commit observations, saved return admission under pressure, responsive-room exclusion, operator authentication/privacy, and missing-measurement compatibility. The existing recovery/backup/upgrade suite remains required because transaction timing touches the shared database wrapper.

The reproducible room capacity trial measures native host costs with staged occupancy and simultaneous reconnects. See health acceptance evidence. Physical hardware thresholds, production load and long-soak qualification, Docker/Coolify execution, automatic restart timing, remote CI, historical telemetry and alerts remain open. This increment does not finish a numbered roadmap milestone or the full engine plan.

#Startup and test-process scope — October 28

The event-loop drift clock starts with the maintenance timer, after content preparation. Time spent preparing a server before it can serve requests is not itself a serving-time stall; later stalls retain the same admission thresholds. The native regression covers both cases. This does not clear RSS, database, queue or already recorded serving-time pressure.

A diagnostic furnished-world process with an unrestricted Node heap reached about 1.1 GB RSS. Repeating with the supervisor's 256 MiB old-space cap observed roughly 450–530 MB RSS, but initial admission still encountered event-loop pressure while the wider qualification workload was running. This is not a sustained memory or capacity qualification. Defaults have not been raised from that observation. Bounded state/replay fixtures exclude imported character art; production art/content and rendered arrival checks remain separate.

The final state/recovery fixtures now construct the common story directly, without first importing and discarding production GLB art. Production defaults still load the furnished arrival, verified by the art/storage checks; the real production-content admission check retains its textured assets and allows the existing pressure cooldown before admission. The smaller fixtures do not qualify the furnished world's memory or startup latency. Existing admission thresholds, supervisor heap cap and configured RSS budget are unchanged.

The retained-health acceptance also confirms the existing conservative cold-host policy: missing observations exclude new arrivals and do not trigger another cold host while current headroom is unknown. A saved traveler can still return; a separate already-running healthy compatible room remains eligible.