Final edition · FPS edition (SCAR) · last updated 10 Oct 2026 10:59 · 32 breakages (7 critical · 10 high · 10 medium · 5 low)
Final edition: project paused 2026-10-10

Where Virtual Matter Breaks: destruction FPS

Engine release/iteration073 (384c388, built 2026-10-05), native client 73.0.1554, CLI 0.6.5. Found while building SCAR, a destruction FPS whose core loop is persistent terrain edits. Same 5W format as the Ashlight report (v33, https://ohge-games.github.io/virtual-matter-breakage-report/): Who / What / When / Where / Why, then impact, repro, mitigation and suggested fix. IDs are F1, F2... Where an entry extends an Ashlight card, the B-number is given. Final edition: project paused 2026-10-10. Every change is published to the play server; "Final state" (below) records the published build and how it was verified, and "Final autopsy" (at the end) is the builder's view of the CLI and MCP tooling.

Edit framing 18QYJ28E5mDi5I1uRKpoY Engine iteration073 · 384c388 Client Native, Windows, 73.0.1554 Tooling virtualmatter CLI 0.6.5 + vm-agent-tools Previous edition Ashlight v33

Final state

Breakage index

#SeverityBreakageStatus
F1CriticalStatic edits reach connected clients through a slow, capped stream; bursts leave clients wrong for minutes (extends B28)Measured; designed around
F2CriticalA connected client can stop receiving static edits entirely until it reconnectsObserved 3 times; mitigated
F3HighClients drop when the server stalls ~15 s or more (terrain save); no reconnectMeasured; mitigated
F4HighNative client loses its connection after a 30-250 s upstream stall, with no auto-reconnectObserved 8 times; open
F5HighClient ray/collision data is LOD-quantised to 0.125-0.25 m a few metres from the cameraMeasured; tolerances sized
F6MediumThe edit guard's budget is per frame: the 4th op in one Lua call throws mid-batchMeasured; mitigated
F7MediumForceStatic() throws on the client (rays and Collide)Confirmed; worked around
F8MediumCLI 0.6.5 files API needs a precondition header on every PUT (HTTP 428 for new files)Fixed in vm-agent-tools
F9LowRe-parenting an object keeps its world transform (child cameras land at the parent's feet)Worked around
F10LowServer:GetStreamingStats() reports STALLING / finalRatioPct 0 for healthy clientsDocumented
F11MediumCast:Ray with a zero-length vector throws and kills the calling loopGuarded everywhere
F12CriticalEvery static edit costs one 30-50 ms server frame whatever its size; >~0.6 edits/s breaks a p99 < 33 ms targetMeasured; governor shipped
F13HighThe server-side screenshot client goes stale after a map-scale rebuild (half the map missing)Observed
F14CriticalA freshly joined client receives gameplay edits 12-45 s late; 4 of 10 never arrived at the 2.5 ops/s budgetMeasured; designed around
F15HighAn object name set after creation (e.g. after Scene:CloneObject) never reaches clients: clones keep the source nameMeasured; worked around (ids)
F16HighScene:CloneObject of a 12-part rig costs ~110 ms of server main thread (~9 ms per voxel part); new objects double the frame time while they streamMeasured
F17CriticalClients draw large terrain regions as grey placeholders with wireframe edges while collision there is correct; the holes follow the viewOpen; the LOD-bubble mitigation did not help and was removed
F18Highimagine() jobs are silently orphaned by any Lua reset (a .lua push): the progress script stays forever, no error, no assetObserved 6 times
F19MediumSmall per-frame Lua garbage (~7k tables/s) produced 10-14 ms client GC spikes, charged to whichever script allocatesMeasured; fixed in SCAR
F20LowScoped zoom (FOV 14) still renders distance-based LOD, so a 4.6x scope magnifies coarse voxelsObserved
F21MediumOn a client, VoxelData.data, .asset and .isLoaded read nil/false even for objects it rendersMeasured; worked around
F22HighA dynamic voxel object streams to a client only near its SERVER position; objects a client poses elsewhere stay empty or coarseMeasured; worked around
F23CriticalServer frame spikes (75-100 ms, 1-2.5 per second) come from streaming to connected clients; zero with no client, and a 4-hour client causes about twice the spikes of a fresh oneMeasured; partial workaround (client restart)
F24MediumScript-created NewUI root windows read visible = false on a fresh native client without any script hiding them; panels that are never re-set stay goneObserved; worked around (re-assert 5 Hz)
F25LowAn object posed on the client in LateUpdate against the camera appears one frame behind it: a held viewmodel trails the view while strafingObserved; worked around (one-frame lead)
F26CriticalEditor vs play: a PLAY client gets pooled child objects unparented and never gets their empty parent objects; the editor client gets the whole hierarchy, so it all works in the editor and breaks after publishMeasured (client diagnostics); worked around (part ids)
F27HighEditor vs play: the published scene does not carry script attachments made in the editor's live scene, so new scripts silently don't run on play serversMeasured; worked around (self-repair at boot)
F28HighEditor vs play: no tool reaches a play client (no client Lua, no screenshot, no errors), and the CLI cannot publish; play-only bugs are invisible to the agentTooling gap; worked around (client diagnostics RPC)
F29Lowvirtualmatter open client self-update fails from Git Bash on Windows (tar: Cannot connect to C: resolve failed)Worked around (run from PowerShell)
F30Mediumscreenshot --client --at moves only the camera; streaming stays with the player, so shots away from the player show holes and placeholder terrain, and the server-side screenshot client can be stale (F13)Observed; no visual A/B possible away from the player
F31MediumEditor vs play: a NewUI window opened with keyboard focus left a browser player unable to close it with keys (Esc / U never reached the game); in the editor's native client the same menu was closed by script during testing, so it never showedObserved; fixed (click-to-close, no focus), published and verified in a browser on the play server
F32MediumThe guide the CLI writes into every folder documents push, fetch and three MCP file tools; the published CLI (0.6.5) has none of themConfirmed 10 Oct; worked around (own push wrapper)

The 5Ws, per breakage

F1

Static edits reach connected clients through a slow, capped stream

CriticalWhy: measured↑ index
Who
every player in a world whose terrain changes during play - the whole premise of a destruction FPS.
What
server-side static edits arrive on a connected client 1-4.5 s later when the stream is idle (6 single 0.8 m removes: 0.9, 4.5, 1.5, 2.5, 3.1, 1.3 s). Under load they queue: a fresh client that is still streaming the map received a single 0.8 m remove somewhere between 30 s and 3 minutes later. After bursts (40 removes at 5/s in one 8 m cell) the client was still wrong at 12-32% of sample points 75 s later, and an 18-crater burst (r 2.5 m) never arrived at all on a client that stayed connected for 15+ minutes (the client showed the cell flat; the server had craters to the pad floor). A client that connected later saw everything at once.
When
whenever edits are made while clients are connected; worst for bursts and large edits, and while a client is still loading detail (a fresh client was still pulling 68-107 KB/s eight minutes after joining a heavily edited 48 x 32 m pad).
Where
static voxel streaming (_AmStreaming). Server:GetStreamingStats() shows each client in a STALLING lag state with stallingSendBPS = 500000; the client receives 24-110 KB/s. A paint over the stuck cell, an LOD bubble, moving the camera 600 m away and back, and AE:ReloadGeometry() did not resync it.
Why
edits are not pushed as small deltas with priority; they wait behind (or are coalesced into) bulk chunk streaming that is rate-capped per client. Under a burst, intermediate versions appear to be dropped and the final version is not always re-sent.

Impact

what a player sees and collides with can differ from the server for seconds to minutes after any dig or blast. Collision divergence drops players into holes that aren't there (or holds them on ground that is gone), and cover differs per player.

Repro

bench/run_bench.py (A1-A4 configs: 40 removes at 5/s, r 0.5-2.5 m, one cell, client camera 9 m above). bench/lat1.py <client> x z for one edit.

Mitigation shipped in SCAR


• every gameplay edit goes through one scheduler: 4 ops/s global, a 6-op burst, 0.8 s spacing per 8 m cell, a 4 s settle window between remove and add, a 24-op queue cap, and 2 ops per frame at most;
• digs are single capsule strokes every 0.9 s (not continuous), and craters are single spheres;
• bullets never edit terrain;
• a divergence detector probes clients every 5 s at recently edited spots and under their feet, teleports a player standing on ground the server doesn't have, and tightens the budget while divergence is open;
• hit occlusion is decided on the server's terrain only.

Suggested fix

version each static chunk and push edits as small high-priority deltas; re-send any chunk whose version a client hasn't acknowledged; expose a per-client "resync region" call and a per-edit delivery callback or acknowledgement.

F2

A connected client can stop receiving static edits entirely

CriticalWhy: observed↑ index
Who
any connected client after a burst of terrain edits.
What
native client #1 received none of 40 removes made at 5/s, nor any later edit (3 pillars of 4 m concrete, several holes) for 15+ minutes. Client #4 showed the same after an 18-op burst. The server-side screenshot client showed every edit.
When
after edit bursts while connected; seen 3 times in one session.
Where
static streaming to that client. Reconnecting fixed it every time.
Why
unknown; consistent with the flow controller (senderRecvLagDeviation pinned at 16383, finalRatioPct = 0) never leaving its stalled state for that client.

Impact

a player in that state plays on a different map for the rest of the session.

Mitigation

the edit budget (F1); the divergence detector counts divergences older than 90 s as unrepaired and tells the player to rejoin (F5 key in the browser).

Suggested fix

recover the flow state, or force a resync of dirty chunks, when a client's streaming has been stalled for N seconds.

F3

Clients drop when the server stalls for ~15 s or more

HighWhy: measured↑ index
Who
anyone connected or connecting while the server saves its terrain.
What
a native client that connected just as SCAR's empty-server maintenance ran (reclaim, then a 17.3 s terrain save) was disconnected and did not reconnect. Its log shows "New max ui client message sending time: 11057 ms", then "Connection to server lost".
When
during any main-thread stall longer than roughly 10-15 s (terrain save B19).
Where
client/server TCP keepalive.

Impact

the empty-server save window is not safe for players who arrive during it, and the native client offers no reconnect.

Mitigation

maintenance now checks for connected AND pending clients on every step and abandons the reclaim/save when anyone arrives. The traversal check that took one frame (7,569 rays) now runs as one row per frame.

Suggested fix

async terrain save (B19); a longer timeout during a known save; automatic reconnect in the native client.

F4

Native client loses its connection with no auto-reconnect

HighWhy: observed↑ index
Who
a native client in a long session.
What
two disconnects not caused by a server stall: 16:40:03 (after "New max ui client message sending time: 29600 ms") and 17:00:06 ("TcpClient::SendMsg: failed to send a message to server" with no preceding gap in the client log; the client was at 88 fps). The window stays open, showing a frozen scene; nothing reconnects.
When
6 and 17 minutes into play. Update (8 occurrences; the 8th at 19:00 after a 51.8 s stall, with no screenshot involved): the client log shows "New max ui client message sending time" climbing to 29.6 s, 51.8 s, 58 s and 253.6 s before "Connection to server lost"; the server then reports the client gone. The client keeps rendering at 88-100 fps throughout, so it is not frozen; its upstream channel is blocked. Three of the seven followed a screenshot --client capture within seconds to minutes; the others came while the client was streaming terrain heavily (150-216 KB/s downstream, footprint 346 MB and growing) during edit benches. A hidden (throttled) browser tab also fell behind and its divergence probes failed.
Where
TcpClient.cpp:197/317; Client.cpp:2654 ("ui client message sending time").

Impact

long tests and long play sessions end silently; the server keeps the player for a while (B49 ghost).

Mitigation

none possible in game code beyond tolerating rejoin (SCAR restores the player's team, class and score by name).

Suggested fix

automatic reconnect with session resume; surface the disconnect reason.

F5

Client ray and collision data is LOD-quantised a few metres from the camera

HighWhy: measured↑ index
What
over an untouched flat pad, server and client rays agree exactly; over dug ground 4-11 m from the client camera, client ray heights are higher than the server's in steps of 0.125 m (and some 0.25 m), even with an LOD priority bubble around the area.

Impact

any client-side check of terrain (movement, divergence probes, impact effects) needs a tolerance of 0.3 m or more; "divergence" measured with tighter tolerances is mostly LOD.

Mitigation

the divergence detector uses 0.6 m; the bench reports both 0.3 m and 0.6 m mismatches.

Suggested fix

an API to read the server's height for a column, or a client query at a forced LOD.

F6

The edit guard's budget is per frame, not per op

MediumWhy: measured↑ index
What
a pad reset issued 4 large ops in one Lua call (remove, add, remove, add; ~491 baseline-m2 each). The 4th threw "frame voxel budget exhausted (~697 already issued this frame...)", leaving half the pad removed and not re-added.

Impact

any batch (map builders, resets, reclaim) must spread ops over frames and handle a mid-batch refusal; a naive loop corrupts terrain.

Mitigation

every SCAR batch runs one op (or 2-4 small ops) per frame and retries "budget" refusals on the next frame.

Suggested fix

an option to queue past the budget (deferred to later frames) instead of throwing.

F7

ForceStatic() throws on the client

MediumWhy: confirmed↑ index

Cast:Ray(...):ForceStatic() and Collide(...):ForceStatic() on a client raise "ForceStatic() only available on server". Client code must use plain rays and filter dynamic objects with Ignore. Suggested fix: allow it on clients (the client has static data) or document it.

F8

CLI 0.6.5 files API requires a precondition on every PUT

Medium↑ index

Uploading a NEW file with no precondition returns HTTP 428. The vm-agent-tools push-only deploy (built against older behaviour) failed for all new files. Fix shipped in vm-agent-tools/cli-ext/vmpush.mjs: createOnly (If-None-Match: *) for new files, etag re-read on conflict.

F9

Re-parenting keeps the world transform

Low↑ index

child.parent = obj keeps the child's world position, so a camera created at the origin and parented to an eye object ended up at the player's feet. Set localPos / localRot after parenting.

F10

Streaming stats read as stalled for healthy clients

Low↑ index

Server:GetStreamingStats() reports senderRecvLagDeviationFlowState = STALLING and finalRatioPct = 0 for a freshly connected client that does receive data. The fields are undocumented, so they cannot be used to detect F2.

F11

A zero-length ray throws

MediumWhy: measured↑ index

Cast:Ray(p, Vec3(0,0,0)):Run() raises "ray is zero length". Inside a util:schedule chain or an Update loop the error stops the chain silently (the B16 bench's sampler died this way when it picked the same player twice). Gameplay code hits it whenever two positions coincide: a grenade exploding exactly at a chest, a bot aiming at a target standing on it, a projectile at rest. SCAR guards every ray whose vector can be zero. Suggested fix: return nil (no hit) for a zero-length ray, as most engines do.

F12

Every static edit costs one 30-50 ms server frame, whatever its size

CriticalWhy: measured↑ index
Who
every player of a world that edits terrain during play.
What
single gameplay edits through SCAR's scheduler on an idle server (17 ms frames): dig capsule r 0.7 -> worst frame 30-33 ms; frag sphere r 1.1 -> 30-34 ms; breach sphere r 1.8 -> 50-52 ms; Bloom (3 small adds) -> 25 ms. A sphere with 3.9x the volume of a dig costs the same frame, so the cost is mostly per op, not per voxel. Each op also adds ~40-110 ms of update work spread over the following frames.
When
every static Vox op (Remove/Add/Paint).
Where
the commit of a static op (the voxel system flush, LOD rebuild and streaming of the touched chunks).

Impact

a p99 frame budget of 33 ms allows about one spike frame per 100 frames (0.6/s). Worst-case stress (8 players firing 214 shots/s with lag compensation costs nothing measurable; terrain at ~3.5 ops/s pushes p99 to 39 ms, max 51 ms). Even 1 op/s gives p99 36 ms. Real 8-player bot matches (fewer edits) held p99 16.8 ms for 5 rounds.

Mitigation shipped

fewer, bigger dig strokes (1.2 s, r 0.8); a global budget of 2.5 ops/s; a frame governor that cuts the edit rate to 40% while more than 1% of the last 300 frames exceed 33 ms; bullets never edit terrain.

Suggested fix

amortise the per-op commit (batch ops per frame into one flush, defer LOD rebuilds), or move it to a worker; expose a per-op cost estimate.

F13

The server-side screenshot client goes stale after a map-scale rebuild

HighWhy: observed↑ index

After SCAR's clean rebuild (clear all static terrain, rebuild 112 x 112 m: 9.3k jobs, then a dressing pass) the CLI's default screenshot (rendered by the server-side client) showed only the southern half of the map, with a floating wireframe of edges over the north, unchanged across 4 captures over several minutes and two camera angles. Server rays (the verify script) found all terrain; a freshly connected native client rendered the whole map correctly. This is B3/B28 at map scale (remove then add everywhere while connected), and it means the agent's main visual check is wrong after any big rebuild. Mitigation: verify with server rays and a freshly connected native client. Suggested fix: resync the server-side client's stream after large edits, or reconnect it per screenshot.

F14

A freshly joined client receives gameplay edits 12-45 s late

CriticalWhy: measured↑ index
Who
every player who joined a world within the last several minutes (in a 4-minute-round game: everyone).
What
bench/gameplay_edit_bench.py on the dressed Scarfield map, native client joined ~2 min earlier, standing 10 m from the edits: single digger (1 stroke/s, one cell) delivered 9/9, latency avg 14.5 s, p90 24 s, max 27 s; the global budget (2.5 ops/s over 6 cells) delivered 6/10 within 60 s (avg 12 s, max 23 s), 4 never arrived; frag craters (1 per 3 s) 8/8, avg 26 s, max 45 s. On the empty off-map pad earlier, a single edit took 1-4.5 s.
When
while the client is still streaming the map: 74 MB received after 8 minutes, 150-216 KB/s, client footprint 323-346 MB and still growing.
Where
static streaming: edits are not prioritised over bulk detail streaming (F1).

Impact

the digger's own hole appears seconds to tens of seconds after the stroke; craters appear long after the blast; collision differs from the server meanwhile.

Mitigation shipped

gameplay never depends on client terrain: exposure, crate pop-up, hit occlusion and validation all use server terrain; the crate surfaces at the hole's rim; dig progress is reported by the server ("CACHE 1.1 m"); client-side dirt/scorch effects play immediately; the divergence detector lifts players out of terrain they are inside of. Lower-resolution static terrain (16 vpm) would quarter the stream, but no such empty terrain asset exists (32/64/128 only).

Suggested fix

push edit deltas ahead of bulk detail streaming; offer a 16 vpm static terrain option.

F15

An object name set after creation never reaches clients

HighWhy: measured↑ index

The SCAR avatar scale bench cloned "Scar Rig 10" 40 times with Scene:CloneObject and renamed each clone ("Scar Bench Rig k") on the server. The server listed 40 bench rigs and one "Scar Rig 10"; the connected native client listed 41 objects named "Scar Rig 10" and none of the new names, minutes later. Client code that finds objects by name therefore never finds a clone. Workaround: pass Object.id (documented as unique across server and clients) and resolve with Scene:GetObjectById. Suggested fix: replicate name changes, or document that names are creation-time only.

F16

Cloning a multi-part voxel rig stalls the server about 110 ms per rig

HighWhy: measured↑ index
What
Scene:CloneObject of a 12-part rig, with each part bound to a baked .am asset, took 96-113 ms of server main thread per rig (about 9 ms per voxel part, max 144 ms). Ten clones in one Lua call stalled the server for about 1.1 s.

After

with 40 extra rigs (480 dynamic voxel objects), the server median frame went from 16.9 to 29.8 ms while the new objects streamed out. Once streaming settled, the median was back to 16.7 ms with all 40 rigs still present (p99 17 ms), so there is no lasting per-object cost.

Why it matters

every clone deep-copies its assets on the main thread (B29). VM reported that B44 engine crashes happen during rig asset copy and free.

Mitigation

SCAR binds each rig part's asset once at pool build, one part per frame, and never rebinds or frees. Bench rigs are deactivated rather than destroyed.

F17

Clients draw terrain as grey placeholders while its collision is correct

CriticalWhy: observed repeatedly↑ index
Who
every player. The playtester reported it several times: "lots of world loading issues while moving; sometimes part of the map isn't generated while turning", then "large chunks of map aren't rendered".
What
whole regions of the 112 m map render as flat grey surfaces with wireframe chunk edges, with floating bushes and boulders. A screenshot pair 3 minutes apart from the same spot showed the east half missing, then the west half missing while the east rendered.
Where
the native client (73.0.1554), after 10-50 minutes of play on a field with ongoing digging. The client sat at 5.8 GB resident at launch and 7.4 GB after 50 minutes (65 GB host), so memory pressure is not the cause.

Collision is fine

client rays over a 6 x 5 grid inside a missing region matched the server's terrain heights within the known LOD offset (F5). The client has the voxel data and is not drawing it.

Mitigation tried and removed

a third LOD priority bubble around the player (Client:AddPlayerLODPriorityBubbleShape, radius 25 m, size 0.6). The SDK controller's bubbles are 2 m and 4 m only. A four-direction test from the player's own camera, with the bubble on, still showed grey untextured placeholder ground in one direction, and distant soldiers looked blobbier with it. It was removed on 9 Oct. A fresh native client showed fewer grey patches than a long-running one (see F23).

Final status

open. Nothing in game code removes the holes.

Suggested fix

keep a radius around the camera resident regardless of view direction, and expose the render streamer's budget.

F18

Lua resets silently orphan imagine() generations

HighWhy: observed ↑ index
What
generations started with imagine(prompt, Object, opts) never finish if any .lua file is pushed while they run. The object keeps its Gen Progress script indefinitely (observed 25+ minutes) and no error is printed. Four weapon jobs were lost this way, then two more.

Also seen

one carbine job ended with no asset and no visible error. The documented "[imagine] generation failed" line was not seen through the CLI.

Contrast

fast = true, hq = true soldier and weapon jobs finished in 2-6 minutes when nothing was pushed.

Impact

an agent that iterates on scripts while models generate loses the models.

Mitigation

no .lua pushes while jobs are pending, and staging below the map (staging in the air left models visible in players' sky).

Suggested fix

resume orphaned jobs after a Lua reset (the docs say the Shape Manager does this; it did not here), or fail them visibly.

F19

Small per-frame Lua garbage produces 10-14 ms client GC spikes

MediumWhy: measured↑ index

SCAR's procedural avatars allocated about 20 small tables per rig per pose (7 rigs at 60 fps, about 7k tables/s). The avatar script measured 87 of 1,015 frames over 2 ms with 14 ms peaks, although its own work averages 0.3 ms. Reusing the tables gave 2 of 1,744 frames over 2 ms with a 2.4 ms peak. The spikes are incremental GC steps charged to whichever script allocates. Rays (0.08-0.13 ms), localPos/localRot writes (0.0008 ms) and Vec3/Quat construction (0.0003 ms) are cheap. A controlled A/B of client fps against injected garbage was inconclusive (25-36 fps either way, with the window unfocused). Suggested fix: document the GC budget, and expose collectgarbage("step") control or a generational GC on clients.

F20

A scoped zoom still renders distance LOD

LowWhy: observed↑ index

At FOV 14 (4.6x), the Marksman scope shows the same voxel LOD as the unzoomed view, so distant targets and terrain look coarse when magnified. The renderer picks LOD by distance, not projected size. Suggested fix: scale the LOD distance by FOV, or expose an LOD bias for the active camera.

F21

Clients cannot see a streamed object's voxel data

MediumWhy: measured↑ index

On the native client, every rig part (rendered and visible) and every viewmodel part reported VoxelData.data == nil, VoxelData.asset == nil and isLoaded == false, while the server reported the bound asset and the volume. A client therefore cannot tell whether an object's voxels have arrived. SCAR's viewmodel waited for data.volume > 0 and never switched on. Workaround: trust the server's binding. Suggested fix: expose a client-side "streamed / LOD level" flag on VoxelData.

F22

Dynamic objects stream only near their server position

HighWhy: measured↑ index

SCAR poses rigs and viewmodels on the client (receiveTransform = false).

Viewmodel

the server pool parked the viewmodel set 40 m under the map. The owning client never received its voxels, so the guns stayed invisible in front of the camera. Moving the set's server position to 6 m under the owner's feet made it appear within seconds.

Rigs

remote rigs whose server root lagged the player (re-centred every 3 s) were reported as "enemy models sometimes not showing". Re-centring at 4 Hz addressed it. The engine decides streaming and LOD from the server-side transform, not from where the client draws the object. Any client-predicted or client-posed object therefore needs its server position kept near where it is drawn (inside the ground if it must stay hidden). Suggested fix: let a client report where it renders an object it owns, or add a per-object "always stream to its owner" flag.

F23

Streaming to clients causes the server's frame spikes, and a long session makes them worse

CriticalWhy: measured↑ index
Who
every player once the world has been running a while.
What
a per-frame capture of all 149 engine timers over 25-30 s attributes the spikes. In spike frames (over 60 ms) these stages are far above normal frames:
• _AmStreaming - Sending: 17.7 ms (normally 0.8);
• VoxelServer Update: 31 ms (normally 0.12);
• AEVVRenderer - Load Phase: 15.6 ms (normally 2.1; the headless renderer, B43);
• VPO Update Part 7: 25.7 ms (normally 10.6).
When
continuously while any client is connected, worse the longer it stays connected.
Where
server-side voxel streaming (sending and VoxelServer update) and the headless renderer's load phase.

Measured by connected client

Client connectedServer spike frames (> 60 ms)Typical spike
Native client, about 4 h, 7.4 GB56-66 per 25 s~100 ms
None (8 bots playing)0 per 25 s (1,386 frames)
Fresh native client, 1-3 min after joining23-27 per 25 s~76 ms

Ruled out by A/B

pausing SCAR's 4 Hz re-centring of rig and viewmodel roots changed nothing (56 vs 66 spikes), and parking every bot's viewmodel set far away changed nothing (59).

Physics is not the cost

10 tumbling RigidBody debris chunks cost nothing measurable with no client (55 fps, 56 ms worst). With the old client connected, the same throw dropped the server to 29 fps with 12 frames over 100 ms. Moving voxel objects cost through streaming, not simulation.

Impact

the server's p99 reaches 100-200 ms during play, though the game's own Lua is about 0.5 ms (bots) to 2 ms. Every client's snapshots and hit registration stall with it.

Workaround

restarting a long-running client halves the spikes. SCAR also keeps moving voxel objects to a minimum (debris capped and short-lived) and never streams HD viewmodel sets to non-owners where avoidable.

Suggested fix

budget per-frame streaming work across frames, and look into why a client's streaming cost grows with session length (requests piling up, or cache churn).

F24

HUD root windows turn invisible on a fresh client with no script involved

MediumWhy: observed ↑ index
Who
any player whose HUD is built from script-created NewUI root windows (ED.RootPanel).
What
after a fresh native client joined (9 Oct, ~05:30), the player reported "my UI is missing". On that client UI:GetWindows() listed all of SCAR's HUD windows, but six (Top, Vitals, Weapon, Feed, Tags, Toasts) read visible = false. Nothing in SCAR's scripts sets those six invisible: they are created visible and never toggled. The windows the HUD re-sets every frame (crosshair, markers) were visible. Setting visible = true from a client run-lua brought the panels back at once (confirmed by a client screenshot).
When
on a freshly started native client, after the game had pushed several script updates (each push resets Lua and rebuilds the HUD). The exact trigger is not known.
Where
NewUI root window visibility on the client.

Impact

a HUD that relies on "created visible, never touched" silently loses its panels.

Workaround

SCAR re-reads the six always-on panels 5 times a second and re-shows any that read invisible (ScarHUD.coreRestores counts restores).

Suggested fix

find what clears visible on root windows (window-state restore from a previous instance after a rebuild?), and document which engine events hide script windows.

F25

Client-posed objects render one frame behind the camera

LowWhy: observed↑ index
Who
first-person viewmodels and anything else posed on the client against the camera.
What
SCAR poses its first-person gun and arms (voxel objects, receiveTransform = false) in LateUpdate from the camera object's world transform, after movement and the camera update. When strafing at 30-40 fps the gun visibly trailed the view, by about a frame of movement (10-17 cm, which is large at 0.4 m from the eye), and "hiccuped" with frame-time variation. Turning showed the same lag, hidden by the deliberate sway.
Where
transform propagation from Lua to the renderer for dynamic voxel objects, compared with the camera's own transform.

Workaround

lead the posed position by the camera's last-frame horizontal motion (ScarVM.LEAD).

Suggested fix

apply LateUpdate transforms in the same frame as the camera, or offer a "render relative to camera" flag for viewmodels.

F26

Play clients get pooled parts unparented, and never get empty parent objects

CriticalWhy: measured↑ index
Who
every player of a published world that builds object pools (rigs, held items, prop groups) under empty parent objects.
What
after publishing, players saw invisible bots, guns floating around the map and the placeholder first-person gun. A one-line diagnostic each client reports to the server showed, on the play client, Avatars rigs=0 and VM set=nil (the editor client: 10 rigs, set 1). The server had all 10 "Scar Rig i" roots with their 14 parts, assets bound, and identical object counts to the editor. The play client never received the empty roots ("Scar Rig i", "Scar VM i", "Scar Lobby Props": no component, nothing synced), so Scene:GetObjectByName found nothing, and the parts arrived unparented, drawn at their raw local offsets near the world origin (the "floating guns").
When
on play servers only. In the editor, the client receives the full hierarchy, so the same code works.
Where
object replication to non-editor clients.

Workaround

the server publishes each pool's part ids in a networked field (Object.id is the same on server and clients); clients resolve parts with Scene:GetObjectById, parent them under a client-only stand-in root and pose them. Prop groups became top-level objects with world transforms. Verified by forcing the id path on an editor client: 10 rigs resolved by id, torsos on their bots, first-person set active. Confirmed on a real play client after the next publish (10 Oct, read from its health line on the play server): VM active=true set=1, Avatars rigs=10.

Suggested fix

replicate the hierarchy (including empty parents) to play clients the same way as to editor clients, or document loudly that play clients receive children unparented, and give the editor a "preview as play client" mode.

F27

Published scenes don't carry script attachments made in the editor's live scene

HighWhy: measured↑ index
What
scripts attached at runtime in the editor (AddScript from an installer, followed by Server:SaveSceneHierarchy()) were present in the editor but absent on the play server after publishing: FindScript("Scar Viewmodel") and FindScript("Scar Replay") returned nil there, while every script file had the new version. The first-person weapon and the round replay simply did not exist in the published game, with no error.
Where
the publish pipeline boots Scenes/InitScene.aeScene; the live editor scene, and whatever SaveSceneHierarchy saved, is not what gets published.

Workaround

Scar:repairScripts() attaches any missing SCAR script at server start (verified on the play server: "attached missing scripts: Scar Viewmodel, Scar Replay").

Suggested fix

publish the editor's current hierarchy, or warn at publish time when the published scene differs from the live one (scripts, objects).

F28

No tool reaches a play client, and the CLI can't publish

HighWhy: observed↑ index
What
run-lua, errors and screenshot work only on the editor framing. A read-only Lua probe can reach the play SERVER, but nothing reaches a play CLIENT: no client Lua, no client errors, no screenshot. Publishing is only a button on the project page (the promised publish CLI command is not in 0.6.5), so each fix needs the maker to click it.

Impact

every play-only bug (F26, F27) shows up as a player's screenshot with no way for the agent to inspect it, which took three publishes to diagnose.

Workaround

every SCAR client sends a one-line health report every 10 s (svDiag: build, fps, phase, viewmodel and avatar state, last errors), kept on the server and readable with the play-server probe.

Suggested fix

run-lua --target client and errors/screenshot for play framings; ship virtualmatter publish.

F29

virtualmatter open self-update fails from Git Bash

LowWhy: observed↑ index

When a new native client build was available, virtualmatter open run from Git Bash printed tar failed (128): tar: Cannot connect to C: resolve failed and did not start: GNU tar reads C: as a remote host. Running the same command from PowerShell (Windows tar) updated and launched the client. Suggested fix: pass --force-local, or extract with a bundled unzipper.

F30

Client screenshots away from the player show unstreamed terrain

MediumWhy: observed↑ index

screenshot --client <id> --at x,y,z --rot ... moves only the camera. Streaming and LOD follow the player, so a shot 30 m from the player shows holes, wireframe chunk edges and coarse placeholder ground; the server-side screenshot client was stale at the same time (F13). A lighting A/B (sun angle) could not be judged at all. Suggested fix: move the streaming focus with a screenshot camera, or wait for streaming to settle before capturing.

F31

A focused UI window swallowed the keys meant to close it (browser play build)

MediumWhy: observed↑ index

SCAR's Armory opened with UI:OpenWindow(window, true, false) (focus) and closed on Esc or U read through Input:KeyDown. A player on the published browser build was stuck in it: neither key closed it, and in a browser Esc only releases pointer lock. The editor testing never caught it because the menu was always closed by script. Workaround: open without focus, and give every modal a clickable close button. Verified on the play server (10 Oct, fifth publish) as a guest player in a browser: U, Esc, the X button and the CLOSE button each closed the Armory, and it reopened every time. One quirk remains: right after a mouse click into the game, the first key press was ignored, and a second U was needed. That happened all 3 times it was tried (synthetic input from an automated browser, so treat it as a lead, not a measurement). Suggested fix: document that focused windows consume keyboard input, and keep a guaranteed escape key that reaches scripts (or closes the focused window) in browser builds.

F32

The guide documents CLI and MCP commands the published CLI doesn't have

MediumWhy: confirmed↑ index
Who
any coding agent that follows the AGENTS.md the CLI writes into every mirrored folder.
What
that guide lists npx virtualmatter push [files] ("one-shot deploy, per-file result"), fetch [files], and the MCP tools patch_file, push_local_files and fetch_files. VM's 8 Oct reply also promised logs [--restarts], publish and status. On 10 Oct, npm's latest virtualmatter was still 0.6.5 (published 2 Oct). virtualmatter push fails with error: unknown command 'push' (exit 1), logs and fetch do the same, and the MCP server lists none of the three file tools.
When
the whole SCAR build (8-10 Oct). The folder's guide was written on 8 Oct, and the platform refreshed the world's AGENTS.md and Skills files again on 10 Oct.
Where
the guide the CLI writes on pull, and the canonical copy at make.virtualmatter.ai/AGENTS.md, compared with the npm release.

Impact

the guide says to prefer push over sync. When push is missing, the only built-in deploy left is sync, the path that broke Ashlight's world (B4, B21, B45, B48).

Workaround

SCAR kept deploying through its own push-only wrapper over the files API (vm-agent-tools, F8).

Suggested fix

publish the release the guide describes, or mark each command in the guide with the CLI version that first has it.

What worked (for the engine team)

Final autopsy: building SCAR through the CLI and MCP

Written by the coding agent (Claude) that built SCAR, when the project was paused (2026-10-10). Ashlight's autopsy (https://ohge-games.github.io/virtual-matter-breakage-report/#autopsy) covered the tooling basics: deploy, publish, crash logs and resets. This one covers what a second, harder project changed, and what an FPS exposed that a survival game didn't.

Scale of the work

What changed since Ashlight

What was genuinely helpful

  1. run-lua on both sides. Client-side Lua found F21 and F24 and fixed a player's missing HUD live. It also let the agent force the play-client code path inside the editor to verify the F26 fix before publishing.
  2. Engine timers through run-lua (AE:GetLogTimers). One 25 s per-frame capture attributed the server stutter to streaming (F23) and cleared the game's Lua. Without it, the stutter would have been blamed on game code.
  3. screenshot --client. Still the only trustworthy image of UI, scope and viewmodel, within F30's limits.
  4. imagine() plus Vox:Copy. A generated soldier became an animatable rig in under a second of engine time. Generated audio covered every gun, dig and UI sound.
  5. open, the native client. A real client for benches and A/B tests, once run from PowerShell (F29).
  6. Hot reload. Still seconds per iteration, with the cost below.

What got in the way

  1. The editor is not the play server (F26, F27, F31).
    • SCAR was developed and tested in the editor all day, and three of the bugs the friends hit could not appear there. A fourth (F31) was hidden by how it was tested.
    • Diagnosing them took three publishes, each needing the maker to click a button, because no tool reaches a play client (F28).
    • The game had to grow its own telemetry (svDiag) to see play clients at all.
  2. A push still resets all Lua, and silently kills imagine() jobs (F18). Six generated models were lost before "no pushes while generating" became a rule, so content and code work cannot overlap.
  3. Visual checks mislead in new ways.
    • The server-side screenshot client went stale after a map rebuild (F13).
    • Client shots away from the player show unstreamed ground (F30).
    • A client cannot tell whether an object's voxels have arrived (F21).

    Several "missing object" alarms were tool artefacts, and one real bug (F22) first looked like one.

  4. Edit delivery is invisible.
    • There is no per-client acknowledgement or pending-edit count.
    • The streaming stats read STALLING for healthy clients (F10).

    Measuring F1, F2 and F14 needed a two-sided bench that polls client rays.

  5. Native client disconnects (F4) ended long benches silently, so bots had to carry the long tests.
  6. The guide is ahead of the release (F32).

What's missing (ranked by how much time it would have saved on SCAR)

  1. Play-client visibility:
    • a "preview as play client" mode in the editor;
    • run-lua --target client, errors and screenshot against a play framing.

    That alone would have caught F26, F27 and F31 before any friend saw them.

  2. virtualmatter publish with per-server status, so a fix reaches players without the maker clicking a button.
  3. Edit-delivery observability: per-client acknowledged chunk versions or a pending-edit count, and a "resync region" call (F1, F2, F14).
  4. A Lua reset that doesn't kill engine async jobs, or a scoped reload of one script (F18; Ashlight's item 4).
  5. Screenshots that wait for streaming at the camera, and a client-side "streamed / LOD level" flag on VoxelData (F13, F21, F30).
  6. Reconnect and session resume in the native client (F3, F4).
  7. The promised CLI and MCP commands, shipped, or the guide gated by version (F32).
  8. Input injection for a managed test client (Ashlight's item 5, still missing). Every feel bug still needed the maker to find it: the ducking spring, strafing judder, viewmodel lag and self-collision.

Tools built for SCAR (in the project's tools/ and bench/ folders)

Bottom line

The engine connection is still excellent: an agent can measure almost anything on the server or on a client it can reach. An FPS shows where that stops. What each player's client actually has (terrain, streamed objects, hierarchy, UI) differs per client and per framing, and the play clients that matter most are the ones no tool can see. Make a play client observable, and simulable inside the editor, and most of SCAR's worst days would have been an afternoon.

Changelog