Virtual Matter Breakage Report

Cost	Measured	Card
Headless client rendered by the server every frame	About 6.2 ms render plus 1.7 ms client scene; an idle server tops out near 58 fps	B43
Fixed cost per active voxel object	About 21 µs per object per frame. An idle edit session already used 11.4–12 ms, leaving about 4 ms for the game at 60 fps	B17
Idle SDK and editor scripts	3.0 ms per frame on an empty server	B38
SDK character controller per NPC	About 0.3 ms; 40 rigged NPCs take a 60 fps server to 13 fps	B1
Moving voxel objects combined with raycasts	15–27 frames of 50–245 ms in every 8 s with 24 moving rigs	B16
Game Lua after optimisation	2.5–4.2 ms per frame with 19–23 enemies (inactive components driven by one manager)	What held up
Worst-case live siege (19–22 enemies, the Matron, the Colossus and a siege wave)	About 55–56 fps, p99 29–30 ms, worst frame 32–37 ms, once spawn-time stalls were removed	B29, Optimization result

Stall	Measured	What the game had to do	Card
`VoxelData:Save()` on the terrain	33–60+ s for the 250 MB map; a later full save of the larger terrain blocked `run-lua` for about 2 minutes	Save only when the server is empty; journal edits in between	B19
Asset deep-copy when a voxel object is created	About 35–95 ms per rig part (about 320 ms per 5-part rig). The spawn ramp's worst frame fell from 893 to 73 ms once rigs were pooled, recycled and their detail baked	A pre-built pool (6, later 14 rigs); no rig builds mid-fight	B29
Asset bind (`vd.asset = …`)	40–360 ms per bind for small assets (154–7,827 voxels), largely regardless of size; 1.2–1.4 s for a sparse 89k-voxel asset	Never bind during play	B42
Mass object creation	+200 objects in one step: a 1.7 s freeze, 9.7 fps	Spread creation over frames	B2
Voxel edits on scaled (≈0.19) rig parts	About 140× the voxels of the same edit at scale 1; at least 6.6 ms of engine commit per op, typically 7–30 ms. Sustained stamping coincided with 4 of the 7 unexplained restarts	A budget of 4 rig ops per second, none on a rig within 80 m of a player	B31, B44
Lua reload after any script save	Every runtime-built object is destroyed and rebuilt: 10–30 s of stutter	Deploys batched and made with nobody online where possible (re-adopting a pool after reload isn't possible)	B32, B8

Cost	Measured
Whole frame	41.6 ms (about 24 fps) at shadow quality 5
Shadows (`VPuzzleObject::Render - Shadows`)	11.8 ms at quality 5; 5.9–6.6 ms at quality 2. In combat, dropping graphics to Low (shadows off) took the client from 22.6 to 31.9 fps
Client Lua (engine profile)	10.6–17.2 ms; client scene update about 12–17 ms
SDK particle updater, written in Lua	About 50 µs per live particle: 11.5 ms at 219 particles
UI render	About 3.5 ms (the game's UI Lua 1.2–1.7 ms)
After the game's fixes (graphics tiers, particle caps, animation LOD)	37.9 fps idle at the hub (from 19.6); 33–37 fps in a fight with a husk group and a hound pack (from 22.6)

#	Severity	Breakage	Break point
B1	Critical	Server NPC capacity	40 NPCs → 13.2 fps
B2	Critical	Dynamic object creation spike	+200 in one step → 9.7 fps
B3	Critical	Client loses static collision	Remove then Add in one region
B4	Critical	Sync overwrites a newer local save	2 saves ~5 s apart
B5	High	Uncapped networked payload	256 MB → 3.57 s ping
B6	High	Uncapped Lua heap	3 GB, main thread blocked
B7	High	`imagine()` silent stalls	>5 concurrent jobs
B8	High	Hot reload orphans live work	Any .lua save; 0.39 fps
B9	Medium	Client particle cap not enforced	8,000 → 12.7 fps
B10	Medium	Scene-wide LOD starvation	~48 prioritizeLod objects
B11	Medium	Transforms that don't replicate	New child scale, plain pos
B12	Medium	Opaque persisted tables	`Config`, `varsTable`
B13	Medium	Edit guard estimates	Thin cylinders, same-frame sums
B14	Low	Editor Tools boot errors	Every boot
B15	Low	Docs vs runtime mismatches	4 confirmed
B16	Critical	Moving voxel objects + raycasts spike the server	50–245 ms frames
B17	High	Fixed per-object cost and thin headroom	~21 µs/object, ~4 ms spare
B18	Medium	Clients can't instance assets	No client-side rig parts
B19	Critical	Saving the terrain freezes the server	33–60+ s for 250 MB
B20	High	Voxel edits aren't persistent by default	Every gameplay edit; procedural props
B21	High	Sync overwrite race, again	2 saves <10 s apart
B22	Medium	Hidden browser tabs throttle the client	6–12 fps
B23	High	Polled input drops fast taps	Press+release in one frame
B24	Medium	Generated audio reaches clients slowly	7 of 14 after ~3 min
B25	Low	Platform popup covers the game's HUD	Browser clients
B26	Medium	Screenshot tool can't show final visuals	Low LOD, no custom lighting
B27	High	`Time.time` restarts on every Lua reload	Every hot reload
B28	Critical	Clients keep terrain the server has removed	Static removes 2–100 m from the camera
B29	High	Every voxel object deep-copies its asset on the main thread	35–95 ms per part; ramp worst 893 → 73 ms
B30	Medium	The SDK character controller jumps 2.5 m and has no slope limit	Walls under ~2.6 m are climbable
B31	Medium	Scaled rig parts: forced rays miss them and their units don't match the world	Part scale 0.19; forced ray hit nothing
B32	Medium	Every Lua reload destroys runtime-built objects	10–30 s of stutter per deploy
B33	Low	No per-object time scale, no UI gradients	Hit-stop and vignettes faked
B34	Low	New scripts can't be attached for about a minute	"Script file does not exist"
B35	Medium	A coding harness can't run automated playtests	~1 in 3 calls "fetch failed"; fly-cam client
B36	High	Left-handed parallelepipeds silently build nothing	327 facets, every home wall; SDK `Explosion.lua`
B37	Medium	Voxel ops never report an empty result	Any degenerate shape
B38	Low	SDK editor scripts cost ~3 ms per server frame when idle	3.0 ms vs <0.6 ms for the game
B39	Low	Server memory climbs on an idle, empty server	Atlas capped at 5,000 MB; +7 MB/min after
B40	Low	Inactive objects ignore forced raycasts	Forced ray returns nil
B41	High	`AE:GetDebugTime()` restarts with the server process	Every server restart
B42	Medium	Baked rig variants are huge; a periodic freeze came with them	329 MB for 111 variants; any bind 40–360 ms
B43	Medium	The server renders a headless client every frame	~8 ms per frame; ~58 fps at idle
B44	High	The server restarts without warning or a reason	7 restarts; one mid-boss-fight on the play server
B45	High	After a server restart, sync silently fails every push	HTTP 403 on every push; 3 deploys lost
B46	High	Server Lua has no stable per-user id	7 of 7 records were placeholders
B47	Medium	`type()` of a Vec3 returns "Vec3", not "userdata"	Every guarded vector dropped
B48	High	`virtualmatter sync` leaves orphans and pushes things it shouldn't	3 syncs at once; 370 MB pulls; 2 Lua resets
B49	Medium	A force-killed client stays connected for 1–2 minutes	Ghost in `GetClients()` ~1–2 min
B50	Low	Client transform race on NPCs	NPC drawn ~5× too big
B51	High	Publishing can boot the wrong scene, then save over the published one	Restart before 390 MB terrain lands
B52	Medium	Play servers boot a different scene from the editor's	InitScene vs PlayScene, no warning
B53	Low	Generated asset names are normalised	"v2" → "V2", "V2L" → "V2l"
B54	Low	Text-to-music sometimes returns a 5–15 s clip	2 of 8 beds regenerated
B55	Low	Networked tables adopt the tables they're given	"table can not have metatable"
B56	Low	MUI panels don't wrap by default, though the docs say they do	`overflow` defaults to "Hidden"
B57	Low	`Widget.sizing` is write-only	Reading it raises
B58	Low	A disabled button is illegible	Blue-grey tint, near-invisible text
B59	Low	A clear background with a frame renders as a grey fill	Transparent + border
B60	Low	`string.upper` corrupted a multi-byte character	"—" mangled
B61	Low	Generated icon art is unreadable below about 24 px	39 generated icons

The 5Ws, per breakage

Server NPC capacity tops out near 40

CriticalWhy: measured↑ index

Who: Every player on a world that runs server-side enemies or companions. It hits any maker who uses the SDK's Examples/Character Controller.lua for NPCs, as the skills docs recommend.
What: Server fps fell from 59.8 to 41.5 at 10 rigged NPCs, 30.4 at 20 and 13.2 at 40. Spawning 20 at once blocked the server for 8.2 s in a single frame. Spawned one at a time, the worst single spawn still took 675 ms.
When: Steady state, not just at spawn. Forty NPCs held 13.2 fps for as long as they lived.
Where: Character Controller:updateControls() on the server: its capsule overlap via Collision GetOverlap. Spawn cost comes from creating about 11 streamed objects per segmented rig.
Why: A benchmark put one updateControls call at 299 µs. Forty NPCs call it every frame, which is 12 ms of a 16.7 ms budget before any AI runs. The profiler showed the NPC script at 28.6 ms per frame for 40 instances.

Impact

A world can't hold the 30–60 server-driven enemies a survival or MMO-style game expects. I capped Ashlight at 24 and throttled distant NPCs to 5 Hz movement. That lifted 40 NPCs to 27.1 fps and 24 NPCs to 36.2 fps.

Repro

-- spawn N CC characters, then:
Scene:SetProfileScriptsUpdate(true)
local t0 = AE:GetDebugTime()
for i=1,50 do cc:updateControls(Vec3(0.1,0,0), false) end
return (AE:GetDebugTime()-t0)*1e6/50  -- ≈299 µs

Suggested fix

Move the server capsule sweep into native code, or batch it across characters per frame. Add a built-in movement LOD so characters far from every client step at a lower rate. Document a per-NPC budget in the character-controller skill.

Creating voxel objects stalls the whole server

CriticalWhy: inferred↑ index

Who: All connected players during any burst of object creation: spawning waves, building, debris, rig assembly.
What: I added objects (one VoxelData plus a 0.3 m sphere each) in steps. Going from 200 to 400 in one step took 1,658 ms and dropped the server to 9.7 fps average and 0.2 fps minimum. Even the 100-object steps hit a 0.5 fps minimum. Afterwards, steady-state cost also grows with count: +300 objects ran at 54.8 fps and +1,000 at 36.9 fps.
When: At creation (the spike), then in proportion to live object count.
Where: Scene:CreateObject + AddComponent("VoxelData") + the first Vox:Add(...):Force(obj), plus per-object streaming to clients.
Why: Creation work appears to run synchronously on the main thread with no per-frame budget. The edit guard throttles voxel volume but not object count. The steady-state cost points to per-object streaming and transform bookkeeping.

Impact

Segmented characters multiply the problem: each Ashlight rig is about 11 objects, so 40 enemies means about 440 objects. Destruction that spawns physical rubble runs into the same wall.

Repro

for i=1,200 do
  local o = Scene:CreateObject("S"..i)
  o.pos = Vec3(i%40, 2, i//40)
  o:AddComponent("VoxelData").syncToClients = true
  Vox:Add(Sphere(o.pos,0.3)):Force(o):Run()
end  -- watch Time.fps for 4 s

Suggested fix

Queue object creation behind a per-frame budget, as the edit guard already does for voxel volume, or warn when one frame creates more than N objects. A lightweight "rig part" object type that shares one stream would make segmented characters affordable.

A client loses collision the server still has

CriticalWhy: inferred↑ index

Who: A connected player standing on terrain that the server rebuilds while they are there.
What: The player fell through a floor that existed on the server. The client's collision and raycasts missed static voxels that the server reported as present.
When: After a large async Vox:Remove (box quarters), followed within a few frames by Vox:Add in the same region. Building in fresh regions never triggered it.
Where: Client static voxel streaming and collision (VVCollision). The client-side character controller collides against client voxel data only.
Why: The remove and add deltas likely arrive or apply out of order, or the add is dropped while the remove is still streaming. None of AE:ReloadGeometry(), AE:AEVV_InvalidateData() or Client:ReloadPage(false) repaired it. ReloadPage did nothing at all in the native client.

Impact

Any destructible or rebuildable world, which is the platform's headline feature, can drop players into the void. I worked around it with settle waits after clears and a fall-catch that teleports players back.

Repro

Vox:Remove(Box(c, Vec3(16,4,16))):ForceStatic():Run()
util:schedule(0.05, function()
  Vox:Add(Cylinder(c, c+Vec3(0,1,0), 7)):ForceStatic():Run()
end)  -- stand a client on c

Suggested fix

Order or version static deltas per chunk so a client never applies an older remove over a newer add. Expose a working "resync region" call. Make ReloadPage work in the native client, or report an error when it can't.

Sync overwrites a newer local save

CriticalWhy: measured↑ index

Who: Makers and coding agents editing through npx virtualmatter sync.
What: I saved Hollow.lua twice, about 5 s apart. The log shows pushed Hollow.lua and then a run of pulled Hollow.lua. The second local edit was gone from disk, with no conflict file and no warning. Separately, a stopped background sync left its node child running; two syncs then produced .remote-conflict files and pulled a 157 MB .am~ temp file into the mirror.
When: When a local save lands while the hot-reload echo of the previous push is still being pulled back.
Where: The CLI's two-way sync: the echo of its own push is treated as a remote change.
Why: The pull compares the remote copy against stale state instead of the content hash it just pushed, so its own echo overwrites a newer local file.

Impact

This is silent loss of source code. An agent that saves a file twice in a row can believe a fix is live when it isn't. I only caught it because I grepped for the change.

Repro

npx virtualmatter sync &
echo "-- a" >> X.lua; sleep 5
echo "-- b" >> X.lua; sleep 20
tail -2 X.lua   # "-- b" is missing

Suggested fix

Track the hash of each push and ignore the matching echo. Never let a pull overwrite a file whose local mtime is newer than the last push; write a conflict file instead. Kill child processes on exit and exclude *.am~.

Networked tables accept any payload size

HighWhy: measured↑ index

Who: Every client of a world whose script writes large values into a util:makeNetworkedTable field.
What: A 16 MB string was written in 94 ms. 64 MB and 256 MB strings were also accepted and delivered. At 256 MB the client's ping rose to 3.57 s, freezing input and replication for every player. It recovered once I cleared the field.
When: On a single write. There is no warning at any size.
Where: Networked table delta replication, server to client.
Why: There is no size cap and no chunking: a large value blocks the reliable channel that carries everything else.

Impact

One careless or hostile script, such as a chat log or inventory dump growing without bound, can lag every player in the session.

Repro

self.net.blob = string.rep("x", 256*1024*1024)
-- watch client ping

Suggested fix

Cap each field (for example 1 MB) and raise a Lua error above it. Send large values on a separate low-priority channel so they can't starve input and transform traffic.

The server Lua heap has no ceiling

HighWhy: measured↑ index

Who: Whole sessions. A runaway script can starve the host process that every player shares.
What: The Lua heap grew to 3,072 MB with no limit, taking the process to 21.7 GB. A run-lua call that blocked the main thread ran past the 60 s CLI timeout. After freeing, Lua dropped to 9 MB, but the process stayed roughly 0.3 GB above its earlier footprint. Over the whole session, server memory rose from 6.1 GB to about 19 GB.
When: When a script allocates without bound, or a long synchronous loop runs inside run-lua.
Where: The server Lua VM allocator, and the run-lua bridge, which runs on the main thread.
Why: There is no allocator limit or instruction-count hook in the sandbox. The allocator keeps freed pages, which is normal, but nothing tells a maker how close the host is to its limit.

Impact

A single infinite-append bug can take the world down for everyone, and an agent's long query freezes the simulation while it runs.

Repro

local t = {}
while collectgarbage("count")/1024 < 3072 do
  t[#t+1] = string.rep("A", 1024*1024)
end

Suggested fix

Set a per-world Lua memory limit through lua_setallocf that raises a catchable error. Add an instruction-count hook so run-lua fails cleanly instead of hanging the main thread.

`imagine()` jobs stall forever, silently

HighWhy: inferred↑ index

Who: Makers and agents generating assets.
What: Of 11 concurrent jobs, 5 finished quickly and 6 sat at 0% for over 20 minutes. Some prompts never progressed at all ("shadow creature … horror", "iron staff with hook"); reworded, they finished in about 75 s. No job ever fired an error, timeout or failure callback.
When: With more than 5 concurrent jobs, or with certain prompt content.
Where: The imagine() job queue and its generation backend.
Why: Probably a concurrency limit of 5 that drops or parks overflow jobs instead of queueing them. The prompt stalls look like a content filter rejecting the job without reporting back.

Impact

A preview blockout sits in the world indefinitely. Agents can't tell "slow" from "dead", so they wait or retry and burn credits.

Repro

for i=1,11 do
  imagine("a wooden barrel "..i, Scene:CreateObject("I"..i), {})
end  -- 6 never progress

Suggested fix

Queue extra jobs visibly with a status field, add a failure callback and a timeout, and return the reason when a prompt is rejected.

Hot reload orphans everything in flight

HighWhy: measured↑ index

Who: Players in a world that is edited live, plus makers running long builds.
What: Saving any .lua file re-runs every script, and anything async is lost: Vox job queues, util:schedule chains and imagine() callbacks. With 40 NPCs alive, one save dropped the server to 0.39 fps for several seconds, and every NPC instance was orphaned.
When: On every save while sync is running, including saves to scripts unrelated to the running work.
Where: The hot reload path: a full Lua reset and re-instantiation of all scripts.
Why: Reload is global rather than per module, and callbacks keep references to the old instances.

Impact

Live-ops edits hitch every player and can break game state. I had to make the terrain builder resumable through Config and avoid saving during builds.

Repro

util:schedule(30, function() print("fired") end)
-- now save any .lua file
-- "fired" never prints

Suggested fix

Reload only the changed module, and remap or cancel orphaned callbacks with a warning. Provide a hook such as OnBeforeReload so scripts can hand off state.

The documented particle cap isn't enforced

MediumWhy: measured↑ index

Who: Players on lower-end clients during heavy effects.
What: The docs say "max 500 particles". The client accepted 500, 2,000 and 8,000 in a single call. Client fps went from about 34–43 to 29.2 at 500, 26.2 at 2,000 (minimum 15.7) and 12.7 at 8,000 (minimum 5.4).

Update: a client profile during play showed the SDK's own Particle Updater script costing about 50 µs per live particle: 11.5 ms of the frame at 219 particles. Game-side caps (200, 140 or 80 by graphics tier, 40 for ambient effects) brought client frames from about 41.6 ms to 28–39 ms together with lower shadow quality.
When: Immediately, on the client that spawns them.
Where: Editor Tools/Particles.lua addParticle on the client.
Why: No cap or eviction exists, and the per-particle simulation runs in Lua.

Impact

Effects-heavy moments, such as explosions or a crowd of fire, drop clients into slideshow territory.

Repro

local ps = Scene:GetObjectByName("Particles"):FindScript("Particles")
for i=1,8000 do ps:addParticle(p, {lifetime={5,6}}) end

Suggested fix

Enforce the cap by evicting the oldest particles, and make it configurable. Long term, simulate particles natively or on the GPU.

B10

Scene-wide LOD starvation

MediumWhy: measured↑ index

Who: Every client in a scene with many prioritized dynamic objects.
What: About 48 objects with prioritizeLod = true (lodBias 5) collapsed the whole scene's LOD on the client into giant blocks, including the hub fire 7 m from the camera. Removing them restored detail. lodBias 3 per character, without the prioritize flag, was fine.
When: When the combined prioritized budget exceeds the client's stream budget.
Where: The client LOD and streaming budget allocator.
Why: The prioritized objects take the shared budget before the rest of the scene gets any.

Impact

Making characters look sharp silently ruins everything else on screen.

Repro

for each of 48 rig parts:
  vd.prioritizeLod = true; vd.lodBias = 5

Suggested fix

Cap the share of the budget that prioritized objects can claim, and keep a floor for static terrain near the camera.

B11

Transforms that don't replicate

MediumWhy: measured↑ index

Who: Makers building attached gear, props or rigs.
What: A child object created on the server with scale 0.089 showed up on clients at scale 1.0, producing a mace 11 times too big. Plain objects without a character controller showed up on clients at (0,0,0).
When: On creation, and on every later change.
Where: Object transform replication.
Why: Only character controllers replicate their position, through RPCs, and scale is never sent. The docs mention position, but the scale case catches people out.

Impact

Every maker ends up building their own transform sync.

Repro

local c = Scene:CreateObject("G"); c.parent = root
c.scale = 0.089   -- client reads 1.0

Suggested fix

Add an opt-in replicateTransform flag that covers position, rotation and scale, or at least send initial values at spawn.

B12

Persisted tables come back opaque

MediumWhy: measured↑ index

Who: Anyone persisting game state.
What: Config:TempSet(key, table) comes back as an opaque sol.LuaTable userdata that can't be indexed or iterated. component.varsTable can't be iterated with pairs, and vt.field returns nil for values set with setSerializeVar.
When: On read-back after a Lua reset.
Where: The Config store and the VarTable bindings.
Why: The bindings return the raw sol object rather than converting it back to a Lua table.

Impact

Silent data loss for anyone who doesn't test a round trip. I serialized everything into Lua-literal strings instead.

Repro

Config:TempSet("k", {a=1})
local t = Config:TempGet("k")
return type(t), t.a  -- userdata, error

Suggested fix

Deep-convert values on get, or reject tables on set with a clear error.

B13

Edit guard estimates are hard to predict

MediumWhy: measured↑ index

Who: Makers and agents building terrain or structures.
What: A thin cylinder (r 17, h 0.55) was estimated at about 3,195 baseline-m² and rejected, while a sphere of r 6.5 was accepted. A sphere of r 10 was rejected because about 505 m² had already been issued that frame. Single spheres passed up to r 8–11, were soft-rejected at r 12–17 and hard-refused at r 24 and above.
When: On Run().
Where: The Vox edit guard.
Why: Cylinder cost is estimated well above the documented formula, and the guard sums all ops in a frame. The sum is sensible, but the documentation doesn't say so.

Impact

Trial and error for anything larger than a room. The rejection message itself is good and actionable.

Repro

Vox:Add(Cylinder(c, c+Vec3(0,0.55,0), 17))
  :ForceStatic():Run()  -- rejected

Suggested fix

Expose Vox:EstimateCost(op), and document the per-shape cost and the same-frame aggregation.

B14

Editor Tools fail at boot

LowWhy: inferred↑ index

Who: Every world on this build.
What: Five errors on every boot: Editor.lua:63 events read-only (memoryPressure), Transform Gizmo.lua:260 onSnapChanged nil, Shape Manager.lua:220 onPermissionChanged nil, Editor.lua:785 onSnapChangedTrigger nil, Events.lua:742 triggerLocally nil.
When: On every Lua reset.
Where: The Editor Tools scripts against the engine's events module.
Why: Version skew between the Editor Tools scripts and the engine. One knock-on effect: the Shape Manager path that resumes imagine() jobs is broken.

Impact

Noise in the error channel that agents must filter, and a broken job-resume path.

Repro

Server:LuaReset()  -- then read errors

Suggested fix

Ship Editor Tools versioned to the engine, and add a CI boot test with zero errors.

B15

Docs and runtime disagree

LowWhy: measured↑ index

Who: Agents and makers who follow the skills docs.
What: Scene:CreateMaterial is documented and used by a skill, but nil at runtime. Button.onPress expects an event-name string; the function form is onPressEx. Generated models face −Z, while the skill says +Z. The particle cap of 500 is documented but not enforced (B9).
When: On first use.
Where: apiEmmyAtomontage.lua and the Skills/ references.
Why: The API dump and the runtime build have drifted apart.

Impact

Wasted turns, plus rigs and characters that face backwards.

Repro

return Scene.CreateMaterial  -- nil

Suggested fix

Generate the API dump from the running build, and add doc tests that call every documented symbol.

B16

Moving voxel objects plus raycasts spike the server

CriticalWhy: inferred↑ index

Who: Every player in a world with server-driven moving characters that also use server raycasts, which describes almost any game with AI.
What: With 24 rigged enemies (6 voxel objects each) moving on the server, 15–27 frames in every 8 s took 50–245 ms, and average fps fell to 40. The game was otherwise identical. Moving the objects without raycasts: 54.9 fps, no spikes. Raycasts without moving: 54.4 fps, no spikes. Both together: the spikes return.
When: When objects are moved through obj.pos at 1–15 Hz while scripts run about 240 static rays per second. The rays use ForceStatic, so they shouldn't touch dynamic objects at all.
Where: Spike frames show NEW/VPO Update Part 7 (80–128 ms), NEW/FixedUpdate VPuzzleObject and NEW/_AmStreaming - Sending (54–74 ms).
Why: Probably a deferred rebuild of the dynamic-object spatial or streaming structures, triggered when a query follows transform changes. An individual ray is only 0.03–1 ms, so the cost lands later in the frame.

Impact

This was the single biggest cause of stutter in the game. The workaround is unusual: keep each enemy's real position in Lua, replicate it through a networked field, and move the engine object only every 10 s or 8 m.

Repro

-- 24 roots with 5 child VoxelData each
-- every frame: root.pos += v*dt
-- 10 Hz each: Cast:Ray(p, down):ForceStatic():Run()
-- watch SUF/Update Frame for 8 s

Suggested fix

Make static-only queries skip dynamic-structure updates. Batch transform-driven rebuilds per frame with a time budget, and expose a "client-driven, server-static" flag for objects whose clients position them anyway (receiveTransform = false).

B17

Fixed per-object cost and very little headroom

HighWhy: measured↑ index

Who: Any world with animated characters built from several voxel parts.
What: Each active dynamic VoxelData costs about 21 µs per server frame, even when it doesn't move: 260 rig parts cost 5.4 ms. Inactive objects are effectively free, and switching off 22 hidden template roots saved about 1 ms. editable = false changes nothing. An idle edit session already uses 11.4–12 ms of update time: editor tools Lua is about 1.8 ms (the undo History script alone 0.9 ms), and the server's own screenshot camera adds about 2.6 ms of render and blit. That leaves roughly 4 ms for the game at 60 fps.
When: Constantly.
Where: Update VPuzzleObject and the edit session's Editor Tools.
Why: Per-object bookkeeping runs every frame regardless of state, and editor and screenshot work run even when nobody is using them.

Impact

I cut each enemy from 11 to 5 voxel objects (a lighter rig with virtual elbow and knee joints) to reach 53–56 fps with 24 enemies, up from 13 fps at the start. A solid 60 fps with 24 enemies grouped around one player isn't reachable on this host, because the idle engine overhead alone is about 12 ms.

Repro

-- 260 parked VoxelData objects:
-- SUF/Update Frame goes from 14.7 ms to 20.1 ms

Suggested fix

Skip unchanged objects in the per-frame update. Render the server's screenshot camera only on request. Profile and trim Editor Tools, or turn them off in play sessions.

B18

Clients can't instance voxel assets

MediumWhy: measured↑ index

Who: Makers who want animated characters without paying B17 on the server.
What: On the client, AE:GetAssets() returns none of the world's rig assets: 0 of the 150 the server lists. The client VoxelData.data is nil, and a template's .asset reads nil. A client can't create local copies of character parts, so every animated limb has to be a server object.
When: Always.
Where: Client asset sync and VoxelData bindings.
Why: Voxel content is streamed per server object instead of shipped as shared assets.

Impact

It blocks the obvious fix for B17: server keeps one root per character, and each client instances the limbs locally.

Repro

-- on a client
local t = Scene:GetObjectByName("Rig Hollow torso")
return t:GetComponentByType("VoxelData").asset -- nil

Suggested fix

Allow client-local VoxelData to reference a server-streamed object's content, or sync asset references so clients can instance them.

B19

Saving the terrain freezes the server for 33 seconds to over a minute

CriticalWhy: measured↑ index

Who: Everyone connected when a world saves its terrain.
What: VoxelData:Save() on the static terrain (a 160 m map at 32 vpm, a 250 MB .am) blocked the main thread past the CLI's 60 s timeout. The CLI returned HTTP 504 and the world was frozen until the save finished. Sync then pulled a 100–120 MB .am~ temp file into the local mirror.

Update: a QA frame sampler happened to straddle one of the Terrain Keeper's automatic saves on an empty server. It recorded a single 33,415 ms frame (10.3 fps averaged over its 8 s window), and every run-lua in that window returned HTTP 504. The same 250 MB map therefore stalls the main thread for anywhere from about 33 s to over 60 s. A later full save, of the larger terrain and with nobody online, blocked run-lua for about 2 minutes (HTTP 504s).
When: On every save of the static terrain. Server:SaveScene() also re-saves it.
Where: Static voxel serialization, which is synchronous and saves the whole file every time.
Why: The save is a single blocking write of the entire common collection, with no async, incremental or per-chunk dirty path.

Impact

A persistent survival world can't save its terrain while anyone plays. Most terrain changes are only safe once the server is empty.

Mitigation shipped

A Terrain Keeper module saves the full terrain only after the server has been empty for 45 s, at most every 10 minutes. Between saves, every gameplay edit is written to a compact journal that survives reloads and restarts, and is replayed onto the reverted terrain after a restart. The first automatic save ran while the server was empty. The Keeper now saves edits that the journal doesn't cover at most once an hour.

Suggested fix

Save only dirty chunks, asynchronously on a worker thread with a copy-on-write snapshot. Expose an edit log the engine replays itself.

B20

Gameplay voxel edits aren't persistent by default

HighWhy: measured↑ index

Who: Players who dig, salvage, build or destroy, which in a voxel game is everyone.
What: After hours of play the static terrain reported isSaved = false. Only the scene hierarchy had been saved every 2 minutes, which excludes voxels. A restart would revert salvaged walls, burned blight, broken ruins and dug holes. Systems that rebuild from saved game state (homes, blight, arena) would then re-add geometry on top of the reverted terrain. Players see walls that pop back up, and growth that stacks.

Update: dynamic objects lose procedural voxels too. After a server restart, all 12 oil drums (save = true, each built from four Vox:Add cylinders with Force(obj)) were back in the scene, active and with their VoxelData component, but with volume 0, zero bounds and no asset: invisible, with no error. The scene's dynamic-prop count went from 37 to 25.
When: On any server restart.
Where: The split between SaveSceneHierarchy (cheap, no voxels) and voxel saves (B19). It applies to static terrain and to procedurally built dynamic objects; only asset-backed VoxelData, or an explicit VoxelData:Save(), survives.
Why: The cheap save path is the documented default. The voxel save path is too expensive to run while anyone is playing.

Impact

"Persistent world" is the game's promise, and the engine's cheap save path quietly breaks it.

Mitigation shipped

Terrain Keeper journal (see B19). Harvesting, digging, blight burning and lean-to shelters all go through it. Rebuild-from-state systems use exact saved positions, so replaying them is idempotent. For procedural props, the fix is to rebuild on boot whenever the VoxelData is empty, not only when the object is missing, or to save the prop as an asset.

Suggested fix

Include voxel deltas in the hierarchy save (journal or dirty-chunk diffs), or warn makers when a world has gone more than N minutes with unsaved static edits.

B21

The sync overwrite race happened again

HighWhy: measured↑ index

Who: Anyone editing through npx virtualmatter sync, especially coding agents that save quickly.
What: Two quick edits to Terrain Keeper.lua: the second was partly lost when sync pulled the echo of the first back over it. One fix was missing from disk with no conflict file. Same mechanism as B4.
When: When two saves to one file land within about 10 s.
Where: CLI two-way sync.
Why: See B4: the pull treats the echo of its own push as a remote change.

Impact

Silent source loss mid-session, again.

Mitigation shipped

All edits are staged outside the mirror and copied in one write per file. I wait 25–30 s, then compare every file against its staged copy before testing. Parallel agents write only to staging folders, and one deployer copies.

Suggested fix

As in B4: track pushed hashes and never let a pull overwrite a newer local file.

B22

Hidden browser tabs throttle the client

MediumWhy: inferred↑ index

Who: Players with the game in a background browser tab, and anyone measuring client performance.
What: The connected WASM client measured 6–12 fps even with no enemies, against 34–43 earlier. The client stays connected while throttled, so its player keeps existing in the world.
When: When the tab or window is hidden.
Where: Browser timer and rAF throttling of the WASM client.
Why: Normal browser behaviour, but the server can't tell an AFK, throttled client from an active one.

Impact

Client-side numbers are only reliable from a focused window, and AFK players still pull enemies.

Mitigation

Client performance tuning is based on code-path costs rather than that client's fps.

Suggested fix

Report a client "visible / throttled" flag to the server (Client:IsAppVisible exists client-side) so games can treat AFK players differently.

B23

Polled input drops fast taps

HighWhy: measured↑ index

Who: Players on slow or throttled clients, and anyone using automation or accessibility tools that send quick taps.
What: The engine's events.keyDown listener received "F", but Input:KeyDown("F"), which the SDK's controllers poll, never returned true. The lantern didn't light.
When: When a key goes down and up within the same rendered frame. That is common at the 6–12 fps of a throttled browser client (B22).
Where: The Input polling API, which samples state per frame instead of edges.
Why: Press and release edges inside one frame cancel out before the poll.

Impact

Players on weak devices lose actions: dodges, flasks, interactions.

Mitigation shipped

The player script queues events.keyDown events and consumes them once per frame. A pressed(k) helper returns the poll OR the queued edge. Verified live: the lantern lit, and the tutorial advanced.

Suggested fix

Latch press edges in Input:KeyDown until the next Lua update, and document the event API as the default.

B24

Generated audio reaches clients slowly

MediumWhy: measured↑ index

Who: Players connected while new sounds or music are added.
What: Fal generated 14 of 14 clips without a failure: 11 sound effects of 20–240 KB, and 3 music tracks of 3.4–4.7 MB. A browser client that connected afterwards listed none of them at first, then 7 of 14 about 3 minutes later.
When: After a montage audio asset is created.
Where: Asset sync to clients (events.assetSyncedIn).
Why: Inferred: assets are streamed in the background at a modest rate, and big MP3s queue behind each other.

Impact

New players hear placeholder or no sounds for minutes.

Mitigation shipped

The audio module re-indexes on every assetSyncedIn event and falls back to similar built-in SDK sounds until the custom ones arrive.

Suggested fix

Prioritise small assets, expose sync progress, and allow preloading a declared asset list at connect.

B25

The platform's popup covers the game's HUD

LowWhy: measured↑ index

Who: Browser players.
What: The website-drawn "Running slow in this browser?" card (about 200×120 px at 1280×720, bottom-left) sits on top of the game's vitals panel until dismissed. Games can't see it or move it.
When: Shortly after joining in a browser.
Where: The make.virtualmatter.ai page chrome around the engine iframe.
Why: The page and the game don't share layout information.

Impact

Health and oil are hidden at the moment a new player most needs them.

Mitigation

A tunable lift for the vitals panel (AshUI.VITALS_LIFT) exists; the default keeps the layout clean.

Suggested fix

Place platform toasts in a reserved corner, or give games a safe-area query.

B26

The screenshot tool can't show final visuals

MediumWhy: measured↑ index

Who: Makers and coding agents verifying graphics and animation.
What: virtualmatter screenshot renders with the server's headless client. It skips the game's custom light pass, streams rigs at very low LOD, and poses characters at whatever rate the game allows there. --target refuses objects without their own voxels, such as rig roots. The browser editor link renders correctly, but it stalls whenever its tab or pane is hidden (B22), and its canvas lives in an iframe that page-level automation can't send held keys to.
When: Every visual check.
Where: The CLI/MCP screenshot path and the browser client.
Why: Inferred: screenshots come from a cheap server-side scene rather than a real client.

Impact

Animation, lighting and UI can only be judged in a visible real client, so verification takes longer and some results are marked unverified.

Mitigation

Geometry is checked with CLI screenshots. UI and lighting are checked in a visible browser client, with player actions driven through client-side Lua and RPCs. Workaround found later: virtualmatter screenshot --client <id> captures through a real connected client (for example the native client opened by the CLI) with the lighting pass on, which sidesteps the headless client running with SSAO and the custom light pass off.

Suggested fix

Allow screenshots from a chosen connected client (--client <id>), keep the custom shader passes, and accept --target on hierarchy roots.

B27

`Time.time` restarts on every Lua reload

HighWhy: measured↑ index

Who: Any script that stores a timestamp somewhere that outlives the Lua state: Config, serialized fields or a ledger.
What: After hours of uptime, Time.time read 92 s, while AE:GetDebugTime() read 59 661 s. The Terrain Keeper stored its last-save time in Config. After a hot reload that time lay in the future ("last full save −64 s ago"), which would have blocked the next save for as long as the world had been running.
When: On every hot reload, Lua reset or restart.
Where: The simulation clock restarts with the Lua state.
Why: The clock is tied to the Lua state, not to the process. That's reasonable, but it isn't documented, and the SDK guide recommends Time.time for timing.

Impact

Cooldowns, respawn timers and regrowth timers that persist across reloads silently stretch or never fire.

Mitigation shipped

The Terrain Keeper first switched to AE:GetDebugTime(), believing it to be wall-clock time; it turned out to be process uptime (B41), and a persistent game clock replaced it. The other game timers were audited: they live only in memory, or persist as "seconds left" rather than absolute times.

Suggested fix

Document the reset, and offer a monotonic per-process clock (for example Time.uptime) for anything that persists.

B28

Clients keep showing terrain the server has removed

CriticalWhy: measured↑ index

Who: Anyone connected while a large terrain edit happens, whether it is far from their camera or right in front of it.
What: A two-pass rebuild of the map's 160 m boundary (about 5 500 static ops) first left, then cleared, debris over the edge band. Afterwards the screenshot client still showed rows of rocks, pillars and brick hanging in the sky over the south and south-west edges. On the server, a 2 m grid of downward rays found no voxels above cliff height there (only the Spire tower), and no objects. The client was drawing terrain that no longer exists.

Update: it also happens close up. After about 600 static edits in the hub (rebuilding the Emberheart and paving the plaza), the screenshot client drew a large blocky "room" filling the view at roughly (0, 0–6, 5–12), inside Cinderhold. On the server, a horizontal ray from (0, 4, 14) toward −Z hit nothing, and downward rays at z = 5–13 all found the plaza floor at y = 0.34.
When: After static removes 70–100 m from a client's camera (the edge rebuild), and after many edits 2–10 m from it (the hub rework).
Where: Client static voxel streaming. The same mechanism as B3, where a client kept collision the server no longer had.
Why: Inferred: the client keeps a cached chunk after the server has changed it, until something forces a reload. Distance from the camera is not the trigger, since phantom chunks also appeared 2–10 m away. A burst of many edits to the same chunks is the common factor in both cases, which points at dropped or out-of-order static deltas, as in B3.

Impact

Players see walls and debris that aren't there: "odd walls that pop up". After any live terrain rework, the world looks different to each client depending on when they connected. In the hub, a phantom block can fill the whole view.

Mitigation

Large reworks run only when nobody is online (the World Dresser and Deep Arena upgrades ran on an empty server). The Terrain Keeper saves afterwards, so fresh connections stream the current terrain. Server-side ray audits are used as ground truth instead of screenshots. The same rule now covers hub and plaza dressing, not only map-scale reworks.

Suggested fix

Version static chunks and have clients re-request any chunk whose version changed, even outside their LOD focus. Expose a working "resync region" call.

B29

Every voxel object deep-copies its asset on the main thread

HighWhy: measured↑ index

Who: Every player whenever a character, prop or piece of debris is spawned.
What: Spawning one 5-part enemy rig froze the server for about 320 ms. Profiling showed vd.asset = template.asset at 33–83 ms per part, and CreateObject + AddComponent at about 11 ms. Scene:CloneObject was no cheaper (56–96 ms per part). Assigning the same asset a second time cost 75 ms again.
When: On every object that gets voxel content from an asset or a clone.
Where: VoxelData asset binding and object cloning.
Why: Inferred: the voxel data is copied per instance, with no sharing or instancing of identical assets.

Impact

Spawning enemies mid-fight stutters everyone. Combined with B17 (per-object cost) and B18 (clients can't instance assets), animated multi-part characters are expensive at every stage.

Mitigation shipped

A spawn pool. Six enemy rigs are pre-built while inactive, one part every 0.35 s when calm and every 1.5 s during fights. A spawn activates a pooled rig. Measured spawn cost: 4–20 ms, down from about 320 ms. Bulwarks now come from their own pre-built pool, and the Matron's 10 parts are staged before her fight (her spawn measured 10 ms), so neither binds assets mid-combat. Dead Hollows are recycled into the pool, and cosmetic voxel detail is baked into variant assets rather than stamped at spawn (see the stress test below).

Suggested fix

Share voxel data between instances of the same asset (copy-on-write), and load or bind assets asynchronously off the main thread.

Stress test: worst case

A siege wave, an Ash Hound pack, a Lantern Thief, the Matron awake and a Colossus fight at once, with 5 fake players and nobody online. Teardown afterwards was verified clean. In the baseline, the ramp-up stalled whenever the pool ran dry and a rig had to be built synchronously (about 320 ms each).

Phase	Server fps	p95	p99	Worst frame	Notes
Baseline, ramp-up	30.4	–	508 ms	893 ms	Synchronous rig builds when the pool ran dry
Baseline, steady (26 Hollows + 2 bosses)	50–51	26–27 ms	–	84–94 ms	Script cost: Hollow 3.1 ms (39 instances), Body 1.1 ms
Round 11 fixes, steady	56.3	21.6 ms	23.8 ms	29 ms	No cold builds mid-fight, throttled pool rebuild, jittered AI think times, lighter static jobs
Round 11 fixes, ramp-up	–	–	341 ms	663 ms	Spikes remained
Round 12, ramp-up	45.4	–	65.5 ms	480 ms	Dead Hollows recycled into the pool, crust pre-worn at pool time, pool rebuild throttled rather than stopped
Round 12, steady siege (22 Hollows)	53.9	25 ms	–	76 ms	The siege no longer starves at 16 Hollows
Round 13, ramp-up	53.3	–	60.9 ms	73 ms	Crust baked into assets: spawns do no voxel ops. The only frames over 100 ms are the bosses' own staged builds at spawn, plus a 25 ms ledger encode
Round 13, steady (22 Hollows + Colossus + Matron + siege)	53.4	–	30.5 ms	51 ms	Baseline steady worst was 84–94 ms; the pool trims back to 14 afterwards
Round 14, ramp-up (no attribution wrappers)	53.7	–	38.6 ms	73 ms
Round 14, steady (23 Hollows + bosses + siege)	54.3	22.5 ms	30.7 ms	123 ms	Script cost per frame: Hollow 3.3 ms (22 instances), Body 1.2 ms, Game 1.0 ms. The next round targets the last ~3 ms
Round 16 + inactive components, steady (22 Hollows + bosses + siege)	54.7	24.6 ms	30.3 ms	32 ms	No frames over 33 ms. Hollow, Body and Beacon server components set inactive and driven by one manager; game script time 8.66 → 4.15 ms per frame
After rounds 17–18, before the fix (22–23 Hollows + Matron + Colossus + siege)	53.8–54.4	–	32–34 ms	64–102 ms	Script time only 2.5 ms; the spikes were the pool building spare rigs mid-fight once the group bonus used every rig
After the pool fix (no new rigs while 18+ Hollows are out)	56.0–56.1	23.7 ms	29 ms	36–37 ms	No frames over 50 ms; peak load settles at 19 Hollows instead of 22

Spike attribution (resolved). Wrapping 1,436 server functions during the ramp showed that every 300–716 ms frame was a spawn running the Hollow crust builder: 7–23 Vox:Add rubble plates on the torso part, which has scale 0.19. Lua took only 12–25 ms of that; the rest was the engine committing voxels. At scale 0.19, a 0.3 m plate is about 140 times as many voxels as the same plate at scale 1 (see the scaled-part op costs under Measured limits). The fix was to bake the crust into three torso variant assets with VoxelData:SaveAs, so spawns do no voxel ops at all, and to have hits paint chips instead of carving them. Pool part binds also showed up (36–72 ms for Hollows and 40–80 ms for Bulwarks), and are now paced. One 469 ms engine-side frame with no voxel ops in round 12 is still unexplained.

Forging (acceptable now): forging armour onto a player rig first hitched at 43.5 fps, with 36 frames over 50 ms and a worst frame of 83 ms, for about 6 s. Every voxel op on a part at about 0.19 scale costs at least 6.6 ms of engine commit, typically 7–30 ms. Pre-baked tiers removed the voxel ops but brought the problems in B42. Armour is now stamped live onto small, empty shell objects at one op per frame plus two idle frames, with no binds and no SaveAs. A full tier-3 set on a dummy takes about 35–40 s at 47 fps, with a worst frame of 48–52 ms.

B30

The SDK character controller jumps 2.5 m and has no slope limit

MediumWhy: measured↑ index

Who: Makers who use the example character controller as shipped, and their level designers. Ashlight's players use it unmodified.
What: Examples/Character Controller.lua defaults to jumpHeight = 2.5 (the apex, in metres) and maxStepHeight = 0.5, and has no slope limit: steepness is handled only by the ground probe and step-up. A wall has to stand more than about 2.6 m above the highest foothold in front of it to stop a player, not 2.5 m.
When: Whenever a level relies on a wall, ledge or steep slope to keep players out.
Where: The example character controller's defaults (jumpHeight, maxStepHeight, ground probe).
Why: The defaults are tuned for a platformer-style jump, roughly twice a typical human jump, and nothing in the controller rejects steep ground.

Impact

Boundaries and "impassable" ruin walls that look tall enough can be jumped. Small interior spaces can't use ledges to gate progress at all: no ledge inside a 3 m crypt stops a 2.5 m jump.

Mitigation

Game-side wall heights. The map edge keeps at least 2.9 m above the highest foothold (a 0.4 m margin over the apex). The Undercroft crypt uses a one-way door instead of a ledge. Lowering charCtrl.jumpHeight to about 1.2 m was considered.

Suggested fix

Ship a human-scale default jump (about 1–1.2 m), add a configurable maximum walkable slope, and document the jump apex next to level-design guidance in the skills.

B31

Scaled rig parts: forced rays miss them and their units don't match the world

MediumWhy: inferred↑ index

Who: Anyone adding voxel detail to a segmented character at runtime (armour, damage, decals), and anyone who tests that code offline first.
What: Armour stamped live onto a Warden rig came out as an oversized, boxy shell of plates that hid the head, although it passed every offline mock. On the live rig, every part (head, torso, shin) has scale 0.1926 and the same position as the root; each part's shape is offset inside its own voxel space. A ray with Force(shinPart) hit nothing at a point where the same ray without Force hit the shin at 1.29 m.
When: When code measures or edits a part using local offsets that assume scale 1, or relies on a Force-filtered ray to find a part's surface.
Where: The SDK's flat-rig convention (parts parented to the root, geometry offset in voxel space) combined with scaled VoxelData; Cast:Ray(...):Force(part).
Why: Inferred: world, voxel and local units differ per part once VoxelData is scaled, and the force filter does not reliably resolve hits against some scaled parts. Nothing reports the mismatch, so the maths silently produces the wrong size.

Impact

Runtime character customisation is error-prone, and offline tests give false confidence unless they model scaled flat parts. The failure is visual and only shows on a live client.

Mitigation

The armour visuals were reverted within minutes, before any player saw them. They were later rebuilt from measurements taken on the live parts, and now stamp slowly onto small shells (B42); offline mocks have to model scaled flat parts.

Suggested fix

Make Force(obj) rays honour the object's scale, and expose helpers that convert between world, local and voxel space per object (for example obj:WorldToVoxel(p)). Document the flat-rig convention, including where a part's geometry sits, in the segmented-character skill.

B32

Every Lua reload destroys runtime-built objects

MediumWhy: measured↑ index

Who: Any game that pre-builds pools or staged assets at runtime, and every player online when a script is deployed.
What: After a hot reload (a file save or Server:LuaReset), the six pooled Hollow rigs were already gone before the game script's Start() ran. They were inactive, parked at y = −60, with save = false. A debug-logged scan of Scene:GetAllObjects() in Start() found none of them. The pool then rebuilt from scratch: one rig ready at t = 4 s, all six at t = 16 s.
When: On every Lua reload.
Where: The reload path: objects with save = false are removed before any script's Start() runs.
Why: Reload appears to reset the scene to its saved state, so only save = true objects survive. Combined with B29 (each rebuilt rig part costs a 35–95 ms asset bind), a reload re-pays the full cost of every pool.

Impact

Every deploy while players are online costs roughly 10–30 s of stutter. QA measured 53.8 fps with a worst frame of 109 ms after one reload. Together with B8, a live edit costs both the work in flight and everything pre-built.

Mitigation

None found in Lua short of persisting pools with save = true, which bloats the saved scene. Adopting the surviving pool after a reload was tried and reverted, since nothing survives to adopt. Deploys are batched and made when nobody is online where possible.

Suggested fix

Offer a runtime-only flag that keeps an object across Lua reloads without saving it to the scene, or let scripts hand objects to their next instance through an OnBeforeReload hook (see B8).

B33

No per-object time scale, and no gradient or vignette in the UI

LowWhy: measured↑ index

Who: Makers building melee combat feel and screen-space damage feedback.
What: Melee hit-stop needs the striker's and the victim's animation to pause for about 60–120 ms while the world keeps running. The engine offers only the global Time.timeScale, which slows every object, the server simulation and every client. Screen-edge damage feedback hits a similar gap: MUI panels are flat colours, with no gradient or vignette primitive.
When: Whenever a game wants local slow-motion or soft screen-edge effects.
Where: Time.timeScale; MUI panel styling.
Why: Time scaling exists only globally, and the UI has no per-panel gradient fill.

Impact

Common combat-feel techniques need workarounds in game code, which cost client time and are easy to get wrong.

Mitigation shipped

The client wraps each character rig's update and feeds it 5% of the frame time during a hit-stop; the rig still follows its root, and its springs hold. The directional damage vignette is built from four edges of two overlapping translucent bands (a stepped gradient), updated only when the quantised alpha changes. Both are deployed and verified server-side.

Suggested fix

Add a per-object (or per-hierarchy) time scale that scripts and animation read, and a gradient fill for MUI panels.

B34

New scripts can't be attached for about a minute after they sync

LowWhy: inferred↑ index

Who: Makers and agents who add a new script file and attach it straight away.
What: AddScript("Ashlight/Combat FX.lua"), run right after the file synced, failed with "Script file does not exist". The same call worked about a minute later, once the server had written the file's .zma metadata (seen as Combat FX.lua.zma appearing in the mirror).
When: For roughly a minute after a new .lua file first syncs.
Where: Asset registration of new script files on the server; Object:AddScript.
Why: Inferred: a script only becomes an attachable asset once its metadata has been written, and that happens on a delay after the file itself arrives.

Impact

Deploy scripts that create and attach a new module in one step fail on the first try, and the error suggests the file is missing rather than not yet registered.

Mitigation

Wait for the .zma to appear in the mirror, or retry AddScript, before attaching a new script.

Suggested fix

Register a script as soon as its file lands, or have AddScript wait for a pending registration. At minimum, say "not yet registered" instead of "does not exist".

B35

A coding harness can't run automated playtests

MediumWhy: measured↑ index

Who: Agents and makers who build through the CLI and want to test gameplay without a human at the keyboard.
What: Two problems. First, over about 10 minutes, roughly 1 in 3 CLI run-lua and errors calls returned "fetch failed" with no HTTP status, then worked on retry. Second, a native client launched with virtualmatter open joins in edit mode: its player character is spawned but not simulated, so neither a client-side charCtrl:setPosition nor server-side writes move it.

Update (QA): after a server restart, about 1 in 6 read-only run-lua calls (0.1–1 s of work each) returned only "fetch failed": no HTTP status and no engine error, unlike the 504 timeout. An identical retry 15–20 s later succeeded every time. There is no way to tell whether the failed call's code ran.
When: The fetch failures were intermittent, during normal building. The edit-mode client happens on every CLI launch.
Where: The CLI's connection to the session; the native client's launch mode.
Why: The CLI-launched client opens the editor view with a fly camera rather than play mode. The cause of the fetch failures isn't visible from outside the platform.

Impact

Boss fights and traversal (for example, falling into the Undercroft vault) can't be playtested automatically from a coding harness, so they need a human playtester. Flaky calls also make long verification scripts fail partway through.

Mitigation

A retry wrapper with backoff for fetch failures. The CLI's sign-in can also expire mid-session ("Not signed in"), after which a run-lua stops to ask for a device login. For Lua that changes the world, read back its effect before retrying, the same rule as for a 504. Boss logic is verified in offline harnesses plus server-side state probes, with the final feel check left to a human.

Suggested fix

Add an open --play mode (or a headless play client) whose character can be driven from run-lua, and have the CLI retry transient network failures itself and report them with a status.

B36

Left-handed parallelepipeds silently build nothing

HighWhy: measured↑ index

Who: Anyone building oriented boxes with Shape.Parallelepiped(center, a, b, c): cliff facets, banners, lids, armour plates.
What: When the three axes are left-handed (dot(a, cross(b, c)) < 0), the Vox op runs, returns no error, and adds, removes or paints no voxels. A parallelepiped is symmetric, so the same box with c negated (right-handed) builds normally. The World Dresser built every north and east edge facet with left-handed axis triples, (1,0,0)/(0,1,0)/(0,0,−1) and (0,0,1)/(0,1,0)/(1,0,0). All 327 of them are missing live: the north and east cliffs, the north and east runs of the NW and SE corners, and 4 plaza banners. Every right-handed facet on the south and west edges stands, and the rubble placed on top of the missing facets floats at y = 8–9. Two sarcophagus lids in the Undercroft and six arch and coping pieces on the Refuge islands are missing for the same reason.

Update: the damage was wider. Every Hearth edge piece (wall, doorway wall, window wall and door) was a left-handed parallelepiped, so player homes have never had walls or doors: they were roofs on foundations. Also missing: 18 of the 38 Spire-lair pieces, the Colossus's left-arm plates, the Matron's robe train, the Abbot's scythe blade, and every crust cavity and crack line in the Deep Arena, so its crust cells never really opened. The SDK's own Script Load Helper/Explosion.lua builds about half of its debris shapes empty for the same reason. Armour and weapon plates were checked and are unaffected: they are built as (t, n × t, n), which is always right-handed (0 of 955 armour plates were left-handed).
When: On every op whose parallelepiped axes form a left-handed basis.
Where: Shape.Parallelepiped as used by Vox:Add, Remove and Paint.
Why: Measured: a mock that drops left-handed parallelepipeds reproduces QA's ray height maps cell for cell. The shape's inside test presumably assumes a positive determinant, so a negative one makes every point test as outside.

Impact

Whole sections of a world go missing with nothing in the error log, and anything placed on top of them floats. It surfaced as the "boulders hanging in the air" over the north and east edges. Worse, it quietly removed core mechanics: homes that were never enclosed undercut the shelter system from launch, and a boss arena whose crust could never open. Any game that uses the SDK's explosion debris inherits the bug.

Mitigation

Normalise handedness in the game's parallelepiped helper before every call: if dot(a, cross(b, c)) < 0, negate c. The north and east cliffs were rebuilt this way (1,287 ops, 0 errors, with nobody online, followed by a full terrain save). A live ray audit afterwards found the lowest crest point at 6.7 m on the north edge (was 0.5 m) and 5.7 m on the east (was 0.0 m), 7.1 m and 6.9 m in front of the Spire, and no floating cells on either side. The four missing plaza banners were laid again. Every builder now goes through the guarded helper, and fixes were verified live by probe: home edge pieces 0 of 3 → 3 of 3, with a boot-time rebuild of any missing piece; Spire-lair buttress tops 0.7 → 5.8 m, keystone placed; Deep Arena rebuilt, with a 1.9–2.4 m cavity under every sampled crust cell (was none).

Suggested fix

Normalise the basis inside Shape.Parallelepiped, or at least raise an error for a left-handed or degenerate basis.

B37

Voxel ops never report an empty result

MediumWhy: measured↑ index

Who: Anyone building worlds procedurally through the Vox API.
What: Neither Run() nor OnFinished reports how many voxels an op changed. A degenerate shape (a left-handed parallelepiped as in B36, a zero-size box, or an op wholly outside its VoxelData) fails silently, and the op queue still reports it as done.
When: On any op that ends up changing nothing.
Where: The Vox op pipeline and its completion callback.
Why: The completion path carries no result count or warning.

Impact

Silent no-ops are found only by auditing the world afterwards, often hours later and by a different agent. B36 went unnoticed through several builds for this reason.

Mitigation

Procedural builds are verified with read-only ray audits (height maps and upward "underside" rays) rather than by trusting the queue's "done". Because ops are asynchronous, a readback in the same run-lua call still sees the old voxels: a repair script once reported a crate's old height although the remove had worked. Verification now runs as a separate call a second later, or through :OnFinished.

Suggested fix

Pass the number of voxels changed to OnFinished, and log a warning when an op changes nothing.

B38

SDK editor scripts cost about 3 ms per server frame when idle

LowWhy: measured↑ index

Who: Every world, whether or not anyone is playing.
What: The script profiler on the live server, with nobody online, showed five SDK and editor scripts taking 3.0 ms per frame between them: Chat 1.25 ms, Click To Play Prompt 0.80 ms, Particle Updater 0.51 ms, Voxelizer 0.29 ms and Shape Manager 0.17 ms. All of Ashlight's game scripts together took under 0.6 ms.
When: Every server frame, including on an empty server.
Where: The platform's built-in Lua scripts that ship with every world.
Why: These scripts update every frame whether or not there is anything for them to do.

Impact

At idle, platform scripts cost at least five times as much as the game's own. With the thin headroom in B17 (about 4 ms spare per frame at 60 fps), that is a large share of the budget spent before any gameplay runs.

Mitigation

None from game code: the scripts belong to the platform.

Suggested fix

Make these scripts event-driven or sleep them when idle (no chat messages, no pending prompts, no particles, no voxelize or shape jobs), and let a world disable the ones it doesn't use.

B39

Server memory climbs on an idle, empty server

LowWhy: inferred↑ index

Who: Long-running worlds, and hosts that pack several worlds per machine.
What: QA memory samples after a restart showed the server process growing from 12.6 GB to 15.9 GB over 75 minutes with nobody online. The largest grower was the Voxel DB atlas, from 3.76 GB to 5.0 GB (about 17 MB per minute). The surface store and the Lua heap stayed flat. Before the previous restart, memory had plateaued at about 20.8 GB.

Update: the atlas then held at exactly 5,000 MB for over 25 minutes, which looks like an engine cap. Total memory reached 16.5 GB and kept growing at about 7 MB per minute after the atlas stopped; the surface store was 5,251 MB and the Lua heap 12.4 MB. Idle performance stayed healthy throughout: 57.7–58.4 fps, 11.3–11.5 ms per update, worst frame 22–27 ms.
When: Continuously after a restart, on an idle server.
Where: The engine's Voxel DB atlas.
Why: The atlas fills a cache up to a fixed 5,000 MB cap (measured). The remaining slow growth of about 7 MB per minute comes from somewhere else, and its source is unknown.

Impact

An idle world gains gigabytes of memory, so its footprint depends on how long it has been up rather than on what players are doing. If the plateau doesn't hold, a long-running world eventually runs out of memory.

Mitigation

The atlas is confirmed capped; the residual growth was still unexplained when the project paused. Nothing in game code touches either. With sessions running, memory grew faster: 13.2 GB after a restart to 16.7 GB 30 minutes later (limit 32 GB).

Suggested fix

Cap the atlas with a documented, configurable budget, and expose its size and eviction counts through Server:GetMemoryUsage().

B40

Inactive objects ignore forced raycasts

LowWhy: measured↑ index

Who: Anyone measuring or decorating pooled or template objects while they are parked out of sight.
What: Cast:Ray(...):Force(obj):Run() returns nil for an object that is not activeInHierarchy, even with the right transform and voxels. This was seen on Rig Library templates and on pooled enemy rigs.
When: Whenever the target object, or any of its parents, is inactive.
Where: Cast:Ray with Force.
Why: Inferred: raycasts only consider active objects, and Force doesn't override that. Nothing in the result says the target was skipped.

Impact

Pools (B29) keep objects inactive precisely so they cost nothing, which is exactly when a game wants to measure and dress them. Together with B31, forced rays are unreliable for any rig that isn't live and at scale 1.

Mitigation

The Hollow pool activates a parked rig (at y = −60) only while it is being dressed, and only between fights.

Suggested fix

Let a forced ray hit its target regardless of active state, or return an explicit "target inactive" result instead of nil.

B41

`AE:GetDebugTime()` restarts with the server process

HighWhy: measured↑ index

Who: Any game that persists timestamps across server restarts: respawn timers, cooldowns, rate limits, save schedules.
What: The server restarted unexpectedly at about 22:24. Memory was at 13.1 of 32 GB, so it wasn't out of memory, and the cause is unknown. Right afterwards, AE:GetDebugTime() read 79 s: it is process uptime, not wall-clock time. Every timer Ashlight had persisted as "wall clock" was wrong after the restart: the Abbot's 45-minute respawn, the siege's once-an-hour limit, the Omen ward and Abbot-stirs timers, and the Terrain Keeper's save gaps. Game state saved every 2 minutes was reloaded from its last disk save, so a one-shot world repair made just before (re-seating a stacked home) was reverted.
When: On every server process restart, planned or not.
Where: AE:GetDebugTime(); the platform's restart behaviour. The CLI's own agent guide lists AE:GetDebugTime() as "wall clock", next to Time.time (simulation time).
Why: The sandbox removes os.time, and no wall-clock or persistent clock replaces it. The only clocks are simulation time, which restarts on every Lua reload (B27), and process uptime, which restarts with the process.

Impact

Timers that persist across restarts silently fire early, or never, after a restart. With no visible cause and no warning before the restart, the last few minutes of game state are lost, along with any repair made in that window.

Mitigation shipped

A persistent game clock: seconds accumulated in the ledger, plus the uptime since boot, with older timestamps clamped to "now". Downtime doesn't count toward any timer. The ledger is saved to disk immediately after any one-shot repair, and a boot pass re-applies home repairs on its own. One edge case showed up after a later restart: the Terrain Keeper logged "last full save 1000008593 s ago" (an edge case in the game's clock clamp) and ran a full terrain save right after boot. That was harmless on an empty server, but a full save freezes the server for 33–60 s (B19), so the edge case matters. Restarts themselves are tracked in B44.

Suggested fix

Expose a real wall-clock time (for example Time.unix), correct the guide's "wall clock" label, and log or announce the reason for every server restart.

B42

Baked rig variants are huge, every bind stalls a frame, and a periodic freeze came with them

MediumWhy: inferred↑ index

Who: Games that bake cosmetic detail into rig assets to avoid runtime voxel cost (B29), their makers' mirrors, and every client that streams the assets.
What: VoxelData:SaveAs on a scaled rig part stores the part at its fine internal resolution, so every baked variant is a full copy. Mace tiers came to about 7 MB each, and the 111 armour and weapon variants to 329 MB of .am assets, all of which sync to the maker's mirror and have to stream to clients. Separately, the patch that switched characters over to the baked armour introduced a freeze of about 700 ms every 8.4 s on an idle server, with the heap growing about 2 MB per freeze and no game Lua running inside it. Reverting that patch fixed it at once (58.2 fps, worst frame 28 ms).

Update: the freeze came from bake code inside the character script, not from the assets themselves. With that code removed, an A/B test showed no freezes in about 7,000 frames both with and without the baked templates loaded. Binding a baked part still costs 100–500 ms each (forging from full baked part copies: 5 frames over 50 ms, worst 502 ms).

Update (armour shells): to shrink the binds, only the armour voxels were baked onto small, separate "shell" objects at the part's scale, about 20–35% of a part. Binding them still blocked the server frame for 40–360 ms, almost regardless of size:

Bound object	Voxels	Synchronous bind (Lua time)
Renown shell	154	98 ms
Greaves shells	329 / 1,175	40 / 82 ms
Helm shells	995 / 3,535	65 / 91 ms
Mail shells	2,468 / 6,924 / 7,827	158 / 277 / 358 ms
Weapon shells	89–96k	1.2–1.4 s
Plain parts, for comparison	head / torso / mace	114 / 265 / 414 ms

Lesson: A bind has a large fixed cost (about 40–100 ms) and lands in a single frame, so "bake and bind" can't make runtime swaps smooth: pre-built objects have to be activated, not bound. The character code for the shells also brought the periodic ~700 ms idle freeze back, so it was reverted too. Thin, spread-out assets cost far more than their voxel count suggests: the 89k-voxel mace shell took 1.2–1.4 s to bind, against 414 ms for the solid 112k-voxel mace.
When: When baking variants of scaled parts; the freeze appeared once the baked variants were wired into the character script.
Where: VoxelData:SaveAs on scaled VoxelData; asset handling for large numbers of voxel assets.
Why: The size follows from the scaled part's internal resolution (see B29 and B31). The bind cost is engine-side and synchronous. The freeze is not proven, but it correlates with heavy SaveAs use: both sessions that ran 75–111 bakes then showed a ~700 ms engine-side frame every ~8 s at idle (heap +2 MB each time), a Lua reload cleared it, and sessions that only bound assets never showed it.

Impact

Baking trades a voxel-commit stall for disk, sync and streaming size, and here also for a periodic server-wide freeze. Neither runtime stamping (B29) nor baking was a clean way to give characters per-tier detail; the eventual answer was slow live stamping onto small shells.

Mitigation

Armour no longer binds or saves assets at all: it is stamped live onto small shells at one op per frame (47 fps, worst frame about 50 ms, over about 35 s per full set), and an idle server showed no freezes over 100 ms in 6,957 frames afterwards. The 111 baked variants (329 MB) and the 75 shell templates (60 MB) were deleted. Hollow crusts keep their three baked torso variants, which are bound once at pool time, never mid-fight.

Suggested fix

Let SaveAs resample to a target resolution, or support instancing and sharing of voxel data between variants. Keep any periodic asset scan incremental, and expose its cost in the engine timers.

B43

The server renders a headless client every frame

MediumWhy: measured↑ index

Who: Every world, at all times.
What: Engine timers on the live server with nobody online: Update Frame (server) 9.8 ms, Render Frame 6.2 ms, Update Frame (FirstScene) 1.7 ms, and Scene/Objects 5.7 ms for 266 active objects (about 21 µs each). FirstScene is the built-in headless client scene that the screenshot tool uses. So about 8 ms of every server frame goes to rendering and updating a client nobody sees, and game update plus that render comes to about 17 ms: roughly 58 fps at idle.
When: Every server frame, whether or not anyone is connected or taking screenshots.
Where: The server process's built-in client (the FirstScene scene and its render frame).
Why: The headless client used for screenshots and agent tooling runs and renders continuously instead of on demand.

Impact

A server can't reach 60 fps even with no players, and the ~8 ms comes straight out of the thin headroom in B17. Together with B38 (idle SDK scripts), much of the frame budget is spent before any gameplay runs.

Mitigation

None from game code. The only lever is fewer active objects, which the game pursues through pooling and inactive components (see What held up). A census found only 101 active objects left, so there is little more to cut.

Suggested fix

Render the headless client only while a screenshot or agent capture is pending, or throttle it to a low rate, and show its cost separately in the engine timers.

B44

The server restarts without warning or a reason

HighWhy: unknown↑ index

Who: Everyone connected, and any game state not yet written to disk.
What: The live server restarted three times in one night with no warning and no reason given: at about 20:13, at about 22:24, and at about 01:10. Memory stood at 13.1 GB and 14 GB of 32 GB at the last two, so none of them looks like running out of memory. The 01:10 restart came during the first live forge measurement; repeating the same forge five minutes later ran to completion without one, so the trigger is unknown.

Update: a fourth restart came at about 01:35 (uptime reset to 46 s), while a background job was stamping small voxel ops onto pooled Hollow rig parts (scale about 0.19) at one op every three frames. Just before it, that "idle" dressing measured 25.1 fps, an average update of 38.9 ms, 32 frames over 50 ms and a worst frame of 596 ms; an offline mock had predicted about 8 ms per op frame. Three of the four restarts happened during live voxel stamping on scaled rig parts or shells: at about 22:24 (the first armour stamping), 01:10 (a forge) and 01:35 (Hollow dressing).

Update: two more restarts came the next morning, with no stamping running: at 07:22, while a playtester in a browser walked from the tutorial island to the hub (gameplay had been very choppy first, and no Lua errors preceded it), and at 07:58 with a test client connected. At that point three orphaned sync processes were running at once and pulling the 370 MB terrain in parallel on every terrain save (B48). That is a likely contributor, but it is unproven.

Update (#7): the play server restarted while a playtester fought the Gate-Warden ("Disconnected for unknown reason"). Sync was off and no deploys were running, so orphaned syncs can't be the only cause; the server came back on the right terrain. The leading theory is runtime voxel ops on scaled rig parts with players connected. Every crusted hit had been painting a ~0.19-scale torso part (about 140 times the voxels of the same shape at scale 1), with no cap across players or Hollows. And the Gate-Warden's knight rebuild, which runs 68 stamping jobs after every Lua reload, stamped regardless of nearby players once the server had been up for 10 minutes.
When: Three of seven during sustained voxel stamping on scaled rig parts, and one more during a boss fight where scaled-part ops were running; two the next morning while orphaned syncs were downloading the terrain (B48); one (about 20:13) under ordinary load.
Where: The server process or its host.
Why: Unknown. Nothing visible from inside the world, or through the CLI, records a cause. The link to stamping on scaled parts is a correlation, not proof: a repeat of the 01:10 forge did not crash.

Impact

Who

Makers and agents running sync for a long session, which is the documented workflow, and everyone in the world they are editing.

What

Stopping a backgrounded npx virtualmatter sync killed only the npx wrapper; the node child kept running. Three syncs were live at once (started at 01:51, 07:26 and 07:44). Between them they:

pulled the 370 MB terrain .am, including the engine's half-written .am~ save temp, in parallel on every server terrain save;
reported a false "changed both here and in the world" conflict on every push (the saved conflict copy was byte-identical to the pushed file), sometimes with no "pushed" line;
pushed partial .vmdownload-* fragments into the world (three were found and deleted) and ping-ponged scene files;
re-pushed unchanged scripts, which caused two full Lua resets mid-session.

When

2026-10-07, over a morning session, after earlier syncs had been "stopped".

Where

The CLI's sync command: its process handling, its file filters and its conflict check.

HighWhy: measured↑ index

Who: Makers who publish a world with large voxel files to a play server, and the players who open the link.
What: Each "Update play servers" restarted the play server before the 390 MB terrain upload had finished, so it booted the old starter scene ("Plane 52m Dark Tile4m"). With "Persisting scene changes" on, the running game's regular scene save then wrote that plane-linked scene back over the published scene on the play server, so every later update booted the broken scene again. The project page's per-server status also stayed "queued" for 15+ minutes after the sync had finished, including after a Restart.

Update: on a later publish, the first "Update play servers" restarted the server and reported finished while delivering none of the changes; only then did the page say "Some of your editor's content never reached every play server". A second update delivered the scripts and music, restarted again, and the repainted 390 MB terrain landed two minutes after the boot, so it only takes effect on the next cold start (play servers sleep 15 minutes after the last player leaves).
When: 2026-10-07, on the first publishes to a Europe play server.
Where: The play-server update path (restart order versus file upload), scene persistence, and the project page's server status.
Why: The update doesn't wait for large files to land before restarting, and nothing stops a running server from persisting a scene that booted on the wrong assets.

Impact

Who: Games that use generated images as UI icons.
What: The 39 generated dark-fantasy icons turn into smudges when drawn smaller than about 24 px on a dark UI.
When: 2026-10-07, while rebuilding Ashlight's whole interface (round 21).
Where: Generated image assets used in MUI icon slots.
Why: Painterly detail doesn't survive being scaled down to a few pixels.

Impact

Small HUD slots (resource rows, the loot feed, the ability bar) can't use the art directly.

Mitigation

The art is drawn only at 24 px and up; smaller slots use coloured shape tokens, and art under 48 px gets a thin ember outline.

Suggested fix

Offer an icon-generation mode that produces simple, high-contrast glyphs, or allow vector or atlas icons for small sizes.

Probe	Load	Result	Verdict
Server raycast	1k / 10k / 50k rays	≈24 µs per ray; 50k = 1.2 s stall	Held
RPC flood	50k, then 500k	50k queued in 72 ms; no visible impact	Held, but no backpressure
Tiny static edits	5,000 in 1 frame	All accepted, 240 ms	Held
Big edits	Sphere r 4→28	Soft reject r 12–17, hard refusal r ≥24	Held, guard worked
Character controller	1 call	≈299 µs on the server	Broke at scale (B1)
Networked write	Changed vs same value	10.1 µs vs 0.1 µs	Fine; write only on change
Networked payload	16 / 64 / 256 MB	All accepted; 3.57 s ping	Broke (B5)
Lua heap	256 MB → 3 GB	No cap; process 21.7 GB	Broke (B6)
Client particles	500 / 2k / 8k	29.2 / 26.2 / 12.7 fps	Broke (B9)
Voxel op on a 0.2-scale part	Add plate 0.28 m / 0.15 m	+114 ms / +33 ms frame time	Expensive (B29)
Voxel op on a 0.2-scale part	Paint plate 0.28 m / remove sphere 0.24 m	+17 ms / +64 ms frame time	Paint is the cheap option
Any voxel op on a ~0.19-scale part	1 op	≥6.6 ms engine commit, 7–30 ms typical	Expensive (B29)
Same Add on a scale-1 object	Plate 0.28 m	+0 ms	Held
Bind a template rig part	1 part	≈60 ms Lua	Expensive (B29)
Bind a small baked shell	154–7,827 voxels	40–358 ms, in one frame	Broke (B42)
Bind a baked weapon shell	89–96k voxels	1.2–1.4 s	Broke (B42)

#	Mitigation in Ashlight	Status
B1/B16/B17	A ray-probed enemy mover with positions held in Lua, engine objects re-centred every 10 s, a 5-part enemy rig, hidden templates switched off, AI at 5 Hz near players and 2 Hz away, and a cap of 24 enemies	Shipped: 53–56 fps worst case
B2	Enemies spawned one per 0.3–0.4 s; boss bodies built across frames	Shipped
B3	Settle waits after terrain clears, and a fall-catch that teleports players back	Shipped (workaround)
B4/B21	Staging folders, single-write copies, compare-after-sync, one deployer	Process
B7	Generation jobs queued at most 4 at a time, with failure tracking. 11 of 11 sound effects generated fine	Shipped
B8	Builders that resume, idempotent rebuilds, and no Lua saves while generation callbacks are pending	Process
B12	Game state serialized as Lua-literal strings in a ledger; the terrain journal kept as its own string	Shipped
B19/B20	Terrain Keeper: full save only when the server is empty, plus a replayed edit journal	Shipped
B9	Particle budgets per effect; no particles in the server's headless client	Shipped
B23	Event-queued key input in the player script	Shipped, verified live
B24	Audio module re-indexes on asset sync and falls back to built-in sounds	Shipped
B26	Server rigs pose at 4 Hz in the headless client; visual checks done in a visible browser client, or with `screenshot --client <id>` through a real client with lighting on	Process
B27	Persisted timers moved off `Time.time`; now on the persistent game clock (B41)	Shipped
B28	Terrain reworks scheduled for an empty server; server-side ray audits; floating props found and grounded (a dead tree hung 35 m up, and an oil drum 1.3 m up after the Pit rim was reshaped)	Process + fixed
Arena	Pit rim reshaped into crags; stair access to all three high-ground shelves verified with a 0.5 m-step flood fill in both directions	Verified live
B29	Pre-built enemy rig pool; spawn 320 ms → 4–20 ms	Shipped, measured
B29 (follow-up)	Bulwark rigs pooled the same way (previously about 300 ms per spawn); the Matron's 10 parts staged ahead of the fight, and her spawn measured at 10 ms live	Shipped, measured
B28 (update)	Phantom geometry also seen 2–10 m from the camera after about 600 hub edits; server rays confirm the plaza is clear. Hub and plaza dressing now follows the empty-server rule too	Process
B23 (follow-up)	Mouse clicks and UI keys are now event-queued too (attacks, build placement, menus)	Shipped
B19 (follow-up)	The Keeper's automatic empty-server save fired twice on its own: the journal was cleared and the terrain marked saved	Verified live
"Odd walls"	The 12 bare ruin shells were rebuilt as dressed ruined buildings (2 168 ops, 0 errors). The hub wall became a scavenged palisade, ash patches became flat paint, the sphere berm became a continuous boundary, and test structures were removed through the journal	Shipped; the edge was later rebuilt as faceted cliffs (B36)
B30	Edge walls kept at least 2.9 m above the highest foothold; the Undercroft gated by a one-way door rather than a ledge	Shipped
B31	Stamped armour visuals reverted before players saw them, then rebuilt from live part measurements as shell stamping (B42); offline mocks must model scaled flat parts	Resolved
B32	Deploys batched and made with nobody online where possible; adopting the pool after reload tried and reverted	Process
B19 (update)	One automatic empty-server save measured at a 33.4 s main-thread stall; the Keeper saves non-journaled edits at most hourly	Measured
B33	Hit-stop by scaling each rig's update time; vignette from stepped translucent bands	Shipped
B20 (update)	Procedural props (oil drums) found empty after a restart; rebuild on boot when the VoxelData is empty, or save props as assets	Fix proposed; not confirmed shipped before the pause
B33 (update)	Hit-stop and directional damage vignette deployed	Shipped, verified server-side
B34	Wait for the `.zma` or retry before attaching a new script	Process
B35	Retry with backoff for fetch failures (read back state before retrying world-changing Lua); boss logic checked in offline harnesses and server-side probes; human playtests for feel	Process
B36	Handedness normalised in the parallelepiped helper; north and east cliffs rebuilt (lowest crest N 0.5 → 6.7 m, E 0.0 → 5.7 m, 0 floating cells); plaza banners restored	Shipped, verified live
B37	Read-only ray audits (height maps, underside rays) after every procedural build	Process
B36 (update)	Guarded helper in every builder; homes 0/3 → 3/3 edge pieces with a boot-time rebuild; lair buttresses 0.7 → 5.8 m; Deep Arena crust cavities 0 → 1.9–2.4 m; armour confirmed unaffected	Shipped, verified live
B29 (siege)	Permanent Hollow pool of 14 for the siege: its warning phase holds 59 fps with a worst frame of 31 ms (was 48.5 fps, 140 ms)	Shipped, measured
B38	None possible in game code	Reported
B39	Watched for a plateau; the atlas capped (see update)	Superseded below
B29 (stress test)	No cold builds mid-fight, throttled pool rebuild, jittered AI think, lighter static jobs: steady 50–51 → 56.3 fps, worst frame 84–94 → 29 ms. Ramp-up spikes (p99 341 ms) traced and fixed in rounds 12–13 (see below)	Fixed later, measured
B39 (update)	Atlas confirmed capped at 5,000 MB; residual growth of ~7 MB/min unexplained at the pause	Open at the pause
B40	Parked rigs activated briefly, between fights, while being dressed	Shipped
B29 (resolved)	Spawn spikes traced to voxel adds on scaled parts; crust baked into 3 variant assets, chips painted, pools paced. Ramp p99 508 → 60.9 ms, worst 893 → 73 ms; steady worst 84–94 → 51 ms	Shipped, measured
B29 (forging)	Forging armour on a player rig: 43.5 fps, worst 83 ms for ~6 s; baked tiers tried and reverted (B42). Now live 1-op/frame stamping onto shells: 47 fps, worst 48–52 ms over ~35–40 s	Acceptable
B29 (round 14)	Ramp p99 38.6 ms, worst 73 ms; steady 54.3 fps, p99 30.7 ms	Measured
B42	Baked-armour character patch reverted (58.2 fps, worst 28 ms after); assets kept for relinking	Reverted
B42 (update)	Freeze traced to the in-script bake code, now removed: 0 freezes in ~7,000 frames with and without templates. Baked binds still 100–500 ms, so armour shells were tried next	Superseded below
B42 (shells)	Armour shells measured at 40–360 ms per bind; shell code re-introduced the periodic freeze and was reverted; 111 obsolete variants (329 MB) deleted. Next: live 1-op/frame stamping onto shells	Reverted
B42 (round 5d)	No binds or `SaveAs` for armour; idle shows 0 freezes over 100 ms in 6,957 frames; 75 shell templates (60 MB) deleted. Freeze correlates with heavy `SaveAs` use (unproven)	Shipped, measured
B44	Restart-proof state: persistent clock, immediate ledger saves, boot repair passes; all survived the 01:10 restart	Process
B29 (round 16)	Hollow, Body and Beacon components inactive, driven by one manager: steady 54.7 fps, worst 32 ms, no frames over 33 ms; script time 8.66 → 4.15 ms	Shipped, measured
B43	None possible in game code; fewer active objects	Reported
B41	Persistent game clock in the ledger; immediate ledger save after one-shot repairs; boot-time home repair	Shipped
B44 (update)	Fourth restart during background Hollow dressing (25.1 fps, worst 596 ms beforehand); dressing reverted (59.7 fps, worst 17 ms); continuous stamping on scaled parts shelved	Shelved
B45	Restart `sync` after every server restart; confirm "pushed" lines per deploy	Process
B46	Joins held until the chat name arrives (up to 20 s); records refused for guest names; playtesters sign in	Shipped, verified
B47	Vectors duck-typed on .x/.y/.z; Reach smoke test 16/16 including the ping broadcast	Shipped, verified
B44 (#5, #6)	Restarts at 07:22 (playtester walking to the hub) and 07:58 (test client); orphaned syncs stopped	Open (cause unknown)
B29 (rounds 17–18)	No new rigs while 18+ Hollows are out: worst case 56 fps, worst frame 36–37 ms (was 64–102 ms)	Shipped, measured
B48	Long-running sync stopped; push-only deploy script on the CLI's files client	Process
B49	Close clients gracefully	Process
B50	Rest transform published in a networked table; client waits for it	Shipped
B44 (#7)	Rig-op budget (4/s, none near players), zero-op crust chips, Warden rebuild never stamps within 80 m; siege smoke rig ops 34 → 0	Shipped, measured
B51	Boot-time terrain relink (`repairTerrainLink`); verified on the play server	Shipped, verified
B51 (update)	A publish reported finished while delivering nothing; retried, then verified the play server's code, terrain and music by probe	Process
B52	Scenes backed up; live world saved as InitScene with its own terrain copy	Fixed by hand
B53	Case-insensitive lookup with a v1 fallback	Shipped
B54	Check durations; regenerate short takes	Process
B55	Plain copies for each networked table	Shipped
B56	Frame helper defaults to `overflow = "Wrap"`; explicit `endLine` elsewhere	Shipped
B57	Sizes kept in game tables, never read back	Shipped
B58	Disabled buttons are a style, not the engine flag	Shipped
B59	No frame on clear elements; only the selected row is framed	Shipped
B60	ASCII-only uppercase helper for titles	Shipped
B61	Art at 24 px and up; coloured shape tokens below	Shipped
B5	No game-side cap exists; networked state kept small and written only on change	Avoided
B6	None possible in game code; no unbounded tables, and long probes kept short	Avoided
B10	None in game code	Open (platform)
B11	The game replicates the transforms it needs itself, through networked tables and RPCs (see also B50)	Shipped
B13	Large builds split into many small ops	Process
B14	Known Editor Tools boot errors ignored when checking engine errors	Process
B15	Symbols checked against the running build before use	Process
B18	Characters stay server-side rigs, pooled (B29)	Worked around
B22	Client tuning based on code-path costs, not a hidden tab's fps	Process
B25	A tunable lift for the vitals panel (`AshUI.VITALS_LIFT`); off by default	Shipped

Executive summary

Summary

For the engine team: renderer and server-threading refactor

1. The server frame budget, as a real game saw it

2. Main-thread stalls: the best targets for a job system

3. The client frame, measured on a real native client

4. Networking and state findings relevant to the refactor

5. Stability: the unexplained restarts

6. Ranked: what the refactor could do for games like this

Breakage index

The 5Ws, per breakage

Server NPC capacity tops out near 40

Impact

Repro

Suggested fix

Creating voxel objects stalls the whole server

Impact

Repro

Suggested fix

A client loses collision the server still has

Impact

Repro

Suggested fix

Sync overwrites a newer local save

Impact

Repro

Suggested fix

Networked tables accept any payload size

Impact

Repro

Suggested fix

The server Lua heap has no ceiling

Impact

Repro

Suggested fix

imagine() jobs stall forever, silently

Impact

Repro

Suggested fix

Hot reload orphans everything in flight

Impact

Repro

Suggested fix

The documented particle cap isn't enforced

Impact

Repro

Suggested fix

Scene-wide LOD starvation

Impact

Repro

Suggested fix

Transforms that don't replicate

Impact

Repro

Suggested fix

Persisted tables come back opaque

Impact

Repro

Suggested fix

Edit guard estimates are hard to predict

Impact

Repro

Suggested fix

Editor Tools fail at boot

Impact

Repro

Suggested fix

Docs and runtime disagree

Impact

Repro

Suggested fix

Moving voxel objects plus raycasts spike the server

Impact

Repro

Suggested fix

Fixed per-object cost and very little headroom

Impact

Repro

Suggested fix

Clients can't instance voxel assets

`imagine()` jobs stall forever, silently

`Time.time` restarts on every Lua reload