How a single bad SetTimeout crashed our 96-slot server for 3 hours

Posting this so others can learn from our pain.

What happened

Last Saturday at peak hours (~80 players online) the server started lagging hard. Resmon was clean, no individual resource above 2ms. CPU on the host machine was at 100% across all cores.

After 40 minutes of "is it the host?" and rebooting the box, we found it: a third-party script we'd installed two days prior had this in its main loop:

CreateThread(function()
    while true do
        Wait(0)
        -- check if a player has a specific item
        for _, ply in ipairs(GetPlayers()) do
            local items = exports.ox_inventory:GetInventory(ply)
            for _, item in pairs(items) do
                if item.name == 'flashlight' and item.count > 0 then
                    -- some FX
                end
            end
        end
    end
end)

GetInventory is server-side, called every frame, for every player, with a nested loop over every item. With 80 players carrying ~30 items each, that's 2400 inventory lookups per frame, 60 times per second.

Why we missed it

Resmon shows client ms. This was bleeding the server thread. Server-side perf is invisible until everything melts.

What we changed

1. Moved to a stateBag-driven check (player toggles flashlight → stateBag fires → react)
2. Added a server-side perf monitor that alerts in Discord when any resource exceeds 5ms tick
3. Mandatory code review for every external script before install — no more drag-and-drop installs

-- Cheap server-side tick logger
local lastTick = GetGameTimer()
SetInterval(function()
    local now = GetGameTimer()
    local delta = now - lastTick - 1000
    if delta > 100 then
        print(('SERVER LAG SPIKE: +%dms'):format(delta))
    end
    lastTick = now
end, 1000)

Lesson: don't trust resmon to catch server-side issues.