The monster that wasn’t

Everything looked right

I checked all the usual suspects for rendering issues: clipping, bad billboarding logic, problems in my port of Doom’s fixed point math, and a bunch of places where the rendering logic tied into the rest of the game. None of it explained how a monster that shouldn’t exist could end up on screen.

I even compared the port’s logic against the reference implementation and found it was pretty much identical, aside from the obvious language differences. As far as I could tell, the game state was totally correct. So where the hell did that monster come from?

Going back to the recording

At this point I realized I was asking the wrong question. I kept looking for a rendering bug involving a monster, but I did not actually know which object was producing those pixels. The video gave me a useful time and place to start: the moment something appeared at the far left edge of the screen. That run had also been saved as a Wax recording, so I could return to that exact moment instead of adding more logs and hoping to catch it again.

A Wax query fn is new code that runs against the state in an existing recording. I ran one at the moment from the video. It took the player’s recorded view and calculated where each object other than the player that was still in the game and in front of the camera would land horizontally. It kept the object whose position was closest to the left edge, then returned the fields I needed to identify it and inspect its state. I had no idea what I expected to find at this point. I just needed one plausible candidate that lined up with the monster I was seeing in Wax but not in the reference.

import Fixed, Mobj, MobjFlags, ThinkerState, Trig, game from DoomWax;

struct EdgeRow {
    int32 tick;
    int32 mobjIndex;
    int32 type;
    int32 x;
    int32 y;
    int32 oldX;
    int32 oldY;
    bool interpolate;
    int32 facingRaw;
    int32 sprite;
    int32 spriteFrame;
    int32 mobjStateRaw;
    int32 tics;
    int32 flagsRaw;
    bool noSector;
    int32 subsector;
    int32 originX;
}

query fn SpriteEdgeCandidate() : EdgeRow? {
    guard (game.simWorld) |sim| else return null;
    guard (sim.runtime.PlayerMobj(0)) |player| else return null;

    // Recreate the recorded player's view of every active Doom object.
    Fixed viewSin = Trig.Sin(player.facingAngle);
    Fixed viewCos = Trig.Cos(player.facingAngle);
    Mobj?[] mobjs = sim.runtime.Mobjs;
    int32 count = sim.runtime.MobjCount < mobjs.size ? sim.runtime.MobjCount : mobjs.size;
    Mobj? best = null;
    int32 bestIndex = -1;
    int32 bestDistance = int32.MaxValue;
    int32 bestOriginX = 0;

    for (int32 i = 0; i < count; i++) {
        guard (mobjs[i]) |mobj| else {
            continue;
        };
        if (mobj.thinkerState == ThinkerState.Removed || mobj.playerIndex >= 0) {
            continue;
        }

        // Transform the object from world space into the player's view.
        Fixed trX = mobj.x.Sub(player.x);
        Fixed trY = mobj.y.Sub(player.y);
        Fixed tz = trX.Mul(viewCos).Add(trY.Mul(viewSin));
        if (tz.data <= 0) {
            continue;
        }

        // Project its horizontal position into screen pixels.
        Fixed tx = trX.Mul(viewSin).Sub(trY.Mul(viewCos));
        Fixed scale = new Fixed.FromInt(160).Div(tz);
        int32 originX = new Fixed.FromInt(160).Add(tx.Mul(scale)).ToIntFloor();
        int32 distance = originX < 0 ? -originX : originX;
        if (distance >= bestDistance) {
            continue;
        }
        best = mobj;
        bestIndex = i;
        bestDistance = distance;
        bestOriginX = originX;
    }

    guard (best) |candidate| else return null;
    return new EdgeRow {
        .tick = game.tickCount,
        .mobjIndex = bestIndex,
        .type = candidate.mobjType as int32,
        .x = candidate.x.data,
        .y = candidate.y.data,
        .oldX = candidate.oldX.data,
        .oldY = candidate.oldY.data,
        .interpolate = candidate.interpolate,
        .facingRaw = candidate.facingAngle.data,
        .sprite = candidate.sprite as int32,
        .spriteFrame = candidate.frame,
        .mobjStateRaw = candidate.mobjState as int32,
        .tics = candidate.tics,
        .flagsRaw = candidate.flags as int32,
        .noSector = candidate.flags.HasValue(MobjFlags.NoSector),
        .subsector = candidate.subsectorIndex,
        .originX = bestOriginX
    };
}

It wasn’t a monster

{
  "mobjIndex": 205,
  "type": 41,
  "mobjStateRaw": 0,
  "sprite": 0,
  "originX": 2,
  "noSector": true
}

The query returned one object almost exactly where the phantom had appeared. Doom calls it a Teleportman: an invisible marker that tells the game where to place something after a teleport. It wasn’t alive. Its state was Null, and it was marked NoSector. In other words, it was explicitly not supposed to appear in a sector’s render list. But there it was, projected right near the left edge of the screen.

Then I noticed that its sprite value was zero. In Doom’s sprite table, zero means TROO, the imp sprite. The marker wasn’t really turning into a monster. The renderer was drawing an object it should never have received, and that object’s default data happened to look like a perfectly ordinary imp. I knew what I was seeing now. I still didn’t know how it had reached the renderer.

How it got there

Now that the query had identified one object and its NoSector flag, the source comparison became much narrower. The reference still gives a NoSector object a subsector because the simulation needs to know where it is. But it deliberately leaves that object out of the sector’s render list.

My Wax port stored all objects in one flat snapshot and rebuilt the render list by checking their subsectors. I had accidentally treated “this object is located in this sector” and “this object should be rendered in this sector” as the same thing. The missing NoSector check was how the teleport marker slipped through.

The missing check

Once I knew what the object was, the fix was obvious. The renderer needed to skip objects marked NoSector before rebuilding the sector’s render list.

foreach (renderMobjs) |thing| {
    if (thing.flags.HasValue(MobjFlags.NoSector)) {
        continue;
    }

    if (thing.subsectorIndex >= 0 && thing.subsectorIndex < renderWorld.subsectors.size && renderWorld.subsectors[thing.subsectorIndex].sectorIndex == sector.sectorNumber) {
        ProjectSprite(thing, spriteLightLevel * maxScaleLight);
    }
}

I ran the same path again. The phantom was gone, and the corrected Wax port matched the reference at the moment where it had appeared. Removing only that check brought it back.