Controlling Time and Framerate

What exactly does this frame interval depend on—what is it that "blocks" until the next frame? What's happening here is that something in the code is waiting for a vertical blank period, an old-timey term from the days of cathode ray tubes (where the electron beam took actual time to be redirected by magnets back to the top of the screen). When rendering waits until the screen is ready to receive another image, we say rendering is vertically synced.

For an experiment, try to find out where the waiting is happening. Take these two lines of code:

let start = Instant::now();
//...
dbg!(start.elapsed());

And wrap them around any few lines of code that you think might be introducing the delay. As a hint, the main part of the delay is not happening because of anything in winit or the event processing code. Really, try it!

• [ ] OK, I tried it!

What you'll find is that the culprit is acquiring the next surface image from the swapchain, whatever that is. Recall that the swapchain is how WGPU negotiates with the graphics card and display adapter to get us images to draw into. We initialize the swapchain with a certain number of images which are our framebuffer render targets (here, framebuffer refers to the actual, displayed framebuffer, not our miniature 2D framebuffer). WGPU needs to do two things with these images:

1. Allow our application to render into them
2. Give them to the operating system to put on the monitor

There are lots of valid choices for balancing these two concerns:

• WGPU could let us draw into the image (1) while it's being rendered (2), but this could cause visible tearing when half the image is from one time point and the rest is from another
• If we had several images, WGPU could let us draw into one while the other is being rendered.
  • We could use them like a queue, filling up images with data and handing them over (then waiting for images to come back from presentation)
    • If our renderer were sometimes slow, WGPU could present the in-progress frame
    • If our renderer were very fast, WGPU could replace the previously-queued images with fresh ones

In WGPU, the presentation mode of a surface dictates which of these approaches are used. Respectively, they are:

• Immediate mode
• If using multiple images:
  • FIFO mode (First In, First Out; the default)
    • Relaxed FIFO mode
    • Mailbox mode

For more information about their tradeoffs, check out this Vulkan tutorial; besides tearing, latency and energy usage are also important considerations.

So if we're stuck waiting for WGPU to give us a swapchain image, that's because the default presentation mode is FIFO. Something like this happens:

1. We draw the first frame very quickly.
2. WGPU gives us the second frame right away.
3. We draw the second frame very quickly.
4. WGPU is still presenting the first frame, so we need to wait for it to be ready again.
5. Finally we quickly draw into the first frame again.
6. … But now we have to wait for the second frame!

Since our renderer is way faster than the presentation interval, after we blow through the queue once we spend most of our time waiting. This is great from the perspective of power consumption, but not great from the perspective of predictable, consistent frame times.

What do you think would happen if you made the number of images in the FIFO queue really high? Would we spend more time waiting, less time, or about the same?

Try changing the presentation mode in the surface configuration to mailbox or immediate (or perhaps, AutoVsync or AutoNoVsync) and see what happens to the framerate and time spent acquiring a swapchain image. Especially take a look at what happens to the movement of your characters. You may also want to search the WGPU documentation for methods and constants involved with presentation mode.

Since rendering takes time (both in terms of our code and the operating system's), updating our game state takes time, and we can't do these things simultaneously (otherwise we may show inconsistent data on screen), we need a way to synchronize or otherwise schedule game updates with respect to rendering.

Our goals here are to achieve a consistent frame rate for smooth motion and to minimize latency. Energy consumption is also worth minimizing. In fact, these three goals trade off against each other: the most energy efficient option is to render as infrequently as possible, while if we render often we can achieve smoothness at the cost of latency by beautifully interpolating between several prior frames (once we know what motion occurred it's easy to average it out). We can even reduce latency by trading away smoothness if we extrapolate previous game states to future predictions, or guess what player or opponent inputs might be (correcting the display if our guess ends up wrong).

The rest of these notes will illustrate two points in this trade-off space. We'll use two notions of time here: wall-clock time (or "real time") and simulated game time (or "time steps").

The simplest way to synchronize updates and rendering is just to perform a render after every update. This bounds latency tightly as we only ever draw the most recently simulated time step, and we draw it as soon as Vulkan says we can.

update_game(&mut game_state);
renderer.render(&game_state);

Oh, that looks familiar!

This technique comes with some drawbacks:

1. If the user has somehow disabled vsync or is using a high refresh rate monitor, we actually have no idea how many game updates will happen per second!
2. Frames don't all take exactly the same amount of time, so we may have jittery, "janky" motion. This is exacerbated if occasionally a frame takes way more or less time than usual.

We might try to mitigate these issues by figuring out how much time has passed and then simulating exactly that much time; but this complicates our simulation code (we need sophisticated integration for position and velocity for instance) and it's hard to imagine a simulation that works well when some timesteps are shorter than five milliseconds and others are longer than 100. Maybe it would make sense to cap the maximum game step duration, and execute several shorter updates in one cycle to avoid instability; but then we're prone to a death spiral where one or two slow simulation frames can lead to cascading slowdowns, grinding the game to a halt.

We're on the right track though—it certainly seems like we need to reason about time explicitly to balance smoothness with latency.

One popular technique (that you'll see in the readings) is to use an accumulator that "fills up" with elapsed time, and when we have enough "stored" time we get to simulate one game step. In this scheme, the renderer (which must run on its own schedule to maintain the display framerate) acts as the clock that determines when the game should update: we say the renderer produces time (adding it to the accumulator), and the simulation consumes it (in fixed-size chunks).

We could implement our accumulator like this:

// During initialization...
let mut acc = 0.0_f32;
let mut prev_t = Instant::now();
// Let's clock the game at 60 simulation steps per second
const SIM_DT : f32 = 1.0/60.0;
// Later on, in our loop...
  let elapsed = prev_t.elapsed().as_secs_f32();
  acc += elapsed;
  prev_t = Instant::now();
  while acc >= SIM_DT {
    update_game(&mut game_state);
    // NOTE: This is when you should swap "new" keys and "old" keys for input handling!
    // Otherwise you'll see several frames in a row where a key was just pressed/released.
    input.next_frame();
    acc -= SIM_DT;
  }
  renderer.render(&game_state);

What happened? Well, even though rendering still drives the simulation, we have decoupled the simulation framerate from the rendering framerate. So we could update the game a hundred times per second, or fifty, or any rate we like. This also means that no matter what the rendering framerate is—30, 60, 144 frames per second, whatever—the game will always progress at the same rate each second.

This does, however, have important implications for latency and smoothness. Imagine that our game is updating 60 times per second. Most of the time, the elapsed rendering duration won't be exactly 1/60th of a second. If it's a little more, that's no problem; we'll update the accumulator and bank the extra time, updating once. But if it's a little less, we might not have enough time saved up to progress the simulation. On average we'll tick 60 times per second, but in the moment we'll see some rendered frames that are identical to the previous frame (no game updates) and some that represent two frames worth of movement (double updates).

An inconsistent update rate can cause visual stuttering, and we might also have problems with latency if we get unlucky with timing (since we often render a slightly stale state, user input might arrive almost a full two rendering frames before the player sees any feedback). We can fix these problems in three ways:

1. Crank up the simulation frame rate. A stream of 0-, 1-, and 2-update frames has more apparent variation than a stream of 3-, 4-, and 5-update frames.
2. Ignore small deviations in render time—if the elapsed frame time is "roughly" 1/60, round it to 1/60 (or 1/15, 1/30, 1/144). This can cause game time to drift from real time over a long enough duration but can be pretty effective.
3. Account for leftover time using interpolation.