Better support for non-60hz display rates

[quinton-ashley]

Goal

Support non-60hz display rates. This includes:

• the 50hz standard used in Europe, Australia, New Zealand, and parts of Africa, Asia, and South America.

• high refresh rate displays (many Androids and gaming monitors)

• ProMotion variable refresh rate displays (latest iPhone models)

I want p5play developers to be able to share their games with the whole world and have it run well on any capable device.

p5play game developers and players shouldn't have to worry about frame rates.

The Problem

The default target frame rate is 60. Currently if you want the target to be something else, like 50hz for example, you'd need to run frameRate(50).

The p5.js draw loop uses requestAnimationFrame but just avoids drawing anything at a higher rate than 60fps by default. Users with higher refresh rate displays are just stuck at 60hz, but if the user's display rate is lower, like 50hz... umm Houston we got a problem! The physics simulation will run 16% slower than real time because by default, p5play updates the physics simulation by 1/60th of a second at the end of each draw call.

By default q5.js will run the draw loop using requestAnimationFrame based on the device's display rate. This would also make the physics simulation too slow on 50hz displays and way too fast on high refresh rate displays.

Personally, as a tech consumer, I've been a big fan of increasing visual fidelity over smoothness, opting for 4K over higher refresh rates. For me the difference between 4K and 1080p is HUGE, but with refresh rates higher than 60hz, I can't really tell the difference. Maybe my eyes are slow or something lol. All the devices I personally own can "only" do 60hz. So my personal biases led me to not consider this major problem until now.

Also it'd be nice if p5play could limit the physics simulation to 30hz, if the user's device isn't capable of achieving 60fps.

I was quite conflicted on how to approach this problem, so I did research on Unity.

How Unity does it

Unity separates the game's physics updates from frame rendering.

https://docs.unity.cn/520/Documentation/Manual/TimeFrameManagement.html#:~:text=Unlike%20the%20main%20frame%20update,the%20last%20physics%20update%20ended.

Unity uses a fixed timestep of 1/50th of a second for physics calculations to ensure consistent physics simulation, regardless of the frame rate.

Unity uses a variable timestep for rendering and general game logic, which is handled in the Update method. This method is called once per frame, so the frequency can vary depending on the display rate.

In between physics updates, Unity interpolates the positions of physics objects for rendering. This means that even if a physics update doesn't happen at the exact same time as a frame update, Unity will estimate the current position game objects based on its previous physics update, projecting from there. This allows the game object to appear to move smoothly at any higher frame rate, even though its actual position is only being updated in the physics sim at a fixed rate of 50hz.

This approach enables Unity to provide consistent physics simulation while still rendering as smoothly as possible based on the performance and capabilities of the device.

I think implementing something like this in p5play would be ideal, but how?

Frames

If the goal is to not require developers or players to worry about frame rates, that means not having a default frame rate. That means maybe frames shouldn't be used as a unit of time measurement anywhere in a game's code. Yikes!

• The collision handling system stores how long players have been colliding in frames.
• Input devices store how long the user has been doing an input action in frames.
• Animations use the ani.frameDelay property to define how many frames an image in the animation is displayed before the next image is displayed.
• Also q5/p5 sketches typically use frameCount as the measure of time.

Switching from frames to seconds would require making tons of breaking changes to p5play.

The case for abstracting frames

@davepagurek suggested an alternative solution:

I guess there's maybe a world where "frames" are like css pixels, and one "frame" as a unit might actually represent multiple real frames under the hood on high refresh rate displays?
not sure if that's more or less confusing though haha
like if everyone defined sketches as if it were 60fps, but sometimes the frame count is a fraction instead of a whole number

More info: https://www.quirksmode.org/blog/archives/2010/04/a_pixel_is_not.html

I do find using frames to be more convenient than milliseconds, a level of precision that's not required for typical user input handling.

I also agree that this issue with higher refresh rates is a bit analogous to the challenge that high pixel density displays posed to developers over a decade ago.

When Apple first introduced high pixel density displays to consumers, I thought the abstract re-branding of pixels was confusing, now it seems perfectly natural. Yet, was that only acceptable because web developers no longer needed to care about real pixels? With retina displays users could zoom in and out without really compromising the visual appearance of text, which Apple had just a few years prior been boasting about always displaying pixel perfect on lower resolution displays.

Have we gotten to that same point with high refresh rate displays above 60fps? Perhaps so.

...continued below

[quinton-ashley]

iOS Safari's default limit is 60fps

I did some testing on an iPhone 13 and to my surprise found that p5play was running at 60fps.

I learned that by default even on iOS devices with ProMotion displays, Safari and WKWebViews are limited to 60fps, assumedly to save battery. This can be changed in settings, although I wouldn't expect that many people do it. Low power mode can also limit some iPhone's display rate to 30hz to save battery.

So it turns out existing p5play v3 projects should just run fine on any current iPhone made for the NTSC region.

Frames

Since frames are small integers, users can use equivalence checks == in their code instead of doing range checks on inconsistent decimal values.

See this example that checks if the user has been pressing the mouse for 10 frames.

if (mouse.pressing() == 10) {
  // run some code one time
}

How could an equivalent check be written if the time was stored in seconds?

if (mouse.pressing() == 0.16666666666) {
  // run some code one time
}

AHHH! It's awful. Also it wouldn't even work with different refresh rates like PAL 50fps. 8 frames would be 0.16 seconds and 9 frames would be 0.18.

ChatGPT says in Unity that developers would have to write code like this:

float mousePressedTime = 0f;
bool codeExecuted = false;

void Update() {
    if (mouse.pressing()) {
        mousePressedTime += Time.deltaTime;
        if (!codeExecuted && mousePressedTime >= 0.166f) {
            // run some code one time
            codeExecuted = true; // Set the flag to true after executing the code
        }
    }
    else {
        mousePressedTime = 0f;
        codeExecuted = false; // Reset the flag if the mouse is not being pressed
    }
}

Oof! That's not going to work for p5play.

In ye old days of retro gaming, no wonder developers choose to keep the code simple instead. They programmed for 60fps and just had the games run slower in Europe lol. But it's +50 years later, so this problem really ought to be solved better in p5play.

Let's assume that we all like being able to check for equivalence with integers and that the time these numbers represent should be loosely equivalent, regardless of display rate. We also want a user's p5play program they developed using a 60fps display to run pretty much the same on any other display and vice versa.

Abstracting frames (revisited)

It seems abstracting frames is actually the way to go. But perhaps not in the way previously described:

like if everyone defined sketches as if it were 60fps, but sometimes the frame count is a fraction instead of a whole number

That'd be fine for frame rates higher than 60hz, but not lower. For example, Let's say p5play is running on a 50hz display, it'd need to convert 50fps frames to the 60fps equivalent by rounding.

for (let i = 0; i < 50; i++) {
    console.log(Math.round(i / 50 * 60));
}

0, 1, 2, 4, 5, 6, 7, 8, 10...

Obviously we're gonna be missing some integers, since 50 is less than 60. If when developing a game with my 60hz display, I check if the user has been pressing the mouse for 9 "frames", it would never be true if a player's draw loop rate is 50hz.

if (mouse.pressing() == 9) {
  // never going to run
}

So this abstract frame would need to be at least as large as the lowest frame rate p5play will support.

Let's say we want to support 30hz and 25hz, even Nintendo still uses such a low frame rate in their biggest games in order for the Switch to keep up. But then our abstract frame would be 1/25th of a second (40ms). Is that precise enough for detecting user input at high levels of play? Not really.

The current record for human button presses per second is around 10-15, achieved via the rolling technique used to play tournament level Tetris. That's one button press every 6ms, so 2.8 button presses per 60hz frame (16ms).

Latency and responsiveness are more important factors to consider. As a musician I can attest that low latency between an action, for example playing a key on a keyboard, and hearing a response is important. Humans can time actions with 10-20ms of precision when they are prepared to act. Higher than expected latency could make players think their game is running slow, even if it maintains a solid high frame rate visually.

[quinton-ashley]

Latency and Responsiveness

Is handling user input in the draw loop even a good idea? Most Unity developers do it. Putting input handling logic in the Update loop that runs once per frame. ChatGPT says for the majority of games, handling input in the Update method provides a good balance between simplicity, performance, and responsiveness.

In p5play, checking user input in the draw loop also enables users to do cool stuff like this:

if (kb.pressing('space') || mouse.pressing()) {
	sprite.color = 'green';
}

Visually there's no point in polling for user input more frequently than the delay between frames. But what about rhythm based games where ~20ms of audio delay would be a major issue? Fortunately there are already event based functions like mousePressed and keyPressed for that to play a sound immediately when an input is detected. Perhaps in contros users should be able to define a input handling function that would run on every controller poll.

Better Name for "Abstract Frame"

On any device with a non-50hz display, the "abstract frame" or "standard frame" will be different from a real displayed frame. Arguably it could be better to give it an entirely different name then. As far as I'm aware there's no precedent for something like this.

ChatGPT suggests beat, pulse, tick, and quantum. Beat and pulse are too generic. "tick rate" is already commonly associated with server updates. "quant" sounds cool but "quantum" It doesn't really have anything to do with time and the implication that it's a really small unit makes it a misnomer. I'll have to think about this more.

I'm open to suggestions!

[ShiMeiWo]

How about putting "base" or "ideal" at the beginning of the words?

ex)

• baseFrequency
• idealFPS

[quinton-ashley]

@ShiMeiWo I wouldn't want to use "ideal" cause it implies other fps rates are not ideal. I think baseFrequency is too long. It'd be nice to have a short name for the abstract frame.

[quinton-ashley]

Dealing with Dropped Frames

If a game program can't reach its target frame rate, because it's taking too long to process and generate a frame, what should it do?

A “dropped” frame happens when a game program cannot generate a new frame in time for it to be displayed. As a result, the previous frame remains on the screen until the next display cycle begins. This can lead to a less smooth and visually appealing gameplay experience, as animations can look choppy. Dropped frames are often an indication of performance bottlenecks that may need to be addressed through optimization techniques or hardware upgrades to ensure smoother gameplay. However, they can also unavoidably occur due to interruptions by other tasks running on a system.

One solution is to skip frames and have the physics engine simulate more time to compensate. Not processing a frame requires skipping that amount of game time. This is known as dynamic frame rate adjustment. It's best for games that have high performance demands and consistently struggle to maintain high frame rates. This is a preferable approach in some cases, such as for real-time online multiplayer games, but typically I look to Nintendo, the leader in the video game industry, to see what choices they make.

Nintendo creates games with a target frame rate in mind. Most Nintendo games will effectively slow down the physics sim relative to real time, in order to match the frame rate. That's because in many games being able to see everything that happens and react to it in slower than real time is preferable to missing it, as long as these slowdowns aren't lengthy or common. In a severe case, let's say a temporary performance hit causes a frame to take a full second to render, if that full second goes by in the physics sim without the player being able to give any input, their player character could miss a jump and die. That'd be unfair! It's better to essentially pause the in-game action, enabling the computer to finish processing, then let players continue in game time from the last displayed frame.

One could argue that Nintendo can opt for this approach because they only make games for their own hardware. They can fine-tune a game to render at a specific size and frame rate.

Other game publishers, must ensure that their games run well on a variety of devices. Most PC games will offer graphical settings that users can edit based on the capabilities of their hardware. Most gamers often choose to push the limits of their system, so that it will just barely run their game at 60fps.

Yet, even if I didn't find Nintendo's approach generally preferable, it's impossible to alter frame timing in p5play v3 without potentially breaking game physics, because for simplicity the draw loop is commonly used, and I recommend it to be used, for everything: physics, input handling, and animation updates.

Future plans

Should it be a goal in p5play v4 to separate the physics simulation from the draw loop? and how should the proposed abstract frame system handle dropped frames?

Currently with p5play v3 if a game can't maintain 60fps, dropped frames are not counted in frameCount. Only after a frame is completed will p5play's post draw function run, which by default runs world.step, updates contacts, and increments user input frame counters. This guarantees that the draw function will be run every time frameCount is incremented. So for example, there will always be a frame where mouse.pressing() == 8 if the user holds the mouse for 8 frames. I like this approach to handling dropped frames.

I will need to do some more research on this topic.