Tropical Square Dance Crisis (Arcade Style Puzzle)

Basic Pixel Art Scaling with OpenGL ES 2

Square Dance is a low resolution pixel art game and it runs on a variety imperfect screen sizes. I've seen tactics for dealing with this here [1], [2] and elsewhere, but I didn't find quite what I was looking for, so I thought it might be interesting to collect up some of what I've tried. My examples here are using LuaJIT and OpenGL ES 2 on iPhone 6's 1334x750 retina resolution.

Square Dance's "base" resolution is 240x160. When a sprite is drawn, its position and size are interpreted as being blitted onto a framebuffer of this size.

Pixel Purity: Integer Scaling Only!

I think most of us would agree that integer scaling is the ideal for pixel art games -- each game-pixel is mapped to an exact square of device pixels 2x2, 3x3 etc. To do this, pick the largest integer multiple of the base resolution that is less than or equal to the device resolution. Usually the resulting quad is centered on the device's display. Base resolution textures can be perfectly scaled up using the GL_NEAREST for GL_TEXTURE_MAG_FILTER.

"Perfect" pixel scaling

Full Size

Here, the pixels are perfect but it results in wasted space both horizontally and vertically. Really not too great in my case...

You might be able to ensure a good or even perfect fit for certain devices by picking a specific base resolution, but generally letter-boxing and/or pillar-boxing results on a device that is not a perfect multiple of the base resolution. In some cases, it is also possible to use the extra pixels to display extra (non-essential) game area instead. In this case, don't scissor and draw more than the minimal area. If you're working in specific format (an emulator?), this may not be an option.

Code:

-- Method 1

fbsize = platform.screen_size()

device_width = fbsize.width
device_height = fbsize.height

base_width = 240
base_height = 160

width_ratio = device_width / base_width
height_ratio = device_height / base_height

-- find the largest integer scale that "fits"
scale = math.floor(math.min(width_ratio, height_ratio))

-- size of the device in base pixels
base_screen_width = device_width / scale
base_screen_height = device_height / scale

-- offset of (0,0) in base pixels
offset_x = math.floor((base_screen_width - base_width) / 2)
offset_y = math.floor((base_screen_height - base_height) / 2)

-- projection to apply to transform base pixels to screen
device_projection = glOrtho(
    -offset_x, base_screen_width-offset_x,
    base_screen_height-offset_y, -offset_y,
    1, -1
)

glViewport(0, 0, device_width, device_height)

-- letter-box / pillar-box to restrict drawing to
-- the virtual screen area
glDisable(GL_SCISSOR_TEST)
glClearColor(0,0,0,1)
glClear(GL_COLOR_BUFFER_BIT)
glEnable(GL_SCISSOR_TEST)

glScissor(
    offset_x*scale,
    offset_y*scale,
    base_width*scale,
    base_height*scale
)

-- draw frame
blit(sprite.texture,
    sprite.x, sprite.y,
    sprite.w, sprite.h,
    0, 0, 1, 1,
    device_projection
)

blit(...)

View Full Example

Accepting Imperfection

For me, the tradeoff for perfection is too steep and I don't want to adjust the basic shape of the screen. I'd like to see either letter-boxing or pillar-boxing, but never both.

On some devices, the device pixels aren't necessarily what's directly displayed on the hardware anyway and some amount of scaling/filtering is applied regardless of an integer scaling factor -- for example non-retina resolutions on retina iPhones, or any resolution on iPhone 6S [3].

iPhone 6 integer scaling with retina disabled.

Full Size

This is basically what I'm aiming for (preserve squares as perceptual squares). Unfortunately, it isn't available everywhere...

Non Integer Upscaling (Bad Ways)

To get non integer scaling, only one thing needs to be adjusted, though the results are not pretty with the default upscaling options and low resolution textures.

Code:

-- Method 2
-- ...

-- don't take the floor for a non-integer scale
-- scale = math.floor(math.min(width_ratio, height_ratio))
scale = math.min(width_ratio, height_ratio)

...

View Full Example (GL_NEAREST)
View Full Example (GL_LINEAR)

GL_NEAREST

Full Size

if GL_TEXTURE_MAG_FILTER is GL_NEAREST, things look sorta right, but some pixels come out wider than others which can be seen here, but is especially strange looking when things are moving.

GL_LINEAR

Full Size

Upscaling low resolution textures with GL_LINEAR magnification, you get a fuzzy resampling that destroys all the hard edges. There are other undesirable edge artifacts here that I haven't bothered to clean up.

Overscaling, Then Downsampling

Although upsamping is crumby, downsampling is not so bad. The idea here is to render to an offscreen framebuffer texture that bigger than the physical screen (and a perfect integer multiple of the base resolution for perfect scaling), then downsample that to the device framebuffer (by texturing a "full screen" quad)

Overscaled Offscreen Framebuffer

Full Size

This is comparable to the result that iPhone gives you when it does scaling for you, (and it works elsewhere / other resoutions). This is a workable, but it requires allocating a fairly large texture (especially on hi-res displays or devices requiring power-of-two textures). Performing these additional large texturing operations can be a performace drag on older hardware.

Code:

-- Method 3

-- ...
device_projection = ...  -- compute device projection as before

-- record the default framebuffer (not always 0, ie iOS)
device_framebuffer = glGetIntegerv(GL_FRAMEBUFFER_BINDING)

-- offscreen framebuffer scale
-- pick the next largest integer scale that is /larger/ than
-- the device scale (well, at least as large)
over_scale = math.ceil(scale)

-- base dimensions scaled up by the overscaling factor
fb_base_width = base_width * over_scale
fb_base_height = base_height * over_scale

-- calculate the width and height of the framebuffer texture.
-- these might be larger than the requested sizes
-- due to power-of-two texture size restrictions.
fb_tex_width, fb_tex_height = next_legal_texture_size(fb_base_width, fb_base_height)

-- set up the texture
fb_texture = glGenTextures(1)
glBindTexture(GL_TEXTURE_2D, fb_texture)
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, fb_tex_width, fb_tex_height, 0, GL_RGB, GL_UNSIGNED_BYTE, nil)
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR) -- <-- this will be used since it's downscaling
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST)
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE)
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE)

-- make the texture the color attachment for the new offscreen framebuffer
framebuffer = glGenFramebuffers(1)
glBindFramebuffer(GL_FRAMEBUFFER, framebuffer)
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, fb_texture, 0)

-- world -> offscreen framebuffer projection
fb_projection = glOrtho(0, fb_tex_width / over_scale, fb_tex_height / over_scale, 0, 1, -1)

-- prepare to draw to the offscreen framebuffer
glBindFramebuffer(GL_FRAMEBUFFER, framebuffer)
glViewport(0,0, fb_tex_width, fb_tex_height)
glScissor(0,0,fb_base_width,fb_base_height)

-- draw as usual, but draw onto the offscreen framebuffer

blit(sprite.texture,
    sprite.x, sprite.y,
    sprite.w, sprite.h,
    0, 0, 1, 1,
    fb_projection
)

blit(..., fb_projection)

-- finally, draw the framebuffer to the device by rendering it's texture
-- into a quad covering the screen. Since the offscreen
-- framebuffer is larger than the device framebuffer, downsampling using 
-- GL_LINEAR will be used. 
glBindFramebuffer(GL_FRAMEBUFFER, device_framebuffer)
glViewport(0, 0, device_width, device_height)

-- letter-box / pillar-box to restrict drawing to
-- the virtual screen area
glDisable(GL_SCISSOR_TEST)
glClearColor(0,0,0,1)
glClear(GL_COLOR_BUFFER_BIT)
glEnable(GL_SCISSOR_TEST)

glScissor(
    offset_x*scale,
    offset_y*scale,
    base_width*scale,
    base_height*scale
)

-- draw the overscaled offscreen framebuffer onto the 
-- device framebuffer.
blit(fb_texture,
     0, 0, base_width, base_height,
     0, 0, fb_base_width / fb_tex_width, fb_base_height / fb_tex_height,
     device_projection
)

View Full Example

Emulated Overscaling + Downsampling

Instead of making a huge framebuffer, we can make a framebuffer that is exactly the base resolution (small) and then use shader magic to treat it as if it had been scaled up to the size of Method 3. The algorithm used by OpenGL ES 2 to downsample is outlined in the spec in section 3.7.7 Texture Minification [4]. The idea is to implement this but treat the texture as if it were scaled by the overscale factor. I found the performance to be better in most cases to Method 3, but with much less texture memory allocated.

Emulated Overscaling

Full Size

Code:

-- Method 4

-- ...

-- calculate the width and height of the framebuffer texture.
-- these might be larger than the requested sizes
-- due to power-of-two texture size restrictions.
fb_tex_width, fb_tex_height = next_legal_texture_size(base_width, base_height)

-- set up the texture
fb_texture = glGenTextures(1)
glBindTexture(GL_TEXTURE_2D, fb_texture)
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, fb_tex_width, fb_tex_height, 0, GL_RGB, GL_UNSIGNED_BYTE, nil)
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST)
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST) -- <-- this will be used since it's upscaling
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE)
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE)

-- make the texture the color attachment for the new offscreen framebuffer
framebuffer = glGenFramebuffers(1)
glBindFramebuffer(GL_FRAMEBUFFER, framebuffer)
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, fb_texture, 0)

-- world -> offscreen framebuffer projection
fb_projection = glOrtho(0, fb_tex_width, fb_tex_height, 0, 1, -1)

-- prepare to draw to the offscreen framebuffer
glBindFramebuffer(GL_FRAMEBUFFER, framebuffer)
glViewport(0,0, fb_tex_width, fb_tex_height)
glScissor(0,0,base_width,base_height)

-- draw as usual, but draw onto the offscreen framebuffer

blit(sprite.texture,
    sprite.x, sprite.y,
    sprite.w, sprite.h,
    0, 0, 1, 1,
    fb_projection
)

blit(..., fb_projection)

-- finally, draw the framebuffer to the device by rendering it's texture
-- into a quad covering the screen as defined before.

-- ...

-- instead of blitting the framebuffer use a special
-- program to emulate downscaling
-- blit(frambuffer, ...)
downscale_blit(fb_texture,
     0, 0, base_width, base_height,
     0, 0, base_width / fb_tex_width, base_height / fb_tex_height,
     over_scale,
     fb_tex_width,
     fb_tex_height,
     device_projection
)

Code:

-- "Downscaling" shader 

-- vertex shader source
fb_v_shader_src =
[[
precision highp float;

uniform mat4 viewProjection;
uniform float superScale;
attribute vec4 vPosition;
attribute vec4 texCoord_in;

varying vec2 texCoord;

void main()
{
    texCoord = texCoord_in.xy;
    gl_Position = viewProjection*vPosition;
}
]]

-- fragment shader source
fb_f_shader_src =
[[
precision highp float;

varying vec2 texCoord;
uniform sampler2D tex;
uniform float superScale;
uniform float pixelWidth;
uniform float pixelHeight;

void main()
{
    // pretend there are this many texels ...
    float wt = superScale * pixelWidth;
    float ht = superScale * pixelHeight;

    // texture minification from section 3.7.7
    float u = wt * texCoord[0];
    float v = ht * texCoord[1];

    float i0 = floor(u - 0.5);
    float j0 = floor(v - 0.5);
    float i1 = i0 + 1.0;
    float j1 = j0 + 1.0;

    float a = fract(u - 0.5);
    float b = fract(v - 0.5);

    vec4 t00 = texture2D(tex, vec2(i0/wt, j0/ht));
    vec4 t10 = texture2D(tex, vec2(i1/wt, j0/ht));
    vec4 t01 = texture2D(tex, vec2(i0/wt, j1/ht));
    vec4 t11 = texture2D(tex, vec2(i1/wt, j1/ht));

    gl_FragColor = (1.0 - a)*(1.0-b)*t00 + a*(1.0-b)*t10 + (1.0-a)*b*t01 + a*b*t11;
}
]]
_fb_program, _fb_vshader, _fb_fshader = init_program(fb_v_shader_src, fb_f_shader_src)
FB_ARG_VPOSITION = glGetAttribLocation(_fb_program, "vPosition")
FB_ARG_TEXCOORD = glGetAttribLocation(_fb_program, "texCoord_in")
FB_ARG_TEX = glGetUniformLocation(_fb_program, "tex")
FB_ARG_PIXEL_WIDTH = glGetUniformLocation(_fb_program, "pixelWidth")
FB_ARG_PIXEL_HEIGHT = glGetUniformLocation(_fb_program, "pixelHeight")
FB_ARG_SUPERSCALE = glGetUniformLocation(_fb_program, "superScale")
FB_ARG_VIEWPROJECTION = glGetUniformLocation(_fb_program, "viewProjection")


-- emulated downscaling blit
function downscale_blit(tex, x, y, w, h, tx, ty, tw, th, super_scale, pixel_width, pixel_height, projection)
    glUseProgram(_fb_program)
    glActiveTexture(GL_TEXTURE0)
    glBindTexture(GL_TEXTURE_2D, tex)
    glUniformMatrix4fv(FB_ARG_VIEWPROJECTION, 1, GL_FALSE, projection)
    glUniform1f(FB_ARG_PIXEL_WIDTH, pixel_width)
    glUniform1f(FB_ARG_PIXEL_HEIGHT, pixel_height)
    glUniform1f(FB_ARG_SUPERSCALE, super_scale)

    glVertexAttribPointer(
        FB_ARG_VPOSITION, 2, GL_FLOAT, GL_FALSE, 0,
        _quad(x,y,w,h)
    )
    glEnableVertexAttribArray(FB_ARG_VPOSITION)

    glVertexAttribPointer(FB_ARG_TEXCOORD, 2, GL_FLOAT, GL_FALSE, 0,
        _quad(tx,ty,tw,th)
    )
    glEnableVertexAttribArray(FB_ARG_TEXCOORD)

    glDrawArrays(GL_TRIANGLE_STRIP, 0, 4)
end

View Full Example

Evaluation vs iPhone Scaling

These are (inverted) image diffs between the screenshots above. Methods 3 and 4 are pretty darn close to what iPhone's scaling does. They show an error along the left-hand side and subtle differences at color boundaries if you look closely. iPhone may align the first lefthand pixel with a device pixel. I'm too lazy to see if this accounts for the difference.

Method 3 vs iPhone Diff

Full Size

Method 4 vs iPhone Diff

Full Size

Method 3 vs Method 4

Full Size

Method 3 and Method 4 do not differ significantly as far as I can tell.

Other Options Not Explored

Another option is to oversize all the textures so that they are all larger than the largest supported scaling factor and apply GL_LINEAR for GL_TEXTURE_MIN_FILTER. This works also, but possibly wastes even more texture memory than method 3, requires preprocessing textures and isn't as general (what's the maximum size screen again?)

Method 4 could be applied as textures are rendered to the device framebuffer rather than writing to an offscreen framebuffer. I did experiment with this a little but I found it a bit more finnicky -- creating difficult small errors on the edges of each sprite rather than on the very edges of the screen. It also probably wants premultiplied alpha in the textures as they require a slightly more complex resampling to combine that I'll leave as an exercise to the reader -- see [5]

Method 4 might be improved by removing "Dependent Texture Reads"[7]. I'm not sure if the 4 texture coordinates can be accurately replaced by "varyings" in the vertex shader. Dependent textures do slow down older devices quite a bit.

Other complex reconstruction algorithms (thanks bdsowers) [8]

Any better ways you've heard of? Better write-ups? Let me know!

Further Reading and References

[1] https://forums.tigsource.com/index.php?topic=36020.0
[2] https://forums.tigsource.com/index.php?topic=5185.0
[3] https://www.paintcodeapp.com/news/iphone-6-screens-demystified
[4] https://www.khronos.org/registry/gles/specs/2.0/es_full_spec_2.0.25.pdf#page=90
[5] http://www.realtimerendering.com/blog/gpus-prefer-premultiplication/
[6] http://www.david-amador.com/2013/04/opengl-2d-independent-resolution-rendering/
[7]https://developer.apple.com/library/ios/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/BestPracticesforShaders/BestPracticesforShaders.html#//apple_ref/doc/uid/TP40008793-CH7-SW15
[8]https://en.wikipedia.org/wiki/Pixel_art_scaling_algorithms