Image for post
Image for post

Sharper Mipmapping using Shader Based Supersampling

Mipmapping is ubiquitous in real time rendering, but has limitations. This article is going to go into what mipmapping is, how it’s used, and how it can be made better.

Inscrutable Text

Okay, story time! On one of the first VR projects I worked on, Wayward Sky, we ran into a curious problem. We had a sign in the game with a character’s name written across it. Not an unusual thing in itself. The problem was the name was almost completely illegible when playing the game.

Image for post
Image for post
Bilinear Filtering (200% Pixel Scale)
Image for post
Image for post
Anisotropic 8x Bias -1.0 (200% Pixel Scale)
Image for post
Image for post
Anisotropic 8x Bias -1.0 2x2 RGSS (200% Pixel Scale)

MIP Mapping

Multum in Parvo

First a bit of discussion about what mip mapping is, and why it’s a good thing. It exists to solve the problem of texture aliasing during minification. In simpler terms, reduce flickering, jaggies, and complete loss of some image details when an image is displayed smaller than its original resolution.

Image for post
Image for post
Extreme Supersampling (200% Pixel Scale)
Image for post
Image for post
No mipmaps, Bilinear Filtering (200% Pixel Scale)
Image for post
Image for post
Image for post
Image for post
Colored Mipmaps, Bilinear Filtering (200% Pixel Scale)

Isotropic Filtering

Lets go over the basics of texture filtering. Really there’s two main kinds of texture filtering, point and linear. There’s also anisotropic, but we’ll come back to that. When sampling a texture you can tell the GPU what filter to use for the “MinMag” filter and the “Mip” filter.

Image for post
Image for post
Bilinear Filtering (200% Pixel Scale)
Image for post
Image for post
Trilinear Filtering (200% Pixel Scale)

Anisotropic Filtering

Anisotropic filtering exists to try to get around the blurry ground problem. Roughly speaking it works by using the mip level of the smaller texel to pixel ratio, then sampling the texture multiple times along the non-uniform scale’s orientation. The mip level used is still limited by the number of samples allowed, so low Anisotropic levels will still become blurry in the distance to prevent aliasing.

Image for post
Image for post
Anisotropic Filtering quality level comparison (200% Pixel Scale)
Image for post
Image for post
Anisotropic Filtering 8x (200% Pixel Scale)
Image for post
Image for post
“Ground Truth” Supersampling (200% Pixel Scale)
Image for post
Image for post
Anisotropic Filtering 8x vs “Ground Truth” (200% Pixel Scale)

Going Sharper-ish

The Modest Proposal

So now we know that mip mapping can cause a texture to be displayed a little blurry, even with the best texture filtering enabled. The “obvious solution” is to disable mip mapping all together. We already saw how effective that is. But this is often the “solution” artist are likely to end up with, mainly because it may be the only option the have immediately available to them. For small amounts of scaling, this is actually quite effective. The problem is of course past that it quickly goes south. It also usually results in some programmer shouting at said artist some time later when it comes time to do performance optimizations. I’ve shown this to you before near the start of the article as the “just sample the full resolution image” example. To remind you of what that looks like…

Image for post
Image for post
No mipmaps, aka “The Horror” (200% Pixel Scale)

Leveraging Conservative Bias

A better solution is to use mip biasing. Mip biasing tells the GPU to adjust what mip level to use one way or another. For example a mip bias of -0.5 pushes the mip changes slightly further away. Specifically it would move the transition back half way between the transition points it would use normally. A mip bias of -1 would push the mip level one full mip back, so where the GPU would originally be displaying mip level N, it’s now N-1. For some engines you can set the bias on the texture settings directly, and no modifications to shaders are needed. But direct texture asset mip biasing isn’t available on all platforms, for example it’s not supported on Apple devices running Metal, so custom shaders may be needed depending on your target platform(s). It can also be annoying to tweak as it’s not value exposed in the Unity editor by default. And the LOD Bias setting in Unreal Engine 4 isn’t the same thing. Luckily biasing in a shader is easy to add, and well supported across a wide range of hardware.

half4 col = tex2Dbias(_MainTex, float4(i.uv.xy, 0.0, _Bias));
// per pixel screen space partial derivatives
float2 dx = ddx(i.uv);
float2 dy = ddy(i.uv);
// bias scale
float bias = pow(2, _Bias);
half4 col = tex2Dgrad(_MainTex, i.uv.xy, dx * bias, dy * bias);
Image for post
Image for post
Anisotropic Filtering 8x Bias -0.5 (200% Pixel Scale)
Image for post
Image for post
Anisotropic Filtering 8x Bias -1.0 (200% Pixel Scale)

The Super Solution

So how do we increase the image clarity without introduce aliasing? Well, we might look to other anti-aliasing techniques. Specifically back at Super-Sample Anti-Aliasing (SSAA), and Multi-Sample Anti-Aliasing (MSAA).

Supersampling

Supersampling for rendering means rendering or “sampling” at a higher resolution than the screen displays, then averaging those values. This step of averaging the values is called downsampling. Since we’re sampling an existing texture there’s no higher resolution rendering, just sampling the texture multiple times. As mentioned earlier, the “ground truth” example was rendered using Supersampling with a crazy high sample count.

// per pixel screen space partial derivatives
float2 dx = ddx(i.uv.xy) * 0.25; // horizontal offset
float2 dy = ddy(i.uv.xy) * 0.25; // vertical offset
// supersampled 2x2 ordered grid
half4 col = 0;
col += tex2Dbias(_MainTex, float4(i.uv.xy + dx + dy, 0.0, _Bias));
col += tex2Dbias(_MainTex, float4(i.uv.xy - dx + dy, 0.0, _Bias));
col += tex2Dbias(_MainTex, float4(i.uv.xy + dx - dy, 0.0, _Bias));
col += tex2Dbias(_MainTex, float4(i.uv.xy - dx - dy, 0.0, _Bias));
col *= 0.25;
Image for post
Image for post
Anisotropic Filtering 8x Bias -1.0 2x2 OGSS (200% Pixel Scale)

Multi Sample Anti-Aliasing

MSAA, or more specifically 4x MSAA, has another trick. MSAA is similar to Supersampling in that it’s rendering at a higher resolution than the screen can necessarily display, but different in what it renders at a higher resolution isn’t necessarily the scene color, but rather the scene depth. The difference is inconsequential for this topic, and I’ve gone into more detail in another post, Anti-aliased Alpha Test, so we’ll skip that for now. What is important is 4x MSAA uses a Rotated Grid pattern, sometimes called 4 rooks.

Image for post
Image for post
from A Quick Overview of MSAA
// per pixel partial derivatives
float2 dx = ddx(i.uv.xy);
float2 dy = ddy(i.uv.xy);
// rotated grid uv offsets
float2 uvOffsets = float2(0.125, 0.375);
float4 offsetUV = float4(0.0, 0.0, 0.0, _Bias);
// supersampled using 2x2 rotated grid
half4 col = 0;
offsetUV.xy = i.uv.xy + uvOffsets.x * dx + uvOffsets.y * dy;
col += tex2Dbias(_MainTex, offsetUV);
offsetUV.xy = i.uv.xy - uvOffsets.x * dx - uvOffsets.y * dy;
col += tex2Dbias(_MainTex, offsetUV);
offsetUV.xy = i.uv.xy + uvOffsets.y * dx - uvOffsets.x * dy;
col += tex2Dbias(_MainTex, offsetUV);
offsetUV.xy = i.uv.xy - uvOffsets.y * dx + uvOffsets.x * dy;
col += tex2Dbias(_MainTex, offsetUV);
col *= 0.25;
Image for post
Image for post
Anisotropic Filtering 8x Bias -1.0 2x2 RGSS (200% Pixel Scale)
Image for post
Image for post
2x2 RGSS vs “Ground Truth” (200% Pixel Scale)

Closing Thoughts

And there we go! We’ve solved the problem! For a modest increase in shader cost we have near ground truth quality texture filtering! This means noticeably clearer text and images with little to no aliasing at significantly less performance impact than the ground truth.

Image for post
Image for post
Bilinear vs RGSS (200% Pixel Scale)

Caveats & Additional Thoughts

Performance and Limitations

Q: Since this is so much better, shouldn’t we be using this everywhere on everything?!

Colored Mipmaps

Q: Those colored mipmaps were fun! Can we see what that looks like for Trilinear and Anisotropic?

Image for post
Image for post
Bilinear (200% Pixel Scale)
Image for post
Image for post
Trilinear (200% Pixel Scale)
Image for post
Image for post
Anisotropic 8x (200% Pixel Scale)

Giffy

Q: Why are you using gifs?! You should be using videos instead, they’re waaay better!

Taking Credit

Q: This is so awesome, you should totally patent it / name it after yourself!

Signed Distance Fields

Q: What about SDF Font rendering?

MORE SAMPLES

Q: If 4 samples are good, more must be better!

Texture Compression

Q: Will this help make compressed textures look sharper?

Gamma Color Space

All the above example images are produced using Linear color space. The reasons for this are due to the fact that averaging of the samples needs to be done in linear space. If you’re using Gamma color space, like for mobile, know that your textures may become overly dim. Even if you convert the samples from gamma to linear space in the shader, bilinear filtering itself is in the wrong color space and will lead to some amount of dimming that is not possible to correct for.

Kaiser-filtered Mipmaps

Another common technique for improving the sharpness of mip mapping is to use a Lanczos, Kaiser, or some similar wider sinc based kernel & sharpening when generating the mipmaps themselves. Certainly Photoshop does something like this by default when downscaling images. And both Unity and Unreal, as well as many other mip mapping utilities, have options for enabling Kaiser or some other sharpening technique. These work by increasing the contrast of the mipmaps to help retain important details. You can use this in combination with the presented Supersampling technique, but I personally find its use unnecessary, and potentially makes things worse. Absolutely enable it on textures that aren’t using Supersampling and see if you like the effect.

Image for post
Image for post
Unity’s Kaiser vs Box filtering vs Nvidia Kaiser, Anisotropic 8x (200% Pixel Scale)
Image for post
Image for post
Unity’s Kaiser vs Box filtering vs Nvidia Kaiser, RGSS (200% Pixel Scale)
Image for post
Image for post
Unity 2019 and earlier Kaiser vs Unity 2020.1 Kaiser vs legacy Nvidia Kaiser, Anisotropic 8x (200% Pixel Scale)
Image for post
Image for post
Unity 2019 and earlier Kaiser vs Unity 2020.1 Kaiser vs legacy Nvidia Kaiser, RGSS (200% Pixel Scale)
Image for post
Image for post
Unity 2019 vs Unity 2020.1 vs Nvidia 2020.1.3 vs Nvidia legacy Kaiser filters, Anisotropic 8x (200% Pixel Scale)
Image for post
Image for post
Ground Truth vs legacy Nvidia Kaiser vs new Nvidia Kaiser, RGSS (200% Pixel Scale)

Blurring at 1:1 Scaling

The above examples are all using a 256x256 texture and rendering to a 256x256 target. When the texture is “full screen” it’s perfectly sharp. If you use the code snippets I actually presented above, they will not be perfectly sharp but instead be slightly blurry. I correct for this in the shader I used by limiting the sample offsets by a mip level calculation:

// per pixel partial derivatives
float2 dx = ddx(i.uv);
float2 dy = ddy(i.uv);
// manually calculate the per axis mip level, clamp to 0 to 1
// and use that to scale down the derivatives
dx *= saturate(
0.5 * log2(dot(dx * textureRes, dx * textureRes))
);
dy *= saturate(
0.5 * log2(dot(dy * textureRes, dy * textureRes))
);
// rotated grid uv offsets
float2 uvOffsets = float2(0.125, 0.375);
float4 offsetUV = float4(0.0, 0.0, 0.0, _Bias);
// supersampled using 2x2 rotated grid
half4 col = 0;
offsetUV.xy = i.uv.xy + uvOffsets.x * dx + uvOffsets.y * dy;
col += tex2Dbias(_MainTex, offsetUV);
offsetUV.xy = i.uv.xy - uvOffsets.x * dx - uvOffsets.y * dy;
col += tex2Dbias(_MainTex, offsetUV);
offsetUV.xy = i.uv.xy + uvOffsets.y * dx - uvOffsets.x * dy;
col += tex2Dbias(_MainTex, offsetUV);
offsetUV.xy = i.uv.xy - uvOffsets.y * dx + uvOffsets.x * dy;
col += tex2Dbias(_MainTex, offsetUV);
col *= 0.25;

Even Sharper

Q: It still looks kind of blurry to me. What can I do to make it sharper?

Max Anisotropic Filtering Quality

Q: Why are all the images using Anisotropic Filtering set to 8x rather than 16x if the performance is the same?

Unity’s Shader Graph

Q: How do I implement this in Shader Graph?

Temporal Jittering

For some small, but free, additional perceived visual quality you can jitter the rotated grid. For VR projects I pass the frame count to shaders and flip the x component of the uvOffsets value. It’s not huge, and for some people it may make things just look like they’re vibrating, but on small text I found it could help with clarity.

Quad Antialiasing

Q: What anti-aliasing are you using to get the edges of the quad looking so nice?

Ground Truth

As with most things, the definition of “ground truth” is fuzzier than you might expect. I mentioned above that the ground truth images I present are using 6400 anisotropic samples per pixel. This is actually a 1.5 pixel wide 80x80 ordered grid with the contribution of each pixel scaled by a smoothstep curve. What makes that ground truth? Why 1.5 pixels? Why 80x80? Why smoothstep?

Notes:

  1. ^This is actually calculated for each 2x2 pixel quad using screen space partial derivatives. GPUs always render the 4 pixels of each 2x2 group at the same time, and can compare the values calculated between them. By default, the values returned by derivative functions, like ddx and ddy, return the same value for all 4 pixels in the group as they’re only calculated using the first pixel in the group and the pixels to the side and below, leaving the last pixel’s values ignored. Using ddx_fine instead will get you the “real” per pixel derivatives, actually taking into account that last pixel, but they’re still limited to the derivatives within the pixel quad.
  2. ^As an aside, that small amount of blurring on the floor is actually due to modern GPUs using an approximation of Anisotropic filtering rather than the reference implementation. Almost every GPU family uses a slightly different approximation, with the actual implementation details being something of a trade secret. To be fair to them, the above texture is just about the worst case scenario, and with modern approximations there’s very little appreciable difference between what they’re doing and the “real thing”.

Tech Artist & Graphics Programmer lately focused on Unity VR game dev.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store