Feb 3, 2020
Generating Perfect Normal Maps for Unity (and Other Programs)
And how to tell what’s wrong with your content pipeline.
Last Update: 7/17/20
Lets say you have an awesome high poly model an artist has been working on that you’re looking forward to getting into your game. They’ve been showing off screen grabs of the model in their content tools and it looks fantastic. Even after baking the normal maps to a low poly version it still looks great.
Then they bring it into the game and … it looks a little off.
Surfaces that should be flat are still slightly rounded, some are even dimpled or worse have odd streaks across them. Sound familiar? Some of you probably shrugged and thought “oh well, I guess that’s as good as it’s going to look” and moved on. Maybe painted in some extra dirt to hide the worst cases.
Really something in your content pipeline isn’t quite right. What might not always be obvious, but it can be fixed!*
*Assuming you’re okay with making changes to your content pipeline in some cases.
Preface
This is an article about how to export perfect normal maps from other applications for use in Unity 5.3 and newer. This is not about how to get good normal maps to begin with. For that there are plenty of tutorials out there, so I won’t be touching that topic at all.
Table of Contents
- The Basis of Tangent Space
- Recognizing Your Enemy
Flipping Out Over an Obvious Error
sRGBeing Funny
Too Different to be Normal
MikkTSpace vs Autodesk
Mesh Export & Substance Painter
MikkTSpace vs MikkTSpace?
Triangulate, Triangulate, Triangulate! - The List For Perfect Normal Maps for Unity
- More on MikkTSpace
- Tangent Space in Different Applications
Unity 5.3+
Unreal Engine 4
Godot
PlayCanvas
Filament
3ds Max
Maya
Blender
xNormal
Substance Painter
Marmoset Toolbag 3
3DCoat
Houdini
Modo - Additional Thoughts
The Basis of Tangent Space
The problem is with how tangent space normal maps work. Specifically, how that tangent space (or tangent basis) part of tangent space normal maps work. I’m not going to go too deep into the technical aspects of what they’re doing. I’ve done that a little already elsewhere, as have others, but in short tangent space is what determines the orientation of the direction in a normal map is supposed to represent. What actual direction “forward”, “up” and “right” in the normal map actually is. That orientation is roughly the orientation of the texture UVs and the vertex normals.
The more important thing to understand is the program being used to bake a normal map from a high poly mesh to a low poly mesh needs to calculate its tangent space exactly the same way as the program the normal map is being used in. Because different programs may use tangent space differently, normal maps baked for a mesh aren’t guaranteed to be universal between applications. Even for applications that use the same tangent space!
Recognizing Your Enemy
Flipping Out Over an Obvious Error
The most common error is the “OpenGL” vs “Direct3D” issue. This is also called “Y+” and “Y-” normal maps respectively. I’m including this here just for completeness. It’s likely most people who come across this already know about it, or at least find a way to fix it. Because normal maps generated in one orientation are basically unusable if used in an application designed to use the other.
This can be recognized if the lighting seems to be inverted, or coming from the wrong side, but only for some light directions. Certain angles or faces might look totally fine until you move the light.
This one is fairly straight forward. Most applications that bake out normal maps have a nice big button or drop down someplace to switch between OpenGL / Direct3D or have a Y orientation setting. Figure out which orientation your final application uses and make sure you’re baking to that. Unity is OpenGL, Y+. Unreal is Direct3D, Y-. They can be easily distinguished by just looking at the normal map texture itself. If the look like they’re being lit from “above” in the green channel, then they’re OpenGL orientation. If from below then it’s Direct3D.
sRGBeing Funny
Another common mistake is using a texture that’s marked as being an sRGB texture. This changes how the GPU samples a texture, telling it to transform the color values from sRGB space, the one people are used to picking color values in, to linear color space. You don’t want this for normal maps. In Unity if you set a texture’s type to “Normal map” it automatically disables this option, but you can also turn off the “sRGB (Color Texture)” option and it’ll work properly as a normal map. At least in Unity 2017.1 and newer.
The way to recognize this one is normal maps will be heavily shifted in one direction. In this model almost every face looks like it’s facing toward the light. Obviously it won’t always end up that way. With a more detailed normal map all the parts will be there and be reacting to light almost correctly, but just shifted at an angle. It can sometimes look like the entire mesh’s surface is twisted.
Too Different to be Normal
Another common issue is using different mesh normals for baking and the in game asset. Just like the tangents, these need to match exactly for things to work. This one can be common if artists aren’t exporting their meshes with proper vertex normals. This means any application that opens the mesh will have to calculate them on their own. There’s no one right way to do this as it depends on the mesh, so make sure you (or your artists) are exporting with the final normals!
This is another one that’s generally pretty easy to notice. If your model is supposed to look like it has some sharp edges, but the shading is curving away from those edges, it means you have a normal map that was baked with a smoothed normal but being rendered on a mesh with a hard edge. If the shading is curving towards the edge it means it was baked with a hard edge and rendering on a soft edge. Really both can be true on the same mesh since you can choose what edges are smooth or hard, and even to what degree. But basically you’re guaranteed for them to be wrong if you let another application guess at what was intended. The above model is also an extreme case and it’s often not that obvious as large parts of the model may look totally normal. But generally if you have a few edges that look especially odd it’s probably due to a vertex normal mismatch.
MikkTSpace vs Autodesk
Update: The below is true for 3ds Max 2020 and earlier. For 3ds Max 2021 Autodesk added proper support for baking MikkTSpace! See the application list at the end of the article for more information.
A more subtle issue is if you’re baking normal maps from 3ds Max or Maya, they’ll always be wrong for Unity. They’ll also be wrong for Unreal too, or really any other program but the program you baked them in. 3ds Max normal maps won’t work in Maya or vice versa either! Each of these applications calculate tangent space in their own unique ways that does not match other applications. They’ll look perfect when used for themselves, and if you’re generating normal maps for use in those programs, by all means bake out your normal maps using them. Almost nothing else will bake proper normal maps for those applications, so you kind of have to.
The only solution for this case is do not bake your normal maps out using these applications if you intend to use them elsewhere. If you need something free, use xNormal. It’s an industry standard tool for doing this. If you’re using Substance elsewhere in your content pipeline, use that.
The easiest way to notice this as being the problem is if some otherwise flat faces have some slight dimpling or curvature to them. This can be very subtle in some cases, especially on organic objects. So this is getting into the issues that go unnoticed far more often than not.
Many applications outside of Autodesk have moved to use a specific way of calculating tangent space which is referred to as MikkTSpace. Both xNormal and Substance, as well as Blender, Unity, Unreal, Godot, Marmoset, Houdini, Modo and several others, use MikkTSpace by default or at least support it in some way, greatly increasing their compatibility. Basically over the last decade it’s basically become the industry standard for anywhere tangent space normal maps are regularly used. Supposedly 3ds Max will add support for it at some point, but it’s unclear if they plan on letting you bake normal maps using MikkTSpace, or just support viewing / rendering with them.
Mesh Exports & Substance Painter
The other issue with 3ds Max and Maya is if you export meshes from them you have the option to export with “tangents and binormals”. These won’t be in proper MikkTSpace tangents, but based. Some applications will override these and recalculate a correct MikkTSpace by default (like xNormal, Unity & Unreal), but other applications will use the mesh’s tangents as they are if they exist and will only calculate MikkTSpace if the mesh doesn’t have any tangents. Substance Painter for example will use the tangents as they are on the mesh if they exist, which means Painter will produce normal maps that look very similar to those baked by 3ds Max! Similar, but not exactly the same, so if you import them back into 3ds Max they won’t look quite as good as those baked in Max.
MikkTSpace vs MikkTSpace?
The next problem is there’s more than one version of MikkTSpace! I know! It wouldn’t be a standard if it wasn’t confusing in some way. MikkTSpace defines both how the data should be stored on the mesh per vertex, and how the data is used when rendering. All MikkTSpace applications will calculate identical per vertex mesh data. But there are two different ways to use that data when rendering. MikkTSpace defines how to calculate a tangent vector and sign per vertex based on the UV orientation. But when using tangent space the bitangent is recalculated either per vertex or per pixel. That too needs to be the same for both baking and viewing.
xNormal and Substance both have an option for baking normal maps using either method. These options are listed as “compute tangent space per fragment” in Substance and “compute binormal in the pixel shader” in xNormal. You don’t want that on if you’re baking for Unity’s built in rendering paths. But you possibly do want it for the SRPs, depending on which one you use!
This is visually similar to using a completely different tangent space, but will usually be much more subtle. Some faces may look like the normal is slightly twisted rather than having a large dimple in it. The solution here is straight forward, don’t check that option if you’re baking for Unity’s built-in rendering paths. Do check it if you’re baking for Unreal or Unity’s HDRP.
There’s a lot of stuff out there that says that Blender and Unity are identical, but this has in fact was never true! Blender uses MikkTSpace & OpenGL orientation, like Unity, but uses per fragment binomals like Unreal.
The good news is while currently the LWRP, URP, and Shader Graph shaders all use per vertex, the HDRP’s built-in shaders use per pixel. In a future update to Shader Graph (7.2.0) it should be switching to per pixel, and URP’s built-in shaders sometime after that. And the default rendering paths can be modified with custom shaders if one chooses that route.
Triangulate, Triangulate, Triangulate!
The last one I’m going to go into is different triangulation. If you export your meshes without pre-triangulating them, there’s no guarantee other applications that take that model will triangulate it the same way. This also breaks normal maps! So make sure you triangulate your meshes, either as part of the export settings, or using a triangulate modifier.
This will present itself very similarly to using a totally wrong tangent space, like normal maps baked from 3ds Max. But generally it’s a slightly sharper almost crease rather than a soft dimple. In this mesh you can see an X shape showing up on the outside faces. This one can be tricky and annoying when dealing with a content pipeline where multiple people may touch a model through multiple applications as it’s often very useful to not triangulate a mesh and keep nice clean looking polygons. For that case you’ll either just need to deal with that loss of convenience, or make sure the last tool in the pipeline before the model gets exported for the game is also used to generate the mesh used to bake the normals. Like the tangent space, and the vertex normals (which are part of the tangent space), the triangulation also needs to exactly match.
The List For Perfect Normal Maps for Unity
So, in summary, to generate tangent space normal maps for low poly meshes from high poly source models for use in Unity:
- Do not bake normal maps using 3ds Max, Maya, or any application that doesn’t have an option for MikkTSpace tangents.
- Export your meshes as triangles, with normals, and without tangents.
- Choose OpenGL or X+Y+Z+ for your normal map orientation when baking (also when creating a project for Substance Painter).
- Do not enable “per pixel” / “per fragment” bitangent options when baking for the built-in rendering paths (forward or deferred) or LWRP.
- Do enable “per pixel” / “per fragment” bitangent options when baking for HDRP, and soon for URP.
- Leave the model import settings for Unity as their default with Normals set to “Import”, and Tangents set to “Calculate Mikktspace”.
And that’s it! Perfect baked normal maps every time!
Okay, yeah, well, we’ll just ignore that last bit. As “perfect” as you’re going to get with 8 bit per channel compressed texture formats. A dithered normal map can look a little better than the streaks, but unfortunately not a lot of tools offer that. Otherwise 16 bit normal maps are an option. But that’ll have to be another article.
More on MikkTSpace
The semi-official website mikktspace.com does a good job of explaining the basic benefits of MikkTSpace. Namely it works well for both either recalculating or transferring tangent data between programs with different mesh handling while keeping the results consistent. The site’s text is a copy of a now gone Blender wiki page that Morten S. Mikkelsen himself wrote. Unfortunately it only mentions the per pixel bitangent version and not the per vertex version used by Unity and xNormal’s default. As Blender only used per pixel derivatives it makes sense that’s the only one mentioned.
Here’s some comments from the original mikktspace.h file on the two options:
For normal maps it is sufficient to use the following simplified version of the bitangent which is generated at pixel/vertex level.
bitangent = fSign * cross(vN, tangent);…
To avoid visual errors (distortions/unwanted hard edges in lighting), when using sampled normal maps, the normal map sampler must use the exact inverse of the pixel shader transformation.
The most efficient transformation we can possibly do in the pixel shader is achieved by using, directly, the “unnormalized” interpolated tangent, bitangent and vertex normal: vT, vB and vN.
pixel shader (fast transform out)
vNout = normalize( vNt.x * vT + vNt.y * vB + vNt.z * vN );
where vNt is the tangent space normal. The normal map sampler must likewise use the interpolated and “unnormalized” tangent, bitangent and vertex normal to be compliant with the pixel shader.…
Should you choose to reconstruct the bitangent in the pixel shader instead of the vertex shader, as explained earlier, then be sure to do this in the normal map sampler also.
So the original example is reconstructing the bitangent in the vertex shader and passing it to the pixel shader as another interpolated value. This is how Unity’s built-in shaders currently handle normal maps. Doing it in the pixel shader is almost an offhand comment rather than the “main” example. So arguably the per vertex method is the “original” one. Plus since the implementation for Unity 5.3 (and xNormal?) was done by Morten S. Mikkelsen himself, and both use per vertex, it adds further evidence to support that theory. But then Blender used per pixel and Morten was involved with that as well. Really it was probably just the option that caused the least change to existing Unity shaders and neither method was seen as being obviously superior to the other at the time for real time.
The benefit of the per pixel implementation is you remove two interpolated values for the cost of a little extra ALU (pixel shader math). This means instead of 3 Vector3 values, the tangent, bitangent, and normal, you only need the a Vector4 and Vector3, the tangent w/ sign, and normal. On modern GPUs is actually a little faster. It is definitely faster if being done on the CPU as it is for may non real-time renderers. To that end it makes sense that most the industry chose to reconstruct the bitangent at the pixel rather than per vertex.
Humorously, while I was writing this article Morten was at the same time pushing to switch the Unity SRPs to use per pixel bitangents instead! It’s just unfortunate that Unity’s built-in rendering paths are now left with this legacy of being something of the odd one out.
Tangent Space in Different Applications
While this article is Unity focused since, as noted, it’s something of an odd goose today, I wanted to put all my research to good use. To that end I’ve compiled a list of a bunch of different applications with information about each of them related to their tangent space usage.
Unity (5.3+)
Orientation: OpenGL Y+
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: per vertex or per pixel*
Additional Notes:
By default will override mesh tangents on import to use MikkTSpace. Has options to use mesh’s tangents for power users, but this doesn’t guarantee compatibility with normal maps baked from other tools, especially when using built-in shaders. Can be changed with fully custom shaders. And the HDRP uses per pixel bitangents for it’s built in shaders. At the time of this writing the URP and Shader Graph shaders are also slated to switch to per pixel bitangents as well. Prior to Unity 5.3 used Lengyel tangents, which can still be used by selecting the Legacy Tangents option on the mesh importer, but the built-in shaders no longer properly support that tangent space.
Note: prior to URP 7.2.0, Shader Graph handling of normal maps is straight up broken! It doesn’t properly follow either MikkTSpace or even Lengyel’s tangent space usage, and there’s no way to fix this from within Shader Graph. Upgrade to 7.2.0 when you can.
Update:
URP 8.1.0 Shader Graph uses per pixel MikkTSpace normal maps, but the URP Lit shader still uses per vertex.
URP 8.2.0 uses per pixel MikkTSpace normal maps for both Shader Graph and the URP Lit Shader.
HDRP 8.0.1+ uses per pixel MikkTSpace for both Shader Graph and HDRP Lit shader.
Unreal Engine 4
Orientation: Direct3D Y-
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: per pixel
Additional Notes:
By default will override mesh tangents on import to use MikkTSpace. Has options to use mesh’s tangents for power users, but this doesn’t guarantee compatibility with normal maps baked from other tools.
Godot
Orientation: OpenGL Y+
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: per vertex
Additional Notes:
Can calculate mesh tangents on import to use MikkTSpace if no tangent data exists. Their documentation suggests it is better to export your meshes with tangent data, but since most tools won’t export the correct mesh tangents, I would ignore that advice. Even if you use the mesh’s original tangents there’s no guarantee the normal maps will look right still.
PlayCanvas
Orientation: OpenGL Y+
Tangent Space: Lengyel/Schüler*
Bitangent Calculation: per vertex / derivative based
Additional Notes:
Offers several different ways of handling tangent space. Tangents generated by code are not using MikkTSpace. Has options for using per vertex bitangent that either has the tangent space vectors normalized in the pixel shader, “fastTbn” which does not normalize the tangent space in the fragment shader (which you do want as that matches MikkTSpace’s per vertex implementation), and a derivative based TBN borrowed from Christian Schüler which ignores the mesh’s tangents entirely and calculates the tangent space entirely in the pixel shader. Currently defaults to the derivative based normal mapping approach, for which I know of no tool that can bake accurate normal maps. It might be possible to import meshes with existing normal and MikkTSpace tangent data and use the “fastTbn” material option to have it match Unity’s built-in rendering paths, but I believe that would have to be done via code and not the editor interface.
Filament
Orientation: OpenGL Y+
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: per vertex
Additional Notes:
May change to per pixel bitangents in the near future depending on perf.
3ds Max
Orientation: Direct3D Y-*
Tangent Space: Proprietary & MikkTSpace*
MikkTSpace Bitangent Calculation: Configurable (per vertex or per pixel)
Mesh Export Options:
Export your FBX files with “Triangulate” on, but with “Tangents and Binormals” turned off. Normals will always be exported. The “Split per-vertex Normals” can remain off.
Additional Notes:
By default 3ds Max assumes Direct3D orientation for normal maps, but it has options to flip the X and Y orientation when using normal map textures on materials. Baking will always use Direct3D orientation.
Update:
3ds Max 2021 adds new Bake to Texture that includes options to export with a configurable orientation, and to set both per vertex and per pixel bitangent MikkTSpace. You must have the Scanline renderer selected as your current renderer for this to work. Baking with MikkTSpace selected with other renderers will produce completely flat normal maps or crash 3ds Max.
Maya
Orientation: OpenGL Y+
Tangent Space: Proprietary
Mesh Export Options:
Export your FBX files with “Triangulate” on, but with “Tangents and Binormals” turned off. Normals will always be exported. The “Split per-vertex Normals” can remain off.
Additional Notes:
I have seen some people say Maya supports MikkTSpace in some capacity now, but I don’t know to what extent.
Blender (2.57+)
Orientation: OpenGL Y+
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: per pixel
Mesh Export Options:
Export your FBX files with Smoothing set to “Normals Only”. You can export with or without “Tangent Space” if your target application also uses MikkTSpace, like Unity or Unreal.
Additional Notes:
Blender does not have an option to triangulate when exporting to FBX, so make sure you’re using a Triangulate modifier on your mesh before hand. Normal maps baked from Blender won’t be 100% compatible with Unity’s built in rendering paths due to not supporting per vertex bitangents. But they will work perfectly for Unity’s HDRP and URP which now use per pixel bitangents (see notes on Unity above). For Unreal you only need to flip the Y channel, which can be done from the baking options as of Blender 2.8+. Though some versions of Blender the axis orientation was inverted and -Y meant +Y!
xNormal (3.17.5+)
Orientation: Configurable
Tangent Space: MikkTSpace*
MikkTSpace Bitangent Calculation: Configurable (per vertex or per pixel)
Normal Map Baking Options:
Has options for flipping all 3 axis of a normal map on export. For Unity you want X+ Y+ Z+. And there is an option for using per vertex or per pixel (aka per fragment) bitangents. xNormal defaults to per vertex bitangents, but it can be changed by going to the Plugins Manager (the power plug icon in the lower left), clicking on “Tangent basis calculator”, “Mikk — TSpace”, and click on Configure. That’ll show a pop up window with one option, “Compute binormal in the pixel shader”. That should be off for Unity, on for Unreal.
Additional Notes:
While xNormal only ships with support for MikkTSpace, through its plugin system it can theoretically support any tangent space out there. Currently the only tangent space plugin I know of is for Unity 4 tangent space baking, which isn’t useful anymore. So realistically it only supports MikkTSpace.
Substance Painter
Orientation: Configurable (OpenGL or Direct3D)
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: Configurable (per vertex or per pixel)
Normal Map Baking Options:
Has options for baking to either OpenGL or Direct3D orientation. And an option for using per vertex or per pixel (aka per fragment) bitangent, controlled by the “Compute tangent space per fragment” setting. Make sure you have OpenGL selected and Compute tangent space per fragment off for the project and for the export settings when exporting for Unity’s built-in rendering paths, and the inverse when exporting to Unreal or Unity’s HDRP.
Additional Notes:
Substance Painter will default to using whatever tangent data the imported mesh uses. Unless you’re exporting from Blender, you do not want this! But there are no options to disable this feature. Make sure you’re exporting your meshes with out tangents before importing into Substance Painter. Substance’s documentation previously erroneously listed Blender as matching Unity’s bitangent calculations, but that’s been fixed!
Marmoset Toolbag 3
Orientation: Configurable
Tangent Space: Configurable
MikkTSpace Bitangent Calculation: per pixel
Additional Notes:
Appears to have options for viewing or baking normal maps in just about every setup in existence. This includes 3ds Max and Maya tangent space presets! However the “Mikk / xNormal” tangent space option is using per pixel bitangents with no option to change it. This means Marmoset Toolbag 3 cannot view or bake normal maps for Unity’s built-in rendering paths properly. May change in a future version as they are aware of the issue.
3DCoat
Orientation: Configurable
Tangent Space: Configurable
MikkTSpace Bitangent Calculation: ?
Additional Notes:
Has a lot of options for both viewing and baking of normal maps. For sure it’s able to match Blender & UE4, so it supports per pixel bitangents, but I don’t know if the Unity preset uses per vertex bitangents or not.
Houdini (16.0.514+)
Orientation: Configurable
Tangent Space: Configurable*
MikkTSpace Bitangent Calculation: per pixel
Additional Notes:
Supports its own tangent basis, or MikkTSpace, both for viewing and baking, but only the per pixel version. By default it does not appear to use MikkTSpace for rendering, but meshes can be modified with built in nodes to use MikkTSpace tangents since Houdini 17.5.
Modo (10+?)
Orientation: Configurable
Tangent Space: Configurable
MikkTSpace Bitangent Calculation: Configurable (per vertex or per pixel)*
Additional Notes:
Seems to support multiple tangent spaces rendering and baking, and options for “Per-Pixel Bitangent” for export. Unsure if configurable for viewing, though it has a built in Unity material type which may use per vertex. I have not tested.
Knald
Orientation: Configurable
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: per pixel
Additional Notes:
Defaults to MikkTSpace, and will override imported mesh tangents, but can optionally use the mesh’s existing tangents if required. Normal map orientation can be flipped with a project wide setting.
SketchFab
Orientation: Configurable
Tangent Space: Proprietary?
Additional Notes:
Appears to Shader is reconstructing the bitangent in the fragment, and normalizing the tangent space vectors, making it incompatible with both the older Lengyel and all forms of MikkTSpace. Documentation reference’s Crytek’s paper on normal mapping, but also does not implement their tangent space either. Appears to use the mesh tangent data if it exists, otherwise calculates its own tangents on upload. You cannot bake accurate normal maps for SketchFab with any of the other software on this list.
Additional Thoughts
One big caveat is if the low poly model you’re baking to is a flat plane (like for a tiling texture), none of this matters and you can bake using whatever tool you please… as long as you remember to output using the correct orientation or invert the appropriate texture channel(s) afterwards.
Most of the issues listed are also generally only noticeable on hard surface models. Organic shapes can hide the more subtle differences, like the per vertex vs per pixel bitangents.
If anyone has other programs they want me to add to this list, or corrections, and can help do the testing, please let me know!
