Generating Perfect Normal Maps for Unity (and Other Programs)

And how to tell what’s wrong with your content pipeline.

Behold the majesty of … this … thing … the traditional normal map test shape!
Aren’t those sides supposed to be flat?

Preface

This is an article about how to export perfect normal maps from other applications for use in Unity 5.3 and newer. This is not about how to get good normal maps to begin with. For that there are plenty of tutorials out there, so I won’t be touching that topic at all.

Table of Contents

  1. The Basis of Tangent Space
  2. Recognizing Your Enemy
    Flipping Out Over an Obvious Error
    sRGBeing Funny
    Too Different to be Normal
    MikkTSpace vs Autodesk
    Mesh Export & Substance Painter
    MikkTSpace vs MikkTSpace?
    Triangulate, Triangulate, Triangulate!
  3. The List For Perfect Normal Maps for Unity
  4. More on MikkTSpace
  5. Tangent Space in Different Applications
    Unity 5.3+
    Unreal Engine 4
    Godot
    PlayCanvas
    Filament
    3ds Max
    Maya
    Blender
    xNormal
    Substance Painter
    Marmoset Toolbag 3
    3DCoat
    Houdini
    Modo
  6. Additional Thoughts

The Basis of Tangent Space

The problem is with how tangent space normal maps work. Specifically, how that tangent space (or tangent basis) part of tangent space normal maps work. I’m not going to go too deep into the technical aspects of what they’re doing. I’ve done that a little already elsewhere, as have others, but in short tangent space is what determines the orientation of the direction in a normal map is supposed to represent. What actual direction “forward”, “up” and “right” in the normal map actually is. That orientation is roughly the orientation of the texture UVs and the vertex normals.

Recognizing Your Enemy

Flipping Out Over an Obvious Error

The most common error is the “OpenGL” vs “Direct3D” issue. This is also called “Y+” and “Y-” normal maps respectively. I’m including this here just for completeness. It’s likely most people who come across this already know about it, or at least find a way to fix it. Because normal maps generated in one orientation are basically unusable if used in an application designed to use the other.

Direct3D orientation normal maps in Unity, which expects OpenGL orientation

sRGBeing Funny

Another common mistake is using a texture that’s marked as being an sRGB texture. This changes how the GPU samples a texture, telling it to transform the color values from sRGB space, the one people are used to picking color values in, to linear color space. You don’t want this for normal maps. In Unity if you set a texture’s type to “Normal map” it automatically disables this option, but you can also turn off the “sRGB (Color Texture)” option and it’ll work properly as a normal map. At least in Unity 2017.1 and newer.

Normal map set to use the Default texture type with sRGB enabled

Too Different to be Normal

Another common issue is using different mesh normals for baking and the in game asset. Just like the tangents, these need to match exactly for things to work. This one can be common if artists aren’t exporting their meshes with proper vertex normals. This means any application that opens the mesh will have to calculate them on their own. There’s no one right way to do this as it depends on the mesh, so make sure you (or your artists) are exporting with the final normals!

Normal maps baked on a smoothed mesh, displayed on a hard edged mesh
Normal maps baked on a hard edged mesh, displayed on a smoothed mesh

MikkTSpace vs Autodesk

Update: The below is true for 3ds Max 2020 and earlier. For 3ds Max 2021 Autodesk added proper support for baking MikkTSpace! See the application list at the end of the article for more information.

3ds Max baked normal map used in Unity

Mesh Exports & Substance Painter

The other issue with 3ds Max and Maya is if you export meshes from them you have the option to export with “tangents and binormals”. These won’t be in proper MikkTSpace tangents, but based. Some applications will override these and recalculate a correct MikkTSpace by default (like xNormal, Unity & Unreal), but other applications will use the mesh’s tangents as they are if they exist and will only calculate MikkTSpace if the mesh doesn’t have any tangents. Substance Painter for example will use the tangents as they are on the mesh if they exist, which means Painter will produce normal maps that look very similar to those baked by 3ds Max! Similar, but not exactly the same, so if you import them back into 3ds Max they won’t look quite as good as those baked in Max.

MikkTSpace vs MikkTSpace?

The next problem is there’s more than one version of MikkTSpace! I know! It wouldn’t be a standard if it wasn’t confusing in some way. MikkTSpace defines both how the data should be stored on the mesh per vertex, and how the data is used when rendering. All MikkTSpace applications will calculate identical per vertex mesh data. But there are two different ways to use that data when rendering. MikkTSpace defines how to calculate a tangent vector and sign per vertex based on the UV orientation. But when using tangent space the bitangent is recalculated either per vertex or per pixel. That too needs to be the same for both baking and viewing.

xNormal “pixel shader binormals” in Unity’s built in forward renderer

Triangulate, Triangulate, Triangulate!

The last one I’m going to go into is different triangulation. If you export your meshes without pre-triangulating them, there’s no guarantee other applications that take that model will triangulate it the same way. This also breaks normal maps! So make sure you triangulate your meshes, either as part of the export settings, or using a triangulate modifier.

Substance Painter baked normal maps in Unity on a mesh exported from 3ds Max as quads

The List For Perfect Normal Maps for Unity

So, in summary, to generate tangent space normal maps for low poly meshes from high poly source models for use in Unity:

  • Export your meshes as triangles, with normals, and without tangents.
  • Choose OpenGL or X+Y+Z+ for your normal map orientation when baking (also when creating a project for Substance Painter).
  • Do not enable “per pixel” / “per fragment” bitangent options when baking for the built-in rendering paths (forward or deferred) or LWRP.
  • Do enable “per pixel” / “per fragment” bitangent options when baking for HDRP, and soon for URP.
  • Leave the model import settings for Unity as their default with Normals set to “Import”, and Tangents set to “Calculate Mikktspace”.
Yay! Perfect Normal Map!
… with reflections …? Uh … *cough*

More on MikkTSpace

The semi-official website mikktspace.com does a good job of explaining the basic benefits of MikkTSpace. Namely it works well for both either recalculating or transferring tangent data between programs with different mesh handling while keeping the results consistent. The site’s text is a copy of a now gone Blender wiki page that Morten S. Mikkelsen himself wrote. Unfortunately it only mentions the per pixel bitangent version and not the per vertex version used by Unity and xNormal’s default. As Blender only used per pixel derivatives it makes sense that’s the only one mentioned.

Tangent Space in Different Applications

While this article is Unity focused since, as noted, it’s something of an odd goose today, I wanted to put all my research to good use. To that end I’ve compiled a list of a bunch of different applications with information about each of them related to their tangent space usage.

Unity (5.3+)

Orientation: OpenGL Y+
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: per vertex or per pixel*
Additional Notes:
By default will override mesh tangents on import to use MikkTSpace. Has options to use mesh’s tangents for power users, but this doesn’t guarantee compatibility with normal maps baked from other tools, especially when using built-in shaders. Can be changed with fully custom shaders. And the HDRP uses per pixel bitangents for it’s built in shaders. At the time of this writing the URP and Shader Graph shaders are also slated to switch to per pixel bitangents as well. Prior to Unity 5.3 used Lengyel tangents, which can still be used by selecting the Legacy Tangents option on the mesh importer, but the built-in shaders no longer properly support that tangent space.

Unreal Engine 4

Orientation: Direct3D Y-
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: per pixel
Additional Notes:
By default will override mesh tangents on import to use MikkTSpace. Has options to use mesh’s tangents for power users, but this doesn’t guarantee compatibility with normal maps baked from other tools.

Godot

Orientation: OpenGL Y+
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: per vertex
Additional Notes:
Can calculate mesh tangents on import to use MikkTSpace if no tangent data exists. Their documentation suggests it is better to export your meshes with tangent data, but since most tools won’t export the correct mesh tangents, I would ignore that advice. Even if you use the mesh’s original tangents there’s no guarantee the normal maps will look right still.

PlayCanvas

Orientation: OpenGL Y+
Tangent Space: Lengyel/Schüler*
Bitangent Calculation: per vertex / derivative based
Additional Notes:
Offers several different ways of handling tangent space. Tangents generated by code are not using MikkTSpace. Has options for using per vertex bitangent that either has the tangent space vectors normalized in the pixel shader, “fastTbn” which does not normalize the tangent space in the fragment shader (which you do want as that matches MikkTSpace’s per vertex implementation), and a derivative based TBN borrowed from Christian Schüler which ignores the mesh’s tangents entirely and calculates the tangent space entirely in the pixel shader. Currently defaults to the derivative based normal mapping approach, for which I know of no tool that can bake accurate normal maps. It might be possible to import meshes with existing normal and MikkTSpace tangent data and use the “fastTbn” material option to have it match Unity’s built-in rendering paths, but I believe that would have to be done via code and not the editor interface.

Filament

Orientation: OpenGL Y+
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: per vertex
Additional Notes:
May change to per pixel bitangents in the near future depending on perf.

3ds Max

Orientation: Direct3D Y-*
Tangent Space: Proprietary & MikkTSpace*
MikkTSpace Bitangent Calculation: Configurable (per vertex or per pixel)
Mesh Export Options:
Export your FBX files with “Triangulate” on, but with “Tangents and Binormals” turned off. Normals will always be exported. The “Split per-vertex Normals” can remain off.
Additional Notes:

By default 3ds Max assumes Direct3D orientation for normal maps, but it has options to flip the X and Y orientation when using normal map textures on materials. Baking will always use Direct3D orientation.
Update:
3ds Max 2021 adds new Bake to Texture that includes options to export with a configurable orientation, and to set both per vertex and per pixel bitangent MikkTSpace. You must have the Scanline renderer selected as your current renderer for this to work. Baking with MikkTSpace selected with other renderers will produce completely flat normal maps or crash 3ds Max.

Maya

Orientation: OpenGL Y+
Tangent Space: Proprietary
Mesh Export Options:
Export your FBX files with “Triangulate” on, but with “Tangents and Binormals” turned off. Normals will always be exported. The “Split per-vertex Normals” can remain off.
Additional Notes:
I have seen some people say Maya supports MikkTSpace in some capacity now, but I don’t know to what extent.

Blender (2.57+)

Orientation: OpenGL Y+
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: per pixel
Mesh Export Options:
Export your FBX files with Smoothing set to “Normals Only”. You can export with or without “Tangent Space” if your target application also uses MikkTSpace, like Unity or Unreal.
Additional Notes:

Blender does not have an option to triangulate when exporting to FBX, so make sure you’re using a Triangulate modifier on your mesh before hand. Normal maps baked from Blender won’t be 100% compatible with Unity’s built in rendering paths due to not supporting per vertex bitangents. But they will work perfectly for Unity’s HDRP and URP which now use per pixel bitangents (see notes on Unity above). For Unreal you only need to flip the Y channel, which can be done from the baking options as of Blender 2.8+. Though some versions of Blender the axis orientation was inverted and -Y meant +Y!

xNormal (3.17.5+)

Orientation: Configurable
Tangent Space: MikkTSpace*
MikkTSpace Bitangent Calculation: Configurable (per vertex or per pixel)
Normal Map Baking Options:
Has options for flipping all 3 axis of a normal map on export. For Unity you want X+ Y+ Z+. And there is an option for using per vertex or per pixel (aka per fragment) bitangents. xNormal defaults to per vertex bitangents, but it can be changed by going to the Plugins Manager (the power plug icon in the lower left), clicking on “Tangent basis calculator”, “Mikk — TSpace”, and click on Configure. That’ll show a pop up window with one option, “Compute binormal in the pixel shader”. That should be off for Unity, on for Unreal.
Additional Notes:

While xNormal only ships with support for MikkTSpace, through its plugin system it can theoretically support any tangent space out there. Currently the only tangent space plugin I know of is for Unity 4 tangent space baking, which isn’t useful anymore. So realistically it only supports MikkTSpace.

Substance Painter

Orientation: Configurable (OpenGL or Direct3D)
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: Configurable (per vertex or per pixel)
Normal Map Baking Options:
Has options for baking to either OpenGL or Direct3D orientation. And an option for using per vertex or per pixel (aka per fragment) bitangent, controlled by the “Compute tangent space per fragment” setting. Make sure you have OpenGL selected and Compute tangent space per fragment off for the project and for the export settings when exporting for Unity’s built-in rendering paths, and the inverse when exporting to Unreal or Unity’s HDRP.
Additional Notes:

Substance Painter will default to using whatever tangent data the imported mesh uses. Unless you’re exporting from Blender, you do not want this! But there are no options to disable this feature. Make sure you’re exporting your meshes with out tangents before importing into Substance Painter. Substance’s documentation previously erroneously listed Blender as matching Unity’s bitangent calculations, but that’s been fixed!

Marmoset Toolbag 3

Orientation: Configurable
Tangent Space: Configurable
MikkTSpace Bitangent Calculation: per pixel
Additional Notes:
Appears to have options for viewing or baking normal maps in just about every setup in existence. This includes 3ds Max and Maya tangent space presets! However the “Mikk / xNormal” tangent space option is using per pixel bitangents with no option to change it. This means Marmoset Toolbag 3 cannot view or bake normal maps for Unity’s built-in rendering paths properly. May change in a future version as they are aware of the issue.

3DCoat

Orientation: Configurable
Tangent Space: Configurable
MikkTSpace Bitangent Calculation: ?
Additional Notes:
Has a lot of options for both viewing and baking of normal maps. For sure it’s able to match Blender & UE4, so it supports per pixel bitangents, but I don’t know if the Unity preset uses per vertex bitangents or not.

Houdini (16.0.514+)

Orientation: Configurable
Tangent Space: Configurable*
MikkTSpace Bitangent Calculation: per pixel
Additional Notes:
Supports its own tangent basis, or MikkTSpace, both for viewing and baking, but only the per pixel version. By default it does not appear to use MikkTSpace for rendering, but meshes can be modified with built in nodes to use MikkTSpace tangents since Houdini 17.5.

Modo (10+?)

Orientation: Configurable
Tangent Space: Configurable
MikkTSpace Bitangent Calculation: Configurable (per vertex or per pixel)*
Additional Notes:
Seems to support multiple tangent spaces rendering and baking, and options for “Per-Pixel Bitangent” for export. Unsure if configurable for viewing, though it has a built in Unity material type which may use per vertex. I have not tested.

Knald

Orientation: Configurable
Tangent Space: MikkTSpace
MikkTSpace Bitangent Calculation: per pixel
Additional Notes:
Defaults to MikkTSpace, and will override imported mesh tangents, but can optionally use the mesh’s existing tangents if required. Normal map orientation can be flipped with a project wide setting.

SketchFab

Orientation: Configurable
Tangent Space: Proprietary?
Additional Notes:
Appears to Shader is reconstructing the bitangent in the fragment, and normalizing the tangent space vectors, making it incompatible with both the older Lengyel and all forms of MikkTSpace. Documentation reference’s Crytek’s paper on normal mapping, but also does not implement their tangent space either. Appears to use the mesh tangent data if it exists, otherwise calculates its own tangents on upload. You cannot bake accurate normal maps for SketchFab with any of the other software on this list.

Additional Thoughts

One big caveat is if the low poly model you’re baking to is a flat plane (like for a tiling texture), none of this matters and you can bake using whatever tool you please… as long as you remember to output using the correct orientation or invert the appropriate texture channel(s) afterwards.

Tech Artist & Graphics Programmer lately focused on Unity VR game dev.