Shadows in 3D Mario games

Hi everybody! I'm a Technical Artist and I’d like to talk about the tricks of rendering dynamic shadows in 3D Mario games.

Projected textures:
This is one of the simplest ways of doing shadows. From your shadow casting object place a plane directly underneath it on the ground with a texture representing a shadows shape.

One way of placing this plane is by doing a raycast straight down from the object to find the position and normal/angle for it.
If the ground is anything more complex than a flat infinite plane then this technique is probably going to result in glitches and errors (although maybe not that noticeable).
Mario 64 is one of the earliest examples I can think of that uses this technique and to avoid making the shadow extend beyond a cliff edge they are comparing the zdepth value of the ground (that’s rendered before the shadow plane) and if the ground and the plane is anything but equal in value it doesn’t render. This doesn’t work for more modern games since the ground is usually not constructed of really big triangles but rather more organic tessellated geometry.

Above: Most things in Mario 64 have a disc shaped shadow but some creativity is used for certain objects, like the spikey tree.

A more advanced way of doing this could be to use some sort of deferred rendering where the plane isn’t really a plane but a projection that always conforms to the ground below the shadow casting object. If you’re using deferred rendering anyways then this is probably a good idea.

Stencil buffer:
The stencil buffer, if you’re not familiar with it, is basically a buffer/texture covering the whole screen where we can add values that we refer to when rendering stuff later. Basically creating masks for whatever we want to render. I believe the values are integers ranging from 0-255 (correct me if I’m wrong about the specifics).

To create a simple disc shaped shadow we are actually going to render an invisible cylinder twice. Both times, the cylinder is rendered, it’s positioned exactly at the same spot. What’s happening is that we’re rendering the front faces of the cylinder first changing the stencil buffer from 0 to 1 for those pixels. Then we render the back faces and set the stencil buffer back to 0 for those pixels. This will make us have a disc shaped mask at the ground where the cylinder is intersecting the ground.
As a final step we render a shadow masked by the stencil buffer, where the value is 1.

To do more complex shadows with this technique it’s all about creating a more complex geometry and how the stencil buffer calculation is done.

In Super Mario Galaxy 1&2 this technique is used heavily in conjunction with projected plane shadows. Most stuff are either a cylinder or projected box with variations but Mario himself is a bit more advanced projecting some geometry more closely representing his shape.

Below: Example from Mario galaxy. Notice how Marios stencil shadow is actually created from two stencil shapes, one slightly thicker than the other. Probably to mimic softness. This screenshot is taken from a hi-res emulator and would look smoother in it's original resolution. Most other things in the game uses a cylinder to create disc shaped shadow.

The earliest instance of this technique in a Mario game is in Mario 64 DS in which they use an 8 sided cylinder to create a shadow. Trees and other static objects still used a textured plane though but since they don't move unexpected glitches doesn't occur.

Below: Yoshi uses a stencil buffer shadow compared to the tree which is a texture on a plane.

Below: Mario without and with the cylinder shape drawn in.

Below: Notice how both Mario odyssey and Mario 64 DS have the same "artifact" when doing stencil buffer shadows.

Shadow mapping:
This technique is the most popular and frequently used one today. We’re basically rendering the whole scene from the lights perspective but we’re only interested in the depth of the scene. When rendering from the camera we compare the depth rendered from the light source to the depth rendered through the camera and by doing this we can calculate if the pixel rendered is in shadow or not.

The lights depth texture needs to be rendered in high enough resolution so as to not make visible edges from the pixels. Usually some kind of filtering is applied to make the shadows a bit blurry looking.

An overwhelming amount of games use this technique, and is probably the one you are most accustomed to.

Shadows in Mario Odyssey:
Mario Odyssey is quite fun as an example actually since it not only uses the Shadow mapping technique but also the stencil buffer technique as well as the projected planes technique.
- The stencil buffer technique is used for Mario (his shadow is always facing straight down so the player has a good time landing jumps).
- The projected planes are used for anything we need to know the exact position of, pickups mostly, and that doesn’t want the pixel perfect crispness of stencil buffer shadows. These shadows are also projected straight down.
- Shadow mapping is used for basically everything else. This makes all the shadows for other things always correspond correctly to the shape of the geometry (no trickery or advanced math needed).

Below: Notice the shadow mapping technique used on the pipes in the background, the projected shadow/texture for the power moon and the stencil buffer shadow used under Mario himself

Below: Notice how the bouncing trash bags are casting two shadows in slightly different directions. One projected blob straight down and one shadow mapped shadow aligning with the sun direction in this level.

Cheers!