CTOForADay.com

Upcoming games like this shot from Bungie's upcoming Halo 2 include well-defined character images like the one you see here (click here for a bigger shot).

These future games rely on a wide range of new technology to deliver a broader, more realistic experience for players. Technological advance, both in terms of the purely mathematical tools which allow these engines to create visual representations, and in the tools that the developers produce in order to use those new mathematical tools to create stunning graphics and animations, have undergone some recent advances:

• Shaders, the basis for all future advances in rendering technology;
• Advances in lighting and shadowing technology;
• and High Dynamic Range, or HDR, rendering.

Shaders

Historically, in rendering systems like that of Pixar's Renderman one didn't apply two-dimensional textures to an object and walk away. Shaders, then and now, are about applying algorithms to perform a specific function - whether that's making something look like it has real fur, or fish scales, or whatever one is trying to model. You don't model scales; there's no point in artists trying to carve each niche in an object like a sculptor does stone. The model, smooth and well-designed, gets a texture applied to it like a coat of paint, but the relief on that object - the bumps, the natural features in the surface - are provided by shaders that affect the coloring, the surface, the deformation of that surface in any direction. Fundamentally, the shader has control over what happens when that part of that surface is 'seen' by the virtual camera.

You've seen this in Pixar films - the motes of light in the water in Finding Nemo, or the waves on the surface of the water, and the way that the light affects it. All of these things are little computer programs, written in a language called a shader language, which affects what happens when the camera sees it, and controls how it is lit, how it looks, how it moves.

You can see it in (view larger image) Tron 2.0 (by Monolith, the people who brought us the joys of Cate Archer in No One Lives Forever) when you see the way that the little lines of light on people glow - that line is there because a shader program is running on your graphics processor, and adding that effect.

Two kinds of programs exist - one, called a vertex program, that allows the application to deform and affect that triangle, and another, called a pixel shader, which happens for each pixel on-screen, to each thing that is visible. Vertex programs can make trees sway in the wind; pixel shaders can do everything from simulate lighting and shadows to performing complex algorithmic textures like fur and skin, scales, and wood.

Given the right set of algorithms, these things make the world seem less square; they reduce the visibility of the mechanics and limitations of 3D, and make the virtual world more realistically simulate real materials in the real world.

These advances in materials technology are responsible for the most fundamental leap, and are the enabling technology that make it possible for many of the next-generation capabilities to exist. As such, the performance of these are critical, and manufacturers of graphics cards are having to work very hard at making these complex programs run efficiently on what, only a few years ago, were systems incapable of performing these kinds of operations. We have only just barely touched the surface of what this technology is capable of doing; the best games out there are incapable of using some of the most primitive shaders from early Pixar films; in future years, the technology will catch up to modern digital filmmakers' capabilities, and make the things we see on film today, with their months of rendering and expensive render farms, possible in the near future, on your computer, in real-time.

Advances in Lighting and Shadowing

There are a range of technologies which will be used to create more realistic lighting. Some of these include:

• Spherical Harmonics

  Spherical Harmonics aren't graphics specific, actually. In graphics terms, though, they refer to surfaces that look different depending on an angle - whether that angle is the view angle, or the angle of the light source to the surface. They're a function of the Legendre polynomial, which most people are familiar with through chemistry - the description of the way electron shells look as you increase the shell depth is a Legendre polynomial.

  Imagine taking a sphere, wrapping it around a lightsource, and marking which directions the light goes in and its intensity. Like modern sample-based pro audio reverb systems, which simulate the reverberation in a room by getting someone to record a special sound in that room, and then recording the room's effects on that sound, Spherical Harmonics do for light what that technology does for sound. Record the way the light actually behaves through complex, offline, production-time software, and ship that recording for use in real-time. The effect is to create light sources that look, act, and behave spookily realistically - the light cast from a candle's guttering, or a lantern's swinging - by casting that pre-recorded information into the room.

  It can also be used to record the effects of ambient light on an ordinary object; instead of wrapping the light source, wrap the object and record how each part of the surface of the sphere contributes to the shadows in and around the object itself.

  The goal of Spherical Harmonics in graphics is to record information used in very expensive (CPU-wise) means of rendering things in an information format that can be used cheaply during real-time to reconstruct images which look and behave in similar ways. The expensive models - radiosity, etc. - are referred to as 'global illumination', and record not only the way that light affects a space, but the ways in which light on the things in that space affect the things around it. Put objects into a room, and the objects in the room reflect that light onto the walls - so having things in a room dramatically impacts how that room is lit. Since we are used to seeing this in the everyday world, images without this look flat, fake.

  Spherical Harmonics, when implemented throughout an engine for a broad range of objects and spaces, can create a world which, through simulating the natural world's lighting effects, more subtly convince us that the space we are looking at acts, and feels like, a space in the real world.

  This can be used in a broad number of ways. It can record the way an object shadows itself when light direction changes; it can also record things like caustics, which is something a reflective/refractive surface does to light when it reflects off of or passes through it. (Look at an empty wineglass in the sun; the strange, bright light that hits the table after passing through the glass, that comes out in a funny shape, is a caustic.

  

  The magic of all of this, and the importance of one of the following elements, is to remember that the calculations can be done on extremely detailed models, and then applied to far lower resolution models that are shipped with the game; the effects on lighting and shadowing, then, are those of the very detailed version of the model, and make the object appear to be much more accurately lit and shadowed than it could be with the visible polygons alone.

• Parallax Mapping

  Referred to by Tim Sweeney from Epic as "Holographic Texture Mapping", Parallax is about making flat, two-dimensional things look three-dimensional. There are a variety of ways, historically, of doing this - bump mapping, for example, made the surface look a bit bumpy where you wanted it to look bumpy, but didn't really deform the object. Deformation maps allowed you to actually make edges look bumpy as well as the surface, but were limited and never really took off in terms of hardware capabilities.

  Using the ability for shaders to deform and perform lighting, these "Holographic Texture Maps" - holographic in that they make something appear to be three-dimensional on the surface of a flat object - have the ability to physically affect the surface in a way that allows for self-shadowing, and looks realistic both at a distance, and more importantly, in close-up. (Bump mapping historically ceases to look realistic at very close range).

  The technology involved in this has evolved over time; Parallax mapping is so called because it approximates the parallax that can be observed on uneven surfaces. Displacement mapping, the most straightforward way to do this, does the trick by triangle subdivision - creating a more complex, 3D surface. Relief Texture Mapping, another mechanism, looks great, but costs a great deal of cpu. In View-Dependent Displacement Mapping (VDM), a per-pixel rendering technique, similar to that used by parallax, swaps out a single representation for a number of maps, each tuned to providing the correct view from a specific angle.

• Normal Mapping

  Normal mapping, in theory, is a simple concept, once you understand what a normal is. Take a triangle; imagine it as a flat surface in your mind, and draw a line perpendicular to the surface. The flat surface, if lying flat on a desk, would have a line coming out vertically to it at 90 degrees, perfectly vertical.

  That line is the normal - and it's used in lighting calculations to figure out how to bounce light off of something. Remember all that school stuff about how the angle of incidence is equal to and opposite of the angle of reflection? And you had an arrow that went down at one angle, and bounced off at another? This arrow is the one that's always exactly in the middle of that - the normal.

  All lighting calculations depend on knowing this number. Graphics systems, provided a bunch of polygons, can happily provide them for you - but there's another option, here. You can provide your own. With the advent of per-pixel lighting, where you are able to run software for each pixel to be lit, you can use a high resolution map which you make out of a very, very detailed version of your object to be rendered, and apply it to a very low resolution version of the object. In essence, you can make the object appear as though it has hundreds of thousands of polygons - but is in fact made up of only a thousand.

  Inserting shadows and lighting through mechanisms like this bypass the actual limitations of the number of polygons in the object, and give the visual illusion (after all, all of 3D graphics is a visual illusion) of a much higher polygon count at a much cheaper cost (in terms of data and computation). The much-awaited Half-Life 2 game makes heavy use of normal mapping to give the world a very detailed appearance without sacrificing performance on today's generation of hardware; future games, when the NV4x and ATI R4x chips are the low-end (probably 2006) will perform these operations quickly and cheaply.

  

High Dynamic Range (HDR) Rendering

The core of HDR is an understanding that the human eye is about more than seeing millions of colors; it reacts to its surroundings. (view larger image)

When you're in a dark room, your eye is initially blind to much of the definition; after a short period of time, the eye adjusts to its darker surroundings, and much of the detail that had been lost becomes visible in shades of black which, in the light of day, are indiscernible from one another.

Neon signs seem to leap out and bleed into their surroundings in darkened streets; lights emit halos that, unlike lens flares, seem to brighten the surroundings. Gone are the days of trying to simulate a camera, with complicated lens flares and cinematic trawl, in are the goals of simulating what it's like to be standing there, at that moment in time, in that simulated place. When the lights come on, we are initially blinded - colors wash to white and light blooms out; light, instead of being white, has real intensity beyond the visible range of the eye that can be adjusted to, adjusted for, and simulated in realtime.

HDR allows rendering engines to simulate the eye's natural capabilities to see much more than just the color ranges - but its ability to adapt to variations in lighting to adjust the sensitivity of this spectrum. The engine can think in enough color detail to allow the engine to simulate the human eye's sensitivities to darkness and light - bright lights bloom and bleed over surroundings, shadows in the darkness become easier to see when the eye adjusts to darkness, the eye can attenuate and focus itself to become more adapted to its surroundings, and with it, the 'digital' eye that represents the camera in 3D gaming can behave similarly.

HDR will make images look more natural, by allowing the rendering pipeline the depth of visual representation that the human eye provides, and by allowing these engines to simulate the same kinds of behavioural traits that the human eye exhibits.

References

The following document references are available for those who either have a fetish for hardcore math or are looking for pretty pictures that might give them a better idea of what some of this stuff will actually do, visually.

Shaders

• cgshaders.org, a site dedicated to the Cg shader language from NVidia. Cg is a metalanguage for shaders - you write your shaders in this language, and it will compile and optimize them down to the API-specific versions best suited for the platform you'll be running on (GL uses ARB pixel and vertex shaders, while Direct3D has HLSL.)
• ATI RenderMonkey, a tool for writing and testing shaders in.
• Apple's OpenGL Shader Builder is a part of their standard dev tools package available with MacOS/X, and comprises one of three tools in what is actually an exceptionally high-powered toolchain for GL development. This toolchain also includes a driver monitor capable of monitoring usage and statistical analysis to assist in finding problems with a rendering pipeline, and a GL profiler which hooks into the OS and can trace, track, insert breakpoints and examine call stacks, as well as pure graphics profiling.

Spherical Harmonics, Radiance Transfer, Etc.

• Spherical Harmonic Lighting: The Gritty Details
  Robin Green, R&D Programmer, Sony Computer Entertainment America, January 16, 2003.
• Spherical Harmonic Lighting using OpenGL
• Beyond3D Forums, UnrealEngine3 Interview discussion on Spherical Harmonics
• Pike Sloan, Microsoft Research

  Mr. Sloan has a range of papers up, covering everything from Precomputed Radiance Transfer and Bi-Scale Radiance Transfer to the various uses for Spherical Harmonics (SH); these are not papers for the weak in math. The Spherical Harmonics paper listed below covers what they were originally used for, how to use them to get objects to shadow themselves, and how to use them to compute how light affects a surface. The Precomputed Radiance Transfer paper mentioned below is about how to construct incredibly complex and realistic light environments in rooms during the design phase, and apply that information to objects in the space in real-time during play. More papers of interest are available from his site, listed above.

  • Fast, Arbitrary BRDF Shading for Low-Frequency Lighting Using Spherical Harmonics
    Jan Kautz, Peter-Pike Sloan and John Snyder, Eurographics Rendering Workshop 2002 ,June, 2002
  • Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Lighting Environments
    Peter-Pike Sloan, Jan Kautz, and John Snyder, SIGGRAPH 2002, July, 2002

Parallax Mapping (sometimes called Offset Mapping, or Holographic Texture Mapping)

Normal Mapping

• Ionization.net: A Tutorial on Normal Mapping
  Contains some very stark images that show the difference between ordinary low-poly models and low-poly models with normal maps attached to them.
• Creating Normal Maps
  Monitor Studios: Ben Cloward. A tutorial on creating normal maps, and the principles of how they work.
• Recovering normal and displacement maps from existing geometry in Maya
  A drone.org tutorial on normal and displacement mapping, including some brilliant pictorial images that really show off the capabilities of a system like this. A woman's torso is provided in low and high definition, and a final image shows the low definition image, with the map information from the high definition image provided.
• Kaldera, by Manuka: A 3D Studio Max Plugin
  Contains another visual representation of the effects of normal mapping on low-resolution images.

HDR

High Dynamic Range imaging

• High Dynamic Range Images
  Timo Jyrinki